DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
Authors: Changyi He
Yifu Ding
Jinyang Guo
Ruihao Gong
Haotong Qin
Xianglong Liu
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Knowledge Distillation
Primary area: deep_learning->large_language_models

Abstract: Although knowledge distillation (KD) is an effective approach to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a large LLM (i.e., the teacher model), it still suffers from high training cost. Existing LLM distillation methods ignore the difficulty difference among different samples, making the distillation of easy samples unnecessary. This leads to high distillation cost. In this paper, we propose difficulty-aware knowledge distillation (DA-KD) framework for efficient knowledge distillation, in which we dynamically adjust the distillation dataset based on the difficulty of samples. We further observe existing KD loss cannot perform well when most of samples are difficult in the distillation dataset because of unstable optimization and the neglect of hard samples. Therefore, we also propose a new KD loss called bidirectional discrepancy loss (BDL) for effective KD. Extensive experiments demonstrate that our DA-KD framework is effective and efficient. Without bells and whistles, DA-KD can outperform existing state-of-the-art KD methods by 2\% with half training cost and even surpass the teacher model with 4.7$\times$ compression.

DA-KD: Difficult--ware Knowledge Distillation for Efficient Large Language Models
Changyi He
Yifu Ding Jinyang Guo
Ruihao Gong"
Haotong Qin
Xianglong Liu
Abstract
Avagg 1aare-133
Although knowledge distillation (KD) is an effec- tive approach improve the performance of a smaller LLM (i.e., the student model) by trans- ferring knowledge from large LLM (i.e., the teacher model), it still suffers from high training cost. Existing LLM distillation methods ignore the difficulty difference among different samples, making the distillation of casy samples unnec- essary. This leads to high distillation cost. In this paper, we propose lifficulty-awar knowl- edge distillation (DA-KD) framework for efficient knowledge distillation, in which we dynamically adjust the distillation dataset based on the diffi- culty of samples. We further observe existing KD loss cannot perform well when most of samples are difficult in the distillation dataset because of unstable optimization and the neglect of hard sam- ples. Therefore, also propose called bidirectional discrepancy loss (BDL) for effective KD. Extensive experiments demonstrate that our DA-KD framework is effective and ef- ficient. Without bells and whistles, DA-KD can outperform existing state-of-the-an KD methods by 2% with half training cost and even surpass the teacher model with 4.7x compression.
388
Owee2 5..36
Qwes25511.5
Performares
78
Trmining Time (min)
KD-RKI
SeaKD
GND
Distillo
O07
Trining
1963
Figure I. Performance various distillation methods on Dolly dataset in the upper part, and their training time (minutes) and iterations the bottom part.
KD loss
and storage requirements, posing challenges for practical deployment (Yang et al., 2024b). To address this issue, nu- merous model compression methods have emerged, such quantization (Guo et al., 2024; Lv al., 2024), pruning (Wang et al., 2024b; Guo et al., 2022; 2023b) and knowl- edge distillation (Hinton, 2015). Knowledge distillation is an effective approach for constructing smaller and more effi- cient neural networks. It aims to transfer knowledge from a figh-performin teacher model compact student model for efficient inference.
1. Introduction
Recent advancements in Large Language Models (LLMs) such LLaMA and Qwen (Touvron et al., 2023; Bai et al., 2023; Yang et al., 2024a) have garnered significant atten- tion due to their remarkable capabilities and intelligence. However, these models impose substantial computational
Despite the success of KD for effective training of stu- dent models, it still suffers from high training cost, which normally requires hundreds of GPU hours when distill- ing models with billions of parameters (Agarwal et al., 2023). To solve this problem, many efficient distillation approaches were proposed recent years. They use knowl- edge caching (Jin al., 2024; Zheng et al., 2021; Huang et al., 2024), self-distillatio et al., 2021; 1; Naeem et al., 2025), dataset condensation (Zhao et al., 2020; Yin et al., 2024) and so on. However, these approaches focus on traditional downstream tasks such as computer vision or language processing, and few of them have delved deeply into efficient distillation using dataset se-
State Key Laboratory of Complex & Critical Software Envi- ronment, Beihang University "Sehool of Computer Science and Engineering, Beihang University "Scoool of Artificial Intelligence, Beihang University TSenseTime Research 'ETH Zurich. Corre- spondence to: Jinyang Guo <jinyangguo @ buaa.edu.cn>.
(Dong et al., 2023; Zhang
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
lection for generative large models. As a result, it is still open problem how effectively select informative sam- ples for distillation, which makes the efficient distillation for generative large models a non-trivial task.
have emerged as transformative advancements in natural language processing, achieving state-of-the-art performance across broad range of complex tasks. Prior works such as GPT (Radford, 2018 introduced the use of stacked trans- former decoders, laying the foundation for subsequent inno- vations. Building on this, models like the Llama (Touvron et al., 2023; Dubey et al., 2024) and the Qwen (Yang et al., 2024a) series refined transformer architecture improve efficiency and performance. Other approaches (Guo et al., 2023a; 2021; 2020) were also proposed in recent years by improving either training techniques or network architec- ture.
To solve the aforementioned problems, in this paper, we pro- pose a knowledge distillation framework called Difficulty- Aware Knowledge Distillation (DA-KD), which is de- signed for efficient distillation of large language models (LLMs). To achieve efficient distillation, we first construct a Difficult-Aware Data Updating (DiffUp) strategy, which includes a Distillation Difficulty Score (DDS) to measure the sample complexity based on the performance gap be- tween the teacher and student. As a result, we use DDS to filter out easy samples for condensed data volume. To strengthen data diversity and mitigate the forgetting problem when only difficult samples are adopt (Jiang et al., 2023), we also propose a Stratified Data Updating (SDU) strategy to improve the diversity of the dataset by gradually mixing samples with various DDS.
Despite these advancements, LLMs often require substantial computational resources due to their massive parameter sizes. In this work, we aim to train an efficient student model by using knowledge distillation technique.
Knowledge distillation for LLM. Knowledge distillation transfers knowledge from a large teacher model to a smaller student model (Hinton, 2015) for faster inference. Recently, many knowledge distillation works were proposed for LLM. For example, MiniL.LM (Gu et al., 2024) proposed reverse KL divergence (RKL) as the loss function prevent the student from overestimating the low-probability regions of the teacher. GKD (A (Agarwal et al., 2023) additionally intro- duced generalized Lensen-Shannon divergence (ISD), and trained the student on its self-generated outputs to mitigate the
Furthermore, as we select difficult samples in the distillation process, we find the existing KD loss cannot provide robust optimization for the student. When extreme distribution occurs in the student, existing KD loss will encounter the explosion or vanishing gradient problem. Moreover, we also find it is challenging for existing KD loss to pay more at- tention to hard samples. To this end, we propose a new loss function called Bidirectional Diserepancy Loss (BDL) to restrict the gradient without explosion or vanishing. Specifi- cally, our BDL is built upon traditional KL divergence, but further incorporates combined probability distribution from both teacher and student. From our detailed analysis, we show that our BDL can not only effectively stabilize the optimization for the student but also enforce the distillation to pay more attention to the hard samples.
mismatch. DistiLLM (Ko et al., 2024) proposed skew KL divergence (SKL) and skew reverse KL divergence (SRKL) that using a mixture of probability dis- tributions to improve the optimization stability.
However, current KD methods for LLM require a significant amount of training time (Xu et al., 2024; Ko et al., 2024), primarily due to the large-scale training data. In contrast, our DA-KD framework aims at not only effective but also efficient LLM knowledge distillation.
Our main contributions are summarized as follows:
We propose difficulty- aware knowledge distillation (DA- KD) for efficient and effective LLM distillation,
Data selection for LLM training. Data selection is increas- ingly recognized crucial for optimizing the LLM fine- tuning process (Zhou et al., 2024; Wang et al., 2024a). These methods aim to prioritize subsets of data that are most valu- able for training, thus reducing computational costs while preserving or even improving model performance (Tao et al., 2025a). Several recent works have explored data selection for LLM fine-tuning. For example, Chen al. (2023) in- troduced Alpagasus to directly leverage ChatGPT for rating each data sample, and select samples with higher ratings. Cao et al. (2023) proposed a linear rule-based approach named Instruction mining by training a linear function to identify the most effective data samples. Li et al. (2023) proposed the IFD method, where a LLM first learns from a small subset of data to acquire foundational capabilities and then uses this learned model to rate and select data from the original dataset.
We also propose difficulty- data updating strategy to dynamically update the distillation dataset, which consists of a distillation difficult score for sample difficulty measure- ment and a stratified sampling strategy for data diversity.
We propose a new loss function called bidirectional dis- crepancy loss, which can effectively stabilize the student training from difficult samples.
Extensive experiments on multiple benchmark datasets demonstrate the effectiveness of our DA-KD framework.
2. Related Work
Large language model. Recent advancements in artifi- cial intelligence have garnered significant attention (Tao et al., 2022; 2025b). In particular, Large language models
2

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
Difficulty-aware Dota Updating
Bidirectional Discrepancy Distillation
Teacher
(x)
Purtition
Stratified Data Updeting
Initial Dataset
Distillation Difficulty (DS(x) Score
D ord
probability
Pm
Current
DRigh in
D Dlow
Deat Pma Qm)
Student
Luge (x)
Dhigh
D
Filter
probabilit' 40
Student
Qm
D'
rID|
Update
Maxtng
Backoropapation
(eDL(p.9) (p.
Figure 2. Overview of our Difficulty-Aware Knowledge Distillation framework.
Data selection for knowledge distillation. Several studies also explored data selection methods in knowledge distilla- tion. For instance, Li al. (2021) proposed an uncertainty- based data selection method for knowledge distillation of LLMs, while Lan et al. (2025) also introduced data lection approach specifically designed for knowledge dis- tillation. However these methods are either solely based on student model (Li et al., 2021) or solely based on the teacher model (Lan et al., 2025), ignoring their discrepancy. Different from these works, we propose DA-KD framework dynamically adjust distillation dataset in the distillation process based on the discrepancy between teacher and stu- dent.
3.2.1. DISTILLATION DIFFICULTY SCORE
To evaluate the difficulty of sample x quantitatively, we first propose a new difficulty metric called Distillation Difficulty Score (DDS) as the ratio of the loss from student over that from teacher:
DDS( = Lgo (x) (x)
(1)
where Lyo (z) and Cp(x) represent the cross-entropy loss between the sample x and the ground-truth from the student and teacher models, respectively.
This simple but effective DDS captures the distillation dif- ficulty based on the discrepancy between the teacher and student models. Figure 3 gives examples for different cases. When the student exhibits high L (x) on a sample but the teacher has low loss (z) on it, the DDS value becomes large. This situation indicates that the teacher is confident about the sample, whereas the student struggles to do so. Such case highlights the student model requires additional supervision from the teacher. Conversely, the DDS value remains small when both the student and teacher achieve low losses, which means the sample is well-leamed by both models, making further distillation unnecessary. When both the teacher and student incur high losses, the DDS value remains small as well, which means the teacher's under- standing of the sample is inadequate, limiting its ability to provide meaningful guidance. The essence of DDS is analogous to real-world teaching: effective teaching occurs when concentrating difficult knowledge that the teacher understands well but the student finds challenging.
3. Difficulty-Aware Knowledge Distillation
3.1. Overview
Figure 2 shows the overview of our DA-KD framework. Given a large teacher LLM and small student LLM, we first calculate the cross-entropy loss of each sample in the initial dataset, based on which calculate the distillation difficult score (DDS) for each sample. Then, we perform the stratified data updating based on the calculated DDS, in which the distillation dataset will be progressively reduced in this process. In the distillation procedure, we use our bidirectional discrepancy loss as the loss function, which can provide stable optimization and pay more attention to hard samples. We will introduce each component in our DA-KD framework in detail.
3.2. Difficulty-wware Data Updating
Knowledge distillation for LLMs often involves training smaller student model to emulate the teacher model's behavior. However, it is inefficient to use the entire dataset indiscriminately for distillation. To address this, we propose Diffiuult--waare data Updating (DiffUp) to dynamically updates the distillation data based on the sample difficulty to reduce the computational costs of distillation.
3.2.2. STRATIFIED DATA UPDATING
In the distillation process, we propose Stratified Data Up- dating strategy to dynamically select the data each epoch based on the aforementioned DDS. The core idea is to prior- itize difficult samples with high DDS value and gradually reduce the overall data volume in the next epoch for efficient distillation. Suppose we have a training dataset of size N, with the data selection ratio per epoch denoted by r. This im- plies that each epoch, we actually select and use rN data samples for distillation. Initially, the selection ratio r is set to meaning all available data is utilized at the beginning.
3

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
Outguats of Teuchhor and Studknt
Loss
valve
En denotes random selection from finite set. The same procedure is applied to the Diow set, where an element DD Diow is randomly selected from Plow- i.e., Diow ER Plow where PTow represents the collection of subsets in the power set of Diow that contain r/Dhigh elements. Thus, the activated data for this training epoch is DD' Diow U Diiighs and is easy to know that |Diocu iDisgnl T T(Dhight + (1
Imput: How many cents do 1 bave if I have 3 quarters?
Output you have 3 quarters you have 75 cents.
0.0003
3685
Outpot If you have 3 quarters you have cents.
1.1055
Imput Identify which animal species is alive or extinct:
Megalanie, Sea Turtie.
Outgut Sea Turtle is alive, Megalania is extinct.
0.0058
0,9828
r) [Dnighl = r|
Outgut Turtle is alive, Megalania extinct.
0.0057
This stratified sampling strategy ensures the distillation dataset not only prioritizes challenging samples but also retains representative easier examples, which prevents poten- tial biases toward high-DDS samples and allows the student to effectively generalize across the entire data distribution.
Imputt: What ame the worts of House Fowler?
Output Beware Talons
3,8324
0.729
Output: Firstin in Battle
3.7285
Figure 3. Examples of DDS values for different cases.
3.3. Bidirectional Diserepancy Distillation
As distillation progresses, the selection ratio r gradually decreases following a linear or cosine decay schedule. the total number of epochs is E, then the selection ratio at the e-th epoch is:
As we use our DiffUp approach to construct our distilla- tion dataset, the retained training samples are often more challenging, exhibiting greater discrepancies between the teacher and student.
linear decay schedule : r = 1
To achieve robust and stable distillation, we propose a new distillation loss function called Bidirectional Discrepancy Loss (BDL) building upon KI. divergence, but further inte- grate the two measured probability distributions, which can be formally expressed as:
(2)
me
cosine decay schedule r COS
E
This strategy allows the distillation process to focus on the most valuable data, thereby lowering training costs without sacrificing model performance.
DEDL (p. q6) DKL (((1 A)p + Aqo) (1 A)go))
(4)
Stratified sampling for diversity. Inspired by Jiang et al. (2023), to strengthen data diversity and mitigate the forget- ting problem when only difficult samples are adopted, we further propose stratified sampling strategy to mix various samples when updating distillation data. Specifically, at the beginning of each epoch, we first calculate the DDS for the whole dataset and sort them in descending order. Then, we split the dataset into two partitions by number threshold r, creating low-DDS partition Drow with r percentage of samples and high-DDS partition Daigh with r percentage of samples, respectively. Then, we randomly draw sam- ples from both partitions to construct the data subset for distillation for the current epoch.
where and qe denote the probability distributions from teacher and student, respectively. À is the coefficient to balance the contribution of p and go during the mixing. DxL denotes the standard KL divergence.
Remark. Our BDL can perform well difficult samples because it can provide robust optimization for the student. Specifically, let us denote Pm = (1 X)p Ago and Qm Xp + (1 A)ge- we compute the gradient of BDL w.r.t. student parameter A:
V DDDDL (p, qa) = VoDKL(Pm Qm)
Vo Σ Pm (z) log Pm
Formally, let us denote the whole dataset as set D. And the power set P(D) represents all subsets of D. It is obvious that Dright Drow PDD) and Drigh U Drow = D. Next, high represents the collection of subsets in the power set of Drigh that contain (1 T) (Drighl elements, i.e.,
Qm (I)
P (I)
Pm (x)
Σ [Alog Qm
+ -11 (1- 2)
V Vogo(z),
Qm (x))
C(z)
(5)
(3) where |.| returns the number of element of the set, and is to balance the selection ratio in high and low partitions. Before the start of each epoch, we compute and sort to obtain Drigh, and then randomly select an element D Diigh from
VS E p1-T, high [S] = (1 - - Drigh and S C Drighs
where C(x) determines the direction and magnitude of each term in Vogo (æ). Next, we substitute Pm and Qm. into Pm (2)/Qm mm (x) to analyze the influence of the blend distri- bution on the gradient:
Pm
(1 A)p + Ago
(6)
Phigh i.e. D' Dhigh En Pligh and D' high S Dright where
Qm Ap + (1 2)9

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
Assuming that the distribution of do may be extremely small
Algorithm Difficulty-aware data updating in our DA-KD Input: Initial selection ratio r = 1; the whole dataset D with size N = [P]; balancing coefficient т: total train- ing epochs E; teacher and student model 0y and 0g-
or large for hard samples. If go(z)) + O, we have (m
Qm
(1-A)p(a)
Ap(z) A. On the contrary, if (x))) + oo, we have
(m) Agplar - A similar phenomenon can
(x. (I-X)oo(s)
Output: Trained student model 0g.
be observed when p(z) approaches or co.
1: Iterate the model using DD and obtain initial DDS
2: for i 4- 2 to E do
This leads to our first conclusion: BDL stabilizes the train- ing by restricting the gradients without explosion van- ishing. Specifically, the range of C(x) is solely determined by X, and is independent of qo or C(x) falls between log - - 2 2 1 and À log A (see Figure 5 in the ap- pendix for illustration). Therefore, since is a pre-defined value, the range of C(x) is finite and known in advance. This prevents C(x) from becoming excessively large or small due to go or p approaching extreme less active, thereby avoiding gradient explosion or vanishing.
3: Update the selection ratio r based on Eq. (2)
4: (D), (D) + Or(D),,ss()
5: DDS (x)(D) (D)
(.(D)
6: Dord t Descending ordered 'D with updated DDS
7:
Drigh - rN]
Diow -Drallr :]
8: Pipw 1L E ((D 8.1.111 -/hhighl
Phigh t Vt E P((higgh ) s.t.|t| = (1 - T) DDight
Diow Drigh ER Piown Phigh
Furthermore, by analyzing the range and monotonicity of C(x) in Appendix A, we derive our second conclusion: BDL emphasizes the samples that have distinct output proba- bilities by teacher and student model (i.e., difficult samples). Specifically, denoting+ = = P/mm// mm/ then we have
10:
D' D high + +Diow
11:
yr, t Or (D'))
12: LADI. s- DRRI,
13: back.backward()
14: end for
15: return Student model 0g
aC(x) A 8z 2
X).
aC(x) exhibit positive value, indicating when 2 - 82
in Appendix A.
C(x) is monotonically increasing with = Pn/ Q From experiments, À 0.9 excels the best performance. Con- sidering the distribution from both teacher and student are usually small, < 0.9 takes most of cases in the distil- T-0.9
In a word, by gradient analysis, we claim that BDL not only stabilizes the optimization process inherently, but also emphasizes the hard samples that induce distinct activations. Compared to the traditional KL divergence, the blending of teacher and student distributions in the numerator and denominator results in smoother gradients, avoiding unsta- ble updates in extreme cases where the teacher and student distributions differ significantly. At the same time, the gra- dients are regularized based on both distributions, empha- sizing the importance of hard samples during training. We provide a detailed parameter analysis experiment to verify the behavior of BDL under different À to show that BDL ensures more stable and efficient update during training.
lation. Therefore, C(z) can be treated as monotonically increasing with 2. From Appendix A, we also demonstrate = Pmm/mn. Qm is also monotonically increasing with go/p when A >> 0.5. As result, C(x) is monotonically increas- ing with respect to ge/p in our setting. We empirically observe most difficult samples have qo >> p. so we have larger C(x) on these difficult samples. Although this as- sumption does not always hold, we find it is helpful for the distillation of our DA-KD. This property can enlarge the gradient of these hard samples, which enforce our BDL to pay more attention on them.
After using the proposed BDL and the DiffUp methods, we can effectively focus on more difficult but informative samples, and thus achieve efficient distillation. To provide better explanation of the process for our DA-KD framework, we present a description of the procedure in Algorithm
Mcanwhile, when À E (.,). Pm is closer to P, and @m C(x) is constantly negative, making BDL behaves similar to standard KI. divergence. When x= > 0.5, Pir. transits to L. and C(x) takes positive values to make qu increase, making BDL behaves similar to reverse KL divergence. However, x = 0.9 excels the best performance when C(z) is either positive negative, which provides a larger range of C(x), and also brings a bigger Pm.. (see Appendix A). This demonstrates that the combination of KL and reverse KL helps the training and convergence of the student. But a higher may also lead to aggressiveness and gradient explosion. The complete derivation and proof can be found
9o"
qm
p.
4. Experiments
4.1. Experiments Setup
Tasks and Datasets. To validate the effectiveness of our DA-KD framework, we conduct two categories of experi- ments: task-agnostic instruction following and task-specific experiments.
5

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
Table Results of task-agnostic instruction following on Llama2 and Qwen2.5 models. We report the average ROUGE-I. scores across five random seeds. The bold and underline indicate the best and second-bes results, respectively.
Model
#Params
Method
DollyEval
SelfInst
Super-Natural
Unnatural
VicunaEval
Avg.
7B
Teacher
29.61
20.98
34.11
35.56
19.19
27.89
SFT
27.52
19.48
30.93
32.09
18.24
25.65
KD-KL (Hinton, 2015)
27.71
19.19
30.24
31.75
17.73
25.32
KD-RKI. (Gu et al., 2024)
28.60
19.74
35.18
36.26
18.71
27.70
2.7B
SeqKD (Kim Rush, 2016)
27.78
18.64
30.95
32.14
25.53
GKID (Agarwal et al., 2023)
28.72
19.48
33.12
33.74
19.17
26.85
Distilim (Ko al., 2024)
27.93
18.90
32.03
35.42
18.76
26.61
Llama2
DA-KD (ours)
29.04
19.50
35.29
37.39
19.34
28.11
25.85
24.13
28.22
17.41
22.04
KDIKIL (Hinton, 2015)
26.17
15.13
24.97
29.22
17.34
22.57
KD-RKL (Gu al., 2024)
25,66
15.01
27.89
32.39
18.34
23.86
1.3B
SeqKI> (Kim Rush, 2016)
25.98
15.00
25.36
29.83
17.66
22.76
GKD (Agarwal et al., 2023)
26.55
16.51
27.88
30.63
17.66
23.85
Distilim (Ko al., 2024)
27.16
16.52
27.27
31.98
17.70
24.13
DA-KD (ours)
26.75
18.05
32.35
36.09
17.53
26.15
7B
Teacher
31.37
25.98
43.61
40.19
23.15
32.86
SFT
28.05
21.12
34.21
37.06
19.22
27.93
KD-KL (Hinton, 2015)
28.01
21.97
37.24
37.94
19.98
29.03
KD-RKL (Gu al., 2024)
28.40
20.60
37.93
39.46
21.18
29.51
1.5B
SeqKD (Kim Rush, 2016)
28.21
20.98
35.79
36.52
21.46
28.59
GKD) (Agarwal et al., 2023)
28.89
23.78
44.76
39.56
23.80
32.16
Distillm (Ko et al., 2024)
29.38
21.74
40.55
37.41
22.96
30.41
Qwen2.5
DA-KI) (ours)
28.91
24.08
45.73
42,31
24.04
33.01
SFT
25.98
14.89
30.74
32.82
16.54
24.19
KD-KL (Hinton, 2015)
24.59
16.07
30.00
30.95
17.57
23.84
KD-RKL (Gu et al., 2024)
25.09
15.30
29.06
32.01
19.08
24.11
0.5B
SeqKI) (Kim Rush, 2016)
26.28
17.40
29.57
31.64
18.90
24.76
GKD (Agarwal et al.. 2023)
26.24
18.74
38.45
33.31
21.56
27.66
Distilim (Ko al., 2024)
27.72
19.55
42.19
37.80
23.94
30.24
DA-KD (ours)
27.36
19.74
43.63
39.52
21.97
30.44
For instruction following experiments, we choose the databricks-dos (Conover al., 2023) dataset pro- cessed by Gu et al. (2024) for distillation. Then, we evaluate the trained student models on five instruction-folloiig datasets: Dolly evaluation (Conover et al., 2023), Self- Instruct (Wang et al., 2022a). Super-Natural Instructions (Wang et al., 2022b), Unnatural Instruction(Honovic 2022) and Vicuna evaluation(Chiang et al., 2023). The eval- uation metric is ROUGE-L (Lin, 2004). To mitigate the randomness, we report the average ROUGE-L score under five different random seeds.
lowing, we employ LLaMA2-7B (Touvron al., 2023) and Qwen2.5-7B (Yang et al., 2024a) teacher models, with
.3 (Xia et al., 2023) and Qwen2.5-1.5B/0.5 as their respective students. For task- specific experiments, we utilize LLaMA2-7B and Qwen2.5- 7B as teachers, distilling knowledge into LLaMA2-1.3B and Qwen2.5-1.5B. respectively. Additionally, we include LLaMA3.2-33 (Dubey al., 2024) the teacher model with LLaMA3.2-1B as the student.
al.,
Implementation details. We train all models for 10 epochs using batch size of 8. We use the AdamW optimizer and cosine learning rate scheduler. The initial learning rate set as le-5. our implementation, we set 7 and À as 0.1 and 0.9, respectively. We use cosine decay schedule gradually reduce the data selection ratio
For task-specific experiments, we consider two distinct tasks for evaluation: text summarization using SAM- Sum (Gliwa et al., 2019), and mathematical reasoning with GSM8K (Cobbe al., 2021). We use ROUGE-L score and zero-shot accuracy for measuring, respectively.
Comparison Methods. We compare our approach with six methods: SFT, KD-KL (Hinton, 2015), KD-RKL (Gu et al., 2024), SeqKD (Kim & Rush, 2016), GKD (Agarwal et al.,
Models. We evaluate our DA-KD using multiple teacher- student model pairs. For task-agnostic instruction fol-
6

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
2023), Distillm (Ko al., 2024). SFT means directly fine-tune the student without distillation.
Table 2. Results of task-specific experiments Llama2-1.33 Qwen2.5-1 and (lama.22-1B. The bold and underline mark- ings indicate the best and second-best results, respectively,
4.2. Main Results
Model
Method
SAMSum
GSM8K
Task-Agnostic Instruction-Follon Follooiwn......iingg Table presents the results of task-agnostic certuuctioofoolowiing experiments. Our proposed DA-KD method achieves the highest ROUGE- L scores most of the evaluation datasets, consistently outperforming other methods across different model types and sizes. For instance, in the case of the Llama2-2.7B model, our approach achieves an average ROUGE-L score of 28.11,, surpassing the state-of-ther see-theeart art methods KD-RKL (27.70) and GKT) (26.85). Even the smallest Qwen2.5- 0.5B model, our method attains average ROUGE-L score of 30.44, exceeding Distillm (30.24) and all other baselines.
Teacher (7B)
40.84
38.67
SFT
37.11
11.52
KD-KL (Hinton, 2015)
37.80
11.37
Llama2
KD-RKL (Gu et al., 2024)
38.45
9.40
GKD (Agarwal et al., 2023)
38.39
14.18
Distillm (Ko al., 2024)
38.73
12.59
DA-KD (ours)
39.20
14.56
Teacher (7B)
39.70
71.72
SFT
37.74
41.77
KD-KL (Hinton, 2015)
37.91
44,50
Qwen2.5
KD-RKL (Gu al., 2024)
38.75
44,88
GKD (Agarwal et al., 2023)
38.89
50.80
Notably, both Llama2-2.77 and Qwen2.5-1 1.oon25511.50 51,5B outperform their respective teacher models in terms average perfor- mance. We hypothesize that this improvement may come from the more informative training datasets brought by our DiffUP approach and the more stable optimization process caused by BDL.
Distillm (Ko et al., 2024)
39.21
46.70
DA-KD (ours)
40.05
54.66
Teacher (3B)
33.47
37.83
SFT
31.78
12.59
KD-KI. (Hinton, 2015)
29.57
7.28
Llama3.2
KD-RKI. al., 2024)
30.51
14.03
GKD (Agarwal et al., 2023)
30.43
17.82
Moreover, we successfully compress the Qwen2.5 model from 7B to 0.5B (14x reduction in disk storage) with only a 2.42 drop average performance. Furthermore, com- pressing Qwen2.5 from 7B 1.5B (4.7x storage reduction) only preserves performance but even yields slight improvement. Similar results are observed in the Llama2 experiments. It further demonstrates the effectiveness of our DA-KD framework, highlighting the potential of our proposed distillation framework in producing smaller large models with minor computational resources and time costs.
Distillm (Ko et al., 2024)
32.53
20.17
DA-KD (ours)
32.92
22.37
Table 3. Training efficiency comparison.
Traing time (minutes)
Method
Iterations
SFT
3570
62.81
KD-KL (Hinton, 2015)
3570
140.75
KD-RKL (Gu et al., 2024)
3570
141.40
Task-Specific Experiments. Table 2 reports the results of task-specific experiments conducted the Llama2-1.3B, Qwen2.5-1.5B and 1-Bma3.2-1BB for text summarization and mathematical reasoning. Our DA-KD method achieves the best performance across both tasks, demonstrating its effectiveness in diverse task-specific scenarios. Specifically, for the text summarization task on the SAMSum dataset, our approach consistently surpasses previous tfate-oftthe methods. For example, Llama2-1 1.3B and Llama3.2- 1-1--1..111 BB, DA-KD achieves 39.20 and 32.92 respectively, outper- forming Distillm (38.77 and 32.53). Notably, Qwen2.5, the student model distilled by our DA achieves a score of 40.05, surpassing even the 7B teacher model. The re- markable result underscores DA-KD's s ability to empower a lightweight student model to retain and even enhance high-quality text generation capabilities.
SeqKI (Kim & Rush, 2016)
3570
159.73
GKD (Agarwal al., 2023)
3570
408.24
Distillm (Ko al., 2024)
3570
213.34
DA-KD (ours)
1963
106.35
However, performance degradation is still observed for smaller models GSM8K. One possible explanation is the inherent complexity of mathematical reasoning, requiring stronger logical inference and multi-step reasoning capabili- ties. Consequently, distilling such knowledge into smaller model is inherently more challenging, and remains an open problem for future work to enhance knowledge transfer in mathematical reasoning.
Training costs. As our objective is develop an effective and efficient distillation framework for large language mod- els, we also evaluate the training efficiency of our method compared to existing work. Table presents the number of training iterations and time required for end-to-end distil- lation. All the test cases compress Llama2-7B into a 2.7B model using four NVIDIA A800 GPUs.
For the mathematical reasoning task the GSM8K dataset, our method achieves a zero-shot accuracy of 54.66% Qwen2.5, 14.56% on Llama2, and 22.37% on Llama3.2, demonstrating significant improvements over competitors.

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
Table 4. Ahlation study of our Dif method.
Table 5. Comparison results for different distillation loss functions combined with DiffUP. We report the average ROUGE-I scores across five random seeds. The bold and underline markings indi- cate the best and second- best results, respectively.
Method
Dolly
SelfInst
Super-N
Unnatural
Vicuna
Avg.
DiffUp
26.75
18.05
32.35
36.09
17.53
26.15
w/o DDS
26.05
16.70
31.27
33.86
17.53
25.08
w/o SDU
26.65
17.37
31.11
34.56
16.54
25.24
w/o DitlUp 26.73
18.51
32.07
35.64
16.58
25.91
Loss
Dolly
SelfInst
Super-N
Unnatural
Vicuna
Avg.
KL
23.69
14,40
24,97
26,16
15.98
21.04
We take SFT as the bascline performance since it directly fine-tunes the student model on the corresponding dataset without knowledge distillation. Compared to all the other typical and commonly-used distillation frameworks, our DA-KD significantly reduces the computational cost of dis- tillation while maintaining superior performance. Specifi- cally, DA-KD requires only 1,963 training iterations, which is 55% fewer compared to all other distillation methods.
RKL
25.35
14.72
27.31
32.15
18.18
23.54
JSD
24.80
16.86
31.18
31.70
16.79
24.26
SKI.
26.02
17.31
30,76
33.02
17.72
24.96
SRKL
26.43
16.91
29.94
34,56
17.12
24.99
BDL
26.75
18.05
32.35
36.09
17.53
26.15
35
Unnatural
Avg.
Std.
Super-N
In terms of training time, our DA-KD method requires only 26. 1 1% of the GPU hours compared to GKD (106.35 vs. 408.24 24 minutes) and 49.9% compared to Distillm (106.35 vs. 213.34 minutes), demonstrating remarkable efficiency compared to other methods. That is because as the train- ing process goes on, we use fewer data with hard samples while filtering out the most simple ones, which brings sig- nificant time saving for each single epoch. Methods such as GKD and Distillm incur substantial computational costs outputs during train-
Dolly
25
zeho
Vicuna
SolfInst
Values ofA
Figure 4. Hyperparameter analysis of X in BDL.
whole DiffUp method and use the entire dataset during dis- tillation process ("w/o DiffUp"), the performance degrades 0.24 compared ours, indicating that dynamically updating the dataset based on the complexity of samples improves the model's performance.
because they rely on suudent-generaan ing, which increases processing overhead. Although will introduce an extra computational cost when computing DDS in DA-KD (for example, the total time of 106.35 minutes in Table 3 consists of 1 minutes of DDS computation and 78.04 minutes of distillation), we can achieve a reduc- tion of 55% in training iterations by using DDS to filter the distillation data. As a result, we can still achieve an overall reduction in computational cost compared to existing KD methods.
4.3.2. EFFECT OF BDL
Comparisons to Other Distillation Losses. To verify the effectiveness of our proposed BDL, we conduct exper- iments by replacing BDL with other objective functions: KL (Hinton, 2015), KL (Gu et al., 2024), JSD (Agarwal et al., 2023), SKL (Ko et al., 2024), and SRKL (Ko et al., 2024). As shown in Table 5, BDL achieves the highest av- erage ROUGE-L score of 26. 26.15, surpassing SRKI. by 1.16, and has even larger margin compared to other loss functions, which demonstrates our BDL can stabilize the training and also learn more information in difficult samples.
4.3. Ablation Study
In this section, we distill Llama2-7B to 1.3B as an exam- ple and conduct extensive ablation studies to validate the effectiveness of each component of our DA-KD approach.
4.3.1. 1. EFFECT OF DIFFUP
Parameter Analysis on A. We conduct the experiment to investigate the performance with different values of X in BDL, and the result is shown in Figure 4, which is con- sistent to our derivation in Appendix A. When X = 0.5, the gradients will have relatively small coefficients which lead to gradient vanishing and performance degradation. When À E (0.1, BDL behaves similar standard KL divergence and constantly improves the model performance. Moreover, when X E (0.6, 0.9), it consistently leads to per- formance gains, and reaches its optimal when À = 0.9, which demonstrates that the combination of KL and reverse KL helps the model to converge. The results in Figure 4 con- firm the conclusions of our theoretical derivation. However,
To verify the effectiveness of our proposed difficulty metric DDS, we first remove each component in DiffUp. The result denoted as "w/o DDS in Table 4 means we randomly select data for distillation at each epoch instead of according to their DDS, showing that our DDS metric can bring 1.07 improvement on the average ROUGE-L score, highlighting the importance of DDS in ordering and selecting challeng- ing samples for distillation. For comparison denoted as "w/o SDU", we remove the mixing strategy, making all the samples in the distillation dataset being sampled from the high-DDS partition. So SDU brings 0.91 improvement on the average ROUGE-L score. Finally, when we remove the
8

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
determining the optimal X, well as fine-grained tuning of A for the best efficiency, remains an open problem.
ing: Instruction data selection for tuning large language
models. arXiv preprint arXiv:2307.06220, 2023.
Chen, L., Li, S., Yan, J., Wang. H., Gunaratna, K., Yadav, V., Tang, Z.., Srinivasan, V., Zhou, T., Huang, H., et al. Alpagasus: Training a better alpaca with fewer data, arXiv preprint arXiv:2307.08701, 2023.
5. Conclusion
In this paper, we introduce Diffe Knowledge Distillation (DA-KD), a novel framework designed to en- hance the efficiency and effecttivnness of LLM distillation. It includes Difficulty-Aware Data Updating (DiffUp) to filters easy samples using a Distillation Difficulty Score (DDS), and Stratified Data Updating (SDU) to mitigates catastrophic forgetting by mixing samples from all DDS partitions. Additionally, we propose Bidirectional Discrep- ancy Loss (BDL) to stabilize student training by preventing gradient explosion or vanishing when distilling difficult samples. Extensive experiments demonstrate that DA-KD outperforms state-of-the- art knowledge distillation methods and even surpasses the teacher model in several datasets with only half training cost, making it highly effective solution for efficient LLM distillation.
Chiang, W.-L.., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, I.., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-sourge chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicua. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C.,, and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction- tuned 1lm. Company Blog of Databricks, 2023.
One of the limitations of our DA-KD is that its performance depends on the teacher model quality. If the teacher model provides incorrect prediction, the DDS mechanism may misidentify difficult samples, potentially affecting student performance.
Dong, X., Bao, J., Zheng, Y., Zhang, T., Chen, D., Yang, H., Zeng, M., Zhang, W., Yuan, L., Chen, D., et al. Maskelip: Masked seff-distillation advances contrastive language- image pretraining. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10995-11005, 2023.
Acknowledgements
This work was supported by the National Science and Tech- nology Major Project 20022D00114055, Beijing Municipal Science and Technology Project (No. Z231 1231100010323002), National Natural Science Foundation of China (Nos.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Daale, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
62306025, 92367204, 62476018).
Impact Statement
Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Samsum corpus: A human-annotated dialogue dataset for abstrac- tive summarization. arXiv preprint 1rXiv:1911.122 2019.
This paper presents work whose goal to advance the field of LLM knowledge distillation. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gu, Y., Dong, L., Wei, F., and Huang. M. Minillm: Knowl- edge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
References
Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, Bachem, O. Gkd: Generalized knowledge distilla-
Guo, J., Ouyang, W., and Xu, D. Multi-dimeeniional pruning: A unified framework for model compression. In CVPR, 2020.
M., and and tion for auto-regressive sequence models. arXiv preprint arXiv:2306. 13649, 2023.
Guo, J., Liu, J., and Xu, D. Jointpruning: Pruning networks along multiple dimensions for efficient point cloud pro- cessing. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Guo, J., Liu, J., and Xu, D. 3d-pruning: A model com- pression framework for efficient 3d action recognition.
Cao, Y., Kang, Y., Wang, C., and Sun, L. Instruction min-

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
IEEE Transactions on Circuits and Systems for Video Technology, 32(11):8777-8729 2022.
Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L.,, Cheng, N., Wang. J., Zhou, T., and Xiao, J. From quantity quality: Boosting I1m performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. 2023.
Guo, J., Xu, D., and Lu, G. Chanet: Towards complexity and bitrate adaptive deep image compression using a single network. IEEE' Transactions on Image Processing, 2023a.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text sunmarization branches out, pp. 74-81, 2004.
Guo, J., Xu, D., and Ouyang, W. Multidimensional pruning and its extension: A unified framework for model com- pression. IEEE Transactions on Neural Networks and Learning Systems, 2023b.
Lv, C., Chen, H., Guo, J., Ding, Y., and Liu, X. Ptq4sam: Post-training quantization for segment anything. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 15941-15951, 2024.
Guo, J., Wu, J., Wang, Z., Liu, J., Yang, G., Ding, Y., Gong, R., Qin, H., and Liu, X. Compressing large language mod- els by joint sparsification and quantization. In Forty-first International Conference on Machine Learning, 2024.
Naeem, M. F., Xian, Y., Zhai, X., Hoyer, L.., Van Gool, L., and Tombari, F. Sile: Improving vision language pretraining with deff-distillation. In European Conference on Computer Vision pp. 38-55. Springer, 2025.
Hinton, G. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.0253,, 2015.
Radford, A. Improving language understanding by genera- tive pre-training. 2018
Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnat- ural instructions: Tuning language models with (almost) no human labor. arXiv preprint wrXiv:222.00668 2022.
Tao, R., Wang, T. Wu, Z., Liu, C., Liu, A., and Liu, X. Few-shot x-ray prohibited item detection: A benchmark and weak- feature enhancement network. Proceedings of the 30th ACM international conference on multimedia, pp. 2012 2020, 2022.
Huang, Y., Wang, Z., Gong, R., Liu, J., Zhang, X., and Zhang, J. Harmonica: Harmonizing training and infer- ence for better feature cache in diffusion transformer acceleration. arXiv preprint arXiv:2410.0172. 2024.
Tao, R., Le, M., Tan, C., Liu, H., Qin, H., and Zhao, Y. Oddn: Addressing unpaired data challenges open-world deep- fake detection on online social networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 39, pp. 799-807, 2025a.
Jiang, Y., Chan, C., Chen, M., and Wang, W. Lion: Adver- sarial distillation of proprietary large language models. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3134-3154, 2023.
Tao, R., Wang, H., Guo, Y., Chen, H., Zhang, L., Liu, X., Wei, Y., and Zhao, Y. Dual-view x-ray detection: Can ai detect prohibited items from dual-view x-ray images like humans? In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025b.
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, X., Liu, X., and Jin, X. Ragcache: Efficient knowledge caching for retrieval- augmented generation. arXiv preprint arXiv:2404. 12457, 2024.
Kim, Y. and Rush, A. M. Sequence-level knowledge distil- lation. arXiv preprint arXiv:1606.07947 2016.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint w:Xiv:2307.09288 2023.
Ko, J., Kim, S., Chen, T., and Yun, S.-Y. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03880 2024.
Wang. J., Zhang, B., Du, Q., Zhang, J., and Chu, A survey on data selection for Imm instruction tuning. arXiv preprint arXiv:2402.05723 2024a.
Lan, W., Cheung. Y-m... Xu, Q.. Liu, B., HU, Z., Li, M., and Chen, Z. Improve knowledge distillation via label revi- sion and data selection. IEEE Transactions on Cognitive and Developmental Systems, 2025.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D.,, and Hajishirzi, H. Self-instruct: Aligning language models with self- gelf-generated instructions. arXiv preprint arXiv:2212.10560 2022a.
Li, L., Lin, Y., Ren, S., Li, P., Zhou, and Sun, X. Dynamic knowledge distillation for pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 379-389, 2021.
Wang. Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran,
10

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
A. S., Naik, A., Stap, D., al. Super-nattraalnsttutiios::: Generalization via declarative instructions 1600+ nlp tasks. arXiv preprint arXiv:2204.0777 2022b.
Wang, Z., Guo, J., Gong, R., Yong. Y., Liu, A., Huang, Y., Liu, J., and Liu, X. Ptsbench: A comprehensive post-training sparsity benchmark towards algorithms and models. In ACM Multimedia 2024, 2024b.
Xia, M., Gao, T., Zeng, 7.., and Chen, Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprini arXiv:2310.0660 2023.
Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, survey on knowledge
C., Tao, D., and Zhou, distillation of large language models. arXiv preprint vrXiv:2402.13111 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024a.
Yang, G., He, C., Guo, J., Wu, J., Ding, Y., Liu, A., Qin, H., Ji, P., and Liu, X. LImebench: Benchmarking large language model compression for efficient deployment. NeurIPS, 2024b.
Yin, Z., Xing, E., and Shen, Z. Squeeze, recover and rela- bel: Dataset condensation at imagenet scale from a new perspective. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, L., Bao, C., and Ma, K. Self-distillation: Towards efficient and compact neural networks. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44 (8):4388 4403, 2021.
Zhao, B., Mopuri, K. R., and Bilen, H. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05925 2020.
Zheng, X.-Y., Lee, M.-C., and Hong, Y.-W. P. Knowledge caching for federated learning. In 2021 IEEF Global Communications Conference (GLOBECOM), pp. 1-6, 2021. doi: N011090GLOBEEC0M4610020000000000
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.

---

DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
A. Details about the BDL
A.1. Proof for BDL
First, we discuss the monotonicity with respect to qo/p for the following equation:
Pm (1- A)p + Ago
(8)
Qm Ap + (1 X)go
Proof.
To simplify the analysis, let 4u (representing the relative ratio of go and p). Then we have
p
Pm +1 Ar
(9)
Qm X+ (1
Next, differentiate
Pm. with respect to x: Qm
i Pm
X(1 + (1 A)æ) ((1 A) + Xx= X)
Qm
(10)
(A + (1 - ))z)²
21 - (A+ + (1-A)x)22
(11)
where the denominator (A + (1 ) is always positive, and the numerator is a constant 2 1. When > 0.5, pim.
21- - 1 > 0, which implies
> 0. meaning that Pm. is monotonically increasing with respect to z. On the contrary, Ün
Q.
when À < 0.5, Qm
P.. is monotonically decreasing. And when 0.5, Pm is a constant. Due to the monotonic function of Q.. Eq. (9), is easy find the boundary values of Em, As + (i.e., qo p), Qm T while as + (i.c.. Qm
qe < p),
P.. Qm A.
1-1 + 2
log
log 1 2
-20
-60
Figure Curves of the two boundaries of C(x).
Next, we prove the monotonicity of C(x) in Eq. (5) that
Pm (x)
P
C(x) = A log
+ X - (1 X)
(12)
Qm
Qm m (x)
Proof. We first differentiate C(x)
8C(x)
X (1 A);
(13)
Øz
where 2 Pmm/// Qm And the second derivative C(x)
(C(x)
(14)
dz2 22
12

---

analysis above, moderate is DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
which constantly below 0 since À 0. Therefore, when = - i.e., Pu - C(x) reaches its maximum A Alog r-x When À < 0.5, X log - < O, constantly brings negative C(xx). It always makes Qm to increase and match Pm.. which is how standard KL Divergence works. However, when À > 0.5, C(r) E (Alog - + 2 X A log T-x). By drawing the curves of the two boundaries in Figure 5, we can see that when 0.5 X <~ 0.7 (value that around 0.7), C(x) is constantly positive, which is how reversed KI. Divergence works in most cases. However, if ~ 0.7 < < 1, the range of ((x) includes both positive and negative values. Since we blend probabilities qo and p into P.. and Q it can be regarded as combination of KL and reversed KL. And it is easy to imagine that when larger À is applied, the range of C(x) is larger, but if À is close to 1, it behaves similar to reversed KL Divergence.
Qm
In a word, we can draw two conclusions. First, given a certain constant X, C(x) falls in the range between À log 1- and À Alog -ーx_ which provides finite and known coefficients on the gradients Vogo (z). It avoids the gradient explosion issue during training in case of the outliers in student models' output probability. Second, when the predefined À E (0.7, 1), Cxx) can be positive or negative values, allowing the Oo to increase or decrease flexibly, While when being close to 1, has the probability to give an extreme regularization.
X +
A.2. Further Discussion on Parameter
Based on above conclusion, we can understand how the loss function BDL functions within the method: when the difference of teacher and student model enlarges regarding to sample æ. the output probabilities of the two networks differs, leading to larger qa p. And through the blending of probabilities in BDL, the gradients in the backpropagation will be enhanced by a larger coefficient C(ar), which emphasizes the importance of the sample Based required.
Our parameter analysis experiments in Sec. 4.3.2 show that BDL consistently brings improvements of most À except 0.5. It is easy to understand that (xx) has the smallest range near ( when X = 0.5, which leads to the gradient vanishing issue. When À E (0, 0.5), Pm. is closer to 2, and C(x)) is negative, making BDL behaves similar to standard KL divergence. When À (0.5,~ ~ 0.7), Q. reverse KL divergence. However, X p = 0.9 excels the best performance, which demonstrates that the combination of KL and reverse KL helps the training and convergence of the model. Mcanwhile, larger À allows a larger range of C(x), and also brings a bigger P.,. Based on the monotonicity of Pm and C(x), we can draw that samples has distinct output Qm
Qm
Pa transits to 22, and C(x) takes positive values make qe increase, making BDL behaves similar
Qm probabilities from teacher and student will have larger C(x) with a larger X, which means the gradient will be enlarged and emphasized.
Based on this, we hypothesize that our method inherently addresses some intrinsic issues in existing distillation frameworks, such as stabilizing gradients and emphasizing hard samples. However, determining the optimal value of À remains an open question.
13


Reviews:
[{"review_id": "hfnfWHcxOv", "reviewer": "Reviewer_BqdN", "summary": "The paper introduces DA-KD for knowledge distillation tailored to LLMs. It proposes two main innovations. First, a Difficulty-Aware Data Updating strategy computes Distillation Difficulty Score for each sample, to filter out easy examples and focus training on challenging ones, and employs a Stratified Data Updating strategy to ensure diversity. Second, the paper proposes a new loss function, called Bidirectional Discrepancy Loss, which blends the teacher’s and student’s probability distributions to stabilize gradients and emphasize hard samples. Extensive experiments on both task-agnostic instruction following and task-specific scenarios show that DA-KD outperforms existing state-of-the-art KD methods while reduce the training cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for recognizing the strengths of our work, including “addresses an important bottleneck in LLM distillation” and “experimental evidence supports the claims\". Your suggestions are very helpful for improving our paper. We have carefully addressed your concerns below:\n\n> Q1: It would benefit from a discussion of alternative methods that focus specifically on dynamic dataset selection in KD. (Revisit the power of vanilla knowledge distillation: from small scale to large scale. Advances in Neural Information Processing Systems 36 (2023): 10170-10183.)\n\nA1: Thank you for the suggestion. The suggested reference [1] aims to prove that VanillaKD can achieve comparable performance with other KD methods like DKD[2] and DIST[3] when using data augmentation and stronger training strategies instead of data selection. In contrast, we introduce a dynamic dataset selection mechanism to make the distillation process more efficient. We will add the discussion with the work [1] in our final version. \n\n[1] \"Revisit the power of vanilla knowledge distillation: from small scale to large scale.\" Advances in Neural Information Processing Systems 36 (2023): 10170-10183.\n\n[2] \"Decoupled knowledge distillation.\" Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.\n\n[3] \"Knowledge distillation from a stronger teacher.\" Advances in Neural Information Processing Systems 35 (2022): 33716-33727.\n\n> Q2: During the Difficulty-aware Data Updating process, in order to obtain the difficulty scores for each sample, it is needed to calculate the cross-entropy loss of the teacher and student models separately for each sample, which may bring some computational budget.\n\nA2: Although it will introduce extra computational cost when computing DDS, we can achieve a 55% reduction in training iterations by using DDS to filter the distillation data (Table 3 in the main paper). As a result, we can still achieve overall reduction in computational cost compared to existing KD methods. \n\nTo better analyze the efficiency of our DA-KD framework, we list the training time of different distillation methods when distilling Llama2-7B into a 2.7B. Our DA-KD requires only 26.1% of the training time compared to GKD (106.35 vs. 408.24 minutes) and 49.9% compared to Distillm (106.35 vs. 213.34 minutes), demonstrating the efficiency of the proposed method. We will discuss the data updating cost in our final version.\n\n| Method   | Training time (minutes)                                   |\n| -------- | --------------------------------------------------------- |\n| KD-KL    | 140.75                                                    |\n| KD-RKL   | 141.40                                                    |\n| SeqKD    | 159.73                                                    |\n| GKD      | 408.24                                                    |\n| Distillm | 213.34                                                    |\n| DA-KD    | 106.35 (28.31 for data updating + 78.04 for distillation) |\n\n> Q3: It would be helpful to include a discussion on potential limitations.\n\nA3: Thanks for the suggestion! One of the limitations of our DA-KD is that its performance depends on the teacher model quality. If the teacher model provides incorrect answers, the DDS mechanism may misidentify difficult samples, potentially affecting student performance. We will discuss these limitations in our final version and try to solve it in our future work.\n\n> Q4: There are some typos or grammar errors: ... ...\n\nA4: Thank you for pointing it out. We will correct all mentioned typos and proofread carefully.", "claims_and_evidence": "The evidence is provided via quantitative comparisons against several baselines on multiple datasets. Overall, the experimental evidence supports the claims.", "methods_and_evaluation_criteria": "The proposed methods are well motivated. The evaluation criteria (ROUGE-L, accuracy, training time) are appropriate in the context of distillation for LLMs.", "theoretical_claims": "The derivations are technical and appear correct. Overall, the theoretical analysis strengthens the motivation for BDL.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "DA-KD builds on established ideas in KD while incorporating data selection. The framework is positioned as a solution to reduce training cost while maintaining or even improving performance.", "essential_references_not_discussed": "it would benefit from a discussion of alternative methods that focus specifically on dynamic dataset selection in KD. (Revisit the power of vanilla knowledge distillation: from small scale to large scale. Advances in Neural Information Processing Systems 36 (2023): 10170-10183.)", "other_strengths_and_weaknesses": "Strength:\n\nThe approach addresses an important bottleneck in LLM distillation by reducing training cost.\n\nWeakness:\n\nDuring the Difficulty-aware Data Updating process, in order to obtain the difficulty scores for each sample, it is needed to calculate the cross-entropy loss of the teacher and student models separately for each sample, which may bring some computational budget.", "other_comments_or_suggestions": "1. It would be helpful to include a discussion on potential limitations.\n2. There are some typos or grammar errors: \n\n1) in line 130, “the distillation dataset will progressively reduced” should be “the distillation dataset will be progressively reduced”.\n2) in line 161, “only difficult samples are adopt” should be “only difficult samples are adopted”.\n3) in the title of Table 5, “disitllation” should be “distillation”.", "questions_for_authors": "See the weaknesses and comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOoCAhpmzy", "reviewer": "Reviewer_fVGK", "summary": "This paper presents an efficient knowledge distillation method. Specifically, it introduces a Difficulty-aware Data Updating strategy to identify challenging samples and proposes a novel loss function, Bidirectional Discrepancy Distillation, to reduce size of training samples required, thereby improving training efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for recognizing the strengths of our work by stating “simple yet effective”, \"well-written\", and \"comprehensive experiments\". Your thoughtful suggestions help a lot to improve our work. We have carefully addressed your concerns below:\n\n> Q1: DDS Computation & Dataset Partitioning (Weakness 1)\n\nA1: The DDS computation and dataset partitioning occur before each epoch, dynamically updating difficulty scores based on the student model’s state. While it introduces extra computational cost, we can achieve a 55% reduction in training iterations. As a result, we can still achieve an overall reduction in computational cost compared to existing KD methods, which is shown in Table 3 in our paper. \n\n> Q2: Prior Work on Data Selection in KD (Weakness 2)\n\nA2: Thanks for the suggestion and we apologize for the oversight in our discussion of the related works. Prior data selection methods are either solely based on student model [4] or solely based on the teacher model [5], ignoring their discrepancy. In contrast, our DDS metric integrates both teacher and student distributions. It selects samples that teacher can provide informative guidance to student. We will supplement a detailed discussion in our related work. \n\n> Q3：Generalisability of DDS to other domains (Weakness 3)\n\nA3: To evaluate DDS in a different domain, we conduct experiments on CIFAR-100 using ResNet-50 as the teacher and ResNet-18 as the student. We compare three settings: (1) using the full dataset, (2) randomly selecting 50% of the data, and (3) selecting the top 50% of the data based on DDS. The results  below demonstrate the effectiveness of DDS.\n\n| data                             | accuracy |\n| -------------------------------- | -------- |\n| full training dataset            | 74.80    |\n| random 50% of the data           | 73.66    |\n| top 50% of the data based on DDS | 74.04    |\n\n> Q4: Comparison with SE-KD [3] (Questions 1)\n>\n> [3] Song, Yuncheng, et al. \"Self-Evolution Knowledge Distillation for LLM-based Machine Translation.\" 2024.\n\nA4: SE-KD is conceptually related to our work by both proposing a difficulty metric of the samples and mixing the distributions of teacher and student. However, our method is intrinsically different from this work:\n\n- Different objectives\n\n  SE-KD aims to improve the LLM distillation performance, by dividing samples into two categories and learning with different metrics.  \n\n  However, DA-KD designs an efficient KD approach, which gradually reduces the data volume, and thus reduces the computational overhead.\n\n- Different criterion for dividing samples\n\n  SE-KD utilizes KL divergence between the student distribution and a target distribution to quantify the learning difficulty of samples. It does not consider whether a sample is well-learned by teacher. \n\n  However, DA-KD uses the ratio between the student and teacher losses to measure the difficulty of samples. It only selects the samples that are well-learned by the teacher but poorly-learned by the student. \n\n- Different distributions mixture in loss\n\n  SE-KD mixes the student distribution and target distribution unidirectionally, which is similar to the SKL in Distillm[a].\n\n  Differently, DA-KD achieves a bidirectional mixing of teacher and student distributions, leading to a more stable optimization process, which is explained in Sec. 3.3.\n\nWe perform an experimental comparison with SE-KD, using LLaMA2-7B (teacher) and LLaMA2-1.3B (student). The results are listed below. Our DA-KD outperforms SE-KD with less training time (about 16% reduction) while obtains better performance.\n\n[a] \"Distillm: Towards streamlined distillation for large language models.\" arXiv:2402.03898 (2024).\n\n| Method | Training Time (mins) | Dolly | Self  | S-N   | Un    | Vic   | Avg.  |\n| ------ | -------------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| SE-KD  | 86.84                | 25.89 | 17.19 | 32.21 | 36.82 | 16.46 | 25.71 |\n| DA-KD  | 72.12                | 26.75 | 18.05 | 32.35 | 36.09 | 17.53 | 26.15 |\n\n> Q5: Theoretical Explanation of q >> p (Questions 2)\n\nA5: In KD, the teacher model is well-learned and often produces a probability distribution that is smoother and more refined. Conversely, the student, being smaller or less trained, tends to be more overconfident in its incorrect predictions [b], or has less diversity in its output contents [c], resulting in a sharper distribution. This is characterized by high confidence concentrated distribution on fewer options. Therefore, for difficult samples, the probability of the wrong class predicted by student (represented by q) is significantly higher than that predicted by teacher (represented by p).\n\n[b] \"On student-teacher deviations in distillation: does it pay to disobey?.\" NeuIPS 2023.\n\n[c] \"Teaching large language models to reason with reinforcement learning.\" arXiv:2403.04642 (2024). \n\n> Q6: Several typos \n\nA6: Thank you for pointing it out. We will correct all mentioned typos.", "claims_and_evidence": "Yes, they are.", "methods_and_evaluation_criteria": "Yes, proposed methods do.", "theoretical_claims": "Yes, I checked Section 3.3, the remark of the proposed loss function.", "experimental_designs_or_analyses": "Yes, I did. I checked experiments setup and all the experiments results.", "relation_to_broader_scientific_literature": "This paper proposes a data selection strategy for knowledge distillation (KD) and a new loss function.\n\nThe experimental results of this paper are consistent with the conclusions of [1][2], indicating that the model does not require the entire training dataset.\n\nAdditionally, the idea of mixing the probability distributions of the teacher and student before computing the KL divergence, as well as the corresponding experimental results, are similar to those in [3].\n\n[1] Zhou, Chunting, et al. \"Lima: Less is more for alignment.\" NeurIPS, 2023.\n\n[2] Li, Ming, et al. \"Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.”2024.\n\n[3] Song, Yuncheng, et al. \"Self-Evolution Knowledge Distillation for LLM-based Machine Translation.\" 2024.", "essential_references_not_discussed": "The paper states in the related work section that “the application of data selection methods in knowledge distillation remains unexplored.” However, several studies have already explored this direction. For instance, [4] proposed an uncertainty-based data selection method for knowledge distillation of LLMs, while [5] also introduced a data selection approach specifically designed for knowledge distillation. These existing works should be discussed.\n\n[4]Li, Lei, et al. \"Dynamic knowledge distillation for pre-trained language models.\" 2021.\n\n[5]Lan, Weichao, et al. \"Improve Knowledge Distillation via Label Revision and Data Selection.”, 2024.", "other_strengths_and_weaknesses": "Strengths：\n\nThe method is simple yet effective, allowing for a reduction in training time while maintaining performance.The paper is well-written with a clear and logical structure.The authors provide comprehensive experiments to validate the effectiveness of both the data selection strategy and the proposed loss function.\n\nWeakness：\n\n1. In Line 182, the authors state, “we first calculate the DDS for the whole dataset and sort them in descending order.” I would like to clarify whether the computation of DDS and the partitioning of the dataset are performed only once at the beginning of training or updated before each epoch. If it is computed only once, given that DDS values may change as training progresses, how does the method account for this dynamic variation?\n\n2. As mentioned in “Essential References Not Discussed” and “Questions for Authors”, there is a lack of references to and comparisons with some closely related works.\n\n3. The computation of DDS relies solely on the cross-entropy loss between the two models (teacher and student) and the ground truth for a given sample  x . Given this formulation, the data selection method is not inherently limited to LLMs. Would this approach also be effective in other knowledge distillation applications, such as in computer vision models?", "other_comments_or_suggestions": "Several typos need to be corrected :\n\n1. “an knowledge distillation ” to “a knowledge distillation” in Line 053.\n\n2. “an LLM first” to “a LLM first” in Line 097.", "questions_for_authors": "1. The knowledge distillation method for LLMs proposed in [3] is similar to the approach presented in this paper, as both methods divide samples into two categories based on certain strategies and mix the teacher’s and student’s probability distributions in the loss function. Therefore, Please include an experimental comparison with [3].\n\n2. In Section 3.3, the author stated, “We empirically observe most difficult samples have  q >> p .” Could you explain why the samples selected by DDS exhibit this experimental phenomenon? Is there a theoretical explanation for this observation?\n\nI am willing to raise my score if the authors’ rebuttal addresses my concerns.\n\n[3] Song, Yuncheng, et al. \"Self-Evolution Knowledge Distillation for LLM-based Machine Translation.\" 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DgdB3FlgM0", "reviewer": "Reviewer_ofRn", "summary": "This paper proposes Difficulty-Aware Knowledge Distillation to improve the efficiency of LLM distillation. It proposes a dynamic data updating strategy that leverages a Distillation Difficulty Score to filter out easy samples, and a Bidirectional Discrepancy Loss that stabilizes gradient updates during training. The method is evaluated on instruction-following and task-specific benchmarks, demonstrating superior efficiency and performance compared to state-of-the-art KD methods, with up to 2% improvement in performance and 50% reduction in training cost.\n\n\n\n## =================update after rebuttal==============\n\nThanks for the authors' responses. My concerns have been well addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for recognizing our contributions, by stating \"well-motivated and appropriate\", \"experimental design is comprehensive\" and \"demonstrating reduced computational cost while maintaining/improving performance\". Your thoughtful suggestions and questions help a lot to improve our work. We have carefully addressed your questions below: \n\n> Q1: The proposed BDL can be converted to KL (when $\\lambda \\rightarrow 0$) or RKL (when $\\lambda \\rightarrow 1$). How would it compare to combining KL and RKL linearly, i.e., $(1-\\lambda)KL+\\lambda RKL$?\n\nA1: Thank you for your suggestion! We follow your suggestion and perform the experiments. Specifically, we use LLaMA2-7B as the teacher and use LLaMA2-1.3B as the student, and $\\lambda$ is set to 0.9. The results are shown below. We observe our BDL performs better than the linear combination of KL and RKL. We hypothesize this is because BDL blends teacher and student distributions, avoiding the issues of gradient explosion or vanishing and achieving a more stable updating process, which is also explained in Sec. 3.3.\n\n| Loss         | Dolly | SelfInst | Super-N | Unnatural | Vicuna | Avg.  |\n| ------------ | ----- | -------- | ------- | --------- | ------ | ----- |\n| BDL          | 26.75 | 18.05    | 32.35   | 36.09     | 17.53  | 26.15 |\n| (1−λ)KL+λRKL | 26.10 | 15.94    | 28.85   | 32.09     | 18.21  | 24.24 |\n\n> Q2： Is the cross-entropy (CE) loss incorporated when distillation is performed? If so, how is the ratio set?\n\nA2：Thank you for your question. The cross-entropy loss is not used in distillation, because we find directly using KD loss performs the best performance. The experiments of loss function combined with cross-entropy loss  are shown below, which is about 3~4% lower than ours. The similar observation was also found in Distillm[1].\n\n[1] \"Distillm: Towards streamlined distillation for large language models.\" arXiv preprint arXiv:2402.03898 (2024).\n\n| Loss              | Dolly | SelfInst | Super-N | Unnatural | Vicuna | Avg.  |\n| ----------------- | ----- | -------- | ------- | --------- | ------ | ----- |\n| 0.0 * CE+1.0 * KD | 26.75 | 18.05    | 32.35   | 36.09     | 17.53  | 26.15 |\n| 0.1 * CE+0.9 * KD | 25.20 | 16.24    | 28.83   | 30.21     | 17.41  | 23.58 |\n| 0.3 * CE+0.7 * KD | 24.37 | 16.27    | 27.01   | 28.28     | 16.44  | 22.48 |\n| 0.5 * CE+0.5 * KD | 24.19 | 15.46    | 26.46   | 28.60     | 16.55  | 22.25 |\n\n> Q3: Discuss potential limitations of DA-KD.\n\nA3: Thanks for your suggestion! One of the limitations of our DA-KD is that its performance is depended on the teacher model quality. If the teacher model provides incorrect prediction, the DDS mechanism may misidentify difficult samples, potentially affecting student performance. We will discuss these limitations in our final version and try to solve it in our future work.", "claims_and_evidence": "The authors claim that DA-KD not only improves training efficiency by reducing the data volume and iterations needed, but also enhances performance—even surpassing the teacher model in some scenarios. These claims are supported by comprehensive experiments and ablation studies.", "methods_and_evaluation_criteria": "The proposed methods are well-motivated and appropriate for addressing the efficiency challenges in LLM distillation. The use of DDS for selective data updating and the stratified sampling strategy are sensible extensions to traditional KD approaches.\nThe evaluation across multiple benchmarks and metrics (e.g., ROUGE-L, zero-shot accuracy and training time) is solid.", "theoretical_claims": "The paper provides derivations to justify the stability and effectiveness of the bidirectional discrepancy loss. No major correctness issues are found.", "experimental_designs_or_analyses": "The experimental design is comprehensive, covering diverse teacher-student pairs and tasks. The ablation studies convincingly demonstrate the contributions of the DiffUp strategy and BDL.", "relation_to_broader_scientific_literature": "The work is well situated within the current literature on KD of LLMs. It extends prior approaches (e.g., KD-KL, KD-RKL, GKD, Distillm) by introducing difficulty-aware sample selection and a novel loss function. The connection to data selection in LLM fine-tuning is also clearly articulated.", "essential_references_not_discussed": "The paper has cited most of the relevant literature on LLM KD methods.", "other_strengths_and_weaknesses": "Strengths: 1) Novelty: Innovative use of a difficulty-aware data selection mechanism to reduce training cost. 2) Efficiency: Demonstrating reduced computational cost while maintaining/improving performance.\n\n\nWeakness: See questions.", "other_comments_or_suggestions": "Discuss potential limitations of DA-KD.", "questions_for_authors": "1. The proposed BDL can be converted to KL (when \\lambda -> 0) or RKL (when \\lambda -> 1). How would it compare to combining KL and RKL linearly, i.e., (1-\\lambda)KL+\\lambdaRKL ? \n\n2. Is the cross-entropy (CE) loss incorporated when distillation is performed? If so, how is the ratio set?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsHT7ev0At", "reviewer": "Reviewer_nqxA", "summary": "This paper investigates knowledge distillation for large language models (LLMs) and introduces two main contributions: difficulty-aware data updating and bidirectional discrepancy distillation. The difficulty-aware data updating approach estimates sample difficulty using the ratio of cross-entropy losses between the teacher and student models. This information guides a stratified sampling strategy that includes both challenging and easy samples to maintain diverse data representation. The second contribution, bidirectional discrepancy distillation, is a novel loss function that blends teacher and student probabilities, yielding a smoother loss landscape. Together, these components enhance knowledge distillation performance across a variety of scenarios.\n\n\n## update after rebuttal\nUpon reviewing the authors’ rebuttal, I realized that DiffUp was intended to improve efficiency. Part of my initial review was founded on a misunderstanding. I adjusted my score to reflect this.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The ablation studies suggest most performance improvements stem from BDL.\n\nA1: We need to clarify that DiffUp aims to reduce data and achieve efficiency. The BDL aims to improve performance. It is unfair to only compare accuracy improvement. To verify the effectiveness of DiffUp, we randomly select 55% data and distill with BDL. This setting requires the same number of iterations as DiffUp, and the results show the effectiveness of DiffUP.\n| Method                                    | Dolly | Self  | S-N   | Un    | Vic   | Avg.  |\n| ----------------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| 55% random data+BDL                     | 24.52 | 15.27 | 29.34 | 32.73 | 16.08 | 23.59 |\n| DiffUP+BDL | 26.75 | 18.05 | 32.35 | 36.09 | 17.53 | 26.15 |\n\n> Q2: Including more ablation studies would help clarify the contribution of each component.  \n\nA2: The two components of our DA-KD do not both contribute to performance, which has been discussed in Q1. To verify the contribution of each component, we conduct ablation studies on Llama2-1.3B.\n\n| Method             | Iterations | Dolly | Self  | S-N   | Un    | Vic   | Avg.  |\n| ------------------ | ------------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Base       | 3570                | 26.17 | 15.13 | 24.97 | 29.22 | 17.34 | 22.57 |\n| DiffUP             | 1963                | 23.69 | 14.40 | 24.97 | 26.16 | 15.98 | 21.04 |\n| BDL                | 3570                | 26.73 | 18.51 | 32.07 | 35.64 | 16.58 | 25.91 |\n| DiffUP+BDL | 1963                | 26.75 | 18.05 | 32.35 | 36.09 | 17.53 | 26.15 |\n\nWe can see that DiffUp reduces the number of training iterations from 3570 to 1963. BDL improves the performance from 22.57 to 25.91. Combining DiffUP and BDL, we can achieve both efficiency and higher performance. \n\n> Q3: Including a discussion of how BDL compares to existing KL variants would strengthen this paper.\n\nA3: Existing KL variants include two main types, mixing KL and RKL or mixing teacher and student distributions. \n\nThe former is formalized as a linear combination of KL and RKL [1]. It may suffer from gradient vanishing or explosion when facing extreme distributions. However, our BDL blends teacher and student distributions, avoiding the issues and achieving more stable updating (see Sec. 3.3 of the paper).\n\nThe latter category includes SKL [2]. However, it only mixes teacher into student distribution, which may cause gradient vanishing (see Eq. (6) in [2]). However, our BDL also mixes the student into teacher distribution, which can stabilize the gradient for better optimization (see Sec. 3.3 of the paper).\n\n[1] Rethinking kullback-leibler divergence in knowledge distillation for large language models. (2024)\n\n[2] Distillm: Towards streamlined distillation for large language models. (2024)\n\n> Q4: The overall novelty of this work is limited. \n\nA4: Our DA-KD is designed to perform efficient LLM KD. We are happy that other reviewers recognize our contributions as \"addresses an important bottleneck in LLM distillation\", \"simple yet effective\" and \"well-motivated and appropriate\".\n\n> Q5: Have you evaluated BDL independently of DiffUp? \n\nA5: We evaluate BDL independently and compare it with other losses. The results indicate that BDL achieves better performance.\n| Loss | Dolly | Self  | S-N   | Un    | Vic   | Avg.  |\n| ---- | ----- | ----- | ----- | ----- | ----- | ----- |\n| KL   | 26.17 | 15.13 | 24.97 | 29.22 | 17.34 | 22.57 |\n| RKL  | 26.04 | 16.67 | 30.46 | 34.00 | 18.37 | 25.11 |\n| SKL  | 25.87 | 16.89 | 30.42 | 33.66 | 17.47 | 24.86 |\n| SRKL | 26.42 | 17.19 | 31.82 | 35.99 | 16.42 | 25.57 |\n| BDL  | 26.73 | 18.51 | 32.07 | 35.64 | 16.58 | 25.91 |\n\n> Q6: Have you explored alternative sampling strategies?\n\nA6: We have compared with random sampling in Tab 4 of our paper (denoted as w/o DDS). We also compare with the data filtered by the normalized DDS scores as probabilities. Results show stratified sampling performs better.\n| Method                       | Dolly | Self  | S-N   | Un    | Vic   | Avg.  |\n| ---------------------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| stratified sampling  | 26.75 | 18.05 | 32.35 | 36.09 | 17.53 | 26.15 |\n| sampling based on DDS | 26.62 | 17.48 | 32.21 | 34.96 | 17.51 | 25.76 |\n\n> Q7: Have you examined the inclusion of samples generated by the teacher and the student?\n\nA7: We incorporate teacher-generated or student-generated samples into the distillation. Here are the results. We can see a small performance degradation when using teacher-generated or student-generated samples. \n| Method          |  Dolly | Self  | S-N   | Un    | Vic   | Avg.  |\n| --------------- |  ----- | ----- | ----- | ----- | ----- | ----- |\n| DA-KD           |  26.75 | 18.05 | 32.35 | 36.09 | 17.53 | 26.15 |\n| DA-KD + tea-gen | 26.25 | 17.36 | 31.47 | 35.17 | 16.98 | 25.45 |\n| DA-KD + stu-gen  | 27.27 | 17.91 | 31.01 | 35.01 | 18.11 | 25.86 |", "claims_and_evidence": "The paper does not present any theoretical evidence to support its claims, relying instead solely on empirical observations. \n\nWhile the authors identify difficulty-aware data updating as their primary contribution, the ablation studies indicate that most performance improvements stem from the bidirectional discrepancy distillation. This discrepancy should be addressed in the discussion. Furthermore, since the two components appear to function independently, it may be more beneficial to focus on just one of them.", "methods_and_evaluation_criteria": "The evaluation criteria are reasonable.", "theoretical_claims": "No theoretical claims are provided.", "experimental_designs_or_analyses": "The experimental setup appears sound. However, including more ablation studies would help clarify the contribution of each component in the algorithm. In its current form, the draft suggests that the loss function plays the primary role in driving performance, while other factors have a comparatively minor impact.", "relation_to_broader_scientific_literature": "Smoothing the KL divergence has been explored extensively in prior works across various domains. Including a more in-depth discussion of how BDL compares to these existing KL divergence variants would strengthen this paper.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The BDL loss function does not demonstrate substantial novelty, as there are numerous similar loss functions. Given that the performance improvements largely stem from the loss, the overall novelty of this work is limited.", "other_comments_or_suggestions": "None", "questions_for_authors": "1.\tHave you evaluated BDL independently of DiffUp in other scenarios? Have you conducted comparisons with other algorithms?\n\n2.\tHave you explored alternative sampling strategies beyond stratified sampling to ensure diversity? I believe there are numerous potential approaches.\n\n3.\tHave you examined the inclusion of samples generated by the teacher and the student models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a well-motivated and empirically validated approach (DA-KD) to tackle the significant challenge of efficient knowledge distillation for LLMs. The core ideas of difficulty-aware data selection (DiffUp/DDS) for efficiency and a tailored loss function (BDL) for effectiveness on hard samples appear sound and demonstrate strong results. The authors provided convincing empirical evidence and effectively addressed reviewer concerns during the rebuttal, including providing additional experiments and clarifications. While one reviewer noted lingering concerns about the BDL's absolute novelty, the overall package and its demonstrated benefits in terms of both efficiency and performance are compelling. The reviewer consensus, with three Accepts and one Weak Accept post-rebuttal, strongly supports acceptance.
Decision: Accept (poster)