DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Authors: Jongwoo Ko
Tianyi Chen
Sungnyun Kim
Tianyu Ding
Luming Liang
Ilya Zharkov
Se-Young Yun
Venue: ICML 2025 oral (2025)
Keywords: knowledge distillation
efficiency
contrastive approach
Primary area: deep_learning->large_language_models
TL;DR: DistiLLM-2 improves Large Language Model (LLM) distillation by leveraging a contrastive approach that increases the likelihood of teacher responses while decreasing that of student responses.

Abstract: Despite the success of distillation in large language models (LLMs), most prior work applies identical loss functions to both teacher- and student-generated data. These strategies overlook the synergy between loss formulations and data types, leading to a suboptimal performance boost in student models. To address this, we propose DistiLLM-2, a contrastive approach that simultaneously increases the likelihood of teacher responses and decreases that of student responses by harnessing this synergy. Our extensive experiments show that DistiLLM-2 not only builds high-performing student models across a wide range of tasks, including instruction-following and code generation, but also supports diverse applications, such as preference alignment and vision-language extensions. These findings highlight the potential of a contrastive approach to enhance the efficacy of LLM distillation by effectively aligning teacher and student models across varied data types.

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Jongwoo Kot!
Tianyi Chen
Sungayun Kim
Tianyu Ding? 2
Luming Liang? Ilya Zharkov? 2
Se- Se-Young Yun
'KIST AI Miicrosoft
*Work done as a research intern at Microsoft
https: https://github...
Abstract
As the demand for reducing computational overhead grows, knowledge distillation (KD; Hinton et al. 2015) has emerged as a promising technique for compressing LLMs into more lightweight student models. By transferring knowledge from a high-capacity teacher model to a smaller student model, KD can significantly improve the performance of small language models (sLMs) as demonstrated by Llama 3.2 (Meta, 2024) and Gemma-2 (DeepMind et al., 2024).
Despite the success of distillation in large lan- guage models (LL.Ms), most prior work applies identical loss functions to both teacher- and student-generated data. These strategies overlook the synergy between loss formulations and data types, leading to suboptimal performance boost in student models. To address this, we propose M-2 a contrastive approach that simul- taneously increases the likelihood of teacher re- sponses and decreases that of student responses by hamessiin this synergy. Our extensive exper- iments show that DISTILLM-2 not only builds high-performing student models across wide range of tasks, including cestruction-oolloimng and code generation, but also supports diverse applications, such as preference alignment and Dision-language extensions. These findings high- light the potential of contrastive approach enhance the efficacy of LLM distillation by effec- tively aligning teacher and student models across varied data types.
Over the years, research on LLM distillation has largely focused on either by designing new loss or by curating training data. From a loss perspective, several studies sug- gest that Kullback-Leibler (KL) divergence, a common loss for KD, may fail capture the teacher model's complex generative behavior (Wen et al., 2023; Gu et al., 2024). Con- sequently, alternative loss functions, such as skew KL (SKL; Ko et al. 2024), have been proposed to better guide the stu- dent. On the other hand, from data perspective, previous works emphasize how the training data is curated to enlarge the effectiveness of KD. For instance, relying solely on of- fline data (e.g., teacher-generated outputs; TGOs) can be problematic where student's outputs at inference time devi- ate significantly from fixed training samples (Agarwal et al., 2024). To address this mismatch, some works incorporate outputs (SGOs) directly into training (Lin et al., 2020; Xu et al., 2024b). However, these works often overlook the synergy between loss formulations and data types, which might have limited the extent of performance improvement of student models.
1. Introduction
Large language models (LLMs) have continuously improved their text generation abilities by increasing the number of parameters and the amount of high-quality training data. However, LLMs typically require extensive computational resources during inference, which makes them diffieult to be deployed practically. Therefore, compressing them by reducing the number of parameters while maintaining their performance becomes important for using these powerful models effectively.
Recently, contrastive approaches such direct preference optimization (DPO; Rafailov et al. 2023), have gained pop- ularity for their efficacy and efficiency in preference align- ment (Tajwar et al., 2024) or reasoning (Pang et al., 2024), by explicitly employing different learning strategies to han- dle two distinct responses. Despite their success, few works have focused on extending their schema KD for LLMs. While Li et al. (2024b) attempted simply apply DPO by replacing the reference model to teacher model (see Equa- tion 4), we observed that their method is prone to reward hacking, which may limit its broader applicability (see Fig- ure 1). This motivates us to design a scalable contrastive approach to boost LLM distillation.
'KAIST AI. Seoul, Republic of Korea *Microsoft, Redmond, Washington, USA. Correspondence Jong- woo Ko wreggwwc..ekkist... ac..st.ac.kr... Tianyi Chen <tianyi. www.ecccooooccoo.....ooooo
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Contributions. In this paper, we introduce DMSTILLM- 2, which features a novel contrastive approach for KD of LLMs. Our DISTILLM-2 builds contrastive framework upon DistiLLM (Ko et al., 2024), which has shown signifi- cant improvements by using SKL--based loss and balanced SGOs. Our detailed contributions include:
2.2. Preliminary: KD in LLMS and DistiLLM
Loss function of KD in LLMs. Given a prompt and re- sponse pair, denoted as (æz y), KD minimizes divergence between the distributions of a teacher p(z/(z) and student qo (o(y)a) (:) parameterized by 0. Conventionally, KI, denoted as DKI., is the most widely used loss in KD due to its sim- plicity and tractability. The sequence-level distillation using KL is accurately decomposed into a sum of loken-wise distillation (Ko et al., 2024):
Contrastive approach with asymmetric loss dynamics: We analyze the behavior of forward and reverse KL (and SKL) during the training process on responses from the student and teacher models, respectively. This analysis motivated the development of a contrastive approach for LLM distillation (CALD; $3.1), which applies distinct loss functions to different types of training samples. By doing so, CALD effectively incorporates the synergy be- tween loss formulations and data perspectives.
DKL pl = Σ p(u Mo x) log plye 1e x))
T
qo (yh ye x)
We can also define reverse KL as DRKL (a. y: p:plle) (z, y: qo lIp). Despite its tractability, such KL has lim- itations of either mode averagngg mode-collapsing for forward and reverse version, respectively. To address this issue, Ko et al. (2024) proposed skew KL (SKL) and skew RKL (SRKL), defined as follows:
Development of the contrastive approach: Additionally, we introduce optimized dataset curation strategies ($3.2) and curiculum-basee adaptive loss mechanisms ($3.3).
These enhancements to CALD, which are collectively coined to as DISTILLM-2, provide solid guidelines for our contrastive approach for practitioners.
DSe) (a, y: pllg)) Dxi v:plpap + (1 a))o);
D (a, y: pllgo) = (x yi go [1([1 a)p ago)..
Advanced performance and versatility: DISTILLM-2 achieves state-of-the-art performance for sLMs across various text-generation tasks, including instruction- following, mathematical reasoning, and code generation (64). Furthermore, we demonstrate the diverse applica- tions of our proposed KD approach ($6), such as pref- erence alignment with better reference models and its expansion to vision-language models.
Despite the simple modification, demonstrratt higher convergence speed and achieved better performance com- pared to recent baselines, such MiniLLM (Gu et al., 2024) and GKD (Agarwal et al., 2024). This effective- ness has been proven from both empirical and theoretical perspectives. For brevity, we will denote D&a) (I, y: pl) and Diska.(aa (x,, P1l4)) as DiRi, (2r, y) and DiRKI. (2:, y). re- spectively.
Data curation of KD in LLMs. To address the training inefficiency and low quality of SGO, which can lead to inac- curate teacher feedback in on-policy approaches (Lin et al., 2020; Agarwal et al., 2024), Ko et al. (2024) introduced an adaptive off- tiflcollcy approach, which bridges offline and purely on-policy setups, striking a balance between the ef- ficiency and efficacy of KD. This balanced strategy reuses SGO by introducing replay buffer, significantly improving computational efficiency while preserving the effectiveness of on-policy distillation. This approach has proven effec- tive in subsequent works on preference alignment of LLMs (Rosset et al., 2024) as in more generalized version.
2. Backgrounds
2.1. Related Work
KD (Hinton et al., 2015) effectively compresses neural net- works, enabling smaller student models to match the perfor- mance of larger teachers. This technique recently has been adapted to address the scalability challenges of LLMs, en- hancing their viability in compute-intensive environments. ImitKD (Lin et al., 2020) demonstrated the use of SGO as training data for distillation. Building on this, Agarwal et al. (2024) introduced an on-policy approach with objec- tives like reverse KL or Jensen-Shannol divergence (ISD). Wen et al. (2023) explored various f-divergences, including total variation distance and JSD, in auto-regressive LMs, while Gu al. (2024) proposed a policy gradient method to mitigate high variance in RL-based techniques. Recently, Xu et al. (2024b) combined static datasets with on-policy methods using speculative decoding for training data gener- ation. Among these, DistiLLM (Ko et al., 2024) achieved state-of-the-ant performance and greater efficiency by intro- ducing SKL, and an adaptive off off-policy approach. A more discussion of related works is available in the Appendix A.
Summary & Connection to our work. Building on the insights from DistiLLM (Ko et al., 2024) - where SKL (or SRKL) and adaptive off-policy have shown efficacy - we introduce a contrastive approach that further refines these objectives. On the data curation side, we adopt batch approach (Rosset et al., 2024) that collects SGO ahead of every training epoch in our setup, rather than on-policy approach, which samples at every training iteration. This also ensures compatibility with advanced LLM inference techniques, such as vLLMM (Kwon et al., 2023), thereby
2

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
increasing generation efficiency and preserving the core philosophy of the adaptive off off-policy approach. As shown in our preliminary results in Appendix D.I, this greatly reduces the computational cost of gathering training samples with minimal impact on student performance.
Algorithm 1 Training pipeline of DISTILLM-2
1: Input: training iterations T, initial skew coefficient ou.
teacher y, student 46, with parameter Oo. prompt set
2: Output: Student model g0 with trained parameters 0gg
3: for epoch e = 1,2....,E.. do
4: 1 Sample batched on-policy responses */
3. Method: DISTILLM-2
Sample responses Yt: ys from teacher p(-|ar) and student qo... -((xx) for given prompt æ
We introduce DISTILLM-2, a novel approach to LLM dis- tillation, which lies its new loss function as presented in Equation 2. This equips with a contrastive schema si- multaneously accounting for different types of training re- sponses ($3.1), along with dedicated data curation ($3.2)
6: Construct D. = { yt. ys)3 for training dataset
for training epoch
7:
Initialize 9. t 0
8:
for iteration T - 1, ,.....,T T do
(1) Sample mini-batch: B = {(x yi y )
(6)
and curriculum-based adaptive loss design ($3.3).
9:
from D.
DISTILLLM-2 ==
(2)
10:
1 Curriculum- adaptive update for a */
1 2
Σ
(1- (1-B)DDRil SKL (z) ye) + 3Deekl (z, v.))
[11
11:
Update a₁ < 1- - (1 (1-100 -o0 ply.ll))-gevvlle and ag - (1-1111100 -00)
m
(D.Wh.Be)~D
12:
1s Gradual increasing coefficient for SRKL #/
where Dier) (z, y1) and D(oe) (æ. ys) are SKL and SRKL tailored for teacher- and student- generrtee responses, re- spectively; D is the training dataset; /8 is a coefficient in [0, 1] to balance SKL and SRKL terms. In the following subsections, we provide detailed motivations, derivations, and use of the loss function in Equation to formulate our DISTILLM-2 training process as stated in Algorithm 1.
13:
Update 8 + clip( + 1)
14:
1e Improved contrastive loss function ($3.3)%/
15:
Update G. by minimizing LDSSTILLM22
1
(1 ()D (1 (x, y:) + BBD (x, ye)
2
16:
end for
17:
end for
3.1. Contrastive Approach
rectly applying DPO into KD. We observed that DPKD (Li et al., 2024b), which simply applies DPO by substituting the reference model with the teacher model, frequently suffers from reward hacking, leading to degenerate sentences:
MOTIVATION
Recently, contrastive approach in preference alignment, including DPO (Rafailov et al., 2023), which increases the likelihood of the preferee response (yw) while decreasing the likelihood of the dis-preferred response (3/), has demonstrated effective in enhancing LM performance.
Concept.
go (b(())z)) A log p(y)(x))
go (B= (x)
-log a A (Alog
(4)
p plys (x)
inherently small p(g) overly decrease go (s.)
you x))
(31 2))
-log
AAlo gref yw x)) A log Orer( (ver(31 (x)
where zt: and ys are TGO and SGO, respectively. This is because DPKD only focuses on maximizing the gap be- tween p(g:|()
incresse Qo (B/wlar)
decreuse Qu (3xx (az)
[=(y](m) and (e(u. (m) As illustrated in Figure 1(b), p(y.(=)
where a is sigmoid function, gref is a reference model, and is hyperparameter for DPO. This improvement stems from
we observe that this loss dynamics excessively decreases the likelihood of qu (o(y)/z) (e.g.. 91.25 in terms of nega- tive log-likelihood; NI..), causing the student model to lose pre-trained information instead of fitting to teacher re- sponses (e.g., 20.29 in terms of NLL), as replaces 9et with p where p(y, az)) is inherently small. Addressing this limitation requires rethinking and redesigning algorithm to integrate contrastive strategies into LLM distillation.
its dual mechanism: not only does it reduce the likelihood of undesired responses (Tajwar et al., 2024) but it also increases the likelihood of preferred responses, effectively reinforcing alignment with the desired behavior.
Similarly, we can apply this concept into KD to increase the likelihood of go(ye a:) as match that of p(y (a)) and decrease the likelihood of qo (3.(æ) as match that of p(y, (æ) by bringing different types of loss function for each type of response. This approach allows better alignment of TGOs and SGOs in the contrastive manner than simply using single type of loss function.
3.1.2. CONTRASTIVE APPROACH FOR LLM DISTILLATION
To bring contrastive strategy into KD,, we propose new loss function LCALD. using a combination of SKL and SRKL (Ko et al., 2024). Our design stems from the follows.
Challenges of contrastive approach into KD. While the concept itself appealing, there are critical issues di-
Observation on behavior of KL and RKL. Here, we pro-
3

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
(a) Toy Dataset Results
(b) LLM Experimental Results
(c) Superiority of Contrastive Approach
p00 pb agbd)
0.06
ptd)
Gylel [KL]
qulel (RKL)
-100 loo (NLL))
teacher
3.50
2.69
student
1.9
22 29.29
27.5
0.04
173
25.0
156
N
22.5
0.02
20.0
-dh
CALD (KL)
-0-
RKL
-Q-- SRKL
CALD (SKL)
0.00
17.5
60
RXL
SKL
SRKL
CALD DPKD
10
(SKL+1RKL)
Val Iteration
Figure 1. (a) The behavior of KL (orange) and RKL (green) is analyzed for long-tailed toy data introduced in Wu et al. (2024), (b) NLL of student models on teacher (red) and student (blue) responses, using Mistral-7B and Danube2-1.88 1.8B as the teacher and student models, respectively, optimized with diverse loss functions. (e) We propose CALD with SKL. and SRKI. that achieves faster convergence and higher ROUGE-L (Lin, 2004). following the experimental setup of Ko et al. (2024). Detailed setup can be found in Appendix D.I.
vide observation the behavior of KL. and RKL: they can increase and decrease the likelihood of qo for TGOs (KI.) and SGOs (RKI.,, respectively. As shown in Figure I, KL increases qe pe(-la) in regions where p(- [z) are high (i.e., pulling-up effect). For example, this occurs in the head of the teacher distribution in Figure I(a) or for TGOs in Figure 1(b) - This behavior arises because they aim to fo- p(|(z) for the region where p( (z)
clusive use of SKL or SRKL in DistiLLM (see Figure I(c)). Note that while simple KL and RKL also prove effective- ness for CALD, using SKI. and SRKI. as backhone achieves higher efficacy, consistent with Ko et al. (2024).
Mathematical connection to DPKD and DPO. We now reveal that our proposed loss function CCALD can be mathe- matically interpreted as exhibiting similar yet different be- havior to DPKD (or DPO).
cus on reducing the ratio go(-|æ)
are large to minimize weighted average. Conversely, RKL
attempts to reduce the ratio s0(clr)
p(-[az) Consequently, 90
Remark 1. Equation 5 can be re-written as follows:
1 [
Aoog (o(y) (x)
A Alog go (z))
decreases in region where p(-(æ) are small (i.e., pushing- down effect), such as the tail of teacher distribution in Figure ((a) student responses in Figure 1(b). Detailed mathematical explanation can be found in Appendix B.
E pprepplpr) No (0-00()11))
p(ya(z))
p (x)
where (-(-)2)) ap(. xx) + (1 x=) and p(-12>) (6)
(-2x))2)) + (1 a ppp((((z)
Our solution. For implementing CALD, an optimal choice among various KL-based loss functions would be one that demonstrates state-of-the-art results while exhibiting similar behavior to KL and RKL, as observed Figure 1. To this end, we utilize skew KL (SKL) and RKL (SRKL). introduced in DistiLLM (Ko et al., 2024), the backbone loss functions. Specifically,
This indicates CALD enable to increase (v (z) (and im- plicitly qo (y, The detailed derivation can be found in Appendix B.2.
( (zz)) and decrease qo go (ar), simultaneously.
Despite this similarity, there are two critical and non-trin trivial differences between CALD and DPKD (or DPO). First, rather than employing the log-sigmoid function used in DPKD, Equation 6 adopts a linear formulation that al- lows token-level decomposition and explicit weighting by p (z) or qo (yyy/ar) (zz) (as in Equation 1). Second, by in- herently linear dependency between (-(xx) and p(-1æ) (or between p(-[ær) and { (z))), this regularizes the overly de- creasing qe (z)) which resolves the challenges in DPKD. From this, CALD (i.e. DISTILLM-2) outperforms DPO and DPKD by a large margin, as shown in Appendix D.I.
design the loss function for CALD, using SKL. for teacher responses (i.e., yt) where most of p(y (+) >> 0 and using SRKL for student responses, ys: where the most of p(ys (z) ~ O. Formally, our proposed loss function can be written as follows:
CCALD= = 2|D (~)
D ala SKL (a), v1) + PRRKL (ar, Ve).
(5) Despite its simplicity, this loss function implies that the importance of simultaneous consideration of responses type during objective function design. Note that Ko et al. (2024) demonstrated that a vanilla interpolation between D (x. + (1 D the same type of responses, (e.g., either y: or y.). does not improve performance compared to using either SKL or SRKL alone. However, we find that the new approach of using different types of responses for different terms sig- nificantly enhances performance. LCALD achieves faster convergence and greater effectiveness compared to the ex-
3.2. Optimal Data Curation for Contrastive Approach
In the context of datasets for LLM distillation, one common question might be:
(a) (a,-) for all y E [0. over
"How can we effectively utilize well given SGO and high-quality fixed datasets in distillation of LlMs?"
While previous works (Xu et al., 2024b; Li et al., 2024a) have proposed effective strategies for leveraging these two complementary dataset types in an SFT manner, we ob- served that their techniques - such as speculative generation
4

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
(a) Speculative Decoding on SKL / SRKL
(b) High Quality Generation
pl (x)x)) or log (Xx)
1.50
90
Winning Rate (%)
-log prpppllx (x) or 1o
1.50
100
Winning Rate (96)
NLL/Teacher
1.25
80
90
NLL/Student
1.00
1.00
80
WR/Generation
WR/Student (SKL)
0.75
60
0.75
WR/Student (SRKL)
0.50
50
0.0
0.3
0.5
0.7
1.0
Mistral (78)
Llama3 (8B)
Gemmaz Ph13
(9B) (Medium)
Generation Models
Figure 2. Comparison of the winning rates compared the student before KD (WR) of student models with (a) replacing ye (orange) or ys (green) with Mspe. responses from speculative decoding (Cai et al., 2024), varying the hyperparameter e. (b) replacing yt with responses generated using stronger LLMs (c.g., Llama-3, Gemma-2, Phi-3) than the teacher models (i.c., Mistral) for the SKL term. We also show the negative log-like oogllikeldhod (NLL) of the student (cyan) and teacher (blue) models on the replaced responses, along with the corresponding WR (red).
Table 1. Motivation for curriculum approach for a. UF and EI indicate the winning rates (%) of responses on the UltraFeedback and Evol-Instruce iteration 1, as judged by HPTT-4--mii.
In speculative generation, student drafts K tokens, and teacher verify them in parallel for S k: I K based on?:
sets, compared the student model before
go go(Un+tl)enntk) > min(s", e exp( Hap(-H(uu (p(((ly<n+k))
Epoch. 1
Epoch.
Epoch.
where (-) and are entropy function and hyperparameter.
(S)
(S)
6
(56)
(9))
a (F) (5k)
0.3
71.83
76.23
When we replace yt with Yspez, as & decreases (i.e., more acceptance of drafts), the distilled model better aligns with the student distribution go(-|zz). However, as shown with orange bars, its performance is highest with Yspec (i.e., identical y:). This implies that the SKL side, mitigating the training-inference mismatch via SGOs does not always lead to performance improvement. Rather, strong guidance from the teacher response is highly related to the distillation performance.
72.18
81.18
Mistral
0.1
73.46
79.83
0.01
75.75
84.35
0.01
0.1
76.59
73.25
84.67
82.86
70.35
76.38
1.0
0,3
69.89
75.86
Qwen2
70.01
76.21
0.01
0.1
74.30
69.95
78.86
81.23
0.1
75.32
$1.81
0.01
66.62
74,82
Entries marked 12" were cmvited as they were found sub-opeiima in previous epechs.
or the use of nigh-quality responses (which may outperform teacher generations) are less effective in CALD. From our further discussion, we conclude that utilizing teacher and student generations for SKL and SRKL, respectively, may be the optimal strategy for CALD, as it consistently aligns with the core philosophy of CALD.
Conversely, on the responses for SRKL, the distilled model achieves the highest performance with Yspee (i.e., identical to y.), as shown with green bars, although these responses are of the lowest quality. This implies that using low-quality SGO samples on the SRKL side may be beneficial for our contrastive approach. The effectiveness of reduced training- inference mismatch via SGOs can be attributed to this edge of alignments.
0.0
Exploring the trade-offs between teacher and student generations. Previous works (Agarwal et al., 2024) have discussed that while teacher responses provide useful infor- mation, they can cause training-inference mismatches. In contrast, SGOs, though lower in quality, effectively reduce such mismatches, leading to higher efficacy. To explore these complementary perspectives, we use speculative de- c2 to find the key factors for dataset curation.
High-quality does not always guarantee success. One ad- ditional question that arises is whether the success of teacher responses on the SKI. term is due to their higher quality. It is natural to consider if using higher-quality responses from powerful LLMs like ChatGPT would improve performance, similar to black-box KD (Li et al., 2024a). To investigate, we replaced the responses for SKL term with those gener- ated from stronger LLMs (e.g.. Llama3-8B) instead of the Mistral-7B teacher's 's responses. As shown in Figure 2(b), al- though these stronger LLMs generate high-quality answers, the student trained on the teacher's responses still performs
'The original work primarily aims to accelerate generation without sacrificing the quality of generated responses. If we use Yope: instead of for SKL, we may mitigate training-inference mismatch, potentially improving overall performance by aligning the training data more closely with student distribution.
"Alternativey, Yspee for SRKL could improve student perfor- mance by training higher-quality samples. However, as dis- cussed in this section, this approach did not yield the desired results.
'Specifically, applied speculative decoding with typical decoding (Cai et al., 2024), as simplifies interpolation compared to rejection sampling-hare based methods (Ievvithan al., 2023).
5

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Table 2. Comparison winning rates (WR) using pairwise comparison (Zheng et al., 2023) on three instruction www.ecctoon-oloowwiiii following benchmarks. The
baseline is text-dav -oxx--davinc--- 303 in AlpacaEval and spt-3. S-tuuroo in Evol-Instruct and Ultraeeedback. The judges are GPT-4o for AlpacaEval and
GPT-44-minn for UltraFeedback. The best and the second best win rates are in bold and underline.
Qwen2-7B-Inst (M2) - Qwea2-1.5B (Mas
Mistral-7B-Inst
-
Danube2-1.88 -.1ble22-888-. (Ms)
Gemma--2 2-9-ma------usttt (M2)
Gemma-2-22 (Mas)
Method
AlpacaRval WR(%)
Exel-Inst WR(%)
UltraFeed
AVG.
AlpacaEval
Evol-Inst WR(%)
UltraFeed
AVG.
AlpacaEval WR(S)
Evol-Inst WR(%)
UltraFeed WR(%)
AVG.
WR(%)
WR(%)
WR())
WR(%)
WR(%)
WR(%)
Mr
88.41
70.70
69.25
76.12
91.92
73.51
83,59
83.01
95.78
88.76
85.90
90.15
MM
51.06
21.93
30.33
48.17
12.84
20.06
27.02
42.51
16.74
26.60
28.62
KD
57.49
28.23
37.86
41.19
60.21
18.23
41.56
40.00
61.78
32.45
54.37
49:53
SeqKD
58.02
29.11
38.35
41.83
59.76
18.45
42.11
40.11
62.43
33.21
55.18
50.27
ImitKD
59.37
30.58
39.92
43.29
58.34
17.89
40.87
39.03
63.12
31.89
53.92
49.64
GKD
66.07
44.61
57.74
56.14
69.75
24.54
57.74
50.68
81.43
50.57
77.20
69.73
D
66.30
44.61
58.18
56,35
70.16
28.78
58.18
52.37
82.95
51.26
76.68
70.30
Speculative KD
61.52
44.95
56.82
54.43
64.58
38.87
60.04
54.50
78.45
57.11
72.21
69.26
DISTILLM-2
69.88
47.13
$9.05
$8.69
74.04
32.84
62.46
56.45
85.97
59.53
78.99
74,83
better. This suggests that the high log-probability of re- sponses from the teacher model may be more important factor in data curation than their higher quality.
suggests that a values in a moderate range (e.g., 0.1-0.3) are generally robust, we observed that the optimal values can still vary across different setups due to the variation of teacher-student pairs and the dynamic requirements of different training epochs (see Table 1).
Discussion on the observations. These findings align with the motivation of CALD in $3.1: the "pulling-up" effect of SKL is maximized at the head of p(-(z) (i.e., yr), while effect of SRKL is maximized at the tail of p(-la)) (i.e., Vs). First, while speculative generations effective with vanilla KI. in Speculative KD (Xu al., 2024b), they are less effective with our contrastive loss because (1) speculative generations are an interpolation of y: and ys. which may weaken both the "pulling-up" and "pushing-down" effect- -ore core mechanisms underlying CALD; and (2) the contrastive loss already exploits both complementary response types simultancously. reducing the need for interpolation compared to single-loss settings. The second observation also supports our claim that pure teacher generation may be optimal for SKL where they completely align with p(-1y). rather than relying on higher- quality responses, from the perspective of maximizing the "pulling-up" effect at the head of p(-(z)).
Regarding the dynamic of different training epoch, we ob- serve that the optimal values for a for the second or third epoch are either equal to smaller than than those in the first epoch (Table 1). Building on this observation, we propose a curriculum- "easy" samples, where p(-) and (o are sufficiently similar, we select a small a. On the other hand, for "hard" samples, where the difference between p(-) and (o()) is large, we choose a larger a.
the "pushing-dowee
approach for updating For
To implement this, we introduce an updating rule for [0, 1| based on the following approximation:
log p(yaar) (1 (1- (p(y)z) qo x)),
(7)
4o (y(
where t = (1--
x+) Note that this approximation originates from the Mercator series ex- pansion (Zwillinger, 2002): log(1 + x:) = (-1(-1) (="). This series allows the first-order approximation logp(=) = p(x) - 1. The detailed derivation can be found in Appendix. Using this formula, we can compute suit- able a in closed-form for each sample, allocating proper by making (1 - a) (p(-) - (-)) consistent across entire training. The detailed implementation for this updating rule can be found in Algorithm 1.
3.3. Curriculum-basee Adaptive Learning
We introduce two modifications, inspired by our empiri- cal observations, to implement difficulty-based adaptive learning and facilitate the conversion from Equation 6 to Equation 2: curriculum approach for and gradual increasing of coefficient for SRKL.
Curriculum Approach for a. One limitation of SKL (Ko et al., 2024) is that we need to manually determine a, which interpolates between the teacher and student distributions. larger a improves optimization stability and accelerates convergence, but it limits the acquisition of informative knowledge by inherently small gap between p(-) and ap(-) (1 - (-). Conversely, a smaller allows for greater knowledge acquisition but reduces optimization stability and slows convergence (Ko et al., 2024). While previous work
(Linearly) Gradual increasing of coefficient for SRKL. Based on the behavior of TGOs with SKL and SGOs with SRKL in Equation 5, the first term enables the acquisition of advanced information by matching high-probability on TGOs, while the second term suppresses undesirable behav- ior by preventing the matching of similarly low probabilities in SGOs. However, achieving qe (-(12=) = p(-1x: for all y: and y, is challenging with limited dataset sizes due to inher-
6

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Table Comparison results on the GSM8k and MATH henchmarks. The best pass @ / score is highlighted in bold.
Table 4. Comparison results on the HumanFval (HEval) and MRPP
benchmarks. The best pass @ score is highlighted in bold.
Fren2-Math-7B--nee (My)
Owen2.5-Math-778-- (MM))
DS-Coder-6 -nstt (Myy
S0wen2.--Code-77BBB- (M1y
-
Qween- (Mss
- (0wen2.5-Math-1-5. (MMs)
- DS- EB5-Coder-113B (Mg)
- -0wen2.5-Coder--.. (.is)
Method
GSM8K
MATH
AVG.
GSMBK
MATH
AVG.
Method
HEval
MBPP
AYG.
HEval
MBPP
AVG.
Paw01
Puss1
P1
Puss
Pus@1
Puss @1
Paw@1
Paw01
Pus
Paw01
Paw01
1
Mrr
K3,93
41.28
62.61
89.31
44.82
67.07
Mr
85.37
82,54
83.96
75.61
74,60
75.61
Mg
74.53
25.56
50.05
77.35
27.14
52.24
Ms
50.61
72.22
61.42
30.73
60.84
45.79
GKD
75.44
34.16
54,80
80.21
40.54
60.38
GKD
54.88
74.34
64.61
40.85
61.90
51.38
DistILLM
75.59
34,54
55.07
81.05
41.14
61.10
DistILLM
53.65
74,34
64.00
39.63
62.17
50.90
DM-
76.27
42.94
DISTLLMM22
75.66
67.79
42.24
62.70
52.47
Table Component analysis of DISTILLM-2 including (1) ap- plying contrastive loss, (2) increasing 3, and (3) introducing curriculum-basek updates to a. When all components are applied DistiLLM it becomes identical to D
N0TH (Paug31
(I) (2)
Qwen2(1)
(1nube2(1)
Gemma2(1)
AVG.(1)
154
716
100
17c
Abc
(vD)
44.61
28.78
51.26
41.55
Figure 3. Comparison of performance for different sizes of training
45.11
30.73
$1.70
43.6]
datasets across wide range of tasks.
<
45.87
31.88
$6.65
44.30
46.33
31.65
57.68
45.22
DISTILLM-2
< 47.24
32,801
59,53
46.50
Instruct. It outperfomms the second best methods by +2.34%. +1.95% and +4.53% % on average for Qwen2-1.5B, Danube2- 1.8B, and Gemma2-2B, respectively. As these evalua- tion benchmarks cover a wide range of domains relevant to real-world applications, these results demonstrate that DISTILLM-2 can be widely used to build strong sLMs.
ent capacity gap between the teacher and student models, which arises from factors such the number of parame- observe that gradually increasing
ters. Nevertheless, the SRKL coefficient 3 in Equation 2, following the lin- ear schedule shown in Algorithm 1, significantly improves student performance (see Table 5). This improvement is achieved by compromising the imitation of the teacher's 's be- havior, which is relatively hard to achieve, while focusing on directly obtaining feedback from SGOs, thereby effectively reducing the training-inferene mismatch.
4.2. Mathematical Reasoning
Setup. We conduct experiments on two standard mathemati- cal reasoning benchmarks: GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021). For teacher and student pairs, we select the Qwen2-Math-7BB
and Qwen2.5- Math-7B-Inst as teacher models and Qwen2-Math-Lae 1.5B and 1ren2.55Math-1155
4. Experiments
student models, respectively. The student models are trained using 50k randomly selected samples from the MetaMathQA (Yu et al., 2024a) dataset. Specifically, the student models are fine-tuned in a super- vised manner on the entire MetaMathQA for a single epoch.
4.1. General Pntruuction-ooloowing
Setup. We first construct the training datasets, by ran- domly sampling 50k prompts from UltraChat200k (Ding et al., 2023) and use the corresponding teacher and stu- dent generate the responses. After training, we evalu- ate DISTILLM-2 for general purpose instruction-folloving task on AlpacaEval (Li et al., 2023), Evol-Instruct (Xu et al., 2024a), and UltraFeedback (Cui et al., 2024). For evaluation, we adopt L.LM-aa-a-JJddee (Zheng et al., 2023) with GPT-4o or GPT-4o-mini as judge models. For experiments, we use Qwen2-7B (Hui et al., 2024), Mistral-7B (Jiang et al., 2023), Gemma-2-9B (DeepMind et al., 2024) instruction models as teachers and Qwen2-1.555 1.55B, Panube2-1.888 (Singer al., 2024), and Gemma-2-2B as students, respectively.
Results. Table summarizes the effectiveness of DISTILLM-2 compared to recent competitive baselines, including GKD and DistiLLM. In both the Qwen2 and Qwen2.5 experimental setups, DISTILLM-2 achieves higher performance than other baselines on the GSM8K and MATH evaluations. Interestingly, the Qwen2.5 student with DISTILLM-2 demonstrates competitive average per- formance and even outperforms the Qwen2 teacher on the MATH evaluation, which is a challenging milestone.
4.3. Code Generation
Results. We report experimental results in Table 2. This comparison between DISTILLM-2 and other baselines shows that our proposed method performs best in the most of evaluation setups, except for Danube2-1.80 in Evol-
Setup. We utilize prompts from WizardCoder (Luo et al., 2024) dataset which is developed using the Evol- Instruct method (Xu et al., 2024a) code instruction datasets. We apply Qwen2.5-Coder- BB-Int (Hui et al., 2024) and
7

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Table Comparison of preference optimization results (WR %)
Table 7. Evaluation on OK-VQA and Tex two popular bench- mark for visual question answering. We utilized VQA accuracy (Antol al., 2015). The best results are highlighted in bold.
using different reference models from various KD methods.
Qwen2-1.5B-SPP (Mg)
Gemma-2-2B-S05 (Ma)
Method
Evol
Ultra
AVG.
Evol
Ultra
AVG.
VQA Acc. (%)
Mr
Ms
GKD
DistiLLM
DMS
Ma
26.92
33,25
30.09
41.34
50.85
OK-VQA
54.70
36.87
41.83
39.38
44.72
GKD
47.84
65.48
56.66
59.13
82.83
70.98
DistiLLM
48.32
65.18
56.75
57.21
79.95
68.58
TextVQA
42.91
28.34
33.84
31.10
34,98
DISTILLM-2
52.88
67.78
60.33
68.99
83,58
76.29
AVG.
48.81
32.61
37.84
35.24
39.85
DeepSeek-Codent 6. as teacher models and Qwen2.5- Coder- 1.5B and DeepSeck-Codes Coder-1.3B as student models, respectively. Similarly, we train the student models for 2 epochs. We evaluate performance on two standard coding benchmarks: HumanEval (Chen al., 2021) and MBPP (Austin et al., 2021).
Table 8. Performance of the Qwen1.5 series with a 0.5B student model and varying teacher. The last column (A) shows the im- provement over Ko et al. (2024). The performance metric is the winning rate compared the original student models on Evol- Instruct.
Size
MM
GKD
DistiLLM
DISTILLM-2
A
188 (1)
81.07
64.18
65.23
66.31
+1.08
Results. The results are presented in Table 4. Across both HumanEval and MBPP, DISTILLM-2 consistently outper- forms the baseline methods, GKD and DistiLLM. Notably, GKD achieves higher scores than DistiLLM, its effective- ness remains lower than that of DISTILLM-2. This outcome highlights DISTILLM-2 's ability to integrate special- ized alignment strategy effectively incorporating SKL (or SRKL) with harmonized response type.
7B (1)
92.71
71.15
72.11
74.86
+2.75
14B (1)
95.16
72.59
72.11
76.78
+4.67
ity gap between large teacher models and compact student models makes KD more challenging, phenomenon re- ferred to the capacity gap (Mirzadeh et al., 2020). We experimented with diverse size of Qwen- 1 -5-Chat with 1.8B, 7B, and 14B parameters as teacher and SFT of Qwenl.5- 0.5B student to seize the behavior of DISTILLM-2 across the different size of teacher models. As shown in Table 8, DISTILLM-22 demonstrates monotonic improvement and consistently outperforms other baselines the teacher size increases. This result highlights DISTILLM-2's effective- ness in addressing capacity gap issues, whereas the previous version (Ko et al., 2024) struggled with capacity gaps, par- ticularly with the 7B and 14B teacher models.
5. Additional Ablation Study
Here, we provide additional ablation experiments on DISTILLMM22 Our ablation studies are conducted with Qwen2 (or Qwen2.5) from its diversified model sizes. We use GPT-4o-mini as a judge model for all ablation studies due to its cost-efficienyy.
Component Analysis. Here, we conducted a component analysis of DISTILLM-2 's technical components, which in- cluded (1) applying contrastive approach ($3.1 & $3.2), (2) increasing the .9 parameter, and (3) introducing curriculum- based updates to a ($3.3). Table 5 shows a component- wise analysis demonstrating how progressively incorporat- ing these improvements into DistiLLM brings its perfor- mance in line with that of DISTILLM-2.. As each compo- nent is added, we observe incremental performance gains, indicating that all of the examined components enhance DISTILLM-2 's overall effectiveness.
6. Broader Impacts
Furthermore, we present a range of diverse applications for DISTILLM-2, demonstrating its broad versatility and highlighting its potential for future use. We also provide additional applications of DISTILLM-2 in Appendix 6.3 (i.e., recovering quantized model) and 6.4 (i.e., achieving higher inference speed in speculative decoding).
6.1. Additional Results for DISTILLM-2 + DPO
In preference alignment (Ouyang et al., 2022; Rafailov et al., 2023), training usually involves two steps: (1) SFT and (2) preference fine-tuning using either PPO or DPO. While most previous works have concentrated on the second step. we highlight that the first step is also important. In our study, we replace the standard SFT method with KD of LLMs and evaluate its effectiveness by comparing how the policy LLMs perform after the second step, which uses DPO. Specifically, we use the reference model for each setup as trained student in Table 2. The results in Table show that replacing SFT with distillation in the first phase leads to
Training Size. We investigated how varying the training data size affects the performance of DISTILLM-22. In Fig- ure 3, we show its performance across various tasks-such as cnstruction-folloiiin ation- and compare it against baselines including GKD and DiM We observe that our proposed method con- sistently outperforms these baselines, demonstrating the highest effectiveness among all considered LLM distillation methods.
math reasoning, and code gener-
Capacity Gap. It is well known that substantial capac-
8

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Table Performance of 13.55-mmm--iistte at different levels of precision. The best results are highlighted in bold.
Table 10. Comparison of the inference speedup of speculative
decoding using different draft models obtained from various KD methods for the target models Phi3-medine and Phi3.5-mim
Size
AvpacaEval
Evol-Inst
UltraFeed
AVG.
Phi-
SFT
GKD
DistiLLM
LMS
Mr (8Flnat16)
91.80
82.80
80.54
85.05
Ms (INT4)
84.32
74.89
77.60
78.94
3-medium
Spd.
x
x x1.64
x1.71
x1.97
GKD (INT4)
88.57
80.04
79.80
82.80
Acpt. (1)
0.412
0.464
0.469
0.487
DistiLLM (INT4)
89.13
81.77
79.15
83,35
Spd. (1)
x1.24
x x1.58
x1.65
x1.84
3.5-mini
DISTILLM-2 (INT4)
89.17
81.96
79.58
83,57
Acpi. (t)
0.397
0.443
0.452
0.522
higher overall alignment performance in preference fine- tuning. Notably, our DISTILLM-2 achieved a substantially higher WR, more than doubling the WR in Qwen2-1. 1.5B, and showed similar improvement in Gemma2-2B. These indicate that DISTILLMM2 can build effective reference models for the subsequent preference alignment phase.
and verifier models. Building on their work, we evaluate the inference speedup of speculative decoding using Phi3,5- mini and Phi3-medium (Abdin et al., 2024) as verifiers and Llama-68m (Miao et al., 2024) as the drafter trained with various KD. Table 10 summarizes that the inference speedup of drafter with DISTILLM-2 surpasses other drafter mod- els, including trained with SFT and DistiLLM for both thi3.5-mini and Phi3-medium verifiers. These results indi- cate that the DISTILLM-2 enables higher token-level align- ment of distribution compared to other LLM distillation baselines, including Zhou et al. (2024b).
6.2. Expansion to Vision-Language Models
We also applied DISTILLM-2 on the distillation setup of vision-language models (VLMs) to boast the versatility of proposed method that can be applied in a wide range of modalities. We select LLa 1,5-7B (Liu al., 2024) and teacher and
7. Conclusion
TinyLLaVA-1.4D (Zhou et al., 2024a) student model, respectively. For training dataset, utilize the prompt from RILAIF-V-Datase (Yu et al., 2024b) which contains 83K prompts, and evaluate the trained models on two popular benchmark, OK-VQA (Marino et al., 2019) and TextVQA (Singh et al., 2019). Table 7 shows that the superi- ority of DISTILLM-2 over other distillation methods holds true not only in LLM setups but also with VLMs. Although other baselines also demonstrated effectiveness compared to original student models (i.e., Ms), DISTILLM-2 out- performed GKD and DistiLLM by +2.01 % and +4.61% % on average, respectively.
In this work, we introduce DISTILLM-2, a novel distilla- tion framework for large language models that combines contrastive loss, curated data, and curriculum-based leam- ing. By differentiating teacher and student outputs, our method overcomes key limitations of traditional distillation and achieves stronger alignment and generalization. Exten- sive experiments across instruction following, mathematical reasoning, and code generation show that DiSTIL.LM-2 de- livers state-of- the-art performance with improved sample ef- ficiency, reducing reliance on expensive preference-labelee data. Leveraging high-quality explanations and contrastive objectives further enhances reasoning ability and robustness in both language and vision-language models. We believe DISTILLM-2 lays the groundwork for future advances in ef- ficient model alignment and multi- learning, enabling more accessible and capable AI systems.
6.3. Restoring the Performance of Quantized LLMs
Using parameter-effiieen fine-tuning methods, such as LoRA, can help recover the performance of quantized LLMs after post-training quantization (Frantar et al., 2023), intro- ducing only negligible number of additional parameters. Here, we demonstrate the effectiveness of DISTILLM-2 2 in restoring the performance of 4-bit quantized LLMs using LoRA by replacing regular SET with KD baselines. Figure 9 shows that all KD methods can significantly improve the performance of quantized models while adding only a few trainable parameters. Additionally, DISTILLM-2 achieves the best average performance among KD basclines. Distilla- tion can also be straightforwardly applied to pairs of original and compressed models, such as pruned or quantized ver- sions, enabling efficient deployment on mobile devices.
A Acknowledgements
JK, SK, and SY were (partially) supported by Institute of In- formation & communications Technology Planning & Eval- uation (ITPP) grant funded by the Korea govermment (MSIT) (No. RS-2019-II19007 Artificial Intelligence Graduate School Program (KAIST)) and the Institute of Informa- tion & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2024-00457882, AI Research Hub Project) and Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration).
6.4. Inference Speedup of Speculative Decoding
DistillSpee (Zhou et al., 2024b) demonstrate that KD can improve speculative decoding by better aligning the drafter

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Impact Statement
References
This paper presents work whose goal is to advance the field of machine learning, specifically efficiency and efficacy of LLMs-based system. We also have shown four possible applications (but not limited to) as follows:
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404.142) 2024.
High-performed reference model for RLHF ($6.1): : Our DISTILLM-2 can replace vanilla SFT ahead of PPO with reward model or DPO (or IPO, Ming et al. 2024) with chosen and rejected response pairs. While our experiments Table 6 are conducted on DPO, we believe that this approach can be expanded to various types of preference alignment methods, which can significantly contribute to building safe AI.
Agarwal, R.,. Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.,, and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Extension to multi-modal LLMs ($6.2) While our DISTILLM-2 is primarily evaluated on LLMs, we also demonstrate its potential adaptability to VLMs in Table Since many multimodal large language models (MLLMs: Dai et al. 2023; Tang et al. 2024) are built on LLMs, we believe that DISTILLM-2 can be effectively applied wide range of MLLMs.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021.
Azar, M. G., Guo. Z. D.. Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Recovering the compressed LLMs ($6.3) : While network pruning (Ko et al., 2023b) and quantization (Shao et al., 2024) have significantly improved the effi- ciency of LLMs, they often come at the cost of inher- ent performance degradation compared to the uncom- pressed original models. Our experiments demonstrate that DISTILLM-2 can substantially enhance the perfor- mance of quantized LLMs, making them highly compet- itive with their original counterparts. We believe that DiSTILLM-2 can further improve the effectiveness of compressed models, such as quantized LLMs, while also being applicable other compression techniques.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple LLM inference accelera- tion framework with multiple decoding heads. In Forty- first International Conference on Machine Learning, 2024. URL https: http///pppeeeeeeeeeeeeeeeeeeeeet net forum? i id=PEpbUobf.v.
Enhancing Inference speed of LLMs ($6.4) We also show that DISTILLM-2 22 can significantly enhance the effi- cacy of speculative decoding (Chen et al., 2023; Leviathan et al., 2023) by improving the alignment between draft and target models. We believe that DISTILLM-2 can be beneficial for systems that utilize diverse models within single framework improve both efficiency and efficacy.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre,, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01316 2023.
Chen, M., Tworek, J., Jun, H., Yuan, Q.. Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman. G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
While we have already demonstrated the applicability of our work across various domains, we believe that it can be utilized in even broader range of fields. For example, the reasoning ability showcased by DeepSeek-RI (Guo et al., 2025) could be integrated with our methodology to yield more impactful results for sLMs. We encourage future research to explore and discuss the broader implications of this work across diverse domains.
Christiano, P. F., Leike, J., Brown, T.. Martic, M., Legg. S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
10

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint arXiv: 10....10...,, 2021.
International Conference on Learning Representations, 2024.
Guo, D., Yang, D., Zhang, H.,, Song, J., Zhang. R., Xu, R., Zhu, Q. Ma, S., Wang. P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint 1rXiv:2501.12455 2025.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction-tuned 1lm, 2023. URL https: wwwpww.atabriicce. www.bllg/cclcocoocccoccccccoocc cwt.bblcgg///2/////2222222 holly-firste open wow.ercially instruction-tunee
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirly-fitth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: ptttpp:://ppereeeeee wwwppppppneeeeeeeeeeeeeee net. forum? i id=7Bywt2m0se
Cui, G., Yuan, L., Ding, N., Yao, G.. He, B., Zhu, W. Ni. Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled feedback. In Forty-first International Conference on Machine Learning, 2024.
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015. URL https: / /ppi. semant commantccccoo//oo cscholar. org/ Cas/porrusss ID 7200347.
Dai, W. Li, Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. InstructBLIP: Towards general- purpose vision-language models with instruc- tion tuning. In Seirty-eeventh Conference on Neural Information Processing Systems, 2023. URL https: //openreviee net. forum? d=vvoRPYy2.JJ.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. https: / //openreview wwww wtt.://mmmm-o-eeeeeeeeee...
DeepMind, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, T., Shahriari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv e-prints, pp. arXiv-2408, 2024.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L.,, Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2/86, 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S.,, Casas, D. d. 1., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, et al. Mistral 7b. arXiv preprint DxXiv:330006822 2023.
Ding, N.. Chen, Y. Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. In Bouamor, H., Pino. J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3029-3051, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.1865331120033.cmml.m 183. URL https: http:///aaaaaloo..oooooooo emnlp-main.11
Ko, J., Park, S., Jeong, M., Hong, S., Ahn, E., Chang, D.-.., and Yun. S.-Y. Revisiting intermediate layer distillation for compressing language models: An overfitting perspective. In Vlachos, A. and Augen- stein, I. (eds.), Findings of the Association for Com- putational Linguistics: EACL 2023, pp. 158-175, Dubrovnik, Croatia, May 2023a. Association for Compu- tational Linguistics. doi: 10. 1t/16/////1/223..nnnnnnngn/.// 12. URL https: httpss////caaatthtocccs wwwpeloooy.orr//// 2023. Ef Findings-eacl.122 12/.
mammain... mto-mainn.1......
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J.. Guestrinn C., Liang, P. S., and Hashimoto. T. B. Al- pacafarm: simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Ko, J., Park, S., Kim, Y., Ahn, S., Chang, D.-S., Ahn, E., and Yun, S.-Y. NASH: A simple unified frame- work of structured pruning for accelerating encoder- decoder language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computa- tional Linguisties: EMNLP 2023, pp. 6076-6093, Singa- pore, December 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023 1/t.:3//i///....iinngg.... mandings-emnlp 404. URL https: / /aa/aaaaaa /aclaataccooggyooo/ooo .rg. 2023. findings-emtt 404/.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Confer- ence on Learning Representations, 2023, URL https: // //openreview.co. wttp:eiiiwiiineennnnnee..aaaeeee wttt:errrmmtitttnnneeeeeeee
Gu, Y., Dong, L., Wei, F., and Huang, M. Minillm: Knowl- edge distillation of large language models. In The Twelfth

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Ko, J., Kim, S., Chen, T., and Yun, S.-Y. DistiLLM: To- wards streamlined distillation for large language mod- els. In Forty-first International Conference on Machine Learning, 2024. URL https: http://:////peeeeee....o. forum? 1 d= s H
Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6121 6133, Online, November 2020. Association for Computa- tional Linguistics. doi: 10. 10.18633/11/0022eemmlmmmmm 494. URL https: https////caaatthloog... /ttttts:///ccaaathtato 2020.
Ko, J., Dingliwal, S., Ganesh, B., Sengupta, S., Bodap- ati. S. B., and Galstyan, A. ScRA: Self-reviewing and alignment of LLMs using implicit reward margins. In The Thirleenth International Conference on Learning Representations, 2025. https: /pwpnreeee...eeeeeeeeeee. net/ win.ummiiidiIInuyDSSS..
emn p panlllpmaiin mww.. 494.
Lin, C.-Y. ROUGE: A package for automatic evalua- tion of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https: //aclanthologe org 2rrg/w4--- 1ww0-0011//4/1100001000010011111
Kwon, W. Li, Z.,, Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Liu, J., Xia, C. S., Wang, Y., and ZHANG, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023, URL https: forum?id=1que https://openrevier wppp//veenreeeeeeeeeeeeeeemmae.e .nt.tiewwwee/eeeeee............
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Inter- national Conference on Machine Learning, pp. 19274- 19286. PMLR, 2023.
Li, M., Chen, L., Chen, J., He, S., Gu, J., and Zhou, T. Selective reflection-tuning Student-selected data recy- cling for LLM instruction-tuning. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 16189- 16211, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. 10.1844
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang. D. Wizardeoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: /owprr/v////peeeeeeee/eeeeeeeeee .ntts//wweee//eeeee... forum? Souum?dddnnUUSSqSS.
findings-acl.955 URL https: // /////aalaa /aclaanhoooogy.... org/2024 findingsnn-ii
958.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEE/cvf conference computer vision and pattern recognition, pp. 3195-3204, 2019.
Li, M., Zhou, F., and Song, X. BiLD: Bi-directional logits difference loss for large language model distillation. In Rambow, O., Wanner, L., Apidianaki, M., AI-Khalifa, H., Eugenio, B. D., and Schockaert, S. (eds.), Proceed- ings of the 31st International Conference on Computa- tional Linguistics, pp. 1168-1182, Abu Dhabi, UAE, January 2025. Association for Computational Linguis- tics. URL https::/p//.. / /aaclaaaatttt acllanhhlogyy....o..ooooooooooma org.o/00..../20...... coling-main.788 m
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-free reward. arXiv preprint 1rXii:2405.14734 2024.
Meta.
Llama 3.2: Revolutionizing edge AI and vision with open, customizable mod- cls.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of fostouction-oolloiinng models. hitps:/ htt///ggtuub/tt// wttpb//cooo/tttaaataeeaaa /ab/ alpaca_eval, 5 2023.
https: //ai http:://a.mmmaaacoom llama 3--2 connect- wow-ettoooo-iiii--eeeeeeee mobile-devices/. 2024.
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z.. Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. Specinfer: Accelerating large language model serv- ing with tree-based speculative inference and verification. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 932-949, 2024.
Li, Y., Gu, Y., Dong, L., Wang, D., Cheng, Y., and Wei, F. Direct preference knowledge distillation for large lan- guage models. arXiv preprint 4rXiv:2406.1977 2024b.
Lin, A., Wohlwend, J., Chen, H., and Lei, T. Autoregres- sive knowledge distillation through imitation learning. In
12

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Tajwar, F,, Singh, A., Sharma, A., Rafailov, R., Schnei- der, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of LLMs should leverage subopti- mal, on-policy data. In Forty-first International Con- ference on Machine Learning, 2024. URL https: / //openreview.o net wWN.:iidd-bNPx6t0000
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 1177730-27744 2022.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, Ti, W., Lu, L., MA, Z., and Zhang. C. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL wwwp:::tp////cc...t httppp:://pppeeeeee /pppereeeeeeeeeeeeeeeeeeeeeeeeee net/ forum? id-14rn7HpKVk
Pang, R. Y., Yuan, W., He, H., Cho, K., Sukhbaatar, S., and Weston, J. E. Iterative reasoning preference optimiza- tion. In The eighth- Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview.o net www.mmmmmd444EEiie
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., al. Zephyr: Direct distillation of Im alignment. arXiv preprint urXiv:2310.169444 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for task agnostic compression of pre-trane pre-trained transformers. Advances in neural information processing systems, 33: 5776-5788, 2020.
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2023.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi. D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Pro- ceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13506 Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10. 18653v1v1 2023.acl-long 754. URL atttss://aclattheee org org /2gg2223..mm acle223.ccl110 acl long. 754.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint ar 55Xiv:2404.03715, 2024.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview net forum? wwwwwh...........
Wen, Y., Li, Z., Du, W., and Mou, L. f-divergence mini- mization for sequence-level knowledge distillation. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Pro- ceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10817-10833 Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10. 18 2023.acl-long.603 URL https: http::///caaathooorg... or org/2023. ac1-1ong....... 605.
Singer, P., Pfeiffer, P., Babakhin, Y., Jeblick, M., Dhankhar, N., Fodor, G., and Ambati, S. S. H2o-danube-L.. b technical report. arXiv preprint arXiv:2401.16811 2024.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Wu, T., Tao, C., Wang, J., Yang, R., Zhao, Z., and Wong, N. Rethinking kullhack-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.0m 026557,,0........ 2024.
Stiennon, N., Ouyang, L.., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Wu, T., Tao, C., Wang, J., Yang. R., Zhao, Z., and Wong, N. Rethinking Kullback-Leible divergence in knowl- edge distillation for large language models. In Ram- bow, O., Wanner, I.,, Apidianaki, M., Al-Khalifa, H.,
13

---

DISTILLM-22: A Contrastive Approach Boosts the Distillation of LLMs
Eugenio, B. D., and Schockaert, S. (eds.), Proceed- ings of the 31st International Conference on Compu- tational Linguistics, pp. 5737-5755, Abu Dhabi, UAE, January 2025. Association for Computational Linguis- tics. URL https:t/p//.. : acc antho org/2025. coling-mann mainn. 383/.
Zhou, Y., Lyu, K., Rawat, A. S., Menon, A. K., Ros- tamizadeh, A., Kumar, S., Kagy, J.-F,, and Agarwal, R. Distillspee: Improving speculative decoding via knowl- edge distillation. In The Twelfth International Confer- ence on Learning Representations, 2024b. URL https: / //openreview.. ftte://ommmmmddddX322322222
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardLM: Empower- ing large pre-trained language models to follow com- plex instructions. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL ht hts::: //openreview wwt.riieweeeeee.e....e forum? www.ooooiumeeer...ooo C 9NDggh 3NDgH.
Zwillinger, D. CRC standard mathematical tables and for- mulae. chapman and hall/CRC, 2002.
Xu, W., Han, R., Wang, Z., Le, L. T., Madeka, D., Li, L., Wang, W. Y., Agarwal, R., Lee, C.-Y., and Pfister, T. Speculative knowledge distillation: Bridging the teacher- student gap through interleaved sampling. arXiv preprint arXiv:2410. 11255, 2024b.
Yu, L., Jiang, W., Shi, H., YU, J. Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL https: //openreview.e.../// htw.leeww.e///futttiieeo forum? NeNNNNN
Yu, T., Zhang, H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt- 4v trustworthiness. arXiv preprint a:Xiv:2405.17220, 2024b.
Zhang, S., Zhang, X., Sun, Z., Chen, Y., and Xu, Dual- space knowledge distillation for large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-.. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 18164-18111, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/1/2024. emnlp- moulp-main.100 1010. 1010. URL https: http://aaaaatttooooo/o// org/ 2024 emn 1p main.1010:
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 104252 2023.
Zheng, L.,, Chiang, W.-L.., Sheng, Y., Zhuang, S., Wu, Z.,, Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 46595 46623, 2023.
Zhou, B., Hu, Y., Weng, X., Jia, J., Luo, J. Liu, X., Wu, J., and Huang, L. Tinyllava: A framework of scall-scale large multimodal models. arXiv preprint arXiv:2402.1 14289, 2024a.
14

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Supplementary Material
A. Additional Related Works
KD in LLMs. Recently, several works have pioneered the KD for LLMs (Gu et al., 2024; Agarwal et al., 2024; Ko et al., 2024). Unlike small BERT-based models, which have focused on intermediate layer distillation (Wang et al., 2020; Ko et al., 2023a), most works on LLMs have focused on logit-based distillation due to their large number of parameters. Gu et al.
(2024) proposed a policy gradient-based method addressing the high variance issues in RL-based methods. Agarwal et al. (2024) propose on-policy approach of SGO with diverse objectives like RKLD and JSD. Based on these pioneer, numerous works (Xu al., 2024b; Zhang et al., 2024; Li al., 2025) continuously studied improve the performance of KD in LLMs. Li et al. (2025) filter out long-tail noise by utilizing top-k teacher and student logits and leverage internal logit ranking information by constructing logit differences. Zhang et al. (2024) introduced the dual-space knowledge distillation (DSKD) framework, which unifies the output spaces of the two models for KD. Similar to our work, Wu et al. (2025) provided adaptive KL. to balance their early-stage behaviors of KI. and RKL., however, they do not consider about the data perspective of LLM distillation.
Contrastive approach. Actor-critic RLHF frameworks (Christiano et al., 2017; Stiennon et al., 2020; Bai et al., 2022: Ouyang et al., 2022) seeks to align language models to human preferences, but is often unstable during training and memory-intensive (requiring the policy model and reward model to be on device simultaneously). To mitigate this, several algorithms (Rafailov et al., 2023; Azar et al., 2024; Ko et al., 2025), such as direct preference optimization (DPO; Rafailov et al. 2023) and sequence likelihood calibration (SLiC-HF; Zhao et al. 2023), learn the contrastive preference in the offline setting using a closed-form loss function without the need for an critic/reward model. (Azar et al., 2024) argued that without regularization, a policy can easily overfit to deterministic preferences and introduced identity preference optimization (IPO) to directly optimize offline preference probabilities with regularization.
B. Derivation for Mathematical Analysis
B.1. Mathematical Explanation on Behavior of KL and RKL
Here, we provide mathematical explanation for (S) and (S)RKL showed in Figure I(a) and (b). Formally, the f-divergence of two distributions is defined as
[s I p2 (p(y)x))
)
[
dp
Dy(p'r1) = =By=pp
=Rpt
Epp
dp2
where dp' and dp² are the probability densities of probability p¹ and p². The KL is fdiieegennce generated by f(t) = t log t and RKL is f-divergence by f(t) = log t. From Ko et al. (2024), the a-skew KL divergence is f-divergence generated by (t) = t log ( and a-skl RKL is is a f-divergence generated by f(a) = log ((1 - a)t + a). Based on these property, we provide detailed explanation for the empirical observation.
Pulling-up effect of (S)KL: By taking f(t) = flog i, we have lim, f(t) +o0 Because p(- p(-(x) E (0, and
qa (+(x)) E (0, 1), qa (-1x)) cannot be too small when p(-|z) is significantly greater than 0. As result, go (x) is encouraged
to "pull up" its values where (()1z)) is large. Similarly, for SKL, by taking f(1) = l log (a+1-a)): we also have lim;- +oo. Thus, SKL also benefits from the same "pulling-up" property of the KL-like term
Pushing-down effecc of (S)RKL: By taking ft) logt, we have lim4-s0 f(t too, which means (-(-) should be small when p(-(z) is small. As result, (-(-1=)) is encouraged to "push down" its values where p(-|æ) is near zero.
Similarly, for SRKL, by taking (() = log ((1 at)/ + at), we also have lim,-ot [(() +oo Thus, SRKL also benefits from the same "pushing-down" property of RKI.-like term.
While Wu et al. (2025) provided a similar observation, their explanation only holds for a unimodal Gaussian distribution, whereas ours applies to more general problem setup. Additionally, our explanation provides mathematical intuition for both S(R)KI, and KR)
15

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
B.2. Derivation for Remark
Based the definitions of SKL and SRKL, we have
D (0) (x, y.) + S (x, yx) = p(ye (x) log
p(y. (x)
+ qo (3. log
90 x))
(8)
Ya (y)())) (x)
po (y/ (x)
log p (z)
20 y (z) Pu ya 2
= -p(
log
(9)
qo Uv (x)
Furthermore, Yt and Ys are independent, the following holds by the linearity of expectation:
Bp(yi les) log p(ye (x) (V a:) + E (x)1)/
Qa (yx(z)
p( ())
qo go (x)
log
log
log
(10)
(y
To (3r (z))
Po (ys [z)
p a²) qo (z)
Pe (y (xx)
log
log
(11)
90 (ys x))
From this, verify that our Equation be interpreted Equation 6, which behaves similarly the contrastive approach defined in DPO (Rafailov et al., 2023).
B.3. First-order Approximation for Mercator Series
From the Mercator series expansion, following hold:
20
2
x²
x
log(1 + x)
Σ (-1)at
+
(12)
2
3
where the series converges to the natural logarithm whenever - < << By substituting p(y)æ)) - into x, we can write as follows,
logp(y/(>) (p(y|x) 1) (p(y)/z) - 1)² (p(y)æx) -1)
(13)
2
Since the softmax outputs of LLMs, p Iz), satisfy 0 < ((y(z) < by the definition of probability, it follows that -1 < p(y|z) < 0. This holds because the softmax function outputs strictly positive values due to the exponential transformation of real-valued inputs.
Hence, from the first-order Mercator series expansion approximation, we have
p(y(2)
log ap(y ar) (1 ())l)(
log ax) log (ap(y)æ + (1 - a) (v(zz)
(14)
[(p(y) - 1) (ap(v)z) + (1 ())(())22)) - 1)] - (11-a)plylz)) + (1 ())((= = (1 - a) (p(y|<)) go (3)l)))
(15)
(16)
which holds for 0 S a S 1. By choosing first-order approximation, we can express the S(R)KL as a closed- form function of a, p(y(=)) and qo (y/lr) which enables to compute proper a for each sample easily. Instead, as we compromise approximation error for either p(y]æ)) or qo (yyy))) < 1, we apply mini-batch wise allocation and clipping for improving the stability of implementation of curicuuuu-bbsse approach. For clipping, we utilize upper and lower bound 0.1 and 0.01.
C. Detailed Experimental Setup
We elaborate the detailed experimental setup regarding the datasets used ($C.1), training details (SC.2), and evaluation details (SC.3). For all experiments, we implement DISTILLM--2 using the trl framework, as well as for other baselines, including GKD (Agarwal al., 2024) and SKD (Xu et al., 2024b).
16

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
C.1. Dataset Description
We apply DISTILLM-2 on instruction-foolooinng math reasoning, and code generation datasets. We provide detailed descriptions of the datasets used.
UltraChat200k instruction-followinne Tunstall et al. 2023 +): This is a heavily filtered version of UltraChat (Ding et al., 2023), originally used to train Zephyr-7B-3 (Tunstall et al., 2023). It is obtained from the original version, which consists of 1.4M dialogues generated by ChatGPT and spans wide range of topics, by removing the dialogues that contain grammatical errors or where the assistant replies with phrases like "I do not have emotions" or "I don't 't have opinions."
AlpacaEval mnsstruction-following Dubois et al. 2024 This dataset is slight modifications (or simplification) of the AlpacaFarm evaluation set. Dubois et al. (2024) first merged the instruction and input fields into single instruction field, This affects 1/4 of the examples in the AlpacaFarm evaluation set, all of which are from the Self-Instruct (Wang et al., 2023). This dataset contains 805 challenging questions.
Evol-Instruct Evaluation (instraction-following: Xu et al. 2024a 9) Evol-Instruct (Xu et al., 2024a) contains 218 questions, spanning multiple topics generated using the Evol-Instruct procedure.
UltraFeedback cntrruction-folloiinn Cui et al. 2024; Tunstall et al. 2023 *)) This is a large-scale, fine-grained, and diverse preference dataset used for training powerful reward models and critic models. Cui al. (2024) collected about 64k prompts from diverse resources, including UltraChat, ShareGPT, and Evol-Instruction (Xu et al., 2024a). They used these prompts to query multiple LLMs, generating four different responses for each prompt. The responses were annotated using GPT-4 to collect high-quality preferences based on instruction-followingg truthfulness, honesty, and helpfulness.
MetaMathQA (mathematical reasoning: Yu et al. 2024a 9) MetaMathQA is a dataset introduced in Yu al. (2024a) improve mathematical reasoning in large language models. It is created through question bootstrapping, where mathematical problems are rewritten from multiple perspectives, including forward reasoning, backward reasoning, and rephrasing.
GSM8K (mathematical reasoning; Cobbe et al. 2021 ) GSM8K (Grade School Math 8K) is a dataset comprising 8.5K high-quality, linguistically diverse grade school math word problems. It is designed to facilitate question answering on fundamental mathematical problems that involve multi-step reasoning.
MATH (mathematical reasoning; Hendrycks et al. 2021 "): This dataset code generates mathematical question and answer pairs covering various question types at approximately school-level difficulty. It is designed evaluate learning models' mathematical comprehension and algebraic reasoning abilities.
WizardCoder (code generation; Luo et al. 2024 12): WizardCoder dataset is constructed using the Evol-Instruct method, which refines and expands existing code instruction datasets. The process starts with Code Alpaca, 20K-sample forlowiin dataset, and iteratively applies instruction evolution techniques to generate progressively more complex training data. These modifications include adding constraints, increasing reasoning steps, providing misleading code, and enforcing time-space complexity requirements. The final dataset consists of approximately 78K evolved samples, which are used to fine-tune the StarCoder model, significantly improving its performance on code generation benchmarks.
EvaanEval (code generation; Chen et al. 2021 13): The HumanEval dataset, released by OpenAI, consists of 164 programming problems, each containing function signature, docstring, body, and multiple unit tests. These problems were manually crafted to ensure they were not part of the training data for code generation models.
MBPP (code generation; Austin et al. 2021 14); The benchmark includes approximately 1,000 crowd-soureed Python programming problems, desigeed for entry-level programmers and covering programming fundamentals, standard library
Jsonl
1
17

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
functions, and more. Each problem features a task description, a code solution, and three automated test cases. As noted in the paper, portion of the dataset has been manually verified.
RLAIF- V-Daasett (visual question answering; Yu et al. 2024b 19:; RLAIF-V-Datasel is a comprehensive multimodal feedback dataset featuring 83,132 preference pairs with high-quality annotations. The instructions are sourced from a diverse selection of datasets, including MSCOCO, ShareGiT-44V, MovieNet, Google Landmark v2, VQA v2, OKVQA, and Tex Additionally, authors incorporate image description prompts from RLHF-V, utilizing them as long-fon image-captioning instructions.
OK-VQA (visual question answering; Marino et al. 2019 1b): OK-VQA is a large-scale visual question answering (VQA) dataset with over 14,000 questions that require external knowledge to answer. Unlike traditional VQA datasets, it challenges models to retrieve and integrate external information rather than relying solely on image content. As a diverse and challenging dataset, OK-VQA surpasses previous knowledge-hased VQA benchmarks in scale, making it a crucial resource for advancing Al reasoning capabilities.
Te (visual question answering: Singh et al. 2019 17): TextVQA is a dataset designed to benchmark visual reasoning based on text in images. To answer TextVQA questions, models must read and interpret text within images, integrating this textual information into their reasoning process. Unlike traditional VQA tasks, TextVQA requires models to handle both visual and textual modalities, making it a unique challenge in multi-modal learning.
C.2. Training Details
Here, we describe the hyperparameters and implementation details for training with DISTILLM-2, Our hyperparameters are shown in Table 11. For all experiments, we utilize LoRA (low-rank adaptation; Hu et al. 2022), which one of the most popular parameter-efficient fine-tuning techniques, for training efficiency. For all models, we use the maximum batch size that fits on 4 NVIDIA A100 80GB GPUs, while matching the effective batch size with 128 by considering the batch size and gradient accumulation. For all experiments in $4, we first train the student models on training datasets with ground-truth responses using SFT, and then conduct KD for LLMs. Instead, we also provide the results for the student models initialized from instruction models with 2emma-2-2B-it (DeepMind al., 2024) in Appendix D.2. Unlike the previous version (Ko et al., 2024), do not use language modeling loss on pre- pre-training corpus for all experiments.
Table 11. Hyperparameter values used in DISTILLM-2 experiments in $4.
Hyperparameter
fowtron Mathematical Reasoning
Code generation
Fine-tuning method
LoRA (r 16)
Target module for LoRA
all lincar layers for self-attention and MLP layers in Transformer network
Learning rale
5.0 10-3
Effeetive Batch Size
128
Epochs
epochs
2 epochs
epochs
Initial oo
0.1, we do not use curriculum-based update in 1st epoch.
Clipping value &
0.5
C.3. Evaluation
Instruction-followin For evaluating the trained LLMs, we applied single NVIDIA A100 80GB GPU for sampling the responses from each model using a temperature of 0.8, top-p value of 0.95, a max-length limit of 512. For LLM-as-a- Judge (Zheng et al., 2023) evaluation, we use a pairwise comparison prompt which depicted in Figure 4 with setting the temperature of 0.7. For AlpacaEval,, we conducted pairwise comparisons against responses from contt-dvvinii--000 which have been officially released. For Evol-Instruct and UltraFeedhack, we compared generated responses to those from gpt-3 5 turbo which were produced internally. To avoid position bias, we average the results by switching the order of the compared responses.
Mathematical Reasoning & Code generation. For evaluating the trained student models, we applied a single NVIDIA A 110 80GB GPU for sampling the responses from the each model using greedy sampling, a max-length limit of 1024. Specifically. for code generation, our evaluation is conducted on EvalPlus framework (Liu et al., 2023).
18

---

DISTILLM-2: A Contrastive Approach Boosts the Distillation of LLMs
[System]
Please act as an impartial judge and evaluate the quality of the responses provided by two Al assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the responses and provide short explanation. Avoid any position biases and ensure not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[]]] if assistant B is better, and "[[C]]" for tie.
the order in which the responses were presented does not influence your decision. Do
[User Question] {question)
[The Start of Assistant A's Answer] {answer_a)
(The End of Assistant A's Answer]
I'The Start of Assistant B's Answer]
(answer_b)
[The End of Assistant B's Answer]
Figure 4. The pairwise comparison prompt introduced LLM-as- (Zheng al., 2023).
D. Additional Experimental Results
D.1. Comparison on On-policy Setup
Setup. We also compare our methods with on-policy manner algorithms, in the code-base of DistiLLM. We compare the recent on-policy distillation baselines, in- cluding ImitKD (Lin et al., 2020), MiniLLM (Gu et al., 2024), GKD. We also provide the results for the adaptive on-policy setup for D and DISTILL-M-2 Also, we conducted experiments with DPKD and DPO. We follow the experimental setup of Ko et al. (2024), which
Table 12. Application of "batched" on-policy setup compared to fully on on-policy and off-policy setup. We evaluate the student models in databricks-dolly-155. test set with ROUGE-L following Ko et al. (2024).
Minil
GKD
DPO
[)KT)>
Disil.I.
D.s-
policy hanched on-pollcy (ours)
23.84
23.75
6.85
26.12
26.37
25.75
23.21
23.42
6.43
26.11
26,20
off-pollicy
23,41
22.89
23.78
6.43
26.12
26.13
trained GPT-2 (Radford al., 2019) on databricks-dolly-15. (Conover al., 2023). All hyper-purameter setups are from (Ko et al., 2024). Note that we apply same experimental setup for the result in Figure 1(c).
Results. Table 12 show that our "batched" on-policy setup, which shares the same takeaway the adaptive off-polic approach in Ko et al. (2024), does not suffer from severe performance degradation despite its significant efficiency. Also, we observe that DPKD (Li et al., 2024b) performs much worse than its reported values, as it is prone to reward hacking, as we reported in 1..1. Hence, we decide not to include DPKD in our main baselines in $4. While we provide results for DPO, except in the on-policy setup, its performance is worse than DISTILLLM-2 but better than DPKD.
D.2. LLM distillation with Inst models
While the results in Table 2 focus the base model as the student model, we also provide results using the inst model as the student model, specifically Gemma-2-2B. (DeepMind et al., 2024) with Gemma-2-9BB-SmmPO (Meng et al., 2024) as the teacher model. We use the same experimental setup as the base model, except that we train the inst model for only a single epoch of 200 training iterations.
Table 13. Comparison of the teacher model (MMr) and student models
with different KD methods. Note that DISTILLM-2 achieve higher performance than teacher in UltraFeedback evaluation.
Mr
Ms
GKD
DistiLLM
DiSTILLM-2
Evol-Inst
88.76
76.80
79.57
80.28
85.10
UltraFeedback
85.90
79.52
81.94
85.56
88.26
Overall, the student models in Table 13 achieve higher performance compared to the base models (+SFT) in Table 2. Notably, our DISTILLM-2 achieves even higher performance in the UltraFeedback evaluation compared to other student models, demonstrating that LLM distillation remains effective for recent state-of-the-ar models. We belive that these results stem from the fast convergence of the contrastive approach in our DISTILLM-2. even with very limited training iterations.
19


Reviews:
[{"review_id": "4HfAJaiOwb", "reviewer": "Reviewer_nZhc", "summary": "The paper introduces DISTILLM-2, a novel approach for LLM knowledge distillation. Unlike prior work that applies identical loss functions to both teacher- and student-generated data, DISTILLM-2 leverages a contrastive loss function to explicitly increase the likelihood of teacher responses while decreasing that of student responses. Extensive experiments demonstrate that DISTILLM-2 achieves superior performance across multiple tasks, including instruction-following, mathematical reasoning, and code generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We have rephrased your comments for simpler reference and have included our respective responses. Look forward to any further discussions.\n\n***Q1. Use of models up to 9B; interest in evaluating scalability to larger models***\n\nThank you for the insightful comment. In the evaluation, we already demonstrated the effectiveness of DistiLLM-2 over varying sizes of teacher and student models. For teacher models, we have included larger backbones with over 9B parameters, such as **Qwen1.5-14B** in **Appendix D.2** (Table 10) and **Phi-3-Medium-14B** in **Appendix D.4** (Table 12). \n\nFor larger student models, we are working on conducting more such experiments. Although we might not be able to report the results in a timely manner during rebuttal due to the resource cost, we will report them in the future.\n\nMathematically, as explained in Remark 1, DistiLLM-2 proposes a dedicated CALD objective function that could exhibit behavior similar to DPO in preference alignment. Given DPO’s wide validation across diverse architectures and the supporting results from our experiments, we are confident in the generality of the proposed approach.", "claims_and_evidence": "The claims in the paper are well-supported by clear and convincing evidence. \nSections 4.1 to 4.3 present experiments across three tasks—general instruction-following, mathematical reasoning, and code generation—using three different LLM teacher-student pairs. The results consistently demonstrate that DISTILLM-2 outperforms baseline KD methods, providing enough empirical validation for its effectiveness.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes, I reviewed the experimental design and analyses, which appear to be sound and well-structured. However, the largest LLM used in the study is only 9B, which raises concerns about the generalizability of the conclusions to larger-scale models. Additional experiments with larger models would strengthen the validity of the findings.", "relation_to_broader_scientific_literature": "Yes.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Please refer to the raised concern in `Experimental Designs Or Analyses`", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1LEhG4T1P9", "reviewer": "Reviewer_F2H5", "summary": "This paper introduces DistiLLM-2, a contrastive approach for LLM distillation, optimizing student models by increasing teacher response likelihood (SKL loss) and decreasing student response likelihood (SRKL loss). It improves data curation, adaptive learning, and curriculum-based loss weighting, outperforming baselines in instruction-following, math reasoning, and code generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. Below we address each concern in detail. Look forward to any further discussions.\n\n***Q1. Empirical validation of the approximation in Equation (7) for sequence-level probabilities***\n\n**A1.** Thank you for pointing out the potential mismatch between our first-order Mercator approximation in Equation (7) and actual sequence-level probabilities $p(y|x)$. Indeed, this approximation assumes $p(y|x) \\simeq 1$, which can be problematic for long sequences where $p(y|x)$ might vanish. \n\nIn our implementation, we compute probabilities at the token level, clip extreme values to avoid outliers, and average them over the sequence, yielding a reasonably scaled alpha that is shared across all tokens.\n\nThe corresponding code (in `src/distillm_trainer.py`, lines 1161–1164) is:\n```\nanchor = (1 - base_alpha_1) * logp_logq\nlogps_logqs = (\n    (tea_per_token_logps * loss_mask).sum(-1) / loss_mask.sum(-1)\n).exp() - (\n    (per_token_logps * loss_mask).sum(-1) / loss_mask.sum(-1)\n).exp()  # sentence-level\nalpha_1 = torch.clip(\n    1 - anchor / (logps_logqs + 1e-5),\n    min=1e-2, max=base_alpha_1\n).unsqueeze(-1).unsqueeze(-1)\n```\nThis design makes the approximation **invariant to sequence length**, effectively resolving the concern about very small $p(y|x)$ values in long sequences. Notably, we want to highlight that this first-order approximation has advantage in closed-form $\\alpha$ updates for efficient, per-sample adaptation.\n\nAdditionally, we provide empirical results of token-level probabilities on teacher ($y_t$) and student ($y_s$) responses to quantify the first-order approximation using the teacher (Mistral-7B) and student (Danube2-1.8B) models. To better characterize the distribution of token-level probabilities, we report the first (Q1), second (Q2, median), and third (Q3) quartiles for both models:\n\n|         | $y_t$  |      |      | $y_s$  |      |      |\n|---------|------|------|------|------|------|------|\n|         | Q1   | Q2   | Q3   | Q1   | Q2   | Q3   |\n| teacher | 0.82 | 0.88 | 0.94 | 0.79 | 0.85 | 0.92 |\n| student | 0.80 | 0.85 | 0.94 | 0.81 | 0.89 | 0.96 |\n\nThe results show that most of the token-level probability values are close to 1, which ensures small approximation error of our first-order approximation. We appreciate the reviewer’s insightful comment and will update the paper accordingly to better explain this implementation and its practical implications.\n\n***Q2. CALD term appears without explanation – unclear terminology***\n\n**A2.** Thanks for the question. CALD stands for **C**ontrastive **A**pproach for **L**LM **D**istillation, which is first introduced in lines 60-61 in our main manuscript. To improve the readability and clarity, we will also define the term at the beginning of Section 3.1.2. We will additionally refine Section 3 and add more descriptive languages right after the definition (Equation (5)).", "claims_and_evidence": "The claims are supported by clear and convincing evidences.", "methods_and_evaluation_criteria": "The evaluation methods make sense.", "theoretical_claims": "The approximation in Equation 7 seems problematic. In Appendix B.3., the approximation relies on the assumption that $p(y|x) \\approx 1$ and $\\alpha p(y|x) + (1 − \\alpha) q_{\\theta}(y|x) \\approx 1$. However, when y is a sequence, p(y|x) can be quite small due to the accumulation of per-token probabilities, making the approximation unreasonable. Could the authors provide empirical results on how the values of $p(y|x)$ are distributed on real datasets?", "experimental_designs_or_analyses": "The experiments on general instruction tuning, math, and code domains are sound and valid.", "relation_to_broader_scientific_literature": "This paper provides modifications to the previous on-policy KD methods proposed in MiniLLM[1] and GKD[2] and shows better empirical approaches.\n\n\n[1] MiniLLM: Knowledge Distillation of Large Language Models.\n\n[2] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weakness on clarity:\nThe presentation of the method can be clearer. For example, what does the term CALD stand for? It comes out in lines 61-62 without any explanation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m6ZZWYFXZj", "reviewer": "Reviewer_wyun", "summary": "This paper addresses the critical challenge of compressing large language models (LLMs) for practical deployment by focusing on knowledge distillation (KD). The authors highlight the limitations of existing KD approaches, which primarily focus on either optimizing loss functions (like Kullback-Leibler divergence or skew KL) or curating training data (teacher-generated vs. student-generated outputs).\nThey also point out that current methods often overlook the synergistic relationship between loss formulations and data types, limiting the potential performance gains in student models. Additionally, the authors note the rise of contrastive learning methods (like DPO) but observe that direct application of DPO to KD suffers from reward hacking issues.\n\nTo address these shortcomings, the paper introduces DISTILLM-2, a novel contrastive approach for KD of LLMs that builds upon DistilLLM. Its main contributions are as follows:\n\n(1) The introduction of a contrastive approach with asymmetric loss dynamics (CALD): This involves analyzing and leveraging the behavior of forward and reverse KL (and SKL) during training, and applying distinct loss functions to different training samples.\n\n(2) Optimized dataset curation and curriculum-based adaptive loss mechanisms: These enhancements provide practical guidelines for their contrastive approach.\n\n(3) Demonstrated performance: The paper claims state-of-the-art performance for small language models (sLMs) across various tasks, including instruction-following, reasoning, and code generation, and also demonstrates the methods versatility by applying it to preference alignment and vision-language models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. Below we address each concern in detail. Look forward to any further discussions.\n\n***Q1. Lines 152–156 might be confusing – rephrasing suggested***\n\nA1. Thank you for the suggestion. We will revise Lines 152–156 to improve clarity. This paragraph aims to illustrate the relation between contrastive learning in preference alignment and our approach to knowledge distillation: increasing the student model's likelihood of generating outputs aligned with teacher responses, and decreasing it for those resembling weaker student responses, using distinct loss functions.\n\nTo make this point clear, we plan to revise the paragraph as follows:\n\n“Similarly, we incorporate this contrastive concept from preference optimization into KD by assigning different loss functions to different types of responses: encouraging the student model to assign higher likelihood to high-quality responses generated by the teacher ($y_t$) -- $q_\\theta(y_t|x)$ -- while reducing the likelihood of lower-quality student responses ($y_s$) -- $q_\\theta(y_s|x)$ -- that deviates from the teacher.”", "claims_and_evidence": "I feel the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria do make sense for the problem at hand.", "theoretical_claims": "The paper does not have any major theoretical contributions — only a few theoretical remarks / observations. (I have not checked the calculations behind these remarks, but they seem straightforward and they're not a major part of this paper anyway.)", "experimental_designs_or_analyses": "I went through the experimental design and analysis of the paper and it looks sound and valid to me.", "relation_to_broader_scientific_literature": "This work builds upon DistiLLM (Ko et. al., 2024) which is a state-of-the-art approach for distilling LLMs.  The paper proposes a new approach for distilling LLMs, by both combining and refining several known techniques, and by introducing new ideas. It seems to be achieving significant improvements over the state-of-the-art across various benchmarks.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength: The authors have conducted extensive experimentation both to demonstrate the effectiveness of their techniques but also to motivate their ideas.\nWeakness: The paper does not have any major weaknesses.", "other_comments_or_suggestions": "This is minor, but I was a bit confused when reading Lines 152-156 — maybe the authors can rephrase their statement there.", "questions_for_authors": "I have no questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PnwwZffiI2", "reviewer": "Reviewer_WSDW", "summary": "The paper introduces DistiLLM-2, a novel approach to distilling knowledge from large language models (LLMs) into smaller, more efficient student models.  DistiLLM-2 employs a contrastive approach that leverages the synergy between loss formulations and data types, simultaneously increasing the likelihood of teacher responses and decreasing that of student responses.  The authors claim that this method achieves state-of-the-art performance for student LLMs across various tasks, including instruction-following, mathematical reasoning, and code generation.  Additionally, DistiLLM-2 supports diverse applications such as preference alignment and vision-language extensions.  The key innovation lies in the contrastive loss function, which applies distinct loss functions to different types of training samples, effectively incorporating the synergy between loss formulations and data perspectives.  The paper also introduces optimized dataset curation strategies and curriculum-based adaptive loss mechanisms to further enhance the distillation process.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments. We have rephrased the question for ease of reference and provide our corresponding responses below. Look forward to any further discussions.\n\n***Q1. Inclusion of acceptance rates in speculative decoding analysis for completeness***\n\n**A1.** Thank you for the suggestion. To provide a more concrete answer, we evaluated the acceptance rates using the vLLM [1] framework, which supports detailed speculative decoding metrics. The results of acceptance rates are presented as follows:\n\n|          | SFT   | GKD   | DistiLLM | DistiLLM-2 |\n|----------|-------|-------|----------|------------|\n| Phi-3-medium | 0.412 | 0.464 | 0.469    | 0.487      |\n| Phi-3.5-mini | 0.397 | 0.443 | 0.452    | 0.522      |\n\nWe could see that DistiLLM-2 significantly boosted the acceptance rates leading to a higher speed-up via speculative decoding. \n\n[1] Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP. 2023", "claims_and_evidence": "Supported Claims:\n- Improved performance across tasks\n- Contrastive approach effectiveness is analytically supported\n- Data curation strategy for teacher and student generations\n- Showed impact of datset size\n- Showed generalization to new tasks including vision", "methods_and_evaluation_criteria": "Yes.Methods:\n\nContrastive Loss Function: The core idea of using a contrastive loss function that leverages the synergy between loss formulations and data types is well-motivated and addresses what the authors see as the limitations of previous distillation methods. Paper also introduced: Data Curation Strategy and Curriculum-based Adaptive Learning\n\nEvaluation Criteria:\n\nBenchmark Evals: AlpacaEval, Evol-Instruct, UltraFeedback, GSM8K, MATH, HumanEval, and MBPP are widely recognized and represent relevant tasks for evaluating LLM capabilities.\nLLM-as-a-Judge: The use of LLM-as-a-Judge for evaluating instruction-following tasks provides a robust and comprehensive assessment of the generated responses.\nPass@k for Code Generation: The use of Pass@k as an evaluation metric for code generation tasks is standard practice and accurately reflects the best case ability of the student model to generate correct and executable code.\nSpeculative Decodeing: Is a great proxy for how well a student matches a teacher.\n\nIn summary, the proposed methods and evaluation criteria are well-aligned with the problem of LLM distillation.", "theoretical_claims": "AppendixB had all the proofs; all very mathy... I'm not 100% sure I'm super qualified to look thru this, but here we go...\n\n- Behavior of KL and RKL: The paper provides a mathematical explanation for the behavior of KL and RKL, including the \"pulling-up\" effect of KL and the \"pushing-down\" effect of RKL.  The explanation made sense / didn't seen to have issues. \n\n- Derivation for Remark 1: The paper presents a derivation for Remark 1, which establishes a mathematical connection between the proposed CALD loss function and DPKD/DPO (Appendix B.2).  I looked thru the derivation; it didn't seen to have issues. \n\n- First-order Approximation for Mercator Series: idk if I'm super qualified to be checking series but it looked fine.", "experimental_designs_or_analyses": "As mentioned in the methods, it looks good.", "relation_to_broader_scientific_literature": "distillation losses are extremely important to a large set of problems including RL, specdec, teaching student models.", "essential_references_not_discussed": "none that i can think of", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to improve the distillation mechanism for LLM. The paper proposes DistiLLM-2 that leverages a contrastive approach for distillation objectives by employing different loss components for teacher-generated data and student-generated data. In addition, the paper explores adaptive mixing of teacher-student distribution while defining loss function, and proposes a curriculum for adjusting the weight of two loss components in the distillation objective. 

All the reviewers agree that the paper makes significant contributions to the SoTA distillation for LLM. The utility of the proposed distillation method is well justified by an extensive empirical evaluation. Furthermore, all the design choices are well supported by a careful analysis. Overall, this paper will be a great addition to ICML.
Decision: Accept (oral)