AMiD: Knowledge Distillation for LLMs with $\alpha$-mixture Assistant Distribution
Authors: Donghyeok Shin
Yeongmin Kim
Suhyeon Jo
Byeonghu Na
Il-chul Moon
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge distillation
Large language model
Information geometry
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: This paper proposes $\alpha$-mixture assistant distribution, a novel generalized family of assistant distributions, and $\alpha$-mixture distillation, coined AMiD, a unified framework for knowledge distillation using the assistant distribution.

Abstract: Autoregressive large language models (LLMs) have achieved remarkable improvement across many tasks but incur high computational and memory costs. Knowledge distillation (KD) mitigates this issue by transferring knowledge from a large teacher to a smaller student through distributional alignment. Previous studies have proposed various discrepancy metrics, but the capacity gap and training instability caused by near-zero probabilities, stemming from the high-dimensional output of LLMs, remain fundamental limitations. To overcome these challenges, several approaches implicitly or explicitly incorporating assistant distribution have recently been proposed. However, the past proposals of assistant distributions have been a fragmented approach without a systematic investigation of the interpolation path and the divergence. This paper proposes $\alpha$-mixture assistant distribution, a novel generalized family of assistant distributions, and $\alpha$-mixture distillation, coined AMiD, a unified framework for KD using the assistant distribution. The $\alpha$-mixture assistant distribution provides a continuous extension of the assistant distribution by introducing a new distribution design variable $\alpha$, which has been fixed in all previous approaches. Furthermore, AMiD generalizes the family of divergences used with the assistant distributions based on optimality, which has also been restricted in previous works. Through extensive experiments, we demonstrate that AMiD offers superior performance and training stability by leveraging a broader and theoretically grounded assistant distribution space.

Published as a conference paper at ICLR 2026
AMID: KNOWLEDGE DISTILLATION FOR LLMs WITH Q-MIXTURE ASSISTANT DISTRIBUTION
Donghyeok Shin', Yeongmin Kim', Suhyeon Jo', Byeonghu Na', Il-Chul Moon Korea Advanced Institute of Science and Technology (KAIST), Psummary.ai (tlsehdgurO, a sdudrlaa aal0, suhyeon jo, byeonghu. na, kcmoon))kaisst.cc.
ABSTRACT
Autoregressive large language models (LLMS) have achieved remarkable im- provement across many tasks but incur high computational and memory costs. Knowledge distillation (KD) mitigates this issue by transferring knowledge from a large teacher to a smaller student through distributional alignment. Previous stud- ies have proposed various discrepancy metrics, but the capacity gap and training instability caused by near-zero probabilities, stemming from the nigh-dimensional output of LLMs, remain fundamental limitations. To overcome these challenges, several approaches implicitly or explicitly incorporating assistant distribution have recently been proposed. However, the past proposals of assistant distributions have been fragmented approach without a systematic investigation of the inter- polation path and the divergence. This paper proposes a-mixture assistant distri- bution, novel generalized family of assistant distributions, and a-mixture dis- tillation, coined AMiD, a unified framework for KD using the assistant distribu- tion. The a-mixture assistant distribution provides continuous extension of the assistant distribution by introducing new distribution design variable a, which has been fixed in all previous approaches. Furthermore, AMiD generalizes the family of divergences used with the assistant distributions based on optimality. which has also been restricted previous works. Through extensive experiments, we demonstrate that AMiD) offers superior performance and training stability by leveraging a broader and theoretically grounded assistant distribution space. We release the code at https: / //gggtthuuu com/ wooo//aallee aailab kaikkkaistaata /AMiD.
INTRODUCTION
Autoregressive large language models (LLMs) have recently achieved remarkable advances, deliv- cring outstanding performance across a wide spectrum of tasks and application domains (Achiam et al., 2023; Touvron et al., 2023; Team et al., 2024). However, their massive parameter scales impose prohibitive computational and memory costs, which hinder their deployment in practical applications. Accordingly, essential objective for practical deployment is compress these high- capacity models by reducing the parameter count while preserving their strong performance.
Knowledge distillation (KD) (Hinton et al., 2015) is widely adopted compression technique that transfers knowledge from a large teacher model to a smaller student model by aligning their token- level predictive distributions. The selection of discrepancy metric an important research topic in KD for LLMs. Several prior studies have proposed either (1) the use of various forms of diver- gence, including the capability of regulating the (2) employing a combination of these divergences (Agarwal et al., 2024; Wu et al., 2024) as the discrepancy metric. However, these approaches do not fundamentally resolve the large capacity gap between the high-capacity teacher and smaller student models, and the optimization instability due to near-zero probabilities, which is prevalent in the high-dimensional probability space of L.MMs.
trade-off (Wang et al., 2025), or
A practical remedy is to introduce an assistant distribution that interpolates teacher and student distributions to stabilize optimization and bridge this capacity gap. Recently, several methodologies have been proposed that either (1) utilize the discrepancy metric that inherently includes a specific form of assistant distribution (Agarwal et al., 2024; Ko et al., 2024: 2025) or (2) explicitly model the assistant distribution (Shing et al., 2025). However, these approaches have generally been treated as independent recipes in different papers without a systematic study, which hinders the development of general and effective methodologies.

---

Published as a conference paper at ICLR 2026
In this paper, we propose generalized framework that integrates the fragmentarily employed sistant distribution and divergence. First, we interpret the existing assistant distributions from the information theory view, revealing that the existing methodology can be expressed as m-mixture, which mixes two probability distributions via arithmetic mean, and an e-mixture, which mixes them via geometric mean. Next, we present new assistant distribution family, coined o-mixture assistant distribution, by extending the mean concept via the generalized fa mean. The a-mixture assistant distribution introduces new design variable a for the assistant distribution, which adjusts the ge- ometry of the interpolation path. Here, a is an independent parameter distinct from the well-utilized parameter X, which controls the portion of interpolation. The a-mixture assistant distribution not only includes the existing assistant distributions new assistant distribution that were not investigated in KD for LLMSs area.
special case (a = +) but also provides several
Under the concept of a-mixture assistant distribution, we investigate several properties of the a- mixture assistant distribution, which are meaningful in KD for LLMs,, such as the analysis with a-divergence, controllable support via a, and continuity with respect a. Next, we propose new KD framework for LLMs, coined as q-mixture distillation (AMiD), which generalizes the optimiza- tion schemes of prior research by unifying both the assistant distribution and the divergence. AMiD aims to align the a-mixture assistant distribution and either the teacher or student. We theoreti- cally prove the optimality of AMiD, which enables student) even when employing arbitrary divergence, a, and X, under the perfect optimization as- sumption. Furthermore, through gradient analysis when employing f-divergence, we theoretically demonstrate that a adjusts the mode-covering and mode-seeking properties of the student distribu- tion, with both toy experiments and real-world experiment results supporting this finding. Across various evaluation scenarios, our proposed framework AMiD consistently demonstrates superior performance compared methodologies that do not utilize the assistant distribution and those em- ploying limited assistant distribution.
achieve the primary goal of KD (teacher
2 PRELIMINARY
2.1 KNOWLEDGE DISTILLATION FOR LARGE LANGUAGE MODELS
We denote the input prompt and output token sequences as x and y. respectively, where y = (y1, "2.....VI.. VL) V is a token sequence of length L, with each token drawn from the vocabu- lary set V. Given the input x, an autoregressive large language model (LLM) outputs a next-token distribution p(3 (z, v<t). conditioned on both the prompt and the previously generated tokens Vet == (y1, .2...-.-..0 y/ 11). 1). We assume access to two LLMs: a large fixed teacher model p(yn (z, yct), and a smaller student model qo (yn)|, M-1) parameterized by O. The goal of knowledge distillation (KD) for LLMS is to transfer the knowledge of the teacher into the student. Concretely, KD for LLMs is typically formulated as aligning the next-toked distributions of the teacher and student:
min (D.))~ D(p(ul x. Me)),, co(an (z. ve(i)
where 1) denotes the divergence and the dataset D is composed of the predefined dataset (Hinton et al., 2015), or various strategies using the muddent-generated outputs (SGOs): on-policy (Lin et al., 2020), mixed approach (Agarwal et al., 2024; Gu et al., 2024; Xu al., 2024b), and an adaptive off-policy (Ko et al., 2024). For notational brevity, we omit the explicit dependence on æ and y whenever clear from context, writing p == p(yn (2., M-1)) and go == go x. vc1).
The choice of divergence D plays pivotal role in KD for LLMs. The widely used Kullback-Leibler
(KL) divergence DRL (pllg) Ex log [o(R] in KD (Hinton et al., 2015; Kim & Rush, 2016) emphasizes mode-covering, often assigning mass to less informative regions. To mitigate this effect, the reverse KL divergence DRKL (pl140) == Dxo(go (p) is employed for its mode- seeking properties (Gu et al., 2024), which possesses mode-seeking properties, but either choice entails a trade-off between quality and diversity. Recent studies address this by (I) combining divergences, e.g., GKD (Agarwal et al., 2024) with the generalized Jensen Shannon divergence
p(k)
Dess(plgo) = λDRL (pl) Ap + (1 -))o) + (1 A) Ap + (1 A)ge), and (2) extending classical divergences to enable explicit control, as in ABKD (Wang et al., 2025), which adopts the a-3-divergence DAB (Cichocki et al., 2011) as a generic framework.
2

---

Published as a conference paper at ICLR 2026
Meanwhile, several methodologies have recently been proposed to improve the optimiza- tion stability of KD for LLMs. Ko et al. (2024) leverages the skew KL divergence DRi (pll Xp (1 2)oo) and the skew reverse KL divergence DSRKI ( == ( Bkalggol>p Ap - A) TAID (Shing et al., 2025) introduces an adaptive intermediate distri- bution that gradually shifts from the student's initial distribution the teacher distribution, i.e., (PA (p)l(e == DxL(rell90 where = softmax((1 Xx) logit(g)) + X logit(p)) with time- dependent interpolation parameter Ao, detached student logits logit (4a), and teacher logits logit(p).
DsKL (pllga)
2.2 M-MIXTURE AND M-MIXTURE
Mixture models are a standard tool for integrating information from multiple distributions. Informa- tion geometry (Amari, 2016; Nielsen, 2020; Eguchi & Komori, 2022) provides dualistic structure on the manifold of probability distributions, characterized by two affine connections: the mixture connection and the exponential connection. These connections induce two natural ways of interpo- lating between distributions, commonly referred to as the m-mixture and the e-mixture.
Given two probability distributions p and defined on the same measureable space, the m-mixture is defined as a convex combination of and q:
(x) = (1 Op(a) Lg(a),
L [0,
contrast, the e-mixture is defined multiplicatively:
(x) p(x))/(x>'~=
Z(1) (10==(((s'tttt p(=))a(=)* dx
7())
The m-mixture forms a straight line in probability space, while the e-mixture forms one in log- probability space. Some studies leverage m- and e-mixtures, for example, construct paths for annealed importance sampling (Grosse et al., 2013; Masrani et al., 2021).
2.3 GENERALIZED /MMAAN
Generalized f f-mean (Kolmogorov & Castelnuovo, 1930) is a general- ized framework of the mean by using monotonically increasing dif- ferentiable function f R + R. Given set of weights {w, E R+ ny = 1) and the set of corresponding input elements (u, RR, the E,
W1
W2
generalized f fmman is defined as:
f
m {ui}) = - Σ
( Euss(u))
(4)
m
u1
u2
The m applies a nonlinear transformation to the inputs, combines them with weights in the transformed domain, and maps the result back to the original domain. The and geometric mean, have homogeneity, which stands for a scale-free property m ([w:), {c- w;}) = c my (fwi), {u:}) for e >> 0. The generalized f-mean is homoge- neous only when f belongs to the unique class of functions (Hardy, 1952; Amari, 2007):
Figure 1: Illustration of generalized f-mean.
means, such as the arithmetic mean
#1 T( (u) log u, a
-
IR+
(5)
This family includes various notable examples, such as the weighted arithmetic mean for a = -1, the weighted geometric mean for a 1, the weighted harmonic mean for a 3, and min(u:). max(u;) for a + oo and + oc, respectively.
METHODOLOGY
This section introduces a new KD framework for LLMs, coined a-mixture distillation (AMiD), which generalizes both the assistant distribution and the associated optimization scheme. Section 3.1 reveals the connection among the existing assistant distributions and highlights the need for sys- tematic study. Section 3.2 proposes the a-mixture assistant distribution, which provides unified and generalized assistant distribution family via a-mixture distribution. Finally, Section 3.3 extends the assistant-base based KD objective into generic divergence framework.
3

---

Published as a conference paper at ICLR 2026
c a-mixture assistant distribution (Ours)
(Agerwal al.. 20241
DistiLLM 2024)
TAID (Shing etal, 2025)
+
maxip.0e 9a3
(2p² 1)88)
Ap + A)go C./ ()
90
min(p, Re)
43 harmonic
max
quadratic
arithmetic
sqrt-mean
geometric
min
(a) Illustration of a-mixture assistant distributions (a.A)
rgg
with varying ax.
Teacher Student
Teacher Sa mixture
Teacher Ra a mixture
Teacher p a mixture
Teacher mixture
mixture
(b) (
(c) a
(d) a 0
(
(f) a 3
Figure 2: Visualization of the a-mixture assistant distribution family. (a) The a-mixture assistant distribution provides a generalized framework for assistant distributions, with prior studies (Agarwal et al., 2024; Ko et al., 2024; Shing et al., 2025) recoverable special cases. (b-f) Illustration of the a-mixture assistant distribution where p = (0, 0.5²). ge = [333, 12). and = 0.3. For a < 1, the support of the a-mixture assistant distribution corresponds to the union of the supports of p and go. whereas for a 2 1, it corresponds to their intersection.
3.1 MOTIVATION
Our primary motivation stems from the observation that recent studies inherently include the com- position of the teacher distribution p and the student distribution qo. which we will refer as assistant distribution тş in this paper. For example, several studies (Agarwal et al., 2024; Ko et al., 2024) utilize the divergences that include rg = Ap + X)go with À 1], which is an weighted arithmetic mean, also known as m-mixture. Moreover, we have newly discovered that the assistant distribution of TAID (Shing al., 2025) is e-mixture, also known weighted geometric mean.
Proposition 3.1. The assistant distribution of TAID (Shing et al., 2025) is e-mixture of p and qo:"
Tg softmax((1 X) logir(go) + X logit(p)) p pa
Please refer to Appendix A.2 for proof. Using the assistant distribution provides several advantages KD for LLMs. First, the assistant distribution facilitates more effective knowledge transfer be- tween the teacher and the student. In KD, significant capacity gap often arises due to differences in model size (Mirzadeh et al., 2020), and this issue becomes particularly pronounced LLMs (Zhang al., 2023; Sun al., 2025) due the high-dimensional nature. This gap makes difficult for the student to faithfully capture the knowledge encoded in the teacher (Mirzadeh et al., 2020; Shing al., 2025). By introducing the assistant distribution that serves and student, the information transfer might be more efficient (Shing et al., 2025). Second, the assis- tant distribution improves training stability. Due the tigh-dimensional nature LLMs, most of probabilites in and qo are inevitably close to zero. These near-zero probabilities might cause insta- bility in both the loss and the gradient computation when divergences involving density ratios (e.g., KI. divergence) are used (Ko et al., 2024). A suitably constructed assistant distribution yields more stable density-ratio estimates, thereby enhancing the robustness of optimization (Ko et al., 2024).
bridge between the teacher
Despite these advantages, systematic study has examined (1) the distinction between and e- mixture assistant distributions, (2) alternative candidates, (3) their compatibility with diverse diver- gences, and (4) their implications for KD in LLMs, supported by theoretical and empirical analyses. This gap hinders the development general and effective methodologies, so the recent studies often fall into sub-optimal performances by relying paper, we alleviate this gap by unifying the existing assistant distributions into a generalized design principle of assistant distribution.
isolated design of assistant distribution. In this
We omit the time index t and detached notation for the sake of uniformity.
4

---

Published as a conference paper at ICLR 2026
3.2 a-MIXTURE ASSISTANT DISTRIBUTION
As discussed in Section 3.1, the existing assistant distributions can be formulated as the mixture of the teacher distribution and the student distribution via the mean function. To integrate the frag- mentarily employed assistant distributions, we employ the generalized fa-mean (Amari, 2016) and introduce a new assistant distribution family, coined a-mixture assistant distribution as follows:
Definition 1 (a-mixture assistant distribution). Let R and À [0, 1]. For distributions and qo defined either on a discrete support X' indexed by l or on a continuous domain X' with variable T, define the unnormalized c--mittre assistant distribution
(AD(2)) +(1- - A) go(z)) ((>)) (o(=)>~>,
a # 1,
(2)
(2)
if 1
where = = k in the discrete and in the continuous case.
Consequently, the (normalized) a- a-mixture assistant distribution is defined as:
rg ((ai)) .(a.A) (z)
1 r (aaNN)) or [ (laaN) (z) de
Z, Σ
(8)
Zr
(o. contains tunable parameters: a and X. The À de- The a-mixture assistant distribution rg termines the portion of the interpolation between teacher and student model qo. which has been fine-tuned in previous works (Agarwal et al., 2024; Ko et al., 2024; Shing et al., 2025). The other parameter a is new axis of distribution design variable, which was only employed as specialized case (a 1), controls the geometry of the interpolation path, as depicted in Figure 3a. Since the form of the generalized fa-mean is solely governed by a, once is fixed, À only serves to control the portion between p and qe along that determined path. In addition, Theorem 3.2 provides an helpful information geometric perspective of the a-mixture (assistant) distribution:
Theorem 3.2. (Amari, 2007) Given fixed and X, the pla.>) defined as Eq. (8) is unique minizer of weighted sum of Amari's 's a-dviergences Dr (see Appendix A. for the definition):
((a = arg min X D. + A) (glr)))
(a.A) is the internal division distribution of and in terms of a- divergence, which bridges the generalization of the mean concept and the geodesic in information geometry. Due to the generalized fa fa-mean, the existing assistant distributions are recoverable as special instances of rg (a.A); rg
Theorem 3.2 indicates that r
(-1.A) m-mixture (Agarwal al., 2024; Ko et al., 2024) that is
minimizer of a weighted sum of DKL. and ro (1.A) is e-mixture (Shing et al., 2025) that is minimizer
of a weighted sum of DRKL- Furthermore, re provides several new assistant distributions that were not previously used in KD literature, depicted in Figure 2a.
Moreover, the support of a-mixture assistant distribution determined by the range of a: supp(p) nn supp(g) when 1. This property demonstrates the necessity of determining the range of based on the charac- teristies the intersection of and qs. For instance, if and qe overlap significantly, setting > can strengthen the matching within the intersection region. Conversely, if they overlap minimally, setting a 1 indicates that matching occurs across broader range. Although in KD for LLMs, p and qe typically share the same support defined by the vocabulary set, this property remains use- ful because many probabilities very small values close to zero due to the high dimensionality.
supp(r.(a..) (a.2) supp(p) U supp(go) when < 1, and supp(r-(a.))
Figures 2b-2f shows the different behaviors of rf (a.A) among the various a values.
Lastly, we also demonstrate that ((aAA) is continuous function with respect to in Proposition 3.3, even though the ((xAA) is piecewise-defined function. This property enables the design of curriculum-bast adaptive a scheduling, paralleling prior work (Shing al., 2025; Ko et al., 2025) that investigated adaptive strategies for X. Please refer to Appendix A.3 for proof.
Proposition 3.3. (Continuity) Assume that p and qe are not both zero. Then, To (a.X) is continuous function w.r. a under the fixed X [0, 1).
5

---

Published as a conference paper at ICLR 2026
nn m-geodesiic --),,,aa (a.)) Agonvol
e-geadesic (x = Stingg stal,
2004
0
Da Da(gallrga
90
0.2
Taadher
la
p
90
Update
a
p
(a,
0.1
(or 0)
(ar
0 (or 13
0.06
(ar
(z.2)
8
1 2
9
-10
-5
10
(a) Role of and À in the distribution space. (b) Optimization dynamics.
(c) Rolc of for 26-
Figure 3: Visualization of the characteristics of the a-mixture distillation, AMiD. (a) a determines the geometry of interpolation while À controls the portion between p and qo. In general, the cur- vature of the path increases as a moves farther away from -1 (the straight line). (b) rg (a.A) can
be interpreted as the internal division point in terms of a-divergence. Due to the uniqueness, the updated parameter 8' via optimization of AMiD, directly affects the student distribution. (c) Toy experiment with two-modal p and uni-modal qo- controls the property of the optimized student distribution 4 between the mode-covering and mode-seeking, even though we minimize the fixed
divergence (pllig (p)))((.))) In particular, when a S 1, increasing a encourages q to exhibit more mode-covering behavior, whereas using smaller a strengthens mode-seeking behavior.
3.3 AMID: KNOWLEDGE DISTILLATION WITH a-MIXTURE ASSISTANT DISTRIBUTION
In this section, we present token-level KD for LLMs with a-mixture assistant distribution, coined as a-mixture distillation (AMiD), which aims to align rg ((.A)) and either or qø. Specifically, the optimization of AMiD is defined as follows, similar to Eq. (1):
Σ P(p.rooan)
D(ge, ro (a.d))
min ( DD
min E (a
(10)
We highlight that AMID allows the use of arbitrary divergence D and any dataset D (see Section 2.1) since r (a.2) is valid distribution. Furthermore, AMil) generalizes the optimization schemes of prior research by extending both the assistant distribution and the divergence. For example, Dis-
tiLLM (Ko al., 2024) corresponds to DxL (xllr) and DxI(goll7z |r) )1))); and TAID (Shing et al., 2025) corresponds to DKL (rg1.)1))))))) Next, we aim to characterize the optimality of AMiD. Theorem 3.4. (Optimality) Let D be any proper divergence and R.
If [0, 1) and 30 s.t. D(p, (a.))) 0, then D(p, (a.2)) ((.)) 0 if and only if qe-
If' (0, 1] and 30 s.t. D(go, Ta (a.))) then D(ge, T (a.A1) if and only if qo.
Please refer to Appendix B.3 for the proof. Theorem 3.4 demonstrates that even if we minimize the divergence between p (or qo) and r ((a.)) the primary goal of KD is guaranteed i.e., p = qo- It in- tuitive because the interpolation point needs to coincide with one of the endpoints when it coincides with the other (see Figure 3b). Therefore, leveraging the benefits of the assistant distribution, establish optimality. Although Theorem 3.4 establishes theoretical optimality for any choice of D, R. and À (0, 1), the effectiveness of AMiD might depend on selecting an appropriate a value due to the imperfect practical optimization. Please refer to Appendix B.3 for the theoretical insights based a tuning guidelines, Appendix C.4 for overlap- based adaptive a scheduling.
Now, we provide the gradient analysis to investigate the specific role of a. In particular, we consider f-divergence, which is widely used in many areas, including KD for LLMs.
Proposition 3.5. (Gradient analysis) The gradient of f-divergence D, Dy(ellr (a.A) be expressed as:
vaD, (pllr(and) = E (a:))
P
P
- E
(a,))
Vo log
(11)
(a.d)
(a.1)
T9
(1 N)90 1-a
where w =
and (y(()) = f(v) (v)
+(1 \)06 -
Ap

---

Published as a conference paper at ICLR 2026
Table 1: ROUGE-L scores (1) on five task-agnostic instruction instruction-following datasets. Bold and
Underline mean the best and second-best performance of each column, except the teacher, respec-
tively. All results are based on our own re-implementation. We use DAB and X = 0.1 for AMiD. We conduct the evaluation with five random seeds. More results of baselines are in Appendix C.I.
Model
Val. (1)
Dolly Eval (t)
Self Inst (1)
Vicuna (1)
Super NI (1)
UnNI (1)
Avg. (1)
GPT-2 XL (Teacher)
27.14 2015
14.55 2082
16.12 203
27.21 2025
31.41 10.06
23.29
GPT-2 (I.5B)
GPT-2 IB)
GKD
27.06
24.58
10:13
11.78
20.44
14.60
20.37
22.84
20.12
25.04
10.00
19.77
TAID
28.37
25.74
1027
12.91
10.31
17.09
10.18
23.66
10.31
26.82
10:05
21.24
DistiLLM (SKL)
27.88
25.50
=028
12.35
=0.30
16.10
#0.22
23.87
#0.30
26.16
#0.06
20.80
DistiLLM (SRKL)
28.21
25.74
40.00
12.13
16.34
25.40
20.10
26.91
21.30
ABKD
28.61
25.49
10:24
12.52
20.52
17.36
26.07
20.14
27.36
21.76
AMID (Ours)
29.24
26.44
#0.12
13.74
16.76
+0..4
29.71
+:008
30.35
23.40
GPT-2 XL (1.5B)
GPT-2 Medium (0.38)
GKD
27.90
..0:5
12.36
.0.42
15.71
1.0.58
23.83
1.0.26
27.14
1.000
20.82
TAID
29.45
27.01 =0:27
14.53
=0.47
17.58
#:0.30
25.14
1:015
29.79
#0.14
22.81
DistiLLM (SKL)
29.65
26.87
14.11
2:0.29
16.85
25.59
4:022
28.84
22.45
DistiLLM (SRKL)
29.72
26.50
:0:20
13.79
20071
17.14
26.25
20.11
29.31
22.60
ABKD
29.64
26.93 10.17
13.69
10.32
17:45
.0.27
28.15
10.18
30.94
10.06
23.43
AMID (Ours)
30.83
27.34
20.18
15.26
20.46
17.69
#0.27
29.04
+:000
33.15
10.13
24,50
GPT-2 XL (I.5B)
GPT-2 Large (0.8B)
GKD
29.36
26.38
14.44
=0.66
17.02
#0.46
26.64
64
#:016
30.99
4:0.13
23.09
TAID
29.83
26.85
:2032
15.07
20:31
17.02
26.71
1:021
31.09
23.35
DistiLLM (SKL)
29.69
26.12 =0:27
15.69
=0.75
16.91
+:0.3
27.23
#0:18
30.73
23.34
DistiLLM (SRKL)
30.59
27.09
10.40
14,61
1.0.66
16.39
10.27
28.
1.045
31.04
1.006
23.51
ARKD
30.49
27.67 20.34
15.46
=0:81
17.43
#0:25
30.74
#:022
33. 1:0.15
24.88
AMID (Ours)
31.10
27.86
=9.29
16.46
=0.41
16.62
#0,50
32.64
#.0.2
35.64
#:0.0
25.84
Table 2: Experimental results on the task-specific distillation. "Trans." and "Summ." indicate translation and summarization task, respectively, We use DAB and À 0.1 for AMiD.
28
TAID
24
DistiLLM
SFTed Gemma-7B-lk - Gemma-28-It
SFTod Qwen2-7B-It to Qwen2-0.
22
ABKD
AMID (Ours)
DistiLLM (SAKL)
Model
Trans. COMET
Summ. ROUGE-L(1)
GSMBK. Acc
Trans.
Summ. ROUGE-L
GSMBK
(f)
TAID
74.20
34.93
24.49
58.17
31.65
33.28
10
20
DistLLLM
52.83
26.51
00.00
57.23
32.27
35.63
Training epach
ABKD
74.21
34.88
24.26
58.07
31.67
33.13
Figure 4: ROUGE-L
AMID (a # +1)
74.78
35.22
24.94
58.31
32.51
36.24
curve on Dolly dataset.
Please refer Appendix A.5 A.5 for the proof. Proposition 3.5 implies the following properties. First, w controls the magnitude of the instance-wis gradient V log (e(y) xr Vez). While w does not affect the individual gradient direction due S I 1, weighting effect may shift the batch- wise gradient direction. Second, the plays crucial role in enabling w to perform instance-wisg weight- ing based on the density ratio p(yl2z.
(a (=-)2))) This property originates from the unique characteristic of the a-mixture assistant distribution that cannot be achieved by learning rate scheduling. Third, (relatively) adjusts the mode-covering and mode-seeking behavior of the optimized student distri- bution. Let us assume L 2 0, i.e., a S 1.2 When Z qa. a larger a produces a correspondingly larger value of w. Thus, it amplifies the gradient magnitude in regions where the student underes- timates the teacher. As a result, choosing a large a (relatively) encourages the student distribution qe to exhibit mode-covering behavior. In contrast, employing the small in qe results in large w value. It assigns a large gradient magnitude to the area where the student overestimates, ultimately exhibiting that the small (relatively) reinforces mode-seeking property.
2
To support the results from the gradient analysis, we investigate the property of the optimized student distribution o through the toy experiments. Figure 3e shows that the optimized student distribution ς with small a converges to one of the peak, which indicates the mode-seeking. As increasing a, the ÿ gradually have thick tails while moving towards the average of p. which implies the mode- covering. These analyses indicate that the balance between mode-covering covercnn and mode-seeking, often attributed to divergence selection, might be controlled by the a-mixture assistant distribution.
Although can be any real numher, we assume for the sake of simplicity.

---

Published as a conference paper at ICLR 2026
Table 3: ROUGE-L scores (1) with various divergences D and a. We utilize GPT-2 XL (1.5B)
GPT-2 (0.IB). We use À = 0.1 for AMiD. indicates the no-assistant baseline. = -1 and correspond to the assistant distribution employed DistiLLM and TAID, respectively.
Divergence
Assistant (o.A)
Val.
Dolly Eval
Self Inst (1)
Vicuna
Super (1)
UnNi
Avg.
25 25.25
22. 96 2023
10.54 2014
15.33
=015
18.10 #0:25
21.10 20.10
17.61
AMID -5.0)
10:4
28.32
10:
29.52 10:
AMID (a -3.0)
28.47
25,72 =0.17
13.68
=0:19
16.71
=030
27.30 #0:30
29.03
22.49
-1.00
1078
10:30
23.87 10.3
26.
AMID (a -0.5)
27.37
24.17 10.57
12.15
1:44
16.37
2038
24.34 10:30
24.36 10:0
20.28
0.0)
1025
10700
20.09
10:00
22.71 10.11
AMID (a 0.5)
25.56
22.81
10:22
10.77
1:400
16.24
:025
18.96
1:045
22.13 10.00
18. 18
(a 1.0)
25.31
10:13
10:51
18.74
10:40
28.85
26.67 #0.50
12.32 #043
17.48
24.25 #0019
26.56
21.46
AMID (a= -5.00
28.39
26.16 +0.35
12.95 2037
17.39
=0.39
24.54 4-0:22
27.17 20.00
21.65
AMID (a -3.0)
28.75
26.42 =0.12
12.71
17.17
27.00
=:011
28.16
22.30
Dasa (
AMID (a =-100
28.97
22,96 :0.15
12.34 2004
17.27
22, 22.44 #:0.6
25.68 2:0.00
20.83
AMID (a= -0.5)
28.
26.15 =0.30
11.81 =0229
16.54
23.49
#:0.25
25.87
20.77
AMID (a 0.00
28. 80.80
20:22
2011
17.71
=0055
22.72 1:0:44
25.36
20,
AMID (a= 0.5)
28.45
25.42
=0.11
11.45 =0.26
17.31
21.58 #:0.44
24.43
20.04
AMID (a 1.0)
0.16
1:001
2.81 3:002
3:006
1.64 10:00
1.84 3:00
3.94
%
28.61
25.49
12.52
17.36
26.07
27.36
21.76
AMID (a -5.0)
29.24
26.44 11:12
13.74
Lovee
16.76
29.71
1ood
30.35
1:0.0
23.40
AMID (a 3300
29.07
26, 38
13.58
16.11
29.27
30.14
23.10
Dia(pllr)
AMID (a- -1.01
28.70
26. 10 10:24
13.34
1,025
16.71
L03:
26.55
1.0
29.55
1.0.11
22.45
AMID (a -0.5)
28.70
26.37
13.59
17,02
27.06
28.50
22.51
AMID (a -..0)
28.86
25.77 10:34
13.57
1001
16.14
.0366
27.26 Loozh
28.52
100.30
22.25
AMDD 0.5)
28.46
25.80
12.94
16.59
26.29 6:0.22
27.73
21.87
AMID (a 1.0)
24.93
22.36
10:22
9.72
10.55
16.29
.0.00
15.09
16.15
1.0.12
15.92
Avg. ROUGE-L T
Avg. ROUGE-1
22
0.2
23.0
0.45
Neg. Self-Beii SEIU-BLEU
Neg. Self-BL
22.5
0.50
0.4
18
22.0
0.55
-5
33
-1
0
1
-5
11
0
1
(a) (pllpll
0
(b) DAn (allr(
12
Figure 5: Performance curve on ROUGE-L (quality) and Self-BLEU (diversity). Colored dashed
lines: no-assistant baseline (go). We use x = 0.1.
4 EXPERIMENTS
We consider both general instraction-folloiing distillation and task-specific distillation to validate the effectiveness of AMiD. AMiD is primarily compared with methodologies, which inherently include the assistant distribution, such as GKD (Agarwal et al., 2024), DistiLLM (Ko et al., 2024), TAID (Shing et al., 2025); and the state-of-the-art ABKD (Wang al., 2025). Please refer to Tabll 15 in Appendix C. for further performance comparisons with additional baselines. We use with QAB = 0.2, BAB 0.7 and adopt adaptive off-policy training (Ko et al., 2024) default. We explore over the range {-5, -3, -1, -0.5, 0, 0.5, 1.0] and employ 0.1 as default by following prior work (Ko et al., 2024). Additional details on datasets, models, and training details are provided in Appendix B.
4.1 PERFORMANCE COMPARISON
Instruction-followin Experiments. Table reports results on the GPT-2 family (Radford al., 2019) across different model sizes. AMiD consistently achieves the best performance in most eval- uation settings, surpassing prior methods such as GKD, TAID, and DistiLLM,, which also exploit assistant distributions. Notably. AMiD delivers substantial gains on SuperNI and UnNI, benchmarks requiring generalization to diverse and unseen instructions (Wang et al., 2022). These improvements suggest that AMiD promotes superior mode coverage and distributional alignment, thereby enhanc- ing out-of-distribution generalization. Even when the capacity gap narrows for larger models (e.g., GPT-2 Large), AMiD continues to yield significant improvements, demonstrating that the a-mixture assistant distribution benefits not only small students but also stronger ones, thereby validating its scalability and robustness. Figure 4 further shows that AMiD consistently envelopes the baseline's
8

---

Published as a conference paper at ICLR 2026
Table 4: ROUGE-L scores (f) with various SGOs. We utilize GPT-2 XL (1.5B) GPT-2 (0. We
use = 0.1 for these experiments. Qo indicates no-assistant baseline ABKD for these experiments. -1 and = correspond to the assistant distribution employed in
We use DAB for AMiD.
DistiLLM and TAID, respectively.
Datasel 17
Assistant rg
Val. (1)
Dally Eval (1)
Self Inst (1)
Vicuna (1)
Super NI (1)
UnNi (1)
Avg. (1)
27.06
24.81
11.25
15.05
21.78
23.73
19.32
Fixed
AMID (n -1)
27.35
25,34
2075
11.54
1076
15.34
1030
21.77
1
24.45 100
19.69
(liinion et al., 2015)
AMiD (a-
27.09
24.36
12:40
1041
14.37
24.36
10:2
26.28 cow
20.35
AMID (ar * 1
27.84
25.32 20L5
13:44
:0.1
15,44
20.1
27.03
20.12
28.19
2:00
21.88
go
28.25
25.70 4020
13.03
20.7
16.86
20.34
24.67
20.16
27.47 =0u5
21.55
On-policy (Lin et al, 2020)
AMID (a- = -1)
28.60
25,43 2034
12.96
20:5
16.59
20.39
27.35
10.16
29.93 =007
22.45
AMiD (n- 1)
28.60
25. 2035
13.28 10
17.08
24.35
26. 94 1,304
21.35
AMID # 1
1001
14.31
10:2
1,027
1,00
29.08
25.67 2016
12.38 1112
17.15
2052
22.98
1::22
26.20 1:314
20.88
Mixed
AMiD = -1)
#025
11.98 2012
=0.13
+0.17
=006
20.93
(Agarwal et al, 2024)
AMID (ar = 1)
28.06
25 25.68 =0.39
12.81 =0:00
16.97
=0.27
24.91
+0.21
26.52 =:00
21.38
AMiD 1))
29.24
26.46
13.62
16.91
28.13
29.39
22.90
Go
28.61
25:49 =024
12.52
#0.5
17.36
=0.55
26.07
+0.14
27.36 #0010
21.76
Adaptive off-policy (Ko et al., 2024)
AMID -1)
28.70
10
13.34 2015
16.71
26.55 #0.17
29.55
22.45
AMID (a- 1)
27.80
25.78 2011
13.74 19
16.42 2,
26.04 1,n.
27.79 1,00
21.95
AMID +1)
29.24
26.44
2012
13.74
20.40
16.76 =0.24
29.71
4:0000
30.35
=:000
23.40
0.1 (21.64 = 2.58)
23.40
23.10
22.45
22.51
22.25
21.87
15.92
Table 5: Win Rate (WR, %) scores on three instruction-following benchmarks. We uti- lize 10wen..5-14B-Instte as a teacher and Qwen2.5-1 Instrrutttt student.
0.5
23.29
23.05
23.13
22.80
21.87
20.02
17.46
(21.68 2:17)
(21.09 = 5.12)
0.9
23.12
23.25
22.89
22.75
21.99
18.58
15.04
Model
AlpacaEval
WR
Eval Insl-Inst
WR
UltaFeed (R)
Avg. (f)
-5.0
-3.0
-1.0
-0.5
0.0
0.5
1.0
Teacher Sudeet
95.7
64.3
47.2
92.2
40.4
84.9
50.6
90,9
(23.77 0.14)
0.12)
0.34)
00116)
0.19
1.65)
122)
(23.12&
(22.52 52
(22.69
69
122.04
(20.16
(16.14
DisULLM-2 1))
80.2
70.4
61.7
70.8
[a
Disnil
81.3
71.1
63.6
72.0
Figure 6: Relationship between a and À under DAB.
to
validation ROUGE-L curve, indicating both efficient and stable optimization. Additional compar- isons with other baselines and experiments on Ope2 are provided Appendix C.
Task-specific Experiments. To further investigate the effectiveness of AMiD, we consider various task-specific distillation, such as translation, summarization, and reasoning tasks. We adopt the im- plementation of SKD (Xu et al., 2024b) and employ the fixed dataset strategy. As shown in Table 2, using an assistant distribution achieves higher performance compared to the no-assistant baseline ABKD. Nevertheless, in the previous framework, where only a = 1 is available, it exhibits mixed performance depending the task and network. Our proposed framework, AMiD, which extends the range of a, allows us to discover the high- performance assistant distribution. This generalization leads to consistent improvements over the basclines, achieving the best performance on all tasks.
4.2 ABLATION STUDY AND ADDITIONAL ANALYSIS
Balancing Mode-covering and Mode-seeking via As mentioned in Section 3.3, we have demonstrated that even when employing the same divergence, adjusting the a of the o-mixture assis- tant distribution allows to control the mode-covering mode-seeking property of the optimized student distribution. To further substantiate this analysis, we examine the trends of ROUGE-L, rep- resenting quality, and (Negative) Self-BLEU, representing diversity, by adjusting a with a fixed divergence. Figure 5 exhibits a clear quality-diversity trade-off for both KL divergence DRL and a-3 divergence DAB Specifically, as a increases, quality decreases and diversity increases, sup- porting the theoretical analysis that the mode-covering property is enhanced. Decreasing shows the opposite effect, aligning with the mode of teacher distribution, which indicates mode- seeking. These results demonstrate that, even under a fixed divergence, a serves as an effective control knob to balance quality and diversity.
Relationship between and X. To further examine the relationship between a and X, conduct the experiments on = 0.1, 0.9 with (Al(pl))(a.)) (()))) Figure 6 presents the average perfor- mance of five instruction-followe following datasets among the various and À combinations. The analyses
of experimental results are as follows: (1) Across all tested values of X, using a smaller a consis- tently achieves higher performance. This observation aligns with our theoretical analysis, indicating

---

Published as a conference paper at ICLR 2026
Table 6: Performances on mnstrrction-followinng code generation, and mathematical reasoning. We utilize the -went.55-Instruct models, which are appropriate for each task, employing a 7B model as
the teacher and a 1.5B model as the student. For evaluation, we employ LLM- au-uuge (Zheng
et al., 2023) with GPT-4o for AlpacaEval and Evol-Instruct, GPT-4o-mini for UltraFeedback.
Model
Instructioa-ollloinng
Code gencration
Math reasoning
AlpacaEvval WR (f)
Eval Inst
UltraFeedd WR (1)
Avg.
HumannEval
MBPP pass (1)
Avg.
WR (1)
WR (1)
pass I (1)
pasc
GSM8K (t)
Teacher Student
93.7
80.8
88.0
83.1
64.2
46.2
40.0
50.1
70.7
69.3
70.0
74.3
(a = -1
79.5
69.0
62.6
70.4
72.0
74,6
73.3
76.9
M- -5)
80.7
71.0
63.3
71.7
73.2
73,5
73.4
77.4
that smaller a relatively induces a more mode- seeking behavior. (2) When À is too large (A = 0.9), performance slightly degrades and exhibits a larger standard deviation. We attribute this the assis- tant distribution being overly close to the teacher distribution, which 1) limits effective knowledge transfer and 2) makes the optimization more sensitive to curvature variations.
contrast, = 0.5 achieves both high and stable performance, demonstrating that choosing a midpoint provides ro- bustness against curvature changes. (3) Compared to X, the performance is generally more robust to changes in a, as indicated by lower standard deviations. However, the a values close to I show noticeably higher standard deviations. We conjecture that this instability arises because the change of mixing coefficient (A) between the teacher and student distributions makes hard mode-covering.
Compatibility with Divergences. As mentioned in Section 3.3, AMiD allows the use the arbi- trary divergence D since the a-mixture assistant distribution rg (a.A) is a valid distribution. Therefore, we conduct performance comparisons across various combinations of divergences and a values to demonstrate the versatility of AMiD. In Table 3, we observed that AMiD generally achieves higher performance than the no-assistant baseline regardless of divergence in most settings. Notably, the highestt-prforming was discovered in & +1 regions beyond the limited scope of prior works, and its values are generally small. These results confirm the universality of AMiD to generic di- vergences, representing its wide adaptiveness and flexibility. Please refer to Appendix C.3 for the student-assistant cases D(go, rg (a.))).
Universality to SGOs. AMID is generalized framework from the view of assistant distribution rg (a,A) and divergence D, and therefore not constrained by the dataset D. In Table we confirm the universality of AMiD across various student-gend generated output (SGO) strategies. AMiD (a # 1) outperforms the no-assistant baseline and previous mixtures (a =1) by significant margin across almost all metrics. These results indicate that AMiD is compatible with diverse SGO pipelines and remains effective regardless of how the datasets are collected.
Cooperation with Contrastive- Distillation. The proposed a-mixture assistant distribution ro
(a.A) is a theoretically valid probability distribution for any and X. Therefore, the r (a,AA can be combined with the contrastive- based distillation methods beyond the divergence- distillation methods. To validate this applicability, we replace the assistant distribution of DistiLLM-2 (Ko et al., 2025), which utilizes m-mixture (a = -1), with our a-mixture assistant distribution. As shown in Table 6, incorporating our a-mixture assistant distribution exhibits further performance improvements or competitive performances over the base contrastive method, DistiLLM-22.
Scalability to Large Teacher. To demonstrate the extensibility, we conduct an experiment by dis- tilling 1-t.2...-1BB-nttteeee into Qwen2.5-1 1-.52.551BBB-..... ISS--nst under the faaruction-following task. As shown in Table integrating a-mixture assistant distribution exhibits performance improvements over the LM- These results indicate that AMiD remains effective in large teacher model.
CONCLUSION
This work introduces a unified framework for KD in LLMs by proposing the a-mixture assistant distribution and the corresponding distillation method, AMiD. Our approach systematically gener- alizes previous fragmented methods and enables flexible interpolation between teacher and student. Theoretical and empirical analyses congruently demonstrate that the design parameter controls the mode-secking vs. mode-covering behavior. AMiD consistently outperforms prior KD methods across diverse settings and establishes new foundation for assistant- KD for LLMs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the IITP (Institute of Information & Communications Technology Plan- ning & Evaluation)-ITRRC (Information Technology Research Center) grant funded by the Korea government (Ministry of Science and ICT) (IITP-2026-) 27--2026-RS222244 RSS2024-004437266 with 50% contri- bution rate. This research was also supported by AI Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous DatP(IITP) funded by the Ministry of Science and ICT(RS-2022-11222
with 50% contribution rate.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 2xXIv:2303.08774,2 2023.
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In The twelfth international conference on learning representations, 2024.
Shun-ichi Amari. Integration of stochastic models by minimizing a-divergence. Neural compula- lion, 19(10)2278--2766 2007.
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv: 108 2021.
Peter S Bullen. Handbook of means and their inequalities, volume 560. Springer Science & Business Media, 2013.
Mark Chen. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A real-life scenario dia- logue summarization dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 5062- 5074, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011. Andings-acl.n 449. URL https: https//////aaaahhoaaaaooo wwt.oorg//22/... windings-acllll.cccccooo 449/.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
/
Andrzej Cichocki, Sergio Cruces, and Shun-ichi Amari. Generalized alpha-beta divergences and their application to robust nonnegative matrix factorization. Entropy, 13(():134-11) 170, 2011.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 1..111280 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi. Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instructiontuned 1lm. 2023.
Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hefferman, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint 12XX::2277.0667. 2022.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ulrrfeedback: Boosting language models with scaled ai feedback. arXiv preprint 10.013377 2023.

---

Published as a conference paper at ICLR 2026
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-30069, 2023.
Shinto Eguchi and Osamu Komori. Minimum divergence methods in statistical machine learning. No Title, 2022.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github. com/ wpwnlm-researche open 11mm..
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, http::/skkllono0/1/// TeenWebTextCorpus 2019.
io/
Roger B Grosse, Chris J Maddison, and Russ Salakhutdinov. Annealing between distributions by averaging moments. Advances in Neural Information Processing Systems, 26, 2013.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large
language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: http://ppeeevvieeee net. forum? id-
GH Hardy, Inequalities. Cambridge University Press, 1952.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint wrxiv:1503.0253/1 2015.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tun- ing language models with (almost) no human labor. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14409-14428, Toronto, Canada, July URL
2023. Association for Computational Linguistics. doi: 10. 10.18853/ 12222 https: aalaalaantt aawlannhollogyyyyaooooooooolllll wwtp///......... ao002.aaclloone 806/.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the
2016 conference on empirical methods in natural language processing, pp. 1317-1327, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In International Conference on Machine Learning, pp- 24872-24895. PMLR, 2024.
Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se- Young Yun. Distillm-2: A contrastive approach boosts the distillation of llms. arXiv preprint arXiv:2503.07067, 2025.
Andreï Nikolaevich Kolmogorov and Guido Castelnuovo. Sur la notion moyenne. G. Bardi, tip. della R. Accad. dei Lincei, 1930.
Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distilla- tion through imitation learning. arXiv preprint 2xXi:2000907253, 2020.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: / /aaaaaaannntoooooooooooo org / WC 10033/
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306, 085688. 2023.
Vaden Masrani, Rob Brekelmans, Thang Bui. Frank Nielsen, Aram Galstyan, Greg Ver Steeg, and Frank Wood. q-paths: Generalizing the geometric annealing path using power means. In Uncer- tainty in Artificial Intelligence, pp. 1948-1947. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2026
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Frank Nielsen. An elementary introduction to information geometry. Entropy, 22(10):1100, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 37730-27744) 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Makoto Shing. Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba. Taid: Temporally adap- tive interpolated distillation for efficient knowledge transfer in language models. arXiv preprint arXiv:2501. 16937, 2025.
Zengkui Sun, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. Warmup-distill: Bridge the distribution mismatch between teacher and student before knowledge distillation. arXiv preprint 0rXiv:2502.11766 2025.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint Nit.:::2203002925.. 2024.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2, 2024.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Guanghui Wang, Zhiyong Yang, Zitai Wang, Shi Wang, Qianqian Xu, and Qingming Huang. Abkd: Pursuing a proper allocation of the probability mass in knowledge distillation via a-G-ddivrreenc. arXiv preprint arXiv:2505.0435 04560, 2025.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Es- haan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob An- derson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravse- haj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Suuper-atturllnstrucconn tion via declarative instructions 1600+ NLP tasks. In Yoav Goldberg, Zoritssa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pp. 5085-5109, Abu Dhabi, United Arab Emirates, December 2022. URL
Generaliza-
Association for Computational Linguistics. doi: 10. org. 2022. emnlp-mainn 340/.
https: https///acaaathhhoooo
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self eeff-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 13484- 13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18.13/ 21/2023.acl-oong.. URL https: aclanthologl org 2023. walc-loong.... 754/.
Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethinking kullback-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.022 02657, 2024.
13

---

Published as a conference paper at ICLR 2026
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow com- plex instructions. In The Twelfth International Conference Learning Representations, 2024a.
Wenda Xu, Rujun Han, Zifeng Wang, Long T Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yo Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. arXiv preprint arXiv:2410. 1at.:2400.11325,, 2024b.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint urXiv:2309. 12284, 2023.
Chen Zhang, Yang Yang. Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, and Dawei Song. Lifting the curse of capacity gap in distilling language models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4535-4553, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/1/2023.aaaa aolno. URL https: http:///aaaaattooaooooeo wtgg/02333322 acl- acl- 249/.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 1097-1100, 2018.
14

---

Published as a conference paper at ICLR 2026
A DEFINITION AND PROOF
A.1 AMARI'S G-DIVERGENCE
Definition. Let a E R be a real parameter. The a-function fa(u) is defined for a o +1 by
fa(u) = 1 - a² (1 (1-2) 1
4
1+a
(12)
Using this a-function, the a-divergence between two probability distributions p {p:} and & {gr) is defined as
Dalplla
4 a²
1
Ps
Ien
qi
I-o
+ #1.
(13)
The dual divergence obtained by replacing a with -a:
Da Dalplla] = D-a [g]p
(14)
We also define the limiting cases at a = 1 by continuity. When a = +1, the a-function becomes
fa(w) u log u, 1, log u, a =
(15)
Connection to KL and reverse KL. Correspondingly, the a-divergence reduces to
4
log Σ, qi
1, 1,
Da [plla]
D
(16)
P
Σ, Ps log
a
which correspond the KL divergence (a 11) and the reverse KI. divergence (a 1).
Relation to Rényi divergence. Since the Amari a-divergence includes KI. and reverse KI. as the limiting cases, it can be viewed as one-parameter generalization of the KL divergence. Moreover, is closely related to the Rényi divergence. If we define reparameterized order
1
2
then the quantity appearing in the Amari divergence,
Σp,
Σ
P. Ite 1-8
q
exactly the same expression inside the discrete Rényi divergence,
Ra
B 1 log
3 1-9 ΣΦe
q8
(17)
Thus, the Amari and Rényi divergences represent the same underlying aaeaaarmetter family through the invertible reparameterization 3 = (1 + a)/2
A.2 PROOF OF PROPOSITION 3.1
Proposition 3.1. The assistant distribution of TAID (Shing et al., 2025) is e-mixture of p and qo:
= softmax(1111111 X) logit(go) + X logit(p)) p 1-X
6
We omit the time index t and detached notation for the sake of uniformity.
15

---

Published as a conference paper at ICLR 2026
Proof. Note that p softmax(logit(p)) ,, exp(logit(p)) where logit(p) is logit of p and Zp is
normalization constant of p. Therefore, logit(p) = log(p 2p).
r = softmax(Alogit() + (1 - A)logit(g))
(18)
1 exp(Alogit(p) + (1 \)logi((g)
(19)
Z,
1 exp(Alog(p2,) + (1 2)))g()2,))
(20)
Z,
1
+ log Zz
(21)
Zr
2
p²q p¹q'-A 1-1
(22)
Z,
1
-"x where Z' Zr
pp
(23)
Z Zz -1-x
Now, it sufficient to show that Z' Epdgt~>
1
Z'
exp exp(Alogit(p) + (1 -()logggt(())
(24)
Z7 ZiZii
1
exp(logi((p)) "( Z
eexp(ogg((())) 1-A
Σ
(25)
7p
Σ At
(26)
Therefore, ra x -^ and it is valid distribution with normalization.
A.3 PROOF OF PROPOSITION 3.3
Proposition 3.3. (Continuity) Assume that p and qa are not both zero. Then, TB (a,2) is continuous function w.r.f a under the fixed X E [0, 1].
Proof. We begin with the proof for the continuity of the unnormalized assistant distribution pla.)) The o case is trivial since it is composition of continuous functions. For the 1 case, is a well-known fact that the power mean is a continuous function (Bullen, 2013). Especially, we can show that as follows:
lim log rg
(a.))
lim a+1
2
log
Ap
1-a 2 + A)g 2
1-a
(27)
1-
1-xx Ap 2
+ (1 A) q 1-0
logp
log q 1-a
lim a+1
(28)
1-a
2p 2
+ (1 A) q
Alogg + (1 logg
(29)
log(p"g2~>)
(30)
By the continuity of the exponential function, get lim <(a,2) = = We use L L'Hôpital's a(a.)) a->1 P
rule in the second equality. Note that Z Σ p (i) is continuous function w.r.t since it is
finite sum of continuous functions w.r.t Also, since p and cannot be both zero, <(aAA) is not zero, so > 0. Therefore, the rg (a.AA) 1 5 (axx) is contiuous function w.r.t. a.
A.4 PROOF OF THEOREM 3.4
Theorem 3.4. (Optimality) Let D be any proper divergence and R.
16

---

Published as a conference paper at ICLR 2026
E 10,1) and 30 s.1. D ra.d)) 0, then D(p,ro ra.))) 0 if and only if qo-
If E (0, 1] and 30 s.t. Digo, rg (aad)) = 0. then D(gs, T8 (0.1)) = 0 if and only if = qo.
Proof. We first prove that D(p, rg that D(p, ro (a.d))= if and only if p r (a
(a. implies p go- By the definition of divergence, have
Case a 1. In this case,
r
(a.7)
221"00 40
(1-A)
p
p 90
1-1
Zp pp1->= =9- 99 --- T-X p q8
(31)
Z
By integrating both sides, Z = which implies Z = 1. Therefore, = q0
Case a + 1. In this case, r
(a.2)
1 {
2
Ap
I-n
2
+ (1 - A) qn
La
2
1-a
1 Ap 2 + (1 20 p Z
T-a
1-x
1-a
2
7 2
2 = xp 2
+ (1 A)
2
(32)
1-a
1-a
I-0
0 2 2
X) p 2
= (1 - 2) g0
2
(33)
1-a
1-a
1-a
Z 2 X
Cp 2 99
=
2
where
C
(34)
1 - X
2
+ C'I-a p=
(35)
By integrating both sides, CT²s = 1 which implies C = 1. Therefore, p = 90-
D(go, 1 (o,))) is similar.
A.5 PROOF OF PROPOSITION 3.5
Proposition 3.5. (Gradient analysis) The gradient of f-divergence Dy(pllrg (u:2)
be expressed as:
VoD,y (pllr(aa))) = E (a.2)) w. { 4
}}
P
p (1(,(e))) (a.i)
- B,
-B,aa))
Vg log
(11)
[a.1]
r
where
(1 X)90
and (y(() == f(u) co)
Ap +(1 1 X)90
Proof. From the basic calculus, we can derive the following equation for fixed p:
a a r = ( P 5
(())))) - 2/ (2)
(36)
Or
Hence,
VoDr(pll!0 (0:2)) = Σ ty
p(n x. Ner)
Verla (ye x. v2))2))
(37)
pl (a.1)
(yn
x. (e1)
vIEV
E (m,))
P
Vo
log (a,))
(38)
(a.2)
17

---

Published as a conference paper at ICLR 2026
Before deriving the gradient of the log probability of (a.A) let us first derive Vorg follows:
Verlar) Vo ha (> (p) + (1 2) ha (10))) }
(39)
haltfard)) (1- 2) Voha(10)
(40)
(1 1/ (20) Vogo
(41)
(1- Na (1o) do Vo log
(42)
1// (go) go
A1
x(a,2) V log q0
(43)
Wa(FF (=(a.))
10
1-a
= (1-A)
q0
A(o,A)
V logge
(44)
A(a
168
1-a
A)
(aa,))
Ve log qo
(45)
1-a
1-a
Xp 2 + (1 X)
w j(a.)) Vo log 90
(46)
Therefore,
Velogroas (o.1) Vorlar)) ((a.1)
1
Verfa.s
(47)
Zr
w
V log qo - -E, B,aaailw
ww V log qe]
(48)
Lastly, placing Eq. (48) into Eq. (38) and rearranging will yield the final result.
VoDy(plir1 ((.)))
E go,
w
Vo log r
(a.N)
(49)
[a.15
(0.
4
p
{ w
Vo log go - B.gansfw Vo log 20
(50)
]
p
P
(,
(o,))
w
4/S
[a,A]
E (a,N) 6
Ta,A
V log go
(51)
r
rg
B EXPERIMENTAL DETAILS
B.1 TOY EXPERIMENT
We employ a two-modal Gaussian mixture for the teacher p- = 0.7A( + 0.3 0.8) and a
unimodal Gaussian for the student go o2) with 140 0, 1. We optimize 0 (14 o²) by minizing DxL. (pllsf (A))) with Adam optimizer (Kingma & Ba, 2014) with 5000 steps and 5e-2 learning rate.
18

---

Published as a conference paper at ICLR 2026
B.2 DATASETS
databricks-dolly-15k (Conover et al., 2023): An open-source dataset of instruc- tion-response pairs created by thousands of Databricks employees. It covers diverse be- havioral categories defined in Ouyang et al. (2022), including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.
Self-instruct (Wang et al., 2023): A framework for improving cestruction-followii ability by iteratively using model outputs to generate new instructional data. The dataset contains 52K instructions and 82K input-output pairs for tuning. 252 expert-written tasks for prac- tical evaluation, and 50K additional examples from public datasets for benchmarking.
Vicuna (Chiang et al., 2023): A benchmark consisting of 80 challenging open-ended ques- tions originally used to assess Vicuna. It provides a compact but difficult testbed for evalu- ating instruction-following performance.
Super-Natural Instructions (Wang et al., 2022): A large-scale benchmark comprising 1,616 expert-written NLP tasks spanning 76 task categories. Its test set includes 9K exam- ples drawn from 119 tasks, covering a wide spectrum of instruction types.
Unnatural Instructions (Honovich et al., 2023): An Al-generated dataset containing 240K instructions created with minimal human intervention. The collection demonstrates that synthetic data can serve as an effective substitute for human--uuater data. Its core subset includes 60K examples.
AlpacaEval (Dubois et al., 2023): AlpacaEval is derived from the AlpacaFarm evaluation suite but includes simplified formatting. In particular, Dubois et al. (2023) combined the original instruction and input fields into one unified instruction, a change that influences about a quarter of the samples sourced from Self-Instruct (Wang et al., 2023). The dataset ultimately comprises 805 difficult instruction-followin
queries.
Evaluation (Xu et al., 2024a): This evaluation set includes 218 prompts produced through the Evol-Instruct generation pipeline. The questions span a wide range of topics and serve compact benchmark for instruction-folloiing ability.
UltraFeedback (Cui et al., 2023): UltraFeedback a large and detailed preference dataset tailored for building high-quality reward and critic models. It contains roughly 64k prompts collected from UltraChat, ShareGPT, and Enol-Instruct. Each prompt was used to elicit four responses from different LLMs, and GPT-4 subsequently annotated these outputs based on dimensions such as following instructions, factual correctness, honesty, and helpfulness.
MetaMathQA (Yu et al., 2023): MetaMathQA was created to strengthen mathematical reasoning in LLMs. The dataset is generated through bootstrapping method in which each math question is re-expressed using multiple reasoning viewpoints, including forward reasoning, backward reasoning, and paraphrased formulations.
GSM8K (Cobbe et al., 2021): GSMSK consists 8.5K meticulously written grade-school math word problems. Designed to require multi-step inference, it is widely used as stan- dard benchmark for evaluating basic mathematical reasoning skills.
WizardCoder (Luo et al., 2023): WizardCoder is an instruction -uned code dataset built via the Evol-Instruct method. Starting with the 20K-sample Code Alpaca corpus, the cre- ators iteratively evolved prompts by increasing complexity. adding constraints, inserting misleading code, and introducing time/space complexity requirements. The final dataset contains about 78K evolved examples, which were used to fine-tune StarCoder and sub- stantially improve its coding performance.
HumanEval (Chen, 2021): HumanEval provides 164 hand-crafted programming tasks with function signatures, natural lanurae-language descriptions, and unit tests. It is one of the primary benchmarks for assessing code generation and was explicitly designed to avoid overlap with existing training data.
MBPP (Austin et al., 2021): MBPP includes roughly 1,000 Python programming exercises aimed at novice programmers. Each problem comes with textual description, reference code solution, and three automatic test cases. Portions of the dataset were manually vali- dated to ensure consistency and correctness.
19

---

Published as a conference paper at ICLR 2026
B.3 IMPLEMENTATION SETTINGS
Training. For instruction-followinn distillation, we use databricks-dolly-155 (Conover et al., 2023) for the distillation loss and OpenWebTee (Gokaslan & Cohen, 2019) for the pretraining loss. Teacher models include GPT-2 XL (1.5B) with SFT, and students are GPT-2 (0. (0.1B), GPT-2 Medium (O.3B), and GPT-2 Large (0.8B). To test scalability, OpenLLaMA2-7B (Geng & Liu, 2023) is distilled into OpenLLaMA2-3B using LoRA.
For task-specific evaluation, we use Flores-200 (Costa-Jussà et al., 2022) for translation, Dialog- Sum (Chen et al., 2021) for summarization, and GSM8K (Cobbe et al., 2021) for mathematical reasoning. Teacher models are Gemma-7B-It (Team et al., 2024) and Qwen2-7B-Instree (Team, 2024), while Gemma-2B-L and (wee2-00.BBIInsttt the full dataset, while students are trained with about 1,000 samples.
serve students. Teachers are fine-tuned on
We use QABBBAB BAB--AB-divergee divergeece with OAB BAB 0.7 and adopt adaptive off-policy train- ing (Ko et al., 2024) as default. We explore a over the range {-5, -3, -1. O, 0.5, 1.0} and employ À = 0.1 as default by following prior work (Ko et al., 2024). We utilize the AdamW optimizer and cosine learning rate scheduling by following the previous work (Ko et al., 2024; Wang et al., 2025). We search the learning rate the range (0.0005, 0.0001, 0.00005) except for the cooperation with contrastive based distillation experiments, which use the default setting of DistiLLM-2 (Ko et al., 2025).
Theoretical insights based a tuning guidelines. Herein, we provide principled tuning guide- lines grounded in the theoretical properties of First, in Section 3.2, we show that the support of the assistant distribution varies with a: supp(r) supp(r((.A)) supp(p) O supp(go) when a 1. In KD for LLMs, the teacher and student of- ten exhibit capacity gap and produce nigh-dimenssonal outputs with many near-zero probabilities. As a result, they do not share sufficiently large common support general. For this reason, we recommend using a < 1 in most practical settings, as this choice improves training stability and enables more reliable knowledge transfer.
= supp(p) U supp(ąe) when a < 1,
Furthermore, our gradient analysis and (toy) experiments demonstrate that a directly controls the trade-off between mode-covering and mede-seeking behavior of the optimized student. Under the < 1, increasing encourages relatively stronger mode covering, thereby improving output di- versity. Conversely, smaller emphasizes mode seeking, which enhances fidelity to the teacher. Therefore, for enhancing the teacher-student alignment and performance, values. However, since too small a can induce high curvature in the geometry of the interpolation path, which may reduce optimization efficiency, such choices should be used with caution.
suggest using small
Based on these theoretical insights, we explore a over the range {-5, -3, 1, 0.5, 0, 0.5, 1.0). Table 7 provides the detailed configuration for each task.
Table 7: Configuration of hyperparameters for AMiD
Student
Divergence
Daz(p (r)
5.0
Duxt r)
-3.0
GPT-2 Base
Daa(plir
5.0
databricks-dolly-
GPT-2 XLarge
Dicstell))
Instruction following
GPT-2 Medium
Das(plir)
-5.0
GPT-2 Large
Das(pllr)
-3.0
OpenLLaMA227B
LaMA-LB
Dan(sllr)
-3.0
Qwen2.5 7---nstuutt
Qwen2.5 1.su2.5-1.5B- 5.5--tttttttt Instruct
Dosearia-s
5.0
UltraCha1200k
Qwen2.5. 14B
Qwen2.5. Instruct
DowaLLlM
2
Gomma-7B-4
Gemma-2B-It
Dan(pllr)
-0.5
Translation
Flores-200
Qwen2- -Bstrrtuctt
Qwen2-0.5BB Instruct
Daniplir)
Summarization
DialogSum
Qwen2 7B Instruct
Gemma- -B-IIt
Gemma-219 - Qwen2 0.5B Instruct
Dan(plir)
Dan(plir)
33.0
0.5
GSM8k
Gemmun-788
Gemmuu 2B-II
Dantplir)
Mathematical reasoning
Swea2-7B-Instrrun
Qwen2-0.55 111--8---nnsstucc
Danielir)
-3.0
MetaMathQA
Qwen2.5 1aan2.5-Math-75 Instruct
Qwen2.5-Mann
Code generation
WizurdCoder
Qwen2.5- Instructt
Qwen2.5-Codde 1,5B. Instructt
DowaLLLM
-5.0
20

---

Published as a conference paper at ICLR 2026
Evaluation. For evaluating generation quality, adopt ROUGE-L (Lin, 2004) and Self- BLEU (Zhu et al., 2018). ROUGE-L measures the similarity between the generated output and the reference text by computing the Longest Common Subsequence (LCS). Specifically, recall and precision are defined as
LCS(2. V)
LCS(x, 1)
R
PLac
(52)
Lx
Ly
where (a, y) is the length of the longest common subsequence between the reference and the generated text y, and Tax by denote their respective lengths. The final ROUGE-L score is given by the harmonic mean:
2. RLCs PLes
ROUGE-L- = RLCs Pics
(53)
A higher ROUGE-L score indicates that the generated text more closely matches the reference in terms of sequence overlap.
Self-BLEU evaluates the diversity of generated outputs by leveraging the BLEU metric (Papineni al., 2002). BLEU computes the geometric mean of modified n-gram precisions with brevity penalty (BP):
1 (11-r/))
if r
BP
(54)
if r.
BLEU((. R) BP exp
Σ Wn lo, R)
(55)
where è is the candidate length, r is the effective reference length, (n Rt denotes the modified n-gram precision, and Wn positive weights summing one. Building this definition, Self- BLEU is calculated by treating each generated sample s; as the hypothesis and the remaining set (sss as references:
M Σ M
Self-BLEU(S)-
(56)
A higher Self-BLEU score (close 1) indicates that the outputs are highly similar to each other, reflecting low diversity and more deterministic behavior, while a lower score (close to 0) suggests greater diversity across generations.
C ADDITIONAL EXPERIMENTAL RESULTS AND DISCUSSIONS
C.I MORE COMPARISON WITH BASELINES
Table 15 presents the complete results on the GPT-2 family, where we extend the comparison to broader set of baseline methods beyond those reported in the main paper. We observe that AMiD consistently outperforms all competing approaches across different student sizes (O. 0.3B, 0.8B), further validating the robustness of our method. In particular, while methods such as SeqKD, Im- itKD, MiniLLM, and AKI. yield modest improvements over standard knowledge distillation (KD), they still fall short of strong assistant-basee methods like GKD, TAID, and DistiLLM. Among these baselines, ABKD often emerges as the strongest competitor. Nevertheless, AMiD achieves clear performance gains over ABKD nearly every evaluation setting.
C.2 RESULTS ON OPENLLAMA2
Table 8 reports results on the Lae22 family, where 7B teacher is distilled into 3B student. Consistent with findings the GPT-2 series, AMiD achieves the best overall performance across most evaluation benchmarks. In particular, AMiD surpasses prior assistant- based approaches such TAID and DistiL.LM (both SKI. and SRKL variants), as well the strong baseline ABKD.
21

---

Published as a conference paper at ICLR 2026
Table 8: ROUGE-L scores (f) on OpenLLaMA2-7B OpenLLaMA2-3B. Bold and Underline
mean the best and second-best performance of each column, except the teacher, respectively. All results are based on our own re-implementation. We conduct the evaluation with five random seeds.
Model
Val. (1)
Dolly Eval (f)
Self Inst (f)
Vicuna (1)
Super (f)
UnNT (1)
Avg. (t)
Teacher
27.60 2033
18.17 20,0
17.85 10.45
31.05 20.31
32.40 20.28
25.41
OpenLLaMA2-7B
OpenLLaMM2-338
TAID
30.85
26.53 2023
17.73
2:000
18.14
31.93
10.23
31.55
10.12
25.18
DistiLLM (SKL)
33.07
28.63 1002
20.20
1:0066
19.15
10.30
35.31
10.19
34.74
10.10
27.61
DistiLLM (SRKL)
33.18
28.83
#041
20.76
#0.37
19.37
36.82
35.76
28.31
ABKD
33.91
29.43
20.46
20.42
39.51
38.07
29.58
AMID (Ours)
34.39
29.69 2047
20.99
=0:37
21.03
39.06
#0.21
37.31 20.11
29.62
C.3 COMPATIBILITY WITH DIVERGENCES (9I (1) (1r)
Table provides the complementary results when employing the divergence DEL (go trasting the student distribution against the a-mixture assistant. Similar to the findings in the main text (Table 3), AMiD consistently outperforms no-assistant baseline across most evaluation benchmarks, confirming that the proposed method is broadly compatible with different divergence directions.
g (a.))) con-
Table 9: ROUGE-L scores (1) with DxL DxL (gellr GPT-2 2 (O.1B). We use fixed X = 0.9 for these experiments.
and various a. We utilize GPT-2 XL (L.5B)
Divergence
Assistant ig
(orA)
(?)
Dolly (f)
Self (*)
Vicuma (?)
Super
UnNT
Avg. (f)
28.71
26.22
12.57
16.97
24.75 1o:00
26.59
21.42
AMiD
-5.0)
27.54
24.23 5023
12.59
30.32
15,80
=044
24.50
#0.14
26.38
=007
20.70
AMiD
-3.0)
27.96
25.13 10:0
12.80
16.32
25.54
26.86
21.33
(owll ( (n.A),
AMiD (a -1.0)
28.21
25.74 50.20
12.13
30.23
16.34
=0.5
25.40
#0.10
26.91
=0.12
21.30
AMiD -0.5)
28.89
26.01
12.84
17.04
27.43
27.59
22.18
AMiD 0.0)
28.84
26.70 50:3
13.36
40.31
15.95
=0:6
26.23
+0.17
27.70
=0.10
21.99
AMID (a -0.5)
29.02
26.48 10:7
13.73
16.78
26.78
28.65
22.48
AMID (a 1.0)
28.40
26.01
60.4
12.03
$0.30
16.96
=0.1
24.84
+0.24
27.01
=0.11
21.37
C.4 OVERLAP-BASED ADAPTIVE SCHEDULING
The theoretical insights based a tuning guidelines efficiently exclude low potential candidates. How- applying single fixed global value can still be sub-optimal in certain cases.
To address this concern, we introduce a curriculum basudd adaptive a scheduling based on the degree of overlap between token-level teacher distribution p(3)|y< and student distribution go (vllv< z). The intuition that when the teacher and student distributions are highly overlapped, encourage mode-covering align further, whereas when the overlap is low, enhance mode- seeking to find the mode first.
We define token-level overlap ouly, Ey, min(p((vly<iia)) qo (ynlu, Obviously, the overlap value oul,, bounded into [0, 1J. Also, oul, approaches 0 when p(ulu=))))) x) and qo (Milv<t, 2) significantly differ, and approaches 1 when they are well-aligned. Furthermore, oul, can be expressed terms total variation distance 1 TV (p(y1) x), (olllvec x)), which provides same interpretation.
Given the predefined Omin and Omars we set the token-level 0s, as the linearly increasing value along the line passing through (0, Omin) and (1, amarz) i.e. + (Amar aminn) ovl,, Ominn- Under this idea, when the teacher and student distributions differ substantially, the oul, becomes small, leading teacher and student distributions are similar, both ovl, and a have larger values, thereby reinforcing mode-covering This mechanism systematically determines by combining the degree of alignment between the teacher and student distribution, which continuously changes through training, with the theoretical characteristies of o.
smaller assigned a, which strengthens mode-seeking. Conversely, when the
22

---

Published as a conference paper at ICLR 2026
Table 10: ROUGE-L scores (1) on five task-agnostic instruction-following datasets with fixed versus adaptive a scheduling. Bold means the best performance of each column. We use DAB and 0.1 for AMiD.
Assistant
Val. (f)
Dolly Eval (1)
Self Inst (1)
Vicuna (1)
Super NI (f)
UnNI (f)
Avg. (1)
AMiD (Fixed
29.24
26.44
13.74
16.76
29.71
30.35
23.40
AMiD (Adaptive a)
29.31
26.50
14.02
16.59
29.87
30.60
23.52
C.5 ROBUSTNESS TO OPTIMIZER
To investigate whether the effectiveness of AMiD depends on particular optimization setup, we evaluate its performance under the Lion optimizer. Table exhibits the robustness of AMiD w.r.t. the optimizer.
Table 11: ROUGE-L scores (1) on five task agnossicc instruction forllwoinnn- datasets under the Lion optimizer across different a-mixture configurations. Bold means the best performance of each col- umn. We use an and À = 0.1 for AMiD.
Assistant
Val. (1)
Dolly Eval (t)
Self Inst (t)
Vicuna (1)
Super NI (f)
UnNI (1)
Avg. (1)
No assistant
28.38
26.02 #0.88
12.19
20:34
17:24
+0.37
26.29
=0.19
28.88
#0.10
22.12
Lion
AMID (a -))
28.68
25.29 20.66
12.21
#0:25
17.81
40.36
24.82
=0.33
27.87
+0008
21.60
AMID +1)
24.98
22.33
=0.25
15.71
16.21
AMID (o. =1)
27.85
26.14 4021
12.55
40.12
17.25
40.48
28.28
=0226
29.69
1.006
22.78
C.6 ROBUSTNESS TO LEARNING RATE SCHEDULING
To further assess the robustness of AMiD to optimization hyperparameters, we evaluate its perfor- mance under the Noam learning rate schedule, originally designed for transformer architectures. Table 12 also support the robustness of AMiD.
Table 12: ROUGE-L scores (1) on five task-agnostic instruction-followin datasets under the Noam learning rate schedule across different a-mixture configurations. Bold means the best performance of each column. We D a and 0.1 for AMiD.
Assistant
Val. (t)
Dolly (t)
Self Inst
Vicuna
Super (t)
UnNi (t)
Avg. (t)
assistant
25.58 10.17
13.35
10.54
16.43
1.028
28.30
1..24
29.90
10.11
22.76
Noam
AMiD -1)
AMiD (a +1)
28.91
28.33
26.02 30
25.59 =025
13.61
14.07
2:025
#0.49
17.09 10:19
16.25
4034
27.78 10.11
26.42
+0.20
29.49
28.26
20.04
=0.19
22.09
22.93
AMID (a +1)
29.39
26.12 1035
13.07
10.52
16.53
10:46
29.06
10.14
30.86
10.09
23.19
C.7 MITIGATE THE CONFLICT VIA TEMPERATURE SCALING
As discussed in Section 3.2, combining an assistant distribution with narrow support and diver- gence that requires the expectation w.r.t. the assistant distribution can lead training instability and poor knowledge transfer.
Since the primary cause of this issue the narrow support of the assistant distribution, we conjecture that applying distribution softening technique could alleviate the instability even for such problem- atic combinations. To verify this conjecture, we employ temperature flattening technique in various area. The table below shows the performance when using various
1, which is widely used
temperature values under DRKI (p]lr (plin (a.)) with a = 1. The results exhibit that introducing the temperature leads to stable training and can even yield strong performance with appropriately chosen temperature value. However, large temperature causes an over-flattening effect, inducing large shift in the assistant distribution and consequently degrading performance. Overall, we demon- strate that temperature scaling can empirically mitigate the instability associated with problematic combinations under the appropriate temperature value.
23

---

Published as a conference paper at ICLR 2026
Table 13: ROUGE-L scores (1) on five task-agnostic instruction-following datasets under
DRKT. DRKi(pllvo (pllv) with = when applying temperature scaling to soften the assistant distribution.
Assistant
Val. (f)
Dolly Eval (t)
Self Inst (1)
Vicuna (1)
Super NI (?)
UnNI (f)
Avg. (1)
AMiD (o = 1.0)
0.16
4.27
2.81
9.12
1.64
1.84
3.94
AMID (a- 1.0. 1.5)
27.40
24.61
11.55
17.11
21.41
23.39
19.61
AMID
2.0)
28.64
26.83
12.74
17.62
23.56
27.01
AMiD 1.U, = 5500)
28.78
26.78
12.43
17.30
25.87
27.74
22.02
AMID (a = 1.0,T 10.0)
27.33
24.73
12.11
16.95
23.65
26.80
20.85
Table 14: ROUGE-L scores (f) across different combinations of À and the a-mixture assistant distribution.
Assislant
Val. (t)
Dolly Eval (1)
Self Inst (f)
Vicuna (1)
Super NI (f)
UnNI (t)
Avg. (t)
No Assistant
28.61
25.49
12.52
17.36
26.07
27.36
21.76
AMiD (a -5.0)
29.24
26.44
13.74
16.76
30.35
23,40
AMiD (a -3.0)
29.07
26.38
13.58
16.11
29.27
30.14
23.10
AMiD (a -1.0)
28.70
26.10
13.34
16.71
26.55
29.55
22.45
0.1
AMiD (a -0.5)
28.70
26.37
13,59
17.02
27.06
28.50
22.51
AMiD (a 0.0)
28.86
25.77
13.57
16.14
27.26
28.52
22.25
AMiD (a 0.5)
28.46
25.80
12.94
16.59
26.29
27.73
21.87
AMiD (a 1.0)
24.93
22.36
9.72
16.29
15.09
16.15
15.92
AMiD (a -5.0)
29.38
26.41
13.81
16.44
29.19
30.58
23,29
AMiD (a -3.0)
29.38
26.45
13.71
16.43
28.23
30.44
23.05
AMiD (a 1.0)
29.11
26.31
14.09
16.70
28.68
29.89 29.89
23.13
AMID (a 0.5)
28.79
13.85
16,30
27.85
29.46
AMID (a
28.74
25.68
13.01
16,51
26.32
27.83
21.87
AMID (a 0.5)
26.73
24.13
12.06
16.19
22.88
24.82
20.02
AMID (a = 1.0)
22.88
21,31
10.41
14.87
19.35
21,34
17.46
AMID (a -5.0)
29.26
26.64
13.49
16.40
28.65
30.40
23.12
AMiD (a = -3.0)
29.14
26.02
13.62
17.03
28.99
30.59
23.25
AMiD (a = -1.0)
29.32
26.19
13.24
15.83
29.23
29.97
22.89
AMID (a -0.5)
29.12
26.19
13.32
16.40
28.15
29.68
22.75
AMiD (a 0.0)
28.28
25.23
13.01
15.51
27.52
28.66
21.99
AMID (a 0.5)
22.05
19.92
10.76
12.33
25.09
24.78
18.58
AMiD (a 1.0)
21.52
19.17
9.16
13.60
15.53
17.73
15.04
D DISCUSSION OF OPTIMALITY
Theorem 3.4 guarantees the optimality of AMiD, yet experimentally demonstrated extremely poor performance for the reverse KL divergence (pkl(p ((.A) and a in Table 3. We conjecture that it is caused by the conflict between RKL and the support intersection property, which leads to instability. RKL includes the expectation of the assistant distribution E. [-] by definition.
However, when a 1, since supp(rg (a.))) is supp(p) n supp(qe) (see Section 3.2), (o,)) I- is
conducted on unstable and narrow region, and this phenomenon intensifies further in the early stages of optimization. In addition, we experimentally find that the combination DRKL (pllr (a.)) and a = 1 produces highly unstable loss and gradient within a few early steps. In conclusion, while AMiD theoretically guarantees optimality, it might be necessary to employ appropriate divergence and alpha values, taking into account the imperfect optimization.
E THE USE OF LARGE LANGUAGE MODELS (LLMS)
We employed the LLM to polish the paper writing. Specifically, it was used to request grammatical corrections once the author had drafted the text.
24

---

Published as a conference paper at ICLR 2026
Table 15: ROUGE-L scores (f) on five task-agnostic instruction-folloiing datasets. Bold and Underline mean the best and second-best performance each column, except the teacher, respec- tively. All results are based on our own re-implementation. We conduct the evaluation with five random seeds.
Model
Val. (1)
Dolly Eval (f)
Self Inst (f)
Vicuna (1)
Super (f)
UnNT (1)
Avg. (t)
Teacher
27.14 e015
14.55 #082
16.12 #0.31
27.21 +0.25
31.41 #0,6
23.29
GPT-2 XL. (1.5B)
GPT-2 (B.1B)
SFT
25.81
23.54
1042
9.62
10:31
10.56
18.42
10.23
19.33
10.13
17.14
KD
25.25
23.44
10.12
14.93
16.88
18.87
16.85
SeqKD
26.07
24.20
11.12
15.82
19.29
22.74
18.63
ImitKD
23.91
22.02
10.34
15.32
17.34
19.68
16.94
GKD
27.06
24.58 20.3
11.78
14.60
22.84
25.04 10.00
19.77
MiniLLM
24.47
#0.18
12.83 =0.50
16.94
25.58 +0.33
26.38 +:0.7
21.24
AKL
25.62
23.23
1035
11.18
1.021
14.94 1.0.3
19.36
1.0.9
22.41
10.08
18.22
TAID
28.37
25.74
2027
12.91
20.31
17.09
1:018
23.66
10.31
26.82
20005
21.24
DistiLLM (SKL)
27.88
25.50 10.8
12.35
2:039
16.
1:0.2
23.87
10.39
26.16
1:006
20.80
DistiLLM (SRKL)
28.21
25.74
=020
12.13
#0223
16.34
+0.15
25.40
+0.10
26.91
+0.12
21.30
ABKD
28.61
25 49.4991 1024
12.52
10.52
17.36
1.0.55
26.07
10.14
27.36
1.0,00
21.76
AMID (Ours)
29.24
26.44
20.12
13.74
20040
16.76
1:0.44
29.71
::0.08
30.35
::0000
23.40
GPT-2 XL (I.SB)
GPT-2
Medium (0.3B)
SFT
27.96
25.70 10.35
12,60
16.51
24.21
27.51
21.31
KD
26.03
24.27 20:42
10.58
15.59
18.15
20.49
20024
17.82
SeqKD
28.41
26.61
=0.34
13.01
#0.46
16.42
23.44
26.93
+0008
21.28
ImitKD
25.93
24.46
=0.62
12.00 =041
15.56
20.12 +00.34
25.11
+0.16
19.45
GKD
27.90
25.06 1035
12.36
J.042
15.71
1.0.58
23,83
1.0.25
27.14
1.0.9
20.82
MiniLLM
25.80
20:57
14,87
20.35
17.62
1:033
26.78
1:0.26
30.70
2:011
23.15
AKL
27.81
25.57
=0.10
12.06
#9.56
15.98
+:017
22.22
+:0.00
26.17
#0.13
20.40
TAID
29.45
27.01
10227
14,53
10.47
17.58
1.0.00
25.14
10.11
29.79
10.14
22.81
DistiLLM (SKL)
29.65
26.87
10.13
14.11
..0.29
16.85
1.0.5
25.59
1.0.22
28.84
10.00
22.45
DistiLLM (SRKL)
29.72
26.50
20.00
13.79
20.71
17.14
+0.50
26.25
10.11
29.3
20.16
22.60
ABKD
26.93
#0.17
13.69
#6032
17.45
+-0.27
28.15
+0.15
30.94
+0.06
23.43
AMID (Ours)
27.34
10.18
15.26
L0.46
1.0.27
29.04
1...00
33.15
1.0.3
24.50
GPT-2 XL (L.5B)
GPT-2 Large (0.8B)
SFT
28.48
26.17 =0:41
13.78
=1021
16.64
+:0.4
23.76
26.64
+0.12
21.40
KD
28.52
26.27 10026
13.72
1.044
16.43
1.0.5
25.24
10.18
28.94
10.09
22.12
SeqKD)
28.24
26. 2041
13.93
20.56
16.35
+0.20
25.03
+0.27
28.58
20.06
22.01
ImitKD
26.96
23.37 20.40
13.26
1:066
16.00
10033
23.31
10.166
27.59
10.14
20.71
GKD
29.36
26.38 #0.24
14.44
#0.66
17.02
+-0.46
26.64
+00.16
30,99
+0.13
23.09
MiniLLM
26,30 10:35
16,50
10.52
18.14
1.0.69
29.45
100.17
34.40
10.17
24.96
AKI.
27.69
25.45 20.40
13.83
2082
15.85
10.35
25.41
:0.25
28.91
20.05
21.89
TAID
26.85 #0.32
15.07
#0.31
17.02
4-0.45
26.71
+.0.23
31.09
+0.17
23.35
DistiLLM (SKL)
29.69
26.12 =022
15.69
45.75
16.91
=0043
27.23
+0.15
30.73
+0.12
23.34
DistiLLM (SRKL)
27.
10.10
14.61
10.66
16.39
1.0.27
28.44
10.45
31.04
10.06
23.51
ARKD
30.49
27.67
20.34
15.46
2081
17.43
30.74
33.1
10.15
24.88
AMiD (Ours)
31.10
27.86
16.46
16.62
32.64
35.64
25.84
25


Reviews:
[{"review_id": "STSJmYknvw", "reviewer": "Reviewer_Gj8D", "summary": "This paper addresses the capacity gap between teacher and student models and the training instability caused by high-dimensional output in LLM knowledge distillation (KD). It proposes α-mixture assistant distribution and a unified distillation framework AMiD (α-mixture distillation). By systematically expanding the auxiliary distribution and divergence selection, it achieves better performance and training stability.", "questions": "1. Adaptive Selection of α Parameter.  The paper demonstrates that α controls the trade-off between mode-seeking and mode-covering, but it does not provide a systematic method for selecting α in different scenarios (e.g., different task types, model size gaps, or data characteristics). Is there an adaptive strategy to determine α (e.g., curriculum learning-based scheduling, data-driven tuning) instead of manual adjustment? For small student models (e.g., 0.1B) vs. large gaps (e.g., 7B→0.5B), does the optimal α range differ significantly?\n\n2. Extensibility to Larger Model Scales and Complex Tasks. The experiments focus on GPT-2 (up to 1.5B teacher) and OpenLLaMA-7B→3B. Have the authors tested AMiD on larger teacher models (e.g., 10B+ LLMs like LLaMA 3 70B) or smaller student models (e.g., <0.1B)? Additionally, do the performance gains hold for complex tasks such as code generation, mathematical reasoning with multi-step logic, or cross-lingual understanding?\n\n3. Mitigation of Instability in Specific Divergence-α Combinations. The paper notes that the combination of \\(D_{RKL}\\) and α=1 leads to extremely poor performance due to support intersection issues. Have the authors explored mitigation strategies (e.g., adjusting λ, adding regularization terms, or modifying the assistant distribution’s normalization) instead of simply avoiding this combination? Are there general principles to avoid such conflicting pairs?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Generalization and Unification of Fragmented Methods.\n\nProposes a novel α-mixture assistant distribution by extending generalized \\(f_\\alpha\\)-mean to KD, introducing the tunable parameter α. This generalizes prior isolated assistant distributions (m-mixture for α=-1, e-mixture for α=1) into a continuous, flexible family, covering new distributions (e.g., harmonic mean for α=3) not explored in LLM KD before.\n\n\n2. Rigorous Theory and Comprehensive Experiments. Provides formal proofs for key properties (continuity of α-mixture distribution, optimality of AMiD, gradient analysis of f-divergence), establishing a solid mathematical foundation for the framework. Validates AMiD across diverse settings, task-agnostic (5 instruction-following datasets) and task-specific (translation, summarization, reasoning) distillation; multiple model scales (GPT-2 series, OpenLLaMA); various divergences (KL, RKL, AB) and SGO strategies. Results are consistent and statistically robust (5 random seeds), confirming reliability.\n\n3. Addressing Core Limitations and Advancing Practical KD. Directly mitigates the capacity gap between teacher and student models and training instability from near-zero probabilities, two fundamental limitations of prior KD methods. AMiD’s flexibility (compatible with arbitrary divergences and datasets) and robustness (stable performance across λ values) make it adaptable to real-world LLM compression scenarios. The α parameter offers a simple control knob to balance quality and diversity, addressing a longstanding trade-off in generation tasks.", "weaknesses": "1. The lack of a systematic strategy for α parameter tuning limits its practicality. The paper demonstrates that the α parameter can control the balance between \"pattern coverage\" and \"pattern finding\" in the student model, but it fails to provide the basis for α selection and efficient tuning methods. The experiments only demonstrate the effects of fixing α (e.g., α=-5, -3, etc.), without explaining the optimal range and tuning logic of α under different tasks (translation/summarization/inference), different model capacity differences (e.g., 10B→0.1B), and different divergences (D), leading to extensive trial and error for users in practical applications.\n\n2. The paper theoretically proves the optimality of AMiD under \"perfect optimization,\" but two unresolved contradictions exist in practice: a) Conflict between divergence and α: Experiments show that the performance of D_RKL combined with α=1 is extremely poor (Avg. only 3.94 in Table 3), attributed to the narrow intersection of support sets, but no specific criteria are given on how to avoid this conflict;\n\nb) Adaptation of optimizer and hyperparameters: The impact of different optimizers (such as AdamW, Lion) and learning rate scheduling on AMiD is not discussed, while in practice, optimizer selection significantly changes the gradient propagation effect of α-mixture.\n\n3. The ablation experiments do not fully decompose the core contributions of AMiD: The interaction between α and λ is not verified: α is only tested with λ=0.1, without analyzing whether the optimal value of α changes under different λ (such as 0.3/0.7), and the collaborative tuning strategy between the two; the contributions of α-mixture and divergence are not isolated: Is the performance improvement of AMiD due to the auxiliary distribution of α expansion or the flexibility of divergence? Verification needs to be conducted by comparing \"fixed α = ±1 (i.e., baseline auxiliary distribution) + arbitrary divergence\" with \"fixed divergence + α ≠ ±1\".\n\n4. The paper's baseline does not include cutting-edge LLM distillation methods since 2025, resulting in insufficient proof of advancement. For example, it does not compare distillation methods based on contrastive learning or reinforcement learning-based distillation (RL-KD), which may have already surpassed traditional divergence-based baselines in some scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QHjC8pbUKF", "reviewer": "Reviewer_yQkB", "summary": "This paper presents  a unified framework for knowledge distillation with an assistant distribution, where the teacher and student distributions are mixed to bridge the capacity gap between them. Previous distillation mixing strategies (e.g., using arithmetic and geometric mean) become special cases under this framework.", "questions": "See weaknesses about hyper-parameters.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors presents an interesting point of view on existing distillation methods.\n2. The experiments are generally comprehensive, showing consistent improvements.", "weaknesses": "1. The motivation of adjusting alpha is still a bit unclear to me. The authors mention alpha adjusts the mode-seeking and mode-covering properties. However, this can also be addressed in the divergence function, as mentioned in 2.1. I am not too sure about the intuition behind doing this again in the mixing stage.\n\n2. The paper mainly focuses on instruction following, evaluated by rouge. This is slightly concerning, because the approach may be hacking the ROUGE score rather than making the model better at instruction following. The paper showed very limited results on reasoning (GSM8K, omitting ABKD for some reason, and no standard deviations). \n\n3. The paper is not very clear on its hyper-parameters. For example, the author mentions that their approach work with any divergence functions, but it's not clear which one is used for Table 1. Also, the values of alpha/lambda is not shown on the table. The author mentions D_AB being used for Table 2, but how about Table 1. Even in Table 2, it says alpha is not -1/+1, but it doesn't state what alpha is. My main worry is that the authors may have spent lots of efforts on tuning these individually, which gave the approach an unfair advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nwTMcCNuQQ", "reviewer": "Reviewer_RkqC", "summary": "The paper proposes an α-mixture assistant distribution and a corresponding knowledge distillation framework named AMiD for large language models (LLMs). It unifies existing methods (e.g., m-mixture and e-mixture) as special cases within a broader, theoretically grounded family parameterized by α. The framework is supported by theoretical analysis (optimality, gradient behavior, support properties) and extensive experiments showing consistent improvements over prior approaches across various tasks, model scales, divergences, and data generation strategies.", "questions": "1. How should one choose the optimal α in practice? Are there adaptive or task-aware strategies for tuning α during training?  \n2. The paper fixes the divergence (e.g., DAB) while varying α. Have the authors explored joint optimization or tuning of both α and divergence parameters (e.g., α_AB, β_AB)?  \n3. Theorem 3.4 claims optimality for any divergence and α, yet Table 3 shows catastrophic failure for DRKL with α=1. Does this indicate that the “perfect optimization” assumption is too strong, and how should practitioners navigate the theory-practice gap?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Introduces a principled and generalized family of assistant distributions via α-mixture, offering a unified view of prior fragmented methods.  \nProvides solid theoretical grounding, including optimality guarantees, support analysis, continuity, and gradient-based interpretation of mode-covering vs. mode-seeking behavior.  \nComprehensive experiments validate the effectiveness of AMiD across instruction-following, task-specific distillation, different student sizes, divergences, and SGO strategies.  \nThe new design variable α offers practical control over the quality-diversity trade-off, independent of the interpolation weight λ.", "weaknesses": "The writing is not very clear and significantly hampers readability.** Despite strong technical content, the exposition is often dense and poorly structured, especially in Section 3. Key concepts (α-mixture, f-mean, α-divergence) are introduced rapidly without sufficient intuition or gradual buildup, making it difficult for readers to follow.  \nFigures (e.g., Figure 1 and 2) lack detailed captions and fail to fully clarify the geometric impact of α on interpolation paths.  \nLimited discussion on why α ≠ ±1 performs better in practice beyond empirical results; the gap between theoretical optimality and practical instability (e.g., DRKL with α=1) is noted but not deeply analyzed.  \nSome notation is ambiguous or inconsistently used (e.g., r vs. r̃, θ vs. θ′), adding unnecessary cognitive load.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)