Distillation of Large Language Models via Concrete Score Matching
Authors: Yeongmin Kim
Donghyeok Shin
Mina Kang
Byeonghu Na
Il-chul Moon
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Knowledge Distillation
Score Matching
Primary area: generative models
TL;DR: This paper performs distillation between two autoregressive language models via score matching, thereby defining the loss directly at the logit level.

Abstract: Large language models (LLMs) deliver remarkable performance but are costly to deploy, motivating knowledge distillation (KD) for efficient inference. Existing KD objectives typically match student and teacher probabilities via softmax, which blurs valuable logit information. While direct logit distillation (DLD) mitigates softmax smoothing, it fails to account for logit shift invariance, thereby restricting the solution space. We propose Concrete Score Distillation (CSD), a discrete score-matching objective that overcomes both softmax-induced smoothing and restrictions on the optimal solution set. We resolve the training instability and quadratic complexity of discrete score-matching in autoregressive LLMs, and the resulting CSD objective aligns relative logit differences across all vocabulary pairs between student and teacher with flexible weighting. We provide both mode-seeking and mode-covering instances within our framework and evaluate CSD on task-agnostic instruction-following, task-specific, and general chat capability distillation using GPT-2-1.5B, OpenLLaMA-7B, and Gemma-7B-IT, Qwen2.5-7B-IT, and Gemma2-9B-IT teachers. Experiments show that CSD consistently surpasses recent KD objectives, achieves favorable fidelity–diversity trade-offs, and yields complementary gains when combined with on-policy techniques, demonstrating its scalability and effectiveness for LLM distillation. Code: https://github.com/aailab-kaist/CSD.

Published as a conference paper at ICLR 2026
DISTILLATION OF LARGE LANGUAGE MODELS VIA CONCRETE SCORE MATCHING
Yeongmin Kim', Donghyeok Shin', Mina Kang', Byeonghu Na', II-Chul Moon l Korea Advanced Institute of Science and Technology (KAIST), 'summary.ai [alsdudrlal] t1sehdgurO, xasong13, 3, beeenyeeoghi na, icmoon] (arroon)kkkkaaaa @kaist ac. kr
ABSTRACT
Large language models (LLMs) deliver remarkable performance but are costly to deploy, motivating knowledge distillation (KD) for efficient inference. Exist- ing KD objectives typically match student and teacher probabilities via softmax, which blurs valuable logit information. While direct logit distillation (DLD) mit- igates softmax smoothing, it fails to account for logit shift invariance, thereby restricting the solution space. We propose Concrete Score Distillation (CSD), discrete score-matching objective that overcomes both softmax-induced smoothing and restrictions on the optimal solution set. We resolve the training instability and quadratic complexity of discrete score-matching in autoregressive LLMs, and the resulting CSD objective aligns relative logit differences across all vo- cabulary pairs between student and teacher with flexible weighting. We pro- vide both mode-seeking and mode-covering instances within our framework and evaluate CSD on task-agnostic cestruction-oolooviii task-specific, and general chat capability distillation using GPT-2-1.5B, OpenLLaMA-7B. Gemma-7B-IT, Qwen2.5-7B-IT, and Gemma2-9B-IT teachers. Experiments show that CSD con- sistently surpasses recent KD objectives, achieves favorable trade-offs, demonstrating its scalability and effectiveness for LLM distillation. Code: https: /tt:::/gitttte com/ aailab-kaa wab.kaist/sss...
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable generative capabilities across a wide range of tasks (Achiam et al., 2023; Dubey et al., 2024; Liu et al., 2024; Comanici et al., 2025). Such progress has been primarily driven by the vast amount of training data and the unprecedented scale of model parameters (Kaplan et al., 2020). However, when deploying such LLMs in real-world applications, the recurring inference cost becomes prohibitively expensive. Consequently, research into reducing the parameter size of LLMs while preserving performance has become particularly crucial for enabling efficient inference. In this context, knowledge distillation (KD) (Hinton et al., 2015) has emerged promising approach for LLMss, it allows smaller student model to inherit the capabilities of a large teacher model, thereby enabling more efficient inference.
The common paradigm KD for LLMS to align the per-token probability distributions of the student with those of the teacher. Kullback-Leibler (KL) divergence was initially the most widely adopted objective, and the search for more effective probability matching losses has since become a central topic of research. Alternative objectives have been proposed within the framework of f-divergence (Wen et al., 2023; Gu al., 2024: Agarwal et al., 2024), as well as its smoothed variants (Ko et al., 2024; Shing et al., 2025; Ko et al., 2025). However, existing distillation losses primarily targeted the estimated probabilities obtained through the softmax transformation, instead of directly utilizing the raw neural network outputs (logits) from either the teacher the student. As illustrated in Figure Ib, even when the teacher's logit values differ substantially, their corresponding probability values can be nearly indistigguishabl.. Such smoothing hinders the student from faithfully capturing the teacher's 's knowledge, a challenge further exacerbated in modern LLMs with large vocabularies, where most tokens assigned near-zero probabilities (See Figure Ia).
In traditional KD, direct logit distillation (DLD) (Ba & Caruana, 2014; Urban et al., 2017) has been proposed alternative strategy, with advantages in generalization capability and in removing the

---

Published as a conference paper at ICLR 2026
PrO < c.01)) ==
Logit vectors H
Probabilities
Parameter space
Frequentyy (lgg scall)
f+l
Pr(y")
pr(y")
Only
-1
.01
.01
CESD (Ours)
o. 0.0023%
Softmax
-4
-9
.00
.00
OLDD (Raseline)
.99
.99
10-00
11
10"
10
VoDia. (prllgu)
Σ (gu(y) pr(v))
Vofvlal
9txp
[9|.0|) +]. Bbc (01/o)
Probababilty value (log scalc)
VEV
frH
(a) Probability value statistics.
(b) Softmax hides teacher's s knowledge.
(c) Optimal solution sets.
Figure 1: Motivation for logit-level distillation and limitations of prior work. (a) Statistics per- token probabilities for every vocabulary for 16 input-output sequences from the teacher model (GPT-2-1. 53). The probabilities are highly sparse, with only 0.0023% being greater than 0.01. (b) Despite large differences in logits (e.g.. [-1, -4 4] vs. [1, -9,9,,00 6)), softmax yields nearly identical probabilities and gradients. (c) Prior direct logit distillation restricts the solution set.
softmax smoothing (Kim et al., 2021). However, such approaches have not been thoroughly explored in the context of LLMs. This paper identifies a key drawback of DLD: its restriction on the optimal solution set as described in Figure 1c. Considering the softmax activation in inference, it is sufficient for the teacher's ' and student's 's logits to agree up to an additive constant, but the previous solutions of DLD fail accommodate such acceptable slack constant, a.k.a. logit shift invariance. Such restriction on the solution set may hinder the discovery of optimal solutions in distillation, particularly when the teacher and student models have large capacity gap, as often the case with LLMs. Therefore, the goal of this paper is to establish a design space of distillation losses that overcome both the softmax- softmax-induced induced smoothing of teacher knowledge and the restriction on the solution set.
This paper adopts the idea from energy-based models (Song & Kingma, 2021), which design objectives that avoid the constraint of probabilistic models (sum-to-one) by using the score-matching objective (Hyvärinen & Dayan, 2005). We propose Concrete Score Distillation (CSD), discrete form of the score-mater matehing objective (Meng al., 2022) adapted for autoregressive LLM distillation. We address training instability and computational overhead arising when applying the score-matching objective to LLMs, and provide theoretical guarantees of optimality, showing that its solution set is broader than that of DLD from both theoretical and empirical perspectives. The resulting objective reduces to matching the relative logit differences across all pairs of vocabulary items between the student and teacher, while allowing flexible weighting across all vocabulary pairs in linear time with respect to vocabulary size. Furthermore, we present instances within our framework that exhibit both mode-seeking and mode-covering properties.
In our experiments, we conducted task-agnostic Instruction-folloiinn distillation, task-specific distillations (summarization, mathematics, and translation), and general chat capability distillation using GPT-2 (Radford et al., 2019), OpenLLaMA (Geng & Liu, 2023), Gemma (Team et al., 2024a), Qwen2.5 (Team al., 2024b), and Gemma2 (Team al., 2024a) backbones. The proposed CSD consistently outperformed recent probxbilitymmatching objectives as well as direct logit distillation. By appropriately choosing weighting functions, we further demonstrated that our method resides on the frontier of the diversity-fidllity trade-off. Finally, we observed complementary performance gains when integrating our loss with on-policy techniques.
PRELIMINARIES
2.1 KNOWLEDGE DISTILLATION OF LARGE LANGUAGE MODELS
We consider autoregressive large language models (LLMs), consisting of teacher pr and student qe with 0 E O where the student is a smaller and more efficient model. Given an input context c, the student generates
output sequence y (31, ./..... M/) with probability (o(yic) ILe qe (ve (, yet), where L denotes the sequence length, and the teacher's probability is defined analogously. Each token Yt is drawn from the fixed vocabulary set V == {u1, 02,..... }. As in prior works (Lin et al., 2020; Ko et al., 2024), we assume the teacher and student share the same vocabulary set. To compute the token probability go(yelc, c, cx), LLM typically adopts parametric function fo V el 1 - R which maps the input (c, (ct) to a logit vector fo(c, Y yct) RPP, The
2

---

Published as a conference paper at ICLR 2026
logit corresponding to token Y/1 is denoted by fo(c. y<()u.. For brevity of notation, the input arguments of the function fo will be omitted hereafter. Let fr be the parametric function of the teacher. Accordingly, the probability of each token is calculated through the softmax transformation:
exp(folye) go go(vll,, EVED exp(Ja(z)))
exp(fr(ue))
Pr (T(ll,, ΣnEV exp(sr(a))
(I)
The goal of knowledge distillation for LLMS is to align the student's per-token probability distribution with that of the teacher, that the student inherits the teacher's capabilities.
Problem definition:
We assume access to input-output sequence pairs yc ) ~ DD obtained either from fixed dataset or from samples generated by the student or teacher (Lin al., 2020; Ko et al., 2024). For each selected instance (c, y), distillation is performed by selecting specific discrepancy metric and minimizing the discrepancy between the per-token probability distributions with respect to 0:
1 L
(eyy))
Σ
D (pT ye), llae (lc,
(2)
Prior work and motivation: In previous studies, D is most commonly chosen as the KL diver- gence (Hinton et al., 2015), which is formulated as follows (the input of the probability omitted):
yx(l Σ pr (r( log go(ye V.EV
DKL (pT (lo))
(3)
However, DRI. focuses on the teacher's probabilities and is constrained by the softmax. As shown in Figure 1b, although the teacher carries rich knowledge across all vocabulary items at the logit level, much of it is lost after softmax, and the teacher provides nearly identical gradient signals to most minor tokens. Accordingly, in classical KD studies (Ba & Caruana, 2014; Urban et al., 2017), direct logit distillation (DLD) has been widely adopted as a logit-level mean squared error (MSE) loss:
COLD (0: pr, w)
Σ w(y)) (folvel - fr(va)) NEEV
2
where w(-) is strictly positive weighting function'. Kim al. (2021) showed that CDLD provides better generalization and representation capability by taking minority indices into account. Since faithfully distilling logit information crucial for large- vrgabbocayyl LLMs, we investigated the use of DLD for LLM distillation. However, we found that its optimal solution does not permit logit constant invariance, thereby severely restricting the solution set. This observation motivated us to develop a logit-level distillation loss that does not restrict the optimal solution.
2.2 SCORE MATCHING FOR A DISCRETE RANDOM VARIABLE
mooee-mtchingg (SM) (Hyvärinen & Dayan, 2005) was originally proposed in maaee mod- els (Song& & Kingma, 2021) with continuous variables Rd. An energy function Eg Bd IR maps x to a scalar. The corresponding probability and the score-matching objective are given by:
(x(x) exp(- (x)))
LsM (0; Paatas w) Ex l log qx(x) - Vx log Pditt (x) 112
(5)
Zo
where Pdata is the data distribution, Zy = fg exp( Fx )dx is the partition function, and w(()) is weighting function. The term Vx log qo (x) = - V (x) is known as the Stein score, which uniquely identifies the probability distribution without requiring the computation of Zg. Lsm facili- tates the design of losses without considering the normalization constraint of probabilistic models. The probability computation qu here follows, analogously, the form of the LLM probabilities in Eq. (1). The difference is that an LLM outputs energy values fe over all finite states once, whereas an EBM handles continuous variables, so that each imput to Eg yields only a single scalar output.
Inspired by how EBMs design losses beyond the normalized structure of a probabilistic model through sctre-matching, we extend this idea to construct logit-level distillation losses for LLMss However, because the Stein score is defined through derivatives, it cannot be directly applied to discrete random
Throughout this paper, we assume each weighting function sums one the vocabulary for simplicity.
3

---

Published as a conference paper at ICLR 2026
variables. Meng al. (2022) proposed generalized score function, applicable both continuous and discrete variables, named the concrete score: (a( :-
Similar to the Stein score, the concrete score characterizes local changes at the current state, but replaces them with probability ratios between all other point masses. This term is also uniquely identifiable with the underlying distribution. The corresponding concrete score-matching objective is then defined as:
Qa( de
Pdata (z)) qo qo(y) Pdaca (3) )
LesM(0; Pdata w)
ΣΣ WEV
w(y,x)
(6)
where Pdata is the data distribution defined over a discrete state, and w(., is positive weighting function. Previous work language models (Lou al., 2024) typically adopted this loss by directly parameterizing the concrete score (also known as discrete diffusion models) to mimic the data distribution. In contrast, we take this concept as a starting point to design logit-level distillation losses for autoregressive-type language models, which are more dominant in real-world applications.
3 METHOD
This section introduces the proposed Concrete Score Distillation (CSD) objective for knowledge dis- tillation (KD) in autoregressive large language models (LLMs). Section 3.1 discusses the challenges of directly applying feSM to LLMs, so we propose modified objective with theoretical guarantees of optimality and compare the objective with CDLD. Section 3.2 presents an efficient analytic gradient computation for CSD, analyzes its gradient structure, and compares it with that of DKL.
3.1 CONCRETE SCORE DISTILLATION FOR LARGE LANGUAGE MODELS
Tackling training instability: We observe that optimizing the student model to by minimizing Cosm(0;P)) PT, w) leads to training instability, as the likelihood ratio ga(ar) tor approaches zero. In the discrete diffusion model (Lou et al., 2024), single vocabulary item is fed into the neural network 8p. which directly outputs the ratios over the other vocabulary items, thereby avoiding instability. In contrast, autoregressive LLMs compute probabilities for each vocabulary item separately and then take their ratios, making this issue specific to autoregressive LLMs.
ge(ue] can diverge as the denomina-
Training instability is a well-known issue in likelihood ratio estimation (Rhodes et al., 2020). Follow- ing Higuchi & Suzuki (2025), we address it by applying monotonically increasing function to the concrete scores. In particular, adopt the logarithm, which yields the following objective:
(log go(x)c, log pr(cle.y<c)
LCSD (0:PT, =
Σ
w(ye x)
(7)
2
qo (n (y
pr(selc,y
WEEV ZEV
1
Σ
Σ
w (M,, x) (Fo[x) - folu]] fr(z] frlu:))
(8)
2
V.EV MEV
The choice of the logarithm function provides two benefits: (1) it yields MSE loss between logits (i.e., neural network outputs), ensuring stability by avoiding the likelihood ratio computation; and (2)
it naturally leads to the logit-level loss design, which aligns with our motivation.
Logit distillation with vocabuocyyyl relationships: Unlike LOLD, which directly matches student and teacher logits for the same vocabulary item, LesD aligns the logit residuals across different vocabulary items between the student and the teacher. This allows the student not only to be compared against the teacher but also to perform relative comparisons among its own vocabulary items. In contrast to DKL. where softmax normalization implicitly adjusts each vocabulary item relative to all others, our loss explicitly controls the pairwise relationships between student vocabulary items 3/ and x through the weighting function w(y,, xr). Figure illustrates how logit vector fo y ct) TR (e.g.. [1,
felvd
lote
Student concrete score
Teacher concrete score
Figure 2: Schematic for LeSD (Eq, (8)).
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: Gradient computation of Concrete Score Distillation
Input: Student fo. teacher fr. prompt c, prefix << function w(.,)) w1 (1())u(())
Compute the student logit folval = v.cc.ycc)luu) Vy, V.
with no no_grad:
Compute the teacher logit fr[v:] fr, y <xll) Vyc E..
Compute the weighted average logits:
Fw:: = - ( (30) x fe deluch. fg = =E =Everluz(v)) x feluc].detach)
.T - ev (We) fr[ue]., f7 Eylevlusl( (w2) ) STlvel]
Compute the weighted normalized logits:
Fwi [yll solve] - ,
Sg [uel - Solvol -0 Vy E V.
Fur
[e]] fr(uk] fr f [al] fr frllv] --p-- f1 Vye EV.
fa [ee] f [v]]] + (2()) for [uo] fr [il] Vya EV
10
Ugrad (yt) wi (Br)
VoLeso (0:pr, w) = Eyev [Wgred( Vofolyel 12 return VoLcSD (0; pr, w)
-3)) produces a concrete score and how is matched with the teacher's concrete score. The following theorems provide the theoretical guarantee of the proposed objective function.
Proposition 1. (Consistency) Given context с and prefix <ct. assume model capacity |9| + co. For
any w(, .) > define the set of optimal parameters as Ocsp arg mingge LCSD (0; pT, w). Then, :Pr.
for any 0 O'sD. we have LESD : w) = 0, and the following holds for all E V:
qo- (y: c, y.) (y
Please refer Section A.1 for the proof. Proposition shows that consistency holds when matching the log-transformed concrete scores of the student and teacher, and guarantees that our objective leads the student to converge to the target teacher.
Theorem 2.
(Solution Superset) Assume model capacity |+] - co, let the set of optimal parameters 0cSD = arg milagee Lesp (6: pr, w) and ODLD = arg minggee LOLD (6: pr, w), then following holds:
CSD 2 ODDD
Please see Section A.2 for the proof. Theorem 2 implies that all solutions obtainable by LOLD can also be recovered by Lesp- This is because LesD is invariant to constant shifts in logits; for example, when [vlv.] fr [ye] + for all y E V, the probabilities are identical and the LCSD is zero, whereas the LDLD is not optimal. This advantage could be pronounced under limited model capacity, where the larger solution set of LCSD enables more faithful approximation of the teacher's knowledge.
3.2 GRADIENT COMPUTATION AND ANALYSIS
The remaining challenge of the proposed objective LCSD in Eq. (8) lies in its computational cost of O(VV)). Unlike DKL and DDLD. DesD requires a double summation over the vocabulary set V. This formulation is infeasible to implement in standard computational environments due to memory constraints. Nevertheless, we show that the gradient of this objective can be computed in linear time:
Theorem 3. (Efficient Gradient Computation) Assume w of Lcsp (r; pr, w) with respect to 0 could be computed in ((VV) as:
w(ye, ye, .I) wi (w)),, then the gradient
VoLeSp (0; pr,, w))
Σ w(y+)T (Folva) Fr(we) Vololun). MEE
where w(y.) = (w, (v+), wy (v.))) Polvel = (For jyll Fu [va]] T Er(val = (F [vel, [wel, Fioa [val) (val)
T
with far [yv] felvv E (n) [2]]. f [yo] fr(v]] - Bu(x)) [x]] are normalized logits.
The proof is provided in Section A.3. For the actual training time and memory usage, please refer to Table 9 in Section D) These results follow from factorizing the independent variables. Algorithm 1
5

---

Published as a conference paper at ICLR 2026
further details the gradient computation of Eq. (9) step by step, with each step requiring only linear time over the vocabulary. An alternative approach is to use Monte Carlo estimation as described in Algorithm 2 of Section C. Instead of taking weighted sum over all possible states of y with w1. one can draw a single sample of yt according to probability wi and compute the loss in expectation. The Monte Carlo estimation, unlike the analytic gradient form, does not require assuming independence between the two variables of w, allowing it to model the joint weighting function space directly. However, defining a joint weighting function over two discrete vocabulary spaces is generally difficult. We found that independent weighting functions capture various behaviors in Section 4.4. In these cases, Monte Carlo estimation increases the variance within batched samples, which slightly slows the convergence compared to the analytic computation (see Figure 5c).
The gradient of LCSD in Eq. (9) has a structure similar to that of DXL. For intuitive understanding, let us consider the case where the weighting function of CSD is the uniform distribution U. Then, the gradient of each loss becomes:
Gradient analysis:
exp(So (
exp(Sr Evev exp(fr(x])
DK. (pT (l00
E B+EV
Vo.oluel,
ΣzEV exp(fola) ())
normaliized student logil
normalizest tencher logit
VoLesD (0:pr,U)- Σ MEE
2 [
(Jolun) Exev Solad) [V]
[rluv]
Enev ST(U] D
VoJolun)
normalized student logit
normalized teacher logit
In gradient descent, both losses decrease the student's logit folve] where the student's normalized logits are large, and increase So [u]]] where the teacher's normalized logits large. The only difference lies in how the logit coefficients are normalized over the vocabulary set: DKL. inherits the softmax form, which, as noted in Figure Ib, poses major problem for transferring the teacher's knowledge. In contrast, our LCSD uses centering normalization, allowing the student to directly capture the teacher's logit information. Moving beyond the uniform weighting case study, the formulation in Eq. (9) further provides a design space for logit normalization through (W1, Wz), where wi controls the weighting of vocabulary tokens during gradient updates and w2 governs coefficient normalization, with their roles applied again in reverse order (w2, w1).
4 EXPERIMENTS
This section comprehensively validates the effectiveness of the proposed Concrete Score Distillation (CSD) across various experimental setups. Section 4.1 shows results on task-agnostic instruction- following distillation, comparing CSD with alternative loss functions and assessing its performance when combined with on-policy methods. Section 4.2 further examines task-specific settings, including math, summarization, and translation, to evaluate the applicability of CSD. Section 4.3 evaluates scalability by examining whether general conversational abilities can also be distilled. Finally, Section 4.4 establishes the contribution of each component in CSD through ablation studies.
4.1 TASK-AGNOSTIC INSTRUCTION
DISTILLATION
We follow the training setup of DistiLLM (Ko et al., 2024). For the distillation dataset D, we use databricks cerriicssdooll---i dolly-15M (Conover et al., 2023). We first fine-tune the teacher on this dataset and then distill it into student models. We use the detached student probability as the default choice for both wi and W2. and we apply it similarly to the weights DLD. Please refer to Section C.I for further details on backbone, training configuration, baseline, and the evaluation protocol.
Loss-level comparison: To purely analyze the effect of the distillation loss itself, this comparison excludes the use of pretraining losses, initialization with an SFT-tuned student, and any on-policy techniques. Table shows that the proposed CSD objective outperforms the other nine objectives, ranking first on three of the five benchmarks, second one, and achieving the highest average score. SKL (Ko et al., 2024) and AB (Wang et al., 2025) exhibit slightly lower performance than previously reported, likely due to their reliance on pretraining losses on-policy techniques. Figure 3a shows the fidelity-die diversity trade-off based on ROUGE-L and Self-BLEU scores. Traditionally, KL favors diversity, whereas RKL. favors mode-seeking. Within this trade-off, SKI., SRKI., TV, and AB

---

Published as a conference paper at ICLR 2026
Table
Comparison of loss functions for distilling GPT-2-1 2-T-2-1.35 into 1-2T-2-0.22 Every result is from our implementation with the same teacher, purely using the distillation objective. ROUGE-L scores were averaged over five random seeds; best scores are boldfaced, second best underlined.
Loss
Dolly Eval
Self Insirectt
Vicuna Eval
Super -N
UnNI
Avg. (1)
Teacher
27.00 0.39
1.307
16.31 0.32
26.46 0:41
31.10 0.06
22.99
KL
23.52
10.02 0.58
14.57
0.32
16.76
18.55 0.13
16.68
RKL. (Gu 2024)
24.26 0.11
11.19 0.17
15.80 0.26
20.17
0.15
22.99 = 0.14
18.88
Sym-KI.
23.29 0.30
10.24. 0:31
15.25 0.43
17.46 0.11
20.60 = 0:08
17.37
Jeffrey
23,00 1: 0.38
10.82 0.44
15,00 0.50
18.19
0.11
20.07
0.11
17.42
TV (Wen etal, 2023)
23.88
11.03
13.13
24.58
25.24
19.97
GJS (0.9) (Agarwal et at., 2024)
24.10 0.24
11.40 0.29
16.02 0.57
20.28
0.13
22.55 - 0.12
18.87
SKL (0.1)
2024)
24.17 0.34
11.21
15.29 0.34
22.65 0.14
24.69
0.11
19.60
SRKL (0.1) (Ko al,, 2024)
24.53 0:31
12.19
12.19:00.0
15.63 0.22
23.37
0.27
24.28
0.18
20.00
AB (0.2, 0.7) (Wang al., 2025)
24.20::0.1
11.82. 0:29
15.87 + 0.36
21.44
25.59
19.78
CSD (Ours)
24.94
12.06
15.78
24.60
25.88
20.65
21
22
ROUGE-L (5 Benchmark Avg.)
ROUGE4 (5 Benchmark-AAgg.
20
18
18
14
CSD (us)
Jeftrey
18
CSD (ts)
12
CSD (98)
RKL
KL
SKL
CSD (us)
SKL
RKL
SHKL
10
(ts)
SRKL
Sym-KL
AB
-0.6
-0.5
-0.4
-0.3
-022
0.5
0.6
0.7
0.8
0.9
Negative Self Belf-BLEU (5 Benchmark-AAgg.
Distinct-2 (5 Benchmark-Avgy
(a) Fidelity vs. Diversity trade-off.
(b) Decoding temperature ablation [0.2, 1.8).
Figure 3: An in-depth analysis of the distributional behavior of different loss functions.
achieve higher ROUGE-L scores than RKL, but at the cost reduced diversity, reflecting a stronger emphasis fidelity. Diversity, however, remains an important aspect of user experience in instruction- following, and it becomes a valuable metric as it enhances performance when combined with best-of-N sampling. The proposed CSD provides an additional lever control the diveritydiversity trade-off. By default, using the detached student probabilities (S, S) yields the highest fidelity. Replacing one side with uniform (U, S) or with the teacher (T.S) gradually increases diversity. This is likely because the (S, 9) makes the model focus only on regions where the student already assigns a high likelihood, limiting its exploratory ability. The trade-off offered by CSD envelopes those of existing losses, and we expect that even better operating points may exist within the design space of w and W2. Figure 3b presents an ablation on temperature, which enables easy adjustment of the trade-off during inference. Even within reasonable range of decoding temperature, CSD achieves better trade-off points than other losses. In particular, CSD (U, S) demonstrates a well-balanced exchange between diversity and fidelity through temperature adjustment.
Orthogonal improvement with recent on-policy advances: Table reports the performance of recent distillation baselines augmented with the CSD loss, demonstrating its orthogonal applicability. We applied the CSD (S, S) and DLD (S) loss to lmitKD (Lin et al., 2020), GKD (Agarwal et al., 2024), and DistiLLM (Ko et al., 2024). DLD-mean refers the mean-centered DLD variant, as described in Section A.4. The primary distinction among these methods, apart from their losses, lies in the choice of dataset D: ImitKD uses purely reddertteeeertted on on-policy data, GKD combines fixed data with student outputs. and DistiLLM adaptively selects between them based on validation loss. As a result, the average ROUGE-L score improved for both GPT-2-0. 1B and GPT-2-0- 3B students in all settings, compared to both the baseline and the corresponding DLD versions. The best result on each benchmark was also achieved by our method, with particularly strong performance under pure on-policy settings. We also evaluated using GPT-4 as the judge in Figure 4, where our best model was judged superior to other baselines. There may exist CSD variants other than (S. S) that perform better for specific D, but leave this exploration to future work. Finally, applying our best setting to a larger OLen also outperformed baselines, demonstrating the scalability of CSD with respect to model size. We provide comparisons with more baselines in
Table 7 of Section D.
7

---

Published as a conference paper at ICLR 2026
Table 2: Instruction-followin performance of CSD with on-policy techniques for various backbones. D denotes the distillation dataset. ROUGE-L scores are averaged over five random seeds, with the best score for each student highlighted in bold. CSD and DLD use the student probability weighting.
Method
Loss
D
Dally Eval
Self Instruct
Vicuna Eval
Super
UnNT
Avg. (1)
Teacher (GPT-2-1, 5B)
27.00.: 0.19
14.07 0.37
16.31 =: 0.32
26.46 041
31.10.:
22.99
Teacher (OpcnLLaXA-7B)
27.60 0.4
18.17 0.80
17.85 :: 0.48
31.05 :: 0.31
32.40 0.28
25.41
GPT-2-1.5H GPT-2-0.
GKD (Agarwal 2024))
GIS
Mix 22.48
10.08
0.67
15.61
13.88
16.59
15.73
DistiL.M (Ko et al. al. 2024)
SKL
Ada
25.28 0.8
12.04
0.49
16.66
= 0.34
2
24.32
20.09
ImitKD (Lin al, 2020)
KL
On
21.79
10.25
0.37
14.65
= 0.82
17.35 0.12
1..4
16.69
GKD DLD
DLD
Mix
2
12.51
0.62
16.59
0.28 20.87:
22.63
19.58
DistiLLM DLD
DLD
Ada
24.231.021
11.86
0.51
17.69
0.18
19.60
22.77:1..22
19.23
ImitKD + DLD
DLD
On
24.69
12.10
0:40
16.77=
21.58
23.93
19.81
GKD + Ours
CSD Mix
25.50
0.34
12,03
0:65
16.65 = DA4
21.39
0.14
23.48
0.03
19.81
DistiL.LM + Ours
CSD Ada
25.34
0.23
11.93
0.36
16.99
0.29
22.96
0.34
24.72
0.09
20.39
ImitKD + Ours
CSD
On
25.70
0.23
12.40
0.48
17.18
0.52
22.91
0.46
25.47
0.17
20.73
GPT-2-1, SB GPT-2-0.38
GKD (Agarwal at, 2024)
GJS
Mix
25. 15:1
041
11.22
0.33
16.45
= 0.48
17.35
0.29
22.25
0.05
18.48
DistiLLLM et al, 2024)
SRKL
Ada
26.92 023
13.75
0.29
16.90
0.25 26.12
0.27
29165
22.67
ImitKD eLinettll 2020)
KL
On
23.61
12.37
0.26
15.53
0.27 20.20
0.20
24.42+
19.23
GKD DLD
DLD
Mix
26.06:
13.51
16.94
23.91
27.33+
21.55
DistiLLM + DLD
DLD
Ada
25.43
0.37
12.64
0.40
16.91 :: 0.61
22.69 :: 024
25.60
0.10
20.65
ImitKD + DLD
DLD
On
25.82
0.37
13.64
0.33
17.55
0.16
25.51
0211
29.07
0.08
22.32
GKD Ours
CSD Mix
27.11
0.42
13.71
0:45
16.98
0.29
25.49
0.35
30.16
0.13
22.69
DistiLLM Ours
CSD Ada
26.77
13.96
0.62
17.05
0.34
26.29
0.08
29.56
22.72
ImitKD + Ours
CSD
On
27.14
14.85
0.66
16.88
26.28
30.43
23.12
OpenLLaMA-7BB OpenI MA3B
TAID (Shing et.l. 2025)
1KL
Ada
26.53
0.23
17.73
0.69
18.14::.
0.39
31.93
1:
0.3
31.55
0.12
25.18
DistiLLM at.
SKL
Ada
28.63
20.20
19.15
0.32
35.31
34.74
27.61
DistiLLM eto. 2024)
SRKT. Ada
28.831
20.76
0:37
19.37
36.82
35.76
28.31
ImitKD + DLD
DLD
On
29.07
20.07
0.60
20.05
0.37
36.30
35.71
28.24
ImitKD + DLD-mean
DLD
On
28. 036
19.91
0.50
19.58
= 0.55
35.85
0.50
35.49
27.79
ImitKD + Ours
CSD
On
29.63
0.40
21.81
0.47
20.37
0.51
36.49
0.13
36.86
0.10
29.03
Table
Task-specific distillation
from
0.56
CKD
Gemma
IT to Gemma-22 -
GPT4 feecbback Doll Evel
SeaKD
Summarization Translation
GSM8K
ImitKD
0.50
Loss
ROUGE-L
COMET
Accuracy
Teacher
37.09
79.23
60.27
0.46
KL
35.02
73.96
24.03
IS
35.60
74.05
23.73
TV
27.49
73.73
0.00
0.40
Jeffrey
35.29
74.02
23.28
25.86
59.65
0.00
0.35
GP 12
0.122033
SRKI.
RKL
26.68
0.00
45.02
73.10
0.00
0.00
Figure 4: GPT-4 feedback performance, showing the proportion of responses judged correct relative to the golden answers. The teacher's 's score is 0.61.
DLD (5)
0.00
21.52
0.00
DLD-max (T)
32.54
65.28
17.74
CSD (7
35.67
74.14
25.78
4.2
DISTILLATION
We evaluate the effectiveness of CSD across dialogue summarization, low-resource translation, and arithmetic reasoning tasks. Distillation was conducted on 1,000 samples from the DialogSum (Chen et al., 2021), Flores-200 (Costa-Jussà et al., 2022), and GSM8K (Cobbe et al., 2021) datasets, following the experimental setup of Xu et al. (2025) (Please refer to Section C.2 for further details). Table 3 compares performance across the three tasks against baseline loss functions. Under identical experimental conditions, the proposed CSD objective achieved the best results on all tasks. For the arithmetic reasoning task, observed several cases in which certain losses yielded zero accuracy. A
8

---

Published as a conference paper at ICLR 2026
Table 4: Benchmarking result for general chat capability, The best score is highlighted in bold.
Qwen2.5-7B-TT +
Qwen2.5-1.5B-IT
Gemma2-9B-IT
GemmaZ-2B-IT
Benchmark
MT-Bench (0-10)
AlpacaEval (WR)
MT-Beecch (0-10)
AlpacaEval (WR)
Judge
GPT4 GPT4-Turbo
GPT4-Turbo
GPT4
GPT4-Turbo
GPT4-Turbo
Teacher
8,59
7.52
88.69
8.91
7.66
94.60
DPKD (Li eti, 2024)
1.04
1.09
0.32
6,30
4.89
71.18
DistiLLM-2 eK,, 2025)
7.28
5.75
70.42
7.81
6.45
89.91
DLD (T)
7.25
5,56
69.80
5.85
4.45
31.24
DLD (S)
7.28
5.74
67.67
7.58
6.53
89.84
CSD (T.
7.42
5.90
70.42
7.85
6.55
89.92
CSD (S
7.69
5.95
69.64
7.77
6.43
90.05
case study in Tables 11 to 15 of Section D shows that these models often produce excessively long chains of thought without arriving at final answer, indicating a failure learn proper formatting; furthermore, much of the reasoning itself is incorrect. As illustrated in Figure 3a, the RKL, TV, SKL, and SRKL losses exhibit mode-seeking tendencies. which we conjecture may have caused collapses into suboptimal modes under these limited data distillation settings. Similarly, CSD (S. S), which also shows mode-seeking behavior, performed poorly as 21.09, 63.78, and 0.00 on summarization, translation, and GSM8K, respectively. CSD achieved stable performance in this case by using the (T.S) weighting. DLD likewise performed poorly under student weighting and became only marginally stable when teacher weighting was applied. A full ablation is provided in Table 10 of Section D for DLD variants. Across all cases, DLD remained significantly weaker than CSD, likely due to its restricted solution space being more detrimental under limited-data distillation.
4.3 GENERAL CHAT CAPABILITY DISTILLATION
To evaluate general chat performance, we conducted distillation experiments using the latest instruction-tuned models, @wee2.55-Instruct (Team al., 2024b) and Gemma2-Instruet (Team et al., 2024a). We followed DistiLLM-2 (Ko et al., 2025) and performed distillation using the 50K subset of the UltraChat dataset (Ding al., 2023) (Please refer Section C.3 for further details). Ta- ble 4 shows that CSD outperformed both DistiLLM-2 and DLD on MT-Bench (Zheng et al., 2023) and AlaacaEval (Li al., 2023) win rate (against text text-davi nei n -003), demonstrating superior performance. These results further demonstrate the scalability of CSD.
4.4 ABLATION STUDIES
CSD vs DLD: Table presents comprehensive ablation studies on the weighting function choices in CSD and compares them with direct logit distillation. DLD accommodates only single weighting function. When comparing DLD with T, U, and S against CSD with (T, T), (U, U), and (S, S), respectively, method consistently achieved higher average scores. As shown in Theorem hypothesize that the broader solution space positively contributed to this improvement. We also compared CSD against other shift-aware DLD variants such as DL.D-min and DIDDmma, well as ranking-main matthing variants including DLD-std (Sun et al., 2024) and DLD-range. However, none of these methods outperformed the naive DLD baseline. Figure 5a shows the effect of temperature scaling on the weighting function of DLD (s). Under the same temperature scaling, CSD consistently outperformed DLD. Figure in Section shows that DLD restricts solutions those with a residual constant of zero; CSD adapts residual constants per token, providing evidence that it explores broader solution set.
Design choice of CSD: CSD provides more flexible loss design space through two weighting functions. While (S, S) provides high-fidelit generation, (U, S) and (T, .S) have its own benefits. As illustrated in Figure 3a, replacing (S,S) with (U, or (T, S) reduces ROUGE-L but increases diversity, highlighting that CSD can adapt to tasks requiring either mode-covering or mode-seeking properties. As shown in Figure 13 in Section D. the (U, S) weighting substantially reduces the gradient concentration on a small subset of vocabulary tokens that induced by softmax. This allows all vocabulary items to be learned more evenly, which in turn produces the pattern in Figure 9c where the logits residuals are tightly centered around their offsets. When minority-vocabulit logits are well learned, performance improves noticeably in situations where their contribution

---

Published as a conference paper at ICLR 2026
Table 5: Ablation on the logit-level loss design space using 1-T-2200.15 student. U, and S denote teacher, uniform, and detached student probabilities. ROUGE-L scores are averaged over five seeds; best scores are in bold. Please refer to Section A.4 for DLD variants details.
Loss
W1 (-)
(-2(-)
Dolly Eval
Self-Instruct
Vicuna Eval
Super-NI
UnNT
Avg. (t)
0.09 0.02
0.07 0.02
0.18 0.3
0.07 0.01
0.06
0.00
0.09
DLD
11.25 0.30
5.55
0.03
9.10 027
9.02 0.14
8.24
0.07
8.63
S
24.22 0.24
12.01 + 0.40
15.42 0.11
25,44 0.19
24.88
0.19
20.39
T
1.14
0.93 0.05
1.65 0.13
0.85 0.01
0.87 0.01
1.09
DLD-min
U
7.16
4.53 0.20
7.49 0.19
7.20 0.04
5,97 0.05
6.47
23.89
11.11 0.17
15,43
23.78
25.87
20.02
T
0.39
0.32 0.00
0.73
0.22 0.01
0.21
0.37
DLD-max
6.47 0.06
4.81 0.07
6.67: 021
6.80 0.06
5.17
5.98
9.65 :: 0.28
5.81 0.11
8.66:: 0.43
11.73 0.24
11.62.: 0.12
9.49
5.98 0.18
4.80 0.14
7.98 0.15
4.93 0.06
4.85 0.05
5.71
DLD-std
21.45 :: 0.29
10,55 :: 0.59
15,90 0.17
18,85 0.26
20,82 0.09
17.51
S
9.74 0.22
5.67 0.11
12.29 0.27
7.07 0.11
6.97 0.07
8.35
0.85 0.03
0.73 0.04
1.44 0.06
0.53 0.02
0.51 0.00
0.81
DLD-range
U
10.84
7.49 0.14
12.82 0.30
9.97: 0.04
7.89
9.80
S
8.90
4.74+
7.70
8.70
8.90
7.79
T
T
6.82
4
9.16
4.53
4.83
5.91
U
17.21 1:
8.08 0.39
14.27
13.19 0.27
14.07
13.37
CSD (Ours)
24.94
0.29
12.06 0.46
15.78 0.49
24.60 0.31
25.88
0.13
20.65
U
S
24.15 1:
12.25 1: 0:47
15.25
22.55
0.09
25.19
0.12
19.88
T
S
22.77
0.25
10.62 0.32
14.06 0.25
18.81
0.40
21.71
17.59
Expected Calturaton Ereer
0.30
Vartatton ROLIOE (can)) %
30
0.29
CSD (56) (Aro)to)
CSD (8)
0.15
CSD (5S)
CSD (16)
CSD (2) (NC)
OLD
CSD
ULD
C50 (us)
DLD
CSU
0.00
0.
05
20
25
30
35
10
16
12
16
Woight Tomperature
Epoch
Epoch
(a) Weight scaling
(b) Calibration
(c) Training curve
Figure Ablation studies for logit-level distillation loss design space.
becomes more significant, such as under high-temperature sampling. This explains why (U. performs exceptionally well in the high-diversity region of Figure 3b. Figure 5b measures probability calibration performance. While (S,S) provides nigh-fidelity generation, it becomes overconfident. Therefore, in scenarios where probability calibration is preferred, we recommend using the (T, S) weighting. Better calibration potentially improve training stability in small-data settings where the optimization landscape more difficult, as in Section 4.2. Figure 5c compares the method to resolve ((D())) computational cost: 1) using analytic gradient computation from Theorem and 2) Monte Carlo sampling. In both cases, the performance is far superior to KL, but the analytic CSD shows slightly faster training and better convergence. Thus, we recommend using the analytic gradient whenever possible. However, the Monte Carlo variant allows extensions such as joint weighting functions using Lp loss instead of L2, making a viable option for more complex modeling.
CONCLUSION
We introduced Concrete Score Distillation (CSD), novel design space for distillation losses in large language models. CSD simultaneously addresses the challenges of softmax-induced smoothing and restrictions on the optimal solution set, which prior methods have failed to resolve together. Within this framework, we presented instances of both mode-covering and mode-secking, and demonstrated scalability by consistently surpassing prior work across diverse tasks and model backbones up to 7B parameters. We anticipate that even better instances can be discovered within the proposed design space, particularly by refining wi and W2 and adapting them to the type of data (fixed or on-policy). This points to promising directions for future exploration of improved instances.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the ITTP (Institute of Information & Communications Technology Planning & Evaluation)-TTRC (Information Technology Research Center) grant funded by the Korea government (Ministry of Science and ICT) research was supported by AI Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous Data(IITP) funded by the Ministry of Science and ICT(RS-2022. 11220077) (50 %)
00026-6--2024-00--5
(50 %) This
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXxx:2333008744 2023.
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. The Twelfth International Conference on Learning Representations, 2024. URL https::tp//.o http////ppeeeveeeeeeeeeeeeeeeeee
net/ forum? idd aadd
Jimmy Ba and Rich Caruana. Do deep really need to be deep? Advances in neural information processing systems, 27, 2014.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A real-life scenario dialogue sum- marization dataset. Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACI--IJCNLP 2021, pp. 5062-5074, Online, August 2021. A Association for Computational Linguistics. doi: 10.18653/v1/200 htt.6////112211...nnnsaaa..a..a URL httph:tt///cmm aalantholoogyy org/ 22211222111 findings acl. 449/.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https:
/tt.m/ss.oggg/aaaa.////// hrtp://g///o/bo//22////// -0 -3-00---- 30 vicuna/.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2rXiv:2110.14100 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. 2025.
arXiv preprint
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matci Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instructiontuned Ilm. 2023.
Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan. Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint Xrxiv:2207.044 04672, 2022.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 0029-3051, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023. URL https: //gi tiu......................... com/ www/open////eoooocooooooop/eoe.e open research/open.. aama.
11

---

Published as a conference paper at ICLR 2026
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:://Sky/1en007.ggii. 1o/
ToreeabbeetCrrouuu 2019.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
URL https:////// / wpwppeeeeeeewweeeee/pe net forum? forrmmdddShh@peee idd
Rei Higuchi and Taiji Suzuki. Direct density ratio optimization: A statistically consistent approach to aligning large language models. In Forty-second International Conference on Machine Learning, 2025. URL https: wwtp///weeweew.eee///p/e. net/
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 22330 2015.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 14409-14428, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/1/2023..2 act.20000.....1-o...66...66 URL wttp//gg/000 aa2.23..cc-oonn... 806/.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (XXi::0001.0836111 2020.
Taehyeon Kim, Jaehoon Oh, Nak Yil Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback- leibler divergence and mean squared error loss in knowledge distillation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2021.
Yeongmin Kim, Sotiris Anagnostidis, Yuming Du, Edgar Schönfeld, Jonas Kohler, Markos Geor- gopoulos, Albert Pumarola, Ali Thabet, and Artsiom Sanakoyeu. Autoregressive distillation of diffusion transformers. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15745-15756, 2025a.
Yeongmin Kim, HeeSun Bae, Byeonghu Na, and Il chul Moon. Preference optimization by estimating the ratio of the data distribution. In The Thirty-ninn nitet Annual Conference on Neural Information Processing Systems, 2025b. URL https: http://ppnnneeeeeeeeeeeeeeeeeeee wormmttt///eeeeeeeeeeeeee wintrrrmmmiiidiie f59
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18653/11DDI6-111 1139. URL https::////....... httpss////aaaaahhllaaao
Btg/DD6-1139)
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Youn Yun. DistiLLM: Towards streamlined distillation for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 24872-24895 PMLR, 21-27 Jul 2024. URL https: httppp::/ppproeeee /pppceeeioe...... mlr. http///25/5222222 4c. html.
Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang. Ilya Zharkov, and Se- Ye-Young Yun. DistiLLLM-2: A contrastive approach boosts the distillation of LLMS.. In Forty-second International Conference Machine Learning, 2025. URL https: /ttp://oppeeevveeeeee ew net/
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. A objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, 2016.
12

---

Published as a conference paper at ICLR 2026
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction mnstructionfoolloinng models, 2023.
Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, and Furu Wei. Direct preference knowledge distillation for large language models. arXiv preprini arXiv:2406. /9774, 2024.
Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6121-6133, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 11813661/22200000 wttp:////200...mmmmmmmi.....mm. emnlp-ma. me-maiin 494/.
URL www.tttss..oo http://ccaaatthhooooo wwwethooogy..o.. ogg.oo//..000...
Chin- Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL httpt:ttp///.c...t. /tte://aclla acc www.lnnnnooooyy..oooooooo orw.://-0rg/004000000
10t. 100411334.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, pp. 32819-32848. PMLR, 2024.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data, Advances in Neural Information Processing Systems, 35:34532- 34545, 2022.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 37:124198-12423, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https: /a/cccaaaa aclanthologyy wwt.oo/////// 2022.0022..2 wmt - 10. 52/.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905-4916, 2020.
Donghyeok Shin, Yeongmin Kim, Suhyeon Jo, Byeonghu Na, and Il chul Moon. AMid: Knowledge distillation for LLMs with ShalphaS-mixture assistant distribution. In The Fourteenth International Conference on Learning Representations, 2026. URL ptttps://openrevaee J.d=WPPJEEc) dM.
net/ forum?
Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba. TAID: Temporally adaptive interpolated distillation for efficient knowledge transfer in language models. In The Thirteenth International Conference on Learning Representations, 2025, URL htttps::/opeereeeee net/forum? inneruummiimeee wdd
Yang Song and Diederik P Kingma. How to train your maseg models. arXiv preprint 20ti::2001.022883 2021.
13

---

Published as a conference paper at ICLR 2026
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15731-15740, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXivv2403.08299, 2024a.
Qwen Team et al. Qwen2 technical report. arXiv preprint arXiv:2407 10671, 2(3), 2024b.
Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang. Abdelrah- man Mohamed, Matthai Philipose, Matt Richardson, and Rich Caruana. Do deep convolutional nets really need to be deep and convolutional? In International Conference on Learning Represen- tations, 2017. URL https: www.revieneeeeeeeeeeeeeeeeroeewe net/ hwt.//eee///////et
Chaoqi Wang, Yibo Jiang, Chenghao Yang. Han Liu, and Yuxin Chen. Beyond reverse kl: Generaliz- ing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2024.
Guanghui Wang. Zhiyong Yang. Zitai Wang, Shi Wang, Qianqian Xu, and Qingming Huang. ABKD: Pursuing proper allocation of the probability mass in knowledge distillation via a- 3-divergence. In Forty-second International Conference on Machine Learning, 2025. URL https::/cppe wppprreiieepeeeeeeeee.re nnt/ fowummid-wtt5viiaa
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Saperr-Nturallnsttuetooon Generalization via declarative instructions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 5085-5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 11186553/1/2022.cmm me220222.mmlppmmm..... URL https: http//aaaaantttoooooooooe org/ 2022. www.emmww.mmmmeeeemmmmmmmmmeeeee 340/.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). pp. 13484- 13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18.18653/v1/ 2023.ac1- a023.acl-long.755 754. URL https: / aa// aclanthologyy ott//////2322 http://2222....1oooogago 754/
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Anna Rogers, Jordan Goaber,, and Naoaki Okazaki (eds.), )) Pro- ceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10817-10834, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 2023. acl-loncl 605/-
URL https: http://aaaaaatalocooo org/
Taigiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang. Zhe Zhao, and Ngai Wong. Rethinking Kullback-eibllr divergence in knowledge distillation for large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 5737-5755, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https: //ta:///cct //cc///cla////// antho wonthhllgyyyooo..ooo..o org 02 col ing-ma ma n. 383/.
Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen- Then-YYLaee Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher- student gap through interleaved sampling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: httass///pppeviieeeeeeee net forum? 2 d=RgghwWR2ERR.
14

---

Published as a conference paper at ICLR 2026
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging Ilm-as-at aa--ds---jude with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The s/st international ACM SIGIR conference on research development in information retrieval, pp. 1097-1100, 2018.
15

---

Published as a conference paper at ICLR 2026
A PROOFS AND DERIVATIONS
A.1 PROOF OF PROPOSITION 1
Proposition 1. (Consistency) Given context с and prefix <tt, assume model capacity | + co. For
any w(- :) > 0, define the set of optimal parameters as Øcsn arg mingee Lesp (0; pr, w). Then,
for any 8 E CesD we have LESD (6"; Pr. w) = 0, and the following holds for all E V:
qo (y: c, pr (,
Proof. We have the following objective:
w(ye 2²) log go(x)c,ye)) log yr(
CCSD (0, w)
Σ
Σ
(10)
40 c, pr(vilc,
VEV
= 2
w(ye, x)) (folz) - folual - frlal + frlvil)? MeEV
Σ
Σ
(11)
Since the objective is weighted sum of squares with strictly positive weights w(::)) > the loss attains its minimum if and only if each squared term vanishes, i.e.
fo [x]] fo [ue] ST(z] frl, Vy,, EV.
(12)
Then, the probability of student satisfying the following:
exp(fe- [vr])
exp(fo- (Bc))
go- (ye (ullc,yc)) Erey exp (x]))
(13)
Exev exp( [yo] + ST( Jr (v.)))
exp(/r(vi))
(14)
Evevexple exp(ff(<x))
A.2 PROOF OF THEOREM 2
Theorem (Solution Superset) Assume model capacity |0] -> oo, let the set of optimal parameters
OcsD arg mingge LcsD (0; pr, w) and OLDD arg mingge COLD (0; pr, w), then following holds:
Øcsn 2 On
Proof.
We have the following objective for direct logit distillation (DLD):
COLD (0; PT, w) 2
Σ w(y) (Jolve) - fr(v)))²
(15)
B-EV
Since the loss is expressed strictly positive weighted of squares, it achieves its minimum value only when all squared terms are individually zero, i.c..
fegu [vu] = fr(vd,
Vy E.
(16)
Unlike DLD, the optimality condition of our loss is more relaxed. Specifically, is sufficient for d* satisfy the condition in Eq. (12), i.e.,
fe- [vo] frin+
Vy CER.
(17)
At C = 0, our objective recovers the solution set of DLD; for arbitrary choice of C, yields strictly larger optimal solution set. This arises from the fact that the softmax mapping used to express probabilities is invariant under additive constants, whereas DLD explicitly constrains this constant to coincide with that of the teacher, which consequently reduces the solution set.
16

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF THEOREM 3
Theorem 3. (Efficient Gradient Computation) Assume we woyy, x) wi (1))w2(1)), then the gradient
of LcsD (0; pr, w) with respect to 0 could be computed in ([V)) as:
VoLcsD (1)
Σ w(v))² MIE
[Folvo] Fr(vel) Vefelval.
(9)
f lell f0 [ve]] T frlll = (Fi² (fr" [vcl, fr lvol) ( T
where w(v:) = (wI(3)), 2z(2)) fol
20 frlve] Bm(x)]fr[r]]] are normalized logits.
with fø [9c] folval wn [z]] frr [9e]
Proof. We have the following objective:
wi (ye (w) (1os (,(x),y)))) log (r/c, y<c)
LeSD (0; w) = Σ
Σ
(18)
go(ue ye.)
(1t yct)
W:EV ZEV
wi (1)) (Jol)) Solve] fr(a]] sT(U
(19)
WVV 2EV
And its gradient is given by:
VoLCSD (0:p1, w)
Σ
Σ wi (f)) folvel fr|a] frle))) Vo(felx] folve))
BEV
ΣΣ
wi (1) (fo - frlz))
E Σ
wi (1()))22() (folz) fo (x)) vo(folul)
VEEV VEV
ME XEV
(2)) (- [un] + Sr(1)) Vo([o/z))
Σ
Σ
wi (1))w2(x)) (- (-folm)] + sT(U)) Vo (o(lln))
BEEVEE
V.EV ZEV
1
Σ water) Σ Wz (x) (fo[x] fr(z)) vo(fo(x)) = Σ w2(ve) (Fo[ve] Sr(u:)) (o(o)))))
MEE
ZEV
V.EV
4
[
W1 (26) (felyi) fr(ee) vo(felul)
Σ
w1 (ve) (felye) fo (bc)) ve(fe(yd)
M.EV
NEV
Wyy (2) (fo[z] - frle)) x Σ
2
REV
WeV
[fo|a] Exy(x)
fr[æ]] Σ wi (1.) (f(fe())))
WEV
3
wi (1t) (Jolun] sr(ue))
Σ
w2 w2 Vo
VLEV
ZEV
Emy(o)) [folve] frlvell x
W (2) (f(o)())
xEV
Em [folz] -fr[x]] Σ
W ( (o(f(l)))
17

---

Published as a conference paper at ICLR 2026
+ 2 Σ wi (Be) (fulve) - frival - (wz(s))lolat - (oflull))
WIEV
= Σ
wi (yr))
f [wo] ( Vo (o(lolu)))
1+ 3
Σ
W2
(fe(ve) - fr (ur()) - fr(xxl) Vo(folu))
V.EV
= Σ
W2 (t))
foo [ye] - Fir ( (o(felull)
BVEV
2 +
+
Σ
((v))²
(elwe)-fr(un))) Frluvl) vo(Solue))
yr VVEV
A.4 THE VARIANTS OF DIRECT LOGIT DISTILLATION
We define the DLD variants used for our comparisons in Table 2, Table 3, and Table
FDD (0:Pr, w) Σ w(ye) VIEV
(ffolor) min fa [x]) (fr [ve]
min fr(()))
x
CDLD (0: w) = Σ w(yb)
(ffolve) max falæ)) - (frlua) max fr(()))
NIEV
felvil 15ं E folx]]
fr fee
[VT Σ fr(z]
LLDD
(0:Pr, (b
1
Σ
m((r)
zEV
NEV
2
2
NLEV
Σ fo(x]] (b)
Σ folco
Σ fr((]] 1/ Σ frijd
M
ZEV
N'EV
DEV
#'EV
LDLD (0:PT,W)
1
Σ
w(vr)
2 ( [ye] - min. fo (x]1))
-1)
(
2 (frlyel min fr fr((())
- -))
2
maxx So [xr] - minn So [3r]
maxx ST(z]] min, S [x]
J.EN
Σ frice) ))
C (0: w)
1
Σ
w(v+)
foluel
1
Σ
folz)] - frluell
1
2
V
M
VIEV
FEV
2
Caymean (0: w)
1
w(yr
fo(yv]
Σ TEV
w(()
fo [x]
frivd - Σ (w(())((o))
2
E.E
TEV
Here, CNLDE is recovered by CSD as special case shown by the following remark.
Remark 4, Let CSD objective as LOSD (0:pr, pr, w) with x(),, wi (1)))x((1). If (f) wz(-).
VoLESD pr, w) = V mor (0; pr, w1).
Proof. We use Eyer (u [uvc] Eyeev wi (1t) (Folue) ExEN un (z)So(az)) = O.
V Vy- (0:Pr, Σ wi (vx) (For [wr] - fr lua)) (Vefelerl - Σ Wi Volool NED
xEV
Σ wi (vr) fo [vo] fr [ui] Vofolvel
Σ
Wa (ve ( (for (Fo0 [vil fun (ool)
Σ Lee
wi (x) (ofo
v.eV
2u
VoLCSD (0:Prwu)
18

---

Published as a conference paper at ICLR 2026
A.5 WEIGHTING FOR DIVERGENCE BASED LOSS
Unlike the L2 loss, a divergence-based loss does not guarantee convergence to the target distribution when weighting is applied. Here, we examine the effect of applying weighting the KL divergence. Define (3)( pr(m) where ZuT = (vre(pr()) x w(y+)) is partition Zur
function. Then we have:
log pr(
DiL. KL. (prll90) == Σ
go(ve)
NEE
w(y))p((U)) log w(()))
Σ WEV
w()))P(()) log
(o(v))
w w())pr(Bi)
log (Vb))rr + log Zwo log w(rr)
ZwT
ZuT
Zwr x qu()))
VvE
log (T(()) + log ZwT - log w(ye)) (o(())
(Ma)
(V() 2o (0(v))
Zuo Σ (1n) log
+ C
VEEV
Zur DEL (g)) +C,
where C' is constant with respect to 0. Thus, in this case, the student qo does not converge to the teacher distribution but instead converges to py. meaning that the target distribution is altered by the weighting w. In contrast, the proposed CSD theoretically guarantees convergence to the target distribution for any choice w proved by Proposition
B RELATED WORKS
The choice of discrepancy metric between teacher and student probability dis- tributions is central to knowledge distillation for large language models (LLMs). Prior work has predominantly employed either forward KL divergence (Hinton al., 2015) or reverse KL diver- gence (Gu et al., 2024). These divergences, however, exhibit distinct biases: forward KL is inherently mode-covering, while reverse KL mode- seeking. Consequently, optimization under either mea- sure imposes an unavoidable trade-off between fidelity and diversity. To address this limitation, recent studies have explored alternative measures, including (generalized) Jensen-Shannon diver- gence (Wen et al., 2023; Agarwal et al., 2024), adaptive KL divergence (Wu et al., 2025), and a-8 divergence (Wang et al., 2025). Complementarily, Ko et al. (2024) introduced skew KL and skew reverse KL divergences to improve optimization stability. Beyond the KL family, total variation distance has also been investigated (Wen et al., 2023). Amid (Shin et al., 2026) extends the distil- generalized assistant distribution within
Distillation loss:
lation objective by matching the student distribution the f-divergence framework. Broadly, existing approaches extend in two directions: (i) instantiating different generating functions within the f-divergence family, (ii) constructing hybrid objectives that combine multiple divergences. In contrast, we propose a novel logit-level distillation framework grounded in concrete-score matching (Meng et al., 2022), which departs from the f-divergence family and offers both extensibility and originality. Furthermore, we introduce loss design space with multiple instances, including instances that envelope the diversity- fidelity trade-off exhibited by previous methods.
Concurrently, a complementary line of work has examined dataset composition to mitigate the distribution mismatch between training and inference. Several studies have explored on-policy strategies, either using only student-generated outputs (Lin et al., 2020) or combining them with fixed dataset (Agarwal et al., 2024) and teacher-generane 2025). To reduce the computational overhead of on- on-policy training, Ko et al. (2024) proposed an adaptive off- off-policy method with a replay buffer. By contrast, our contribution focuses on developing a novel discrepancy metric, which is orthogonal to these dataset composition strategics and can be seamlessly integrated with them as shown in Table 2.
Distillation dataset:
outputs (Gu et al., 2024; Xu et al.,
19

---

Published as a conference paper at ICLR 2026
Autoregressive model meets score function: BPO (Kim al., 2025b) generalizes the preference optimization objective for LLMs by interpreting it as sequence-leved concrete score matching. Unlike CSD, which formulates concrete score matching at the per-token level over the entire vocabulary to match a student model to a teacher model, BPO defines concrete scores at the sequence level between preferred and dispreferred responses and matches them to preference data. To enable tractable learning from such data, BPO further extends the objective within a Bregman divergence framework. ARD (Kim et al., 2025a), in contrast, extends the distillation of score-based diffusion models to an autoregressive model. Specifically, it employs an autoregressive student to learn, through regression, the integration of the ODE defined by the score function.
C EXPERIMENTAL DETAILS
C.I TASK-AGNOSTIC INSTRUCTION FOLLOWING DISTILLATION IN SECTION 4.1.
Experimental setup: We follow the training setup of Di (Ko al., 2024). For the distillation dataset D, we use databri i cks pats-doll dollyy11 (Conover et al., 2023), containing about 14,000 samples for training, with 500 held out for validation and 500 for evaluation. For comparison with the baseline, we optionally add a pretraining loss using the pretraining dataset OpenWebText (Gokaslan Cohen, 2019) in some cases of Table 2. We first fine-tune the GPT-2-1, 5B (Radford et al., 2019) teacher on the dataset, and then distill it into GPT-2-0- 1B and GPT 00.0-20000000 3B students. Similarly, we distill OpenLLaMA- 7B (Geng& & Liu, 2023) into OpenLLaMA- 3B. We determined the learning rate and batch size by referring to the search ranges used in prior studies (Gu et al., 2024; Ko et al., 2024). We use the detached student probability as the default choice for both w1 and Wy, and analyze alternative choices through ablation studies.
All experiments were conducted primarily on four RTX 3090 GPUs. We searched learning rates in [5e-4, Ic-4, 5e-5] and batch sizes in [8, 16, 32). Each configuration was trained for 20 epochs, saving a checkpoint at every epoch, and evaluated using the checkpoint with the highest validation ROUGE-L score. We used the same five evaluation seeds [10, 20, 30, 40, 50] in prior work to compute the mean and standard deviation of the evaluation metric. The baselines in Table 2 were run with the official code settings of prior work (Ko al., 2024), with additional tuning for the batch size. In the OpenLLaMA experiments, all baselines and ours were standardized to batch size of 8, the maximum supported in our environment. Baselines used the learning rates from their official code, while we fixed the learning rate to le-4 (commonly effective for GPT-2) with CSD, without further tuning. For ablation studies Table 5 and Figure 5, we used the same configuration: learning rate le-4 and batch size 8. For GPT-4 feedback in Figure 4, we use the following templates following prior work (Zheng et al., 2023; Ko et al., 2024) as shown below. We computed the ratio between the model answer and the golden answer for each of the 500 samples from Dolly Eval, and reported the average over all samples. We provide the reference implementation for CSD in Code 1.
Baselines: Since main focus is on the loss function, compared our method with existing objectives using the same teacher checkpoint. The basclines include KL, reverse KL (RKL) (Gu et al., 2024), symmetric KI. (the mean of KI. and RKL., Jeffrey's divergence, Total Variation (Wen et al., 2023), Generalized Jensen-Shannon (GJS) (Agarwal et al., 2024) with smoothing parameter 0.9, Skewed KL (SKL) (Ko al., 2024), Skewed reverse KL (SRKL) (Ko et al., 2024) with smoothing parameter 0.1, and a-3 divergence (AB) (Wang et al., 2025) with parameters (0.2, 0.7). We followed the hyperparameter choices reported in each paper and the implementation of DistiLLM. For KL, we performed full-range hyperparameter search, as in our method. For losses not specified in prior work, we adopted the same settings as for KL.
benchmarks: 1) the test set of Dolly, 2) Self- (Wang et al., 2023), 3) Vicuna Eval (Chiang et al., 2023), 4) Super- Natural Instructions (Super-NI) (Wang et al., 2022), and 5) Unnatural Instructions (UnNI) (Honovich et al., 2023). ROUGE-L (Lin, 2004), which measures similarity to the golden answer, was used as the primary metric. We additionally employed Self-BLEU (Zhu et al., 2018) and Distinet-N (Li et al., 2016) as diversity metrics. Furthermore, GPT-4 feedback (Zheng et al., 2023) was used as a proxy for human judgment. Checkpoints were saved at each epoch, with evaluation performed on the one achieving the best validation ROUGE-L. The decoding temperature was set to by default, and following prior work, reduced to 0.7 for GPT--jugge evaluation.
Evaluation
metries and setups: We evaluated on five
20

---

Published as a conference paper at ICLR 2026
C.2 TASK-SPECIFIC DISTILLATION SECTION 4.2.
Experimental setup:
We verify the effectiveness of CSD across diverse tasks, including dialogue summarization, low-resource translation, and arithmetic reasoning. Distillation was conducted on DialogSum (Chen et al., 2021), Flores-200 (Costa-Jussà et al., 2022), and GSM8K (Cobbe et al., 2021) datasets, following the experimental setup of Xu al. (2025) with fixed dataset. We used Gemma-7B IT (Team et al., 2024a), fine-tuned with SFT as the teacher and Gemma -aaiaa--B-a 2B-IT as the student. We compared with the baselines using the same teacher, changing only the loss function.
For teacher SFT, we trained summarization and arithmetic reasoning for 3 epochs and translation for 10 epochs, using the full datasets. Model evaluation was performed every 16 steps, and the checkpoint with the lowest validation loss was selected. The batch size was fixed to 128 for all tasks, with the learning rate set to le-5. For each task distillation, we distilled both the baselines and our method from the same teacher checkpoint with fixed learning rate of le-5 and batch size of 8, using about 1,000 samples. We trained for epochs on summarization and arithmetic reasoning, and 10 epochs on translation. For the baselines, checkpoints were saved every 25 steps, and the one with the lowest validation loss was used for evaluation. For CSD, since the loss itself cannot be directly computed and training relies its gradient, validation loss unavailable; thus, evaluated using the final checkpoint. For all tasks, we set wi and W2 using the teacher's and student's probabilities. For evaluation, used task-specific metrics: COMET (Rei al., 2022) for translation, ROUGE-L (Lin, 2004) for summarization, and answer accuracy for arithmetic reasoning, all evaluated on each task's test dataset.
C.3 GENERAL CHAT CAPABILITY DISTILLATION IN SECTION 4.3.
Experimental setup: We closely followed the official code of DistiLLM-2 (Ko et al., 2025) for our distillation setup. For 50k UltraChat prompts, we generated samples from both the student and the teacher. We then applied the CSD and DLD matching losses to each pair of generated samples. Although methods such as DPKD (Li et al., 2024) and DistiI.L define different losses depending which model produced the sample, CSD could also benefit from adopting such strategy, suggesting room for further improvement. For evaluation, we followed the SimPO (Meng al., 2024) protocol. We used the same learning rate and batch size DistiLLM-2.
21

---

Published as a conference paper at ICLR 2026
Algorithm 2:
Monte Carlo estimation to compute LOSD in Eq. (8)
Input:
Student fe, teacher fr. prompt c, prefix y<r, function w(...) = wi w1())2((11))
Compute the student logit fo [o fo yul Vy, V.
Compute the teacher logit [r( = (r, [r), Vyu E.V.
Sample Yr according to w1 (-).
Lese (0;PT, w) -vev
w2 (2( (folz) felvo] [r((] [r(v1)
return VoLESD (0; pr, w)
GPT-4 feedback template
[System] Please act as an impartial judge and evaluate the quality of the response provided by an Al assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing short explanation. Be objective as possible. After providing your explanation, please rate the response on a scale to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
[Question] {question)
[The Start of Assistant's Answer] {answer]
[The End of Assistant's Answer]
import torch
import wrrcch.nn.functiinn
CSD_ (sg_llosstttuudt__ooooooo teacher_logite mode)
student_proioe F.coftmax (suudeed_itt din-
tescher_proas F.softmax (tecchrr dim--1)
1f mode "S5";
fatudent_logits. teacher_ torch.sum (tst//stttptpbbeeeee (studens_iogita teacher_logiten dfma-1, keepdimeTruel) .detach student_ crndent_rrbbeddetoot
student_logits
elif mode
"rs":
losa1 (student_logits teacher Beacher_logite torch. ttacceer probs aatudent_lloitee teacher_logital dim--1, (eepdiiree)))) 1 (detach atudent_ p_tol_rrobbaeeee detach
stsdent_logits
los82
Eetudent_logite teacher Loagter_logits torch, sum tudueent......m proba (stucent_logtta Lescher_iogite) dim- L.keepdin=Tcue) detach () teacher_probs student_logite (Lossl loss2) / 2
less
disti
torch.aue (losa, dim=-1) summat fon vooab
return lesa
Code I: CSD loss function implementation
22

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL RESULTS
This section presents additional experimental results. Figure shows the logit and probability statistics of the GPT-2-1 5B teacher, corresponding to Figure 1. Figure 7 illustrates further fidelity-diveriit trade-offs using Distinct-N metrics, corresponding to Figure 3a. Figure presents validation ROUGE- L. scores during training, corresponding Table 1. CSD not only converges to higher point but also achieves faster performance gains in the early stages. Table provides comparisons with additional baselines corresponding
Table 2, and Table 8 compares CSD with the MSE probability- matching objectíve under different weighting schemes. Finally.Tabless 1 to 15 present case studies of model generations for math questions.
3.0
Logit /+[
pr(-)( 0.01)
10a
pr(-)( =0000000)))
2.5
Frequency (log scale)
100
Freqeuncy
2.0
Only 0.0023%
10"
1.5
20a
1.0
10a
0.5
10a
0.0
-10
30
10
10
10"
Logit value
Probababilty value (log scale)
(a) Logit statistics
(b) Probability statistics
Figure 6: Comparison between teacher's logit and probability statisties. While the logits span wide
range from 20 to and convey rich information, the probabilities are mostly concentrated near zero.
ROUGE-L Benchmark Avg.) 1
Benchmark-ANy Avg-))
19
19
(es)
TV
CSO
(us)
Juffrey
Juffee
CSD (ts)
GJS
ROUGE-L
CSO (ts)
GUS
SKL
RXL
SRKLL
17
RML
SRWL
Sym-LL
Sym-L
AB
Distinct-1 (5 Benchmark-Avg..)
Distinct-2 (5 Benchmark-Avg.) t
Figure 7: Fidelity vs. Diversity trade-off with more metrics.
Validation ROUGELL (Dolly)
27
23
21
CSD (Ours)
Jaffee
RXL
SKL
Sym-KL
SRKL
17
10
15
20
Epoch
Figure 8: Validation ROUGE-L scores over training epochs.
23

---

Published as a conference paper at ICLR 2026
2500
2500
1/
96
1/
36
V2
21
M/
36
36
Frequency
1500
Frequency
1500
V1
2NO
ys
1000
1000
500
500
50
0
50
200
150
200
250
300
50
0
50
200
150
200
250
300
frH1
frH
faH
(a) DLD (S) (Avg. ROUGE-L: 20.39)
(b) CSD (S, (Avg. ROUGE-L: 20.65)
2500
2500
Vi
36
2000
1/
2000
1/
37
Frequency
1500
Mi
39
Frequency
1500
1/
30
Us
1000
1000
90
250
95
150
frHl f6H
frH-1 faH-
(c) CSD (U, (Avg. ROUGE-L; 19.88)
(d) CSD (T, (Avg. ROUGE-L: 17.59)
Figure 9: Solution set restriction of direct logit distillation (DLD) and the flexible selection of logit
residual constants in Concrete Score Distillation (CSD). CSD finds broader solution space.
D. ANALYSIS ON THE LOGIT OFFSETS.
Figure shows the logit offsets between the teacher and the student for 10 consecutive tokens within sentence. This demonstrates that DLD converges only solutions with zero residual constants, whereas CSD learns token-dependent residual constants. In other words, CSD explores a wide solution space during the training. Figure 11 also shows how the offset for the same token changes across training epochs. We observe that an appropriate offset for each token is determined early in training. after which the model consistently refines its solution around that offset.
Figure 9c shows that CSD (U, S) is more tightly centered. As analyzed in Figure 13, vocabulary items are learned more uniformly, causing the logits to cluster around the token-wise offset. This indicates that minority vocabulary items are also well learned, which helps explain why the method performs exceptionally well under the high-temperature sampling setting of Figure 3b, where the contribution of minority vocabulary becomes more significant.
Figure 10 presents the averaged KI., probability MSE, ECE errors across training epochs, and Table 6 shows the per- per-instance values corresponding Figure 9. Since the probabilities of specific vocabularies (those with high probabilities the student or teacher) are more important than the full vocabulary in these metries, CSD (T, S) performs well because it leamss with probability weighting from the teacher and student.
In contrast, the generative performance performance only needs good quality in the regions favored by the student, which is often negatively correlated with probability calibration (Achiam al., 2023; Wang al., 2024).
highest with CSD (S, This is because generative
24

---

Published as a conference paper at ICLR 2026
Pr (vs)
DRL. (PTllgo)==
Σ pr( log
(0(
MSE (PT, qo) = Σ (pT(M)) - q0 (v:)))
B.EV
ECE (Pr, qe) = Σ
qo (e()))lr( - qe (vr))
veV
Table 6: Instance-wise probability calibration results for various logit distillation methods correspond
to Figure 9.
5/1
1/2
5/4
1/7
y/8
y
1/1D
AVG
KT. Divergence
(S)
0.61
5.40
16,99
0.00
0.58
0.00
10.14
0.00
17.41
CSD (S, S)
8.98
11.25
12.37
0.02
0.29
0.00
11.57
12.57
0.01
12.16
6.92
CSD (U, S)
6.41
9.90
12.00
0.03
0.48
0.00
10.38
13.84
0.04
12.08
6.52
CSD ST.S)
8.67
0.70
2.61
5.22
0.47
0.00
0.01
0.01
1.59
0.02
1.93
Mean Squared Error
DLD (S)
0.25
1,01
1.61
0.00
0.29
0.00
1.04
1.50
0.00
2.00
0.77
CSD (S, S)
1.13
1.17
1.94
0.00
0.00
0.00
1.32
2.00
0.00
2.00
0.96
CSD (U,
0.60
1.03
1.39
0.00
0.01
0.00
1.48
0.00
1.96
0.75
CSD (T.S)
0.84
0.02
0.08
0.39
0.01
0.00
0.00
0.00
0.03
0.00
Expected Calibration Error
(S)
0.29
0.03
0.00
0.34
0.00
0.04
0.50
0.00
1.00
0.28
CSD (S, S)
0.39
0.18
0.94
0.00
0.05
0.00
0.32
1.00
0.00
1.00
CSD (U.S)
0.28
0.04
0.39
0.01
0.08
0.00
0.01
0.48
0.00
0.96
0.22
CSD SS)S)
0.25
0.11
0.20
0.36
0.08
0.00
0.00
0.00
0.14
0.00
0.11
Experred Callbretion Ener
Mean Squered Error
0.7
Divergence
8.5
8.4
0.20
CSU [ss -
CSD (bs]
CSD (55)
CSD (t5)
0.15
CSD (65)
CSD (ts)
3.5
CSO (us)
2
CSD (us)
DLD
10 Epoch
10
Epoch
Epoch
(a) divergence
(b) MSE
ECE
Figure 10: Averaged probability calibration results during the training.
25

---

Published as a conference paper at ICLR 2026
2030
200
2000
1900
Frequance
100
AB
(a) CSD (S, (Init.)
(b) CSD (S, S) (Epoch 1)
(c) CSD (S,S) (Epoch S)
51
Frequancy
Feeqanccy
140
300
CSD S) (Epoch 10)
(e) CSD SS) S) (Epoch 15)
(f) CSD (S,S) (Epoch 20)
Figure 11: Logit offsets dynamics during the training of CSD.
D.2 ADAPTIVE LOSS WEIGHTING
21
ROUGE-LI (5 Benchmark Avg.)
19
18
CSD (ss)
CSD (ts 88)
CSD (us)
CSD (ss ts)
CSD (ts)
CSD (Entropy)
17
-0.40
-0.35
-0.30
-0.25
-0.20
Negative Self-BLEU (5 Benchmark-Avg.) t
Figure 12: Adaptive loss weighting
Diversity and fidelity form an inherent trade-off in generative modeling; a well-balanced default option also important. Because an adaptive loss can better reconcile this trade-off, conducted additional experiments on the adaptive loss weighting. We provide the results of two naïve scheduling and one confidence-based adaptive loss that interpolates CSD (S, S) and CSD (T, S) by defining wi an interpolation of ps and pr using We found that the following geometric interpolation performs better than linear interpolation in balancing the fidelity-diversity trade-off:
(x) pr(x)) (2(x) = (x))
CSD (TS - SS): a
Current Epech
Total Epoch
CSD (SS TS): Total Epech Current Epach
Total Epoch
CSD (Entropy): = clip
H(p,(z)) 0,
26

---

Published as a conference paper at ICLR 2026
CSD (TS + SS), CSD (SS TS), and CSD (Entropy) combine the strengths of both CSD (S. S) and CSD (T. S). and therefore achieve better performance at intermediate trade-off points as shown time, the loss used at the beginning of training tends to have a stronger influence on the final trade-off position. For example, CSD (TS SS) behaves similarly CSD (T, .S because its early-stage loss is closer S)..
in Figure 12. Because the learning rate is high in the early stages and decreases
Unlike other epoch- bosch scheduling, CSD (Entropy) adaptively a each token every step. Early in training, the entropy H (p.)) is typically larger than H (pr), so becomes close to 1, making the loss similar CSD (S, S). Since ps more diverse than pr at the early stage, the CSD (S. S) weighting provides richer feedback over larger set of vocabulary indices compared to CSD (T, S).
In the later stages of training, Ps ideally becomes closer to PT. making a close to O. In well-trained situation, CSD (S, S) and CSD SS. will show similar behavior, so the exact value of a becomes less important. However, when training does not progress well and p, becomes overconfident without matching pr. focusing solely CSD (S) weighting is undesirable. In such cases, stronger teacher guidance from CSD (T: 5) is needed, which why CSD (Entropy) designed as above. With this design, the model showed more balanced performance.
D.3 GRADIENT COEFFICIENT DIVERSITY
Grad coefficient diversity
7
6
CSD (ss)
5
CSD (us)
4
CSD (ts)
DLD (s)
3
KL
2
1
0
6
8
10 Epoch
12
14
16
18
20
Figure 13: Gradient coefficient diversity.
The limitation of softmax-based divergence losses, pointed out in Figures la and 1b, is that they provide almost no learning signal for minority vocabulary items. In this section, we analyze how broadly CSD learns across different vocabulary items. For CSD, know the gradient coefficient for each vocabulary item from Eq. (9), which is given as follows:
Coeff (y.) = w(y)) (To [yee Fr.
We take the absolute value of this coefficient, normalize by dividing by a constant so that the values sum to one, and then measure its entropy across training epochs. Figure 13 shows that when both weightings come from either the teacher's or the student's probabilities, the model learns only a small subset of vocabulary items, similar to KL. In contrast, CSD (U, S) learns from much broader range of vocabulary.
This demonstrates that expanding the loss-design space beyond the smoothing behavior imposed by softmax can effective, which was the main motivation of this work. Because CSD (U, S) leams uniformly across all vocabulary items, the logits for all vocabularies their respective offsets, as shown in Figure 9c. This also explains its strong performance under high-temperature sampling (where minority vocabulary contributions become more important) as shown in Figure 3b.
well centered around
27

---

Published as a conference paper at ICLR 2026
Table 7: Comparison with more baselines corresponds Table 2.
Method
Loss D
Dolly Bval
Self-Instruct
Vicuna
Super-NI
UnNI
Avg. (t)
Teacher (GPT-2-1.
14.07 a79
16.31
26.46:
31.10= 0.6
22.99
GP7-2-1.53 GPT-2-0. ..
SFT
SFT Fix 10,56 0.2
15.09
0:48
17.13
19.97
0.08
17.25
SeqKD (Kim & Rush, 2016)
SFT
PT 23
11.67::
14.73
0.37
21.04
23.55 0.11
18.97
KD (Hinton et al., 2015)
Fix 23.52:1025
10.02:1 0.58
14.57
0.32
16.76.
18.55
0.13
16.68
Ours Ours
CSD
Fix 22.
12.06
0.46
15.78
0.49
24.60
25.88
0.13
20.65
CSD
On
25.70
12.40 0.48
17.18
0.52
22.91 + 0.46
25.47
0.17
20.73
GPD-2-1.53 -.T---0.33-
SFT
SFT Fix 25.00 0.0
12.23
16.24 0.40
23.42:1
26,99 0.13
20.79
SD (Kim & Rush, 2016)
SFT
Pr 2..7
11.03 0.95
15.27 1: 0.30
18.91
21.78
0.10
18.36
KD (Hinton et al., 2015)
KL
Fix
25.41
1:522
11.15
0.20
15.83 :: 0.26
20.13
0.38
23.57
0.13
19.22
Ours
CSD
On 27.14
0228
14.85
16.88 0.18
26.28 0:21
30.43 0.04
23.12
Table 8: Comparison with probability matching loss with various weighting functions.
Loss
1F(())
w2(+)
Dolly Eval
Self Insiract
Vicuna Eval
Super NI
UnNI
Avg. (1)
T
24.41
11.45 0:25
14.43
0.68
24.08
0.30
25,53 001
19.98
Prob L2
U
15.62 0.37
6.59 0.19
10.63
0.44
10,31 :: 0.34
12,51
0:14
11.13
S
16.43 0.14
6.51 a:
9.73 0.17
10.94 :: 0.31
13.16
0:20
11.35
T
23.65 0.44
10.36 0.19
15.10 0:41
16.18
1 0.36
19.64 0.07
16.99
KL
U
23,52 0.25
10,02 0.58
14.57 4: 0.32
16.76 :: 0.17
18.55 0.13
16.68
S
23.18
10.04
15,06
16.93
19.78
17.00
T
24.04
10.99
14.68 0.19
25,40
0.06
25.24
0.04
20.07
TV
U
23.88
11.03
15.13 0:44
24.58
0.25
25.24 0.06
19.97
S
3.21 0.41
0.51 +: 0.30
0.97 0.13
0.66 = 0.06
0.69
1.21
T
0.06 0.01
0.04 0.0
0.18 0.02
0.03
0.00
0.03 0.00
0.07
SRKL
U
24.53 0.21
12.19.1:000 0.29
15.63 0.22
23.37 0.27
24.28 0.18
20.00
S
0.64 0.04
0.49 0.05
0.94 0.08
0.53 0.02
0.44: 0.00
0.61
T
T
6.82 = 0.16
4.24 0.12
9.16 0.25
4.53
0.02
4.83 0.02
5.91
U
U
17.21 + 0.30
8.08 0.39
14.27
0.40
13.19
0.27
14.07 0.04
13.37
CSD
S
S
24.94
12,06
15.78
24.60
0.31
25.88
0.13
20.65
(Ours)
U
24.15
0.55
12.25 0.47
15.25
0.41
22.55
0.09
25.19
0.12
19.88
T'
22.77
0.25
10.62
0.32
14.06
:: 0.25
18.81
1: 0.40
21.71 a: 0.18
17.59
E THE USE OF LARGE LANGUAGE MODELS
In this work, LLMs were used only for minor writing assistance, such as grammar correction after drafting. In addition, since the research topic is LLM distillation, LLMs were employed as the subject of experiments and also as evaluation models for performance assessment.
28

---

Published as a conference paper at ICLR 2026
Table 9: Distillation configuration, memory usage, and training speed for each paacher-student pair and distillation method. All measurements were obtained using single A100 GPU.
Teacher - Student
GPT-2
GPT-2
OpenI.aaaa Qwen2.5-IT Gemma2-IT
1.5B + O.1B 1.5B - 0.3B
3B
7B 1.5B 9B->
Configuration
Vocab
50,257 512 (256)
50,257 512 (256)
32,000
151,665
256,000
Max sequence len. (prompt len.)
512 (256)
1024 (512) 1024 (512) 32 (2 x 16) 32 (1 x 32)
BatchSize (microbatch accum.)
32
32
32
LoRA
X
X
Efficiency (memory & training speed)
DLD
Memory (MB) (4)
30489.98
50656.78
35341.80
45134.24
46203.35
Elapsed Time (sec batch) ())
0.758
1.033
4.035
26.62
44.50
SKL
Memory (MB) (4)
39129.84
60082.85
41542.05
52234.71
52196.50
Elapsed Time (sec batch) ())
0.803
1.094
4.077
27.88
43.92
KL
Memory (MB) (4)
32845.77
49870.58
35041.99
38032.67
40208.22 22
Elapsed Time (sec batch) (J)
0.770
1.027
4.044
25.84
43.34
CSD (Anal.) Memory (MB) (4)
28919.70
49085.38
34542.05
42766.25
44205.31
Elapsed Time (sec batch) (1)
0.764
1,033
4.041
26.64
42.12
CSD (MC)
Memory (MB) (4)
30490.
50656.91
35341.92
47502.30
48201 .43
Elapsed Time (sec batch) (J)
0.789
1.061
4.063
27.78
43.06
Table 10: Task-specific distillation performance from the Gemma-7B- teacher to the Gemma-2B-I7 student. refer to Section A.4 for DLD variants details.
Summarization
Translation
GSM8K
Loss
ROUGE-L
COMET
Accuracy
Teacher
37.09
79.23
60.27
DLD (T)
0.00
19.00
0.00
DLD (U)
0.00
18.98
0.00
DLD (S)
0.00
21.52
0.00
DLD-min (T)
0.46
48.05
0.00
DLD-min (U)
13.29
53.81
0.00
DLD-min (S)
15.91
52.98
0.00
DLD-max (T)
32.54
65.28
17.74
DLD-max (U)
15.75
24.23
0.00
DLD-max (S)
18.72
60.56
0.00
DLD-std (T)
0.00
35.71
0.00
DLD-std (U)
0.85
35.29
0.00
DLD-std (S)
18.97
58.07
0.00
DLD-mean (T)
0.00
34.21
0.00
DLD-mean (S)
18.78
43.98
0.00
DLD-mean (U)
0.03
32.03
0.00
CSD (T, S)
35.67
74.14
25.78
29

---

Published as a conference paper at ICLR 2026
Table 11:
Qualitative comparison on the GSM8K dataset. Only CSD (Ours) produces correct final answer, other students give incorrect results.
Prompt for model:
Q: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? A:
Correct Answer:
It takes 2/2=<2/2=1>1 bolt of white fiber
So the total amount of fabric is ==1==2+1=303 bolts of fabric ####
Teacher sampled output:
The robe takes 1/2 <2* 1/2=1>1 bolt white fiber.
So it takes 2 + =2+1=3>3 bolts in total. ##### 3
CSD (ours) student sampled output:
The robe takes 1/2 <2* 1/2=1>1 bolt of white fiber. So, takes + <2+1=3>3 bolts in total.
#### 3 (V)
KL student sampled output:
bolts of blue fiber 1 bolt of white fiber <2+1=3>3 bolts in total bolts of blue fiber 2 bolts of white fiber <3+2=5x5 bolts in total #### (X)
Jeffrey student sampled output:
The robe takes 2 2**2=444 bolts of blue fiber:
The robe takes 4 = c4/2-2*2 bolts of white fiber.
The robe takes 4 + 2 <4+2=6>6 bolts in total. ##### 6 (X)
JS student
The robe takes 2=2=42+==404 bolts of blue fiber. The robe takes 4/2=<4/2=252 bolts of white fiber. The robe takes 4+2=<44==6~6 bolts in total.
#*## 6 (X)
30

---

Published as a conference paper at ICLR 2026
Table 12: Qualitative comparison on a GSM8K dataset. The SRKL student's sampled output shows degenerative repetition and drifting arithmetic reasoning.
Prompt for model:
Q: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? A:
Correct Answer:
It takes 2/2=42/2=1>> bolt of white fiber
So the total amount of fabric is 2+1=x2+1=3>> bolts of fabric ##### 3
SRKL student sampled output:
The robe takes 2/2=1/2 bolts of white fiber.
So, it takes 1/2*2=1 bolt of blue fiber.
So, it takes 1+2=3 bolts in total.
So, takes 3*1=3 bolts in total.
So, takes 3*2=6 bolts in total.
So, it takes 6+1=7 bolts in total.
So, it takes 7*2=14 bolts in total.
So, takes 14+3=17 bolts in total.
So, it takes 17*2=34 bolts in total.
So, it takes 34+1=35 bolts in total.
So, it takes 35 bolts in total.
So, takes 35*2=70 bolts in total.
So, takes 70+1=71 bolts total.
So, it takes 71 bolts in total.
So, it takes 71$2=142 bolts in total.
So, takes 142+3=145 bolts in total.
So, it takes 145*2=290 bolts total.
So, it takes 290+1=291 bolts in total.
So, it takes 291 bolts in total.
So, takes 291 #2=582 bolts total.
So, takes 582 bolts in total.
So, it takes 582*2=1 164 bolts in total.
So, it takes 1 1164 bolts in total.
So, takes 11442=2328 bolts in total.
So, takes 2328 bolts in total.
So, it takes 232862=4656 bolts in total.
So, it takes 4656 bolts in total.
So, takes 4656*2=9312 bolts in total.
So, takes 9312 bolts in total.
So, it takes 9312#2=18624 bolts in total.
So, it takes 18624 bolts
31

---

Published as a conference paper at ICLR 2026
Table 13: Qualitative comparison GSM8K dataset. The TV student's sampled output shows degenerative repetition and drifting arithmetic reasoning.
Prompt for model:
Q: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? A:
Correct Answer:
takes 2/2=#2/2=Iw1 bolt of white fiber
the total amount of fabric 2+1==2+1=303 bolts of fabric ##### 3
TV student sampled output:
The robe takes 2 1 bolt of white fiber.
it takes + bolts total.
The robe takes 3 2 = 6 bolts in total.
So it takes 6 bolts in total.
The robe takes 6 bolts in total.
So it takes bolts in total.
The robe takes 6 bolts in total. So it takes 6 bolts in total.
The robe takes 6 bolts in total.
So it takes bolts total.
The robe takes 6 bolts in total.
So it takes 6 bolts in total.
The robe takes 6 bolts in total.
it takes bolts total.
The robe takes 6 bolts in total.
So it takes 6 bolts in total. (repeats)... ...
The robe takes bolts in total. So it takes 6 bolts total. The robe takes 6 bolts in total. So it takes 6 bolts in total. The robe takes bolts in total.
So it takes 6 bolts in total.
The robe takes 6 bolts in total.
So it takes 6 bolts in total.
The robe takes bolts in total.
So it takes 6 bolts in total.
The robe takes 6 bolts in total.
So it takes 6 bolts in total.
The robe takes bolts in total.
So it takes 6 bolts in total.
The robe takes 6 bolts in total.
So it takes 6 bolts in total.
32

---

Published as a conference paper at ICLR 2026
Table 14: Qualitative comparison GSM8K dataset. The SKI. student's sampled output shows degenerative repetition and drifting arithmetic reasoning.
Prompt for model:
Q: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? A:
Correct Answer:
It takes 2/2=<2/2=1>1 bolt of white fiber
the total amount of fabric 2+1==2+1=303 bolts of fabric ##### 3
SKL student sampled output:
The robe takes 2/2=1/2 bolts of white fiber. it takes 2+1/2=3/2 bolts of fiber total.
So it takes 3/2=1.5 bolts of fiber in total.
So it takes 1.5$2=3 bolts of fiber in total.
So it takes 3+1=4 bolts of fiber in total.
it takes 4*2=8 bolts fiber total.
So it takes 8+1=9 bolts of fiber in total.
So it takes 9+1=10 bolts of fiber in total.
So it takes 10+2=12 bolts of fiber in total.
it takes 12+1=13 bolts of fiber in total.
So it takes 13+1=14 bolts of fiber in total.
So it takes 14+4=18 bolts of fiber in total.
So it takes 18+1=19 bolts of fiber in total.
So it takes 19+1=20 bolts of fiber in total.
So it takes 20+2-22 bolts of fiber in total.
So it takes 22+1=23 bolts of fiber in total.
So it takes 23+1=24 bolts of fiber in total.
So it takes 24+1=25 bolts of fiber in total.
So it takes 25+1=26 bolts of fiber in total.
So it takes 26+1=27 bolts of fiber in total.
So it takes 27+1=28 bolts of fiber in total.
So it takes 28+1=29 bolts of fiber in total.
So it takes 29+1=30 bolts of fiber in total.
So it takes 30+1=31 bolts of fiber in total.
So it takes 31+1=32 bolts of fiber in total.
it takes 32+1=33 bolts of fiber in total.
So it takes 33+1=34 bolts of fiber in total.
So it takes 34+1=35 bolts of fiber in total.
So it takes 35+1=36 bolts of fiber in total.
33

---

Published as a conference paper at ICLR 2026
Table 15: Qualitative comparison GSM8K dataset. The DLD (S) student's sampled output shows degenerative repetition and drifting arithmetic reasoning.
Prompt for model:
Q: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? A:
Correct Answer:
takes 2/2=42/2=1>1 bolt of white fiber
So the total amount of fabric 2+1=x2+1= 33+33 bolts of fabric #### 3
DLD (S) student sampled output: reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluet reluct reluct reluct reluct reluct reluct reluct (repeats)....
reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
reluct reluct reluct reluct reluct reluct
34


Reviews:
[{"review_id": "UrQYJHEdGN", "reviewer": "Reviewer_TcP2", "summary": "The paper proposes Concrete Score Distillation (CSD), a discrete score-matching objective for LLM distillation that avoids softmax-induced smoothing and is invariant to constant logit shifts, which are two limitations of common KD and direct-logit losses. It resolves the quadratic cost of pairwise score matching with an analytic, linear-time gradient and provides formal guarantees. Experiments on instruction following and task-specific settings (e.g., summarization, translation) show CSD consistently outperforms recent KD objectives and delivers favorable fidelity-diversity trade-offs.", "questions": "Can you report throughput, peak memory, and step time for CSD vs. KL/SKL/DLD on matched setups, plus scaling with vocab size? (This directly tests the linear-time gradient claim and Algorithm 1 practicality.)\n\nCan you provide defaults, schedules, or an adaptive/learned weighting (e.g., driven by confidence or gradient variance)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proves consistency and that CSD’s optimum set strictly contains DLD’s, leveraging invariance to constant logit shifts\n\n- It derives an Algorithm 1 for keeping compute/memory in line with standard KD\n\n- The method consistently surpasses strong KD baselines across tasks and yields complementary gains when combined with on-policy techniques, suggesting good scalability.", "weaknesses": "- The paper claims linear-time gradients, but lacks systematic reporting of GPU memory, tokens/sec, and wall-clock per step versus KL/SKL/DLD across vocabulary sizes (e.g., 32k to 128k). Adding end-to-end cost curves and scaling ablations would strengthen adoption\n\n- While the paper shows that weights modulate mode-seeking/covering, default choices, schedules, or adaptive schemes are not well-justified; more guidance (or learning the weights) would aid practitioners.\n\n- Since CSD replaces softmax normalization with centered logits, its impact on probability calibration warrants measurement. (Related to shift-invariance.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdT8pluvFt", "reviewer": "Reviewer_Pedf", "summary": "This paper addresses the challenge of distilling large language models. To address some of the limitations of Direct Logit Distillation, the authors propose leveraging score matching for discrete variables. They thus propose a new objective for knowledge distillation named Concrete Score Distillation. The main idea is to replace the dataset data distribution ($p_{\\textrm{data}$) in the score matching objective with the teacher probability $p_T$. To address training instability, they propose using a logarithm function. They also propose an efficient approach for computing gradients. A complete experimental validation is provided for different setups: task-agnostic instruction-following distillation and task-specific distillation.", "questions": "+ A key assumption of the proposed approach is that the teacher and student models share the same vocabulary. We know that, in practice, this hypothesis is not satisfied due to the differences between the different tokenizers. How can this issue be addressed in the proposed approach?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**originality**\n + The idea of using the generalized score function for LLM knowledge distillation is new and original.\n + The proposed approach is technically sound, and the claims are supported by both theoretical aspects and by the experimental part.\n\n**quality**\n + The effectiveness of the proposed approach is validated on multiple KD settings.\n\n**clarity**\n+ The paper is well-written, with a clear formalization of the proposed approach.\nThe approach is also well-motivated.\n+ The annexes of the paper contain important additional information, such as proofs, derivations, and additional experimental details.\n\n**significance**\n+ LLM Knowledge distillation is a crucial issue for the real-world deployment of LLMs. The proposed approach clearly advances the state of the art on LLM Knowledge distillation.", "weaknesses": "+ Some minor remarks on equation 5 and 6 : $p_{\\textrm{data}}$ is not defined. \n+ Proposition 1 and Theorem 2 assume a sufficient model capacity. This should be detailed. What is a sufficient model capacity in this particular case? \n+ In Theorem 3, an assumption is that we can write the weighting function $w(y_t,x) = w_1(y_t) w_2(x)$. The authors do not justify this assumption. This rewriting of the weighting function is important in the proposed approach and in the experimental part. The paper does not explain how to build $w_1$ and $w_2$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hvenVeFBjG", "reviewer": "Reviewer_8hoT", "summary": "This paper proposes Concrete Score Distillation (CSD), a knowledge-distillation objective for autoregressive LLMs that addresses two core issues: (1) probability-matching objectives (e.g., KL) lose fine-grained information at the logit level due to softmax smoothing; (2) while direct logit distillation (DLD) avoids softmax, its objective implicitly forces the student logits to match the teacher exactly, ignoring logit shift invariance (i.e., an additive constant does not change probabilities), which restricts the optimal solution set. CSD adapts score matching from energy-based modeling to a discrete setting and matches pairwise relative logit differences across the vocabulary, thereby being inherently shift-invariant and expanding the attainable solution set. The paper further addresses training instability and computational overhead when applying score matching to autoregressive LLMs by deriving an O(|V|) analytic gradient (from a naïve O(|V|2) formulation). Experiments show that CSD outperforms recent probability-matching and direct-logit objectives on both task-agnostic instruction following and task-specific distillation (summarization, math, and translation), offers a controllable fidelity–diversity trade-off, and yields complementary gains when combined with on-policy techniques, indicating good scalability.", "questions": "1. Stronger DLD comparisons. If DLD is augmented with mean-centering, temperature scaling, or an explicit shift-invariant formulation, does CSD still retain the advantages reported in Tables 1 and 3? Could you include strong DLD in the main results and report variance?\n2. On the solution space. Since CSD allows a global additive shift on logits, how does this offset behave during training—does it converge to a stable value? Are there systematic differences across tokens? Does this flexibility introduce any extra instability compared to DLD?\n3. Scaling characterization. For vocabularies of 32k–100k and sequence lengths of 4k–8k, what are the peak memory, step time, and convergence trade-offs of CSD (analytic vs. MC) versus KL/DLD? Is there a practical heuristic for choosing analytic vs. MC estimators at different scales?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Precise problem framing and strong motivation. The paper clearly identifies fundamental shortcomings of existing approaches—probability matching loses logit information; DLD over-constrains the solution set by ignoring shift invariance.\n2. Theory-grounded contribution. CSD is derived from score matching with key guarantees: Proposition 1 (consistency) and Theorem 2 (solution set is a superset of DLD). This provides principled justification for the objective.\n3. Clear, controllable objective design. Equation (9) reveals a gradient structure based on centering, factorized weighting, and role swapping; the (w1,w2) design smoothly tunes the fidelity–diversity trade-off (as illustrated in Figure 3).", "weaknesses": "1. Underpowered DLD baselines. DLD does not appear in the main comparison table and is shown only once in ablations (Table 4). Stronger DLD variants—e.g., mean-centered, temperature-scaled, or otherwise shift-aware formulations—are missing, as are several recent logit-distillation variants. This makes it hard to assess CSD’s margin over the strongest DLD baselines.\n2. Practical computational burden. Although Theorem 3 reduces complexity from O(|V|2) to O(|V|), real-world LLMs with very large vocabularies (e.g., 50k+) still require computing and normalizing all token logits per step. The paper shows that a Monte-Carlo estimator performs worse, so the efficient implementation currently relies on full-vocabulary passes, which may become a bottleneck at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pJGJfZwmUM", "reviewer": "Reviewer_5noZ", "summary": "The paper introduces Concrete Score Distillation (CSD), a novel training objective for improved large language model's knowledge distillation. Based on the observation that softmax is invariant to constant addition, the proposed method extends direct logit distillation (DLD) by allowing student's logits to be matched to the teacher's logits shifted by an arbitrary constant. The paper further introduces and experiments with various weighting functions to balance the output accuracy and diversity of the trained student. Experiments show that the proposed method outperforms prior approaches compared to divergence-based loss function, and that it achieves complementary gains when used in combination with recent on-policy KD methods.", "questions": "1. In figure 4, \"Ours\" refers to which exact model/setting for the two model sizes? Just CSD, or perhaps ImitKD+Ours, or something else? Could the authors share the GPT evaluation scores of vanilla CSD, GKD+ours, DistilLLM+ours,ImitKD+ours if these were also evaluated?\n1. For results in Table 3, w1,w2 were used as (t,s) (as mentioned in line 879) compared to the \"default\"(as mentioned in line 312) of (s,s) elsewhere. Can the authors elaborate on why were w1,w2 changed for these experiments? If the authors tried (s,s) first, could they share these results?\n1. In figure 8 (a) and (b), could the authors also share the KL-divergence/MSE/some other divergence metric between the teacher and student probabilities corresponding to y1 to y10? This will prove that the student probabilities with CSD maintain similarity to teacher inspite of logit shift.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper, including the algorithm, theoretical analyses and proofs, is well-written and easy to follow.\n1. The idea of enhancing logit matching distillation by accounting for the logit shift invariance is well-motivated and reasonable.\n1. The author's extension to DLD theoretically and empirically achieves the target of logit shift invariance.\n1. The experimental results show improvements compared to the baselines, across multiple model families and model sizes.\n1. The paper is well placed in existing literature, and prior works considered are thorough.", "weaknesses": "1. The authors claim that one of their motivations is that vanilla KLD gives almost identical and very low probabilities and gradients to the low-probability tokens in the tail. While their method, with uniform weighting (w1,w2=U) would indeed achieve their target of \"overcoming the softmax-induced smoothing of teacher knowledge\", most of their results use w1,w2 as either student or teacher probabilities. As these values are very small and almost zero, a very similar smoothing is also present in the proposed method.\n1. Comparison with DLD: since the method is an extension to DLD, I believe it would be more appropriate to compare the method with variants of logit matching methods. Currently, only the comparison between DLD and CSD on GPT2-0.1B for the five benchmarks is given. It would be great to see how CSD compares against DLD at larger model sizes, different datasets, etc.\n1. Weighting function: not using weighting function (corresponding to the (u, u) setting in the paper) performs significantly worse compared to other divergence-based baselines, and the overall performance seems to be heavily impacted by the choice of the weighting function. Furthermore, with S as a weighing function, DLD performance seems to be only slightly worse than the proposed method. This makes me wonder if similar performance improvement can be made to the divergence-based methods by adding similar weighting function to those.\n1. The method is only tested on relatively small datasets (eg. 15000 samples), its effectiveness in longer training horizon is unexplored.\n1. Most of the experiments were performed on a relatively weak model (GPT2), with only some experiments on Openllama/gemma. It is uncertain if the results will hold with newer/stronger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)