Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
Authors: Minsang Kim
Seung Jun Baek
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Distillation
Transfer Learning
Large Language Model
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: TSD-KD is a new knowledge distillation method that maximizes performance by selectively distilling only key reasoning tokens from a large model and guiding the smaller model to learn on its own, instead of forcing it to mimic the entire output.

Abstract: Knowledge Distillation (KD) can transfer the reasoning abilities of large models to smaller ones, which can reduce the costs to generate Chain-of-Thoughts for reasoning tasks. KD methods typically ask the student to mimic the teacher's distribution over the entire output. However, a student with limited capacity can be overwhelmed by such extensive supervision causing a distribution mismatch, especially in complex reasoning tasks. We propose Token-Selective Dual Knowledge Distillation (TSD-KD), a framework for student-centric distillation. TSD-KD focuses on distilling important tokens for reasoning and encourages the student to explain reasoning in its own words. TSD-KD combines indirect and direct distillation. Indirect distillation uses a weak form of feedback based on preference ranking. The student proposes candidate responses generated on its own; the teacher re-ranks those candidates as indirect feedback without enforcing its entire distribution. Direct distillation uses distribution matching; however, it selectively distills tokens based on the relative confidence between teacher and student. Finally, we add entropy regularization to maintain the student's confidence during distillation. Overall, our method provides the student with targeted and indirect feedback to support its own reasoning process and to facilitate self-improvement. The experiments show the state-of-the-art performance of TSD-KD on 10 challenging reasoning benchmarks, outperforming the baseline and runner-up in accuracy by up to 54.4% and 40.3%, respectively. Notably, a student trained by TSD-KD even outperformed its own teacher model in four cases by up to 20.3%. The source code is available at https://github.com/kmswin1/TSD-KD.

Published as a conference paper at ICLR 2026
EXPLAIN IN YOUR OWN WORDS: IMPROVING REA- SONING VIA TOKEN-SELECTIVE DUAL KNOWLEDGE DISTILLATION
Minsang Kim
Seung Jun Back
Korea University
Korea University s jjbaek@kore..ac ac.
SK Telecom kmswinl woorea.koee.......
ABSTRACT
Knowledge Distillation (KD) can transfer the reasoning abilities of large models to smaller ones, which can reduce the costs to generate Chain-of-TTe Siiii-of-Thougee Thoughts for reasoning tasks. KD methods typically ask the student to mimic the teacher's s distribution over the entire output. However, a student with limited capacity can be overwhelmed by such extensive supervision causing a distribution mismatch, especially in complex reasoning tasks. We propose Token-Selective Dual Knowl- edge Distillation (TSD-KD), a framework for student-centric distillation. TSD-KD focuses distilling important tokens for reasoning and encourages the student to explain reasoning its own words. TSD-KD combines indirect and direct distillation. Indirect distillation ranking. The student proposes candidate responses generated on its own; the teacher re-ranks those candidates as indirect feedback without enforcing its entire distribution. Direct distillation uses distribution matching; however, it selectively distills tokens based on the relative confidence between teacher and student. Fi- nally, we add entropy regularization to maintain the student's ' confidence during distillation. Overall, our method provides the student with targeted and indirect feedback to support its own reasoning process and to facilitate self-improvement. The experiments show the state-of-the-art performance TSD-KD on challeng- ing reasoning benchmarks, outperforming the bascline and runner-up in accuracy by up to 54.4% and 40.3%, respectively. Notably, a student trained by TSD-KD even outperformed its own teacher model in four cases by up to 20.3%. The source code is available at
weak form of feedback based on preference
INTRODUCTION
Knowledge distillation (KD) (Buciluă et al., 2006; Hinton et al., 2015) a technique to compress large "teacher" models into smaller "student" models. Especially for reasoning tasks, model compression can reduce the inference costs of generating long Chains-of-Thouggh (Wei et al., 2022; Snell et al., 2024; Yang al., 2025). The off- off-policy KD trains a student on outputs generated by teacher by matching logits (Hinton et al., 2015) or high-prooabilltyy tokens (Kim & Rush, 2016). Á draw bawwbcckoo of off-policy KD is the distribution mismatch between aeceereeeeeeated data and the student's 's own output, which can harm generalization (Agarwal et al., 2024).
Recently, on-policy KD (Gu et al., 2023; Agarwal al., 2024) was proposed in which the student is trained on its own generated output to mitigate distribution shift. A penalty such as KL-divergence is used to impose the teacher's feedback on student's output. However, the methods are teacher- forcing, i.e., forcing the student to match the teacher's distribution on every generated token. A student with limited capacity can be overwhelmed by such extensive feedback. Furthermore, the level of reasoning skills to solve complex tasks differ significantly between teacher and student. For example, problem-solving processes by a college senior and middle school student can inherently differ. Thus, direct KL-type penalty may cause distribution mismatch. We consider targeted and
"Corresponding Author

---

Published as a conference paper at ICLR 2026
indirect distillation; by targeted, we mean focusing distilling "important" tokens only; by indirect, we mean subtle feedback to match the student's 's level of reasoning, giving room for improvemre
A metric for token importance is its entropy measured on the LLM's output distribution over vocabulary at the token position. High entropy means the model is uncertain about its output.
Recent work in RL (Wang et al., 2025) proposes policy updates with rewards on a small set of high-entropy tokens to improve reasoning performance. Those tokens act as important branching points in the reasoning process. We
Entropy comparison (First 200 Tokens)
Qwen2.5-1
Gemma2-26
also measured the token entropy of student models and
Entropy
plotted it over token positions (Fig. 1). Two things are observed: 1) Some tokens have indeed significantly higher entropy than others. Many peaks are several times higher than the mean. 2) High-entropy tokens appear early in the reasoning traces. This motivates the design of KD method for reasoning tasks that capture token entropy and its dependence on token positions.
Avg. entropy (Gemma)
3.0
entrapy (Dwen)
302
330
Token Index
Figure 1: Average per token entropy of reasoning traces for (Yuan et al., 2025).
Contribution. We propose Mekeen-eeeectie Dual Knowledge Distillation (TSD-KD). a student- centric framework for on-policy KD. TSD-KD selectively applies supervision to important tokens and encourages the student to reason in its own words. TSD-KD takes a dual approach combining indirect and direct distillation, both performed in token-selective manner. In indirect distillation, the student proposes candidate tokens on its own at each reasoning step. The teacher gives only preference feedback on those candidates, instead of enforcing its entire output distribution as in traditional KD. This weak form of feedback helps the student self-improve and reduces distribution shift. We limit the distillation the early part of reasoning (Fig. 1). and let the student freely generate reasoning for the rest. In direct distillation, we use KL-divergence type feedback which, however, targeted at tokens the student is relatively uncertain. The distillation focuses on tokens that the student considers difficult, but the teacher considers easy. Finally, entropy regularization is added to the dual distillation. The token entropy is selectively minimized to reduce student's 's uncertainty and to boost its confidence. It acts distillation. Our key principle is to improve the reasoning ability of the student by strengthening its own reasoning process with minimal intervention from the teacher. Experiments on 10 reasoning benchmarks show the state-of of-the--rt performance of TSD-KD by up to 54.4% from the baseline student and up to 40.3% improvement from the second-best baseline. Notably, our student model even surpasses its teacher in four cases of reasoning tasks by up to 20.3%.
regularization that maintains the student's ' confidence under
2 PRELIMINARIES
On-policy Knowledge Distillation (KD). In on-policy KD, the student model generates its own outputs, and the teacher model provides a supervisory signal on them. The student's output may improve as distillation progresses. It is distinguished from traditional off-policy distillation, which relies on fixed data pre-generated by the teacher. Since the student model uses its own generated outputs for training, the supervisory signal well-fits the student's 's distribution. This mitigates the distribution shift in off-policy KD and improves generalization performance (Agarwal et al., 2024).
The supervision can be implemented using metrics that encourage the student's s output distribution to match that of the teacher. We first define the distribution p of autoregressive language models given input z and output response y. Suppose y consists of L tokens denoted by y = (V1, 3/2 .... yL). The model distribution p at token position = 1..... L, is the softmax output of logits, i.c., p(y xr, yet) exp(x+) Σ., (x,) where 2 E IR is the output logit position t.
Model Definition.
(Generalized) KL Divergence. KL divergence is a metric used for KD as follows. Suppose P and Q are the model distributions of teachers and students, respectively. The forward KL-divergence P(-) In addition, reverse KI. defined as (x] The given by D = Σ P(-) log QQ) forward KL leads to mode novercover,ng, and the reverse KL leads to mode-seeking behavior of the student (Agarwal et al., 2024) (also see Appendix A.11 for illustration). The choice between forward and reverse KL presents a key trade-off in KD. The generalized
Divergence (JSD) (Huszár, 2015) can control this trade-off by interpolating between both behaviors. Specifically,
2

---

Published as a conference paper at ICLR 2026
1. Indirect Distillation Guided By Teacher's Preference
1-1. Student's Generation
solve
we subtract 5 from both sides. Next, add 2
Select Early part (Opener) based Token importance
To solve x 3 5, we subtract 5 from both sides--eet,
1-3. Distillation by Preference Ranking
Win: Subtract
To solve x 5, we
Which Better?
Token Position
Lose: Add
Lose: O solve
subtract
To salve we subtract
Token Position t+1
Win: 310 To salve x
= 3
we subtract Better
Win: Both
solve 5, we subtract from
#Bc
Student
Token Position 2+2,
Lose: Right
Teacher
2. Direct Distillation based on Uncertainty Gap
Select
Select
Select
Teacher
Token Index
KD
Student
3. Regularization: Entropy minimization at Uncertain tokens Confidence
Confident Token
Student
Select
Select
Select
Select
Figure 2: Overview of TSD-KD. 1) Indirect distillation. Unlike traditional KD, the student actively suggests reasoning candidates, and the teacher chooses from them. 1.1) The student generates rea- soning response. 1.2) Early part of the response containing important branching reasoning (called opener) is selected. 1.3) The student sequentially generates candidates of partial responses from the opener. Top candidates are proposed to the teacher; for example, ! "To solve x+3=5, we subtract 5" and TTo solve x+3=5, we subtract 3". The teacher provides prefer- ence ranking (prefers 2) for better reasoning. The preference signal used as an indirect form of distillation. 2) Direct distillation performs selective distillation of critical tokens about which the student uncertain but the teacher is confident. 3) Entropy regularization minimizes the entropy of critical tokens, reducing uncertainty and maintaining the student's confidence during distillation.
the divergence ISD(3) for 0 < B < 1 is defined as
(I) It was shown that the gradient of JSD(3) converges to that of forward KL when 3 - 0 and reverse KL when 3 + In GKD (Agarwal et al., 2024), the following objective with JSD(3) is minimized for the student correct its errors in self-generated outputs. Denote the autoregressive models for the teacher by pr and the student by P with learnable parameters 0'. GKD minimizes the following:
Pisp())(PI(Q) 3 Dxi (P 3P ()Q) + (1-11) DxL (1--))Q).
D(prllp)) = Σ Disxxo (e.))l v<r)).
(2)
The output response y given input x for this training can sampled from either the teacher (off- policy) or the student (on-policy). As result, the student will attempt to imitate the token-level distribution of the teacher throughout the output generation.
PROPOSED METHOD
Outline. The objective in (2) represents direct distillation that forces the student to adjust its
distribution to that of the teacher at every token. We advocate for being more student-centric and
In case make reference to the model p3 with the gradient flow disabled, i.c., "stop-gradient", we will omit 0 and simply use ps-
3

---

Published as a conference paper at ICLR 2026
introduce an indirect form of distillation that provides subtle feedback on students' generation. Our method will combine indirect and direct distillation, so that both work in a complementary manner. Importantly, the combined distillation is performed in token-selective manner, focusing tokens that the student considers difficult to predict. In contrast, we provide no or limited feedback for the rest of the tokens, allowing the student to freely generate its reasoning. The key idea is to encourage the student to explain reasoning in its own words to promote reasoning capability. In summary, we propose a student-centric approach combining indirect (Sec. 3.1) and direct distillation (Sec. 3.2) in addition to entropy regularization (Sec. 3.3), all of which are performed in a sekensselective manner.
3.1 INDIRECT DISTILLATION GUIDED BY TEACHER'S S PREFERENCE
Motivation. We introduce an indirect distillation guided by the teacher's preference. The idea is to leverage the teacher as a ranking oracle over the student's ' candidate predictions. At each token position, the student proposes its top-k candidate tokens to the teacher. The teacher re-ranky these tokens, and the student aligns the candidate tokens with the teacher's s ranking. Importantly, it is the student who proposes the candidate tokens; the teacher only re-ranks the student's proposal. Unlike traditional KD, the teacher's ' top choices (e.g.. its own top-k) are unknown to the student. Thus, this an indirect form of distillation. In the following, we describe the detailed algorithm.
Distillation by Preference Ranking. Consider the output at token position t given input x. It is the next token predicted by the student given and y/c.- Let 21 [2),.... ..... 2z [k] denote the logits of k-best candidate tokens predicted by the student in the descending order. Since 2 's are logits, we have [1] 2 2, [2] 2 ... 2 24 [k]. The teacher's ranking of these k candidates is denoted by permutation Ne {i..... }} + {I....,k; k}. Thus, že [Ty [i)] is the student's s logit of the i-th best choice by the teacher. The teacher ranks the tokens by its own logits. The teacher first computes its logits at position given and vę+- The teacher then produces the ordering ne by ranking its logits corresponding to the student's candidate tokens.
For the preference modeling, we adopt the Plackett-Luce (PL) model (Plackett, 1975) widely used for aligning LLMs with human preference (Ouyang et al., 2022: Rafailov et al., 2023). The PL framework models the probability of the ranking of items. Suppose there are k items indexed by I...., & and the i-th item is associated with a reward (score) r(i). The probability that a user prefers (ranks) the items in the order of permutation of {L..... k} is given by
k II
exp(r(+()))
for l = 2,3,4. 4,
Σ exp(r(r(()))
where case A: = 2 is the Bradley-Terry (BT) model (Bradley & Terry, 1952). We use the PL model to formulate the probability that the student's 's token ranking is aligned with that of the teacher's. The likelihood that students' candidates are ranked according to ne, denoted by (PL x. 2.y,)), is
exp(z (T)))
(Te x. y<t = II
(4)
j=1 Ee= (xp(2x(m(())
In (4), we directly use logits 2 as the score for the PL distribution. Its implications will be explained in the next paragraph. We minimize its negative log-likelihood over token positions:
Lindirect - Σ log PPL( I Vet).
(5)
Recent work on on-policy KD (Gu et al., 2023; Agarwal et al., 2024) is analogous to imitation learning (Ross et al., 2011) where student simply mimics the output distribution of the teacher. In contrast, the student in our model learns to reason over its own generated choices. This self-reflective nature not only mitigates distribution shift, but also enables improved generalization and reasoning, especially for complex tasks.
Learning Preferences of Sub-responses. We explain the implication of (4) in the context of preference modeling. Given the response y, define the sub-response of length t as the partial response consisting of the beginning tokens of denoted by y- := (y) B2..... Vt). Thus, y-st is the partial response that represents the reasoning trace of reaching the token position 1. We show
4

---

Published as a conference paper at ICLR 2026
that distribution (4) is ranking model for such partial responses follows. For simplicity, let &
and the student's top-2 2 tokens at position be yt (1)
(2) Consider two sub-responses:
Vt
st = (3) V2....,, yr- ( y = (y1. 11 9r. (2)
(6)
That is, the sub-responses differ only at position t. We show that, under some choices of reward func- tions, (4) the distribution of ranking of candidate sub-responses yاu and y For example, consider two sub-responses (Fig. 2): y =TTo solve x+3=5, we subtract 3", and y solve x+3=5, we subtract 5". The teacher chooses the preferred sub-response for better reasoning, and our model aligns the student with the choice. The following proposition shows our claim for k = 2 for simplicity (BT model (Bradley Terry, 1952)). It straightforward to extend the claim for k 2 (PL model (Plackett, 1975)).
(2) ="To
Proposition 1. Suppose the student's 's reward for response y given input is implicitly defined as
rs(x.y) log ps(u(x) = log ps(yalu<e.
Similarly define the teacher's reward function rT y) == log pr(vlz). Suppose top-2 tokens at position I are Yt (1) (2) and sub-responses up to are given in (6). Consider the preference model -y
where teacher's reward determines the preference label for y and y That is, y is preferred (2) over y given (denoted by y-t 222((2)) if
(r, 222 rT (x, (x 2
(8)
and vice versa. Student's reward determines the preference distribution: the probability that y is preferred over U is given by
exp(rs(x, ( v21)))
ply y (1112)=
(9)
exp(rs (x, v-r)) + exp(rs (æ, y-1)
Then (4) is equivalent to for k = Also, Lindirect is the negative log-tikelihood of PL distribution
(4) aligning the student's 's and teacher's preferences on sub-responses y-xt summed over all I.
See Appendix A.1 for proof. Another implication of Proposition is that the preference distribution at token-level (4) is equivalent to that at sentence-level (9), under the preference reward (7) (preference rewards based on log-model probability are commonly used, c.g., (Rafailov et al., 2023)). This shows that our loss Lindirect has significant computational benefits. LIndirect in (5) computes a running sequence of word (token)-leve losses; however, this turns out to be a running sequence of sentence- level losses under reward (7). Considering that sentence is cumulative sequence of words, loss enables much efficient comparison of partial sentences by reducing it down to word-level comparison.
Token-selective Distillation: Importance of Openers. Although we use an indirect form of distillation, it may still drift the student's distribu- tion towards that of teachers. We hypothesized that it is desirable difficult tokens but not on the rest of the tokens, so that the student freely chooses its own words for the remaining process of reasoning. As shown in Fig. 1, difficult tokens are concentrated on the beginning part of responses. This is reasonable; it is crucial to set the initial direction reasoning correctly, as it establishes the initial branching of reasoning traces. Thus, we focus on the high-entropy tokens in the initial part of responses.
Algorithm Indirect Distillation by Teacher- Guided Preference Ranking
1: Given: Teacher model pr. Student Model
provide preference guidance on
p/s. Prompt dataset X
2: Sample x from X, generate y ~ (-1x)
3: Select the opener of up top-c% cumu-
lative entropy
4: for each token position t in opener do
5: Select top-k logits {2,[1], 2 [k]] by
student (-lvct,,, Re-rank logits to get the order of prefer- ence ग. by teacher pr( (et.
6:
7: Compute PPL(TE Vet)
8: end for
We introduce the concept of opener: it is the con-
9: Compute LpL = E, log PeL 1 ves)
secutive token sequence in the beginning part of
5

---

Published as a conference paper at ICLR 2026
response where cumulative entropy first reaches c%. Specifically, for an autoregressive model p. the entropy of response y at position t is defined as
H((p) = -- VEV
p(u x. yet) log p(e) x; M<t)
(10)
where V denotes the vocabulary (we omit the dependence of H+(-) on x and y for notational simplicity). The opener of response y of length L is the initial token sequence up to position m where m is the minimum integer satisfying E Hz(ps)/ E (ps)s) c%.
In this work, we use a small value = 10% which proves to be sufficient (we provide a study on other values of è in Sec. 4). In addition, we can make adaptive where с is determined based on the estimated difficulty the sample. the sample difficulty is estimated to be high, c is set to larger value, i.e., higher fraction of the sample is distilled to the student model. The detailed method of adaptation and related experiments are provided in Appendix A.7. Finally, our indirect distillation is performed in token-selective manner, i.e., only on the tokens in the opener. A pseudocode is shown in Algorithm 1.
3.2 DIRECT DISTILLATION BASED ON UNCERTAINTY GAP
Our indirect distillation is a student-centric approach to refining the reasoning path early in response generation. However, if the input question is very challenging, the student's 's top candidate tokens at reasoning steps may be off from the correct path. To address this issue, we use direct distillation with the JSD objective in (2). However, we apply direct distillation in a token-selective manner. Specifically, the distillation focuses on tokens for which the student is highly uncertain (high entropy), but the teacher is highly confident (low entropy). Thus, the distillation targets tokens that exhibit large uncertainty gaps. We implement the selective mechanism using a gating function as follows.
Dirrect L
E T. (He(p)) He(pr)) DISD(O) (2.
(11)
gating function
where we use sigmoidal gating (u) == (1 exp(-u/r)) and f > controls the sharpness of the entropy gap. Thus, a, dynamically modulates the strength of the distillation: reinforce the distillation on those tokens with large (:(p)) Hz i.e., the student is uncertain while the teacher is highly confident. Moreover, the gating mechanism tries not distill tokens in which the student is relatively confident. This helps the student generate its reasoning freely. Thus, our distillation is centrrit-eetric and also reduces the distribution shift. Our gating mechanism can be considered as a soft selection of tokens based on the relative confidence between student and teacher. Moreover, the soft selection of tokens has an effect of scaling the gradient of the student model during training, which can accelerate the convergence. We provide related analysis in the Appendix A.2.
We comment on another complementary aspect of indirect and direct distillation. Indirect distillation provides sequence-level guidance in setting up the early reasoning path. Direct distillation provides token-level and targeted guidance in precisely generating important tokens. Thus, the combination of indirect and direct distillation has synergistic effects on student-centric learning.
3.3 ENTROPY REGULARIZATION AND FINAL LOSS
Boosting Confidence with Entropy Regularization. Recent findings show that minimizing the token entropy, i.e., making the probability of high-confident output even higher, improves the reasoning capability of LLMs (Agarwal et al., 2025). It reduces uncertainty in intermediate steps, leading to more reliable reasoning trajectories. Inspired by this approach, we perform the entropy minimization, however, in a token-selective manner. We minimize the entropy of the most uncertain tokens only. That is, the student guided to boost confidence on critical reasoning tokens only. However, there no feedback for the remaining tokens, allowing the student to freely generate reasoning, similar to
our distillation methods. Formally, let I C {1...., T) denote the index set of the top- tokens with the largest entropy values {He (x2)) 1T We minimize the following objective:
LEM Exax
(12)
TET

---

Published as a conference paper at ICLR 2026
The choice of top-10% 10% is motivated by the ratio of important tokens suggested for RL for reasoning (Wang et al., 2025). They showed that policy updates from tokens in the top 10-20% of the entropy values improve performance. For completeness, we provide study varying ratios in Appendix A.4.
The entropy minimization has a regularization effect. Similar to indirect and direct distillation, the focus is difficult tokens. For these tokens, the student is made more deterministic (confident) rather than explorative through entropy minimization. This can improve the effectiveness of distilling challenging knowledge.
Final Loss. The overall training objective is
min Ex_x InnLindirect
+ Diirect
LEM
(13)
where a > 0 controls the relative weight of the indirect distillation. For simplicity, we set the relative weights of direct distillation and entropy minimization equal because they both pertain to entropy.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Overview. In our experiments, we mainly compare on-policy distillation methods. We train our method (TSD-KD) and recent state-of-the-an baselines using on-policy generations from the student model. For completeness, we also evaluated off-policy baselines. For some hyperparameters, we used l: = 10 for indirect distillation. For direct distillation, we set 3 = 0.9 for ISD( suggested by GKD (Agarwal et al., 2024). Detailed hyperparameters are provided in Appendix A.9.
Models. We conduct our main experiments with the Qwen2.5 Instrrcct model family (Qwen et al., 2024), using the 14B parameter model as the teacher and the 1.5B model as the student. To demonstrate generalizability, we also evaluate our approach on Gemma2 (Team al., 2024), with the 9B and 2B models serving as teacher and student, respectively.
Baselines. We compare our method with baselines in two categories. On-policy methods: We benchmark against recent on-policy approaches, including DistilLLM (Ko et al., 2024), MiniLLM (Gu et al., 2023), GKD (Agarwal et al., 2024) with / = 0.9, and Speculative KD (Xu et al., 2025). These methods are trained on the outputs generated from the student. Off-policy methods: We include traditional KD approaches such as Supervised- -DD (Hinton al., 2015) and Sequence-leve KD (Kim & Rush, 2016). Under these methods, the student is trained on a static dataset of responses generated by the teacher.
Training Data. For on-policy methods, we use prompts from UltraInteract (Yuan et al., 2025) and let the student generate responses to the prompts. The responses are used as training data. For off-policy methods, we use the teacher's s responses to the same prompts as training data.
Evaluation. We assess the models on a diverse set of reasoning benchmarks. This includes Mathe- matical reasoning (GSM8K (Cobbe et al., 2021), GSM-Plus (Li et al., 2024), MATH (Hendrycks et al., 2021b), CMMLU-Pr-Matte (Wang et al., 2024)), STEM and scientific reasoning (MMLU- STEM (Hendrycks al., 2021a), ScienceQA (SciQ) (Welbl et al., 2017)), Program synthesis (MBPP (Austin et al., 2021)), I)), Broad reasoning (BBH (Suzgun et al., 2022), MuSR (Sprague et al., 2024)), and Instruction following (IFEval (Zhou al., 2023)). For all benchmarks,
report accuracy following the standard evaluation protocols. More details on the benchmarks are in Appendix A.10.
4.2 RESULTS
Main Results. As presented in Table 1, TSD-KD achieves the best performance across six tasks. TSD-KD significantly outperforms all the baseline methods. The margin of improvement is up to 54.4% over the baseline student. The largest gain is achieved in the challenging MATH benchmark; TSD-KD (26.1) surpasses the second-best method by 40.3%. Surprisingly, it even outperforms its own teacher (21.7) by 20.3%. These results indicate that TSD-KD enables more generalizable reasoning as compared to simply mimicking the teacher's 's distribution.
We evaluated TSD-KD on more challenging reasoning benchmarks, as shown in Table 2. TSD-KD consistently achieves the best performances across all four challenging tasks, where TSD-KD leads
7

---

Published as a conference paper at ICLR 2026
Method
GSM8K
GSM-Plus
MATH
MBPP
IFEval
MMLU-STEM
Owen2.5-144 (Teacher)
80.3
59.7
21.7
78.9
85.9
70,5
Qwen2.5-1.55 (Student)
57.1
38.8
16.9
38.4
53.1
49.5
Sequence-Level KD
56.5
38.2
16.5
40.6
53.7
48.9
Supervised-KD
56.3
37.8
17.9
41.0
51.4
48.4
D
57.2
38.7
18.6
42.1
52.5
47.8
Speculative KD
58.0
39.2
18.5
$2.4
MiniLIM
57.7
39.7
17.8
42.2
54.7
48.2
GKD (3 = 0.9)
57,9
39.9
18.1
41.8
52.3
47.7
TSD-KD
60.1
40.5
26.1
42.1
55.2
50.0
Table 1: Performance comparison between distillation methods in Qwen2.5 (14B 1.5B). Best performance in Bold font. (*) mark for the student model outperforming the teacher.
Method
MMouU-Pro-Math
SciQ
BBH
MuSR
Qwen2.5-144 (Teacher)
77.6
86.4
60.8
60.8
Qwen2.5-1.5D (Student)
34.4
85.5
36.5
38.7
Sequence-Level KD
32.6
87.9
36.6
36.6
Supervised-KD
35.3
86.6
36.4
37.9
DistiLLM
35.2
85.7
36.5
38.7
Speculative KD
34.8
86.7
37.2
37.7
MiniLLM
36.2
84.1
36.0
38.3
GKD (B = 0.9)
35.7
86.6
36.2
38.5
TSD-KD
36.9
93.0
40.2
39.6
Table Performance comparison of more advanced benchmarks Qwen2.5 (14B + 1.5B). Best performance in Bold font. (*) mark for the student model outperforming the teacher.
(+)-)
Lindirect
LEM
GSM8K
GSM-Plus
MATH
IFval
MMLU-STEM
x
57.9
39.9
18.1
52.3
47.7
X
X
58.3
40.3
18.2
54.2
48.1
V
X
58.6
40.4
20.9
54.9
49.4
V
X
59.0
39.8
22.3
54.1
49.3
60.1
40.5
26.1
55.2
50.0
Table 3: Ablation study of each component in Qwen2.5 (14B > 1.5B). The baseline is on-policy ISD with a 0.9 model (except all components).
the runner-up by 9.8%. Remarkably, on complex scientific reasoning tasks in SciQ, the student model trained with TSD-KD surpasses its teacher by significant margin of 7.6%. The results show the effectiveness of our student-centric approach to distilling knowledge pivotal reasoning steps.
Ablation Study. Table 3 presents ablation study on three main components of TSD-KD. The first column represents the use of loken-selective gating a-(-) in direct distillation. The second column represents the use of indirect distillation (Cindirect). The third column represents the use of selective entropy minimization (CEM). The default baseline without any of these components is the vanilla GKD with 8 = 0.9, which is the fully direct KD on all tokens.
The results show the performance gains achieved by adding each component. The component dr (-) yields initial gain of up to 3.6%. Incorporating Lindieet increases performance by up 15.5%; significantly improves performance on general reasoning tasks such as AMMU-STTN Lastly, LEM effective for mathematical reasoning. increasing the MATH score from 18.1 1.1 to 22.3. The fully
8

---

Published as a conference paper at ICLR 2026
Method
GSM8K
GSM-Plus
MATH
MBPP
IFEval
MMLU-STEM
Gemma2-99 9B (Teacher)
57.7
40.8
25.8
40.0
64.7
50.2
Gemma2-22 2B (Student)
53.7
36.2
16.1
39.2
62.6
25,5
Sequence-Leeve KD
53.4
33.8
17.7
39.2
62.4
25.9
Supervised-K0 KD
53.6
36.8
17.9
39.0
63.4
25.5
LM
54.3
35.8
17.8
39.6
62.2
25.2
MiniLLM
54.4
36.8
17.5
40.6*
65.8
25.9
GKD (8 0.9)
54.1
36.9
17.2
40.0
63.5
26.2
TSD-KD
55.3
37,9
18.9
40.4*
66.5*
27.0
Table 4: Performance comparison between different distillation methods in Gemma2 (9B 2B).
Best performance in Bold font. (*) mark for the student model outperforming the teacher.
Method
GSM8K
MATH
GSM Plus
MBPP
IFEVAL
MMLU STEM
Qwen3-8B (teacher)
82.9
22.4
86.5
40.2
Qwen3-1.7B (student)
66.3
21.0
43.1
50.8
75.1
36.2
Sequence KD
65.7
20.8
43.5
51.3
75.7
36.9
Supervised KD
66.5
21.5
43.7
51.2
76.1
36.5
DistiLLM
66.7
21.2
45.5
51.4
78.2
36.8
MiniLLM
67.0
22.6
46.4
52.1
79.2
36.5
GKD
67.2
22.8
46.0
51.7
79.1
36.7
TSD-KD
68.7
28.0
47.2
51.9
79.8
37.5
Table 5: Performance comparison between different distillation methods in Qwen3 (8B 1.7B). Best performance in Bold font. (*) mark for the student model outperforming the teacher.
integrated TSD-KD model achieves performance improvement up 44.2% over the baseline. Our results indicate strong synergistic effect among the key components of TSD-KD.
Generalization to Gemma2 Models. We evaluate our method on a different base model and extend our evaluation to Gemma2 (Team et al., 2024). We apply the same set distillation methods to a Gemma2-9B teacher and Gemma2-25 student. Table shows that TSD-KD achieves the highest performance across 5 tasks. While some competing methods, such as GKD and MiniLLM, show strong performance GSM8K and MBPP, TSD-KD performs the best on the majority of tasks, including MATH and IFEval.
Importantly, the Gemma2-2B student model trained by TSD-KD surpasses the Gemma2-9B teacher, on both the IFEVAL and MBPP benchmarks. This outcome student surpasses teacher, similar to the findings for Qwen2.5, underscores the effectiveness of student-centric learning. The results of TSD-KD for Gemma2 validate its effectiveness as general framework for distillation.
Generalization to Qwen3 Models. We further evaluate TSD-KD on Qwen3 LLMs, using the 8B model as the teacher and the 1.7B model as the student. As shown in Table 5, TSD-KD achieves state- of-the-art performance in most of the benchmarks. Moreover, on the challenging MATH benchmark, the student model from our method outperforms the teacher model by large margin (+25.0 ))); this improvement is even greater than the case of Qwen 2.5 (+20.3%) on MATH. These results confirm that our method consistently improves reasoning performance across multiple benchmarks, demonstrating its effectiveness on more competitive reasoning models such as Qwen3-8B.
Analysis of Openers in Indirect Distillation. The opener is key concept in loken-selective indirect distillation. The length of the opener is defined as the token position from the beginning until reach e% of the cumulative entropy. There are two settings for the hyperparameter c: fixed or adaptive. We examine the effect of c when we use fixed c.
Fig. 3 shows the average performance the benchmarks from Table with varying We observe that the peak performance is achieved at c = 10%. This indicates that the teacher's 's feedback should focus relatively small portion early in the student's reasoning trajectory.

---

Published as a conference paper at ICLR 2026
Interestingly, performance worsens 100%, which is equivalent to applying distillation to all tokens. This result strongly supports our key hypothesis: a strategic selection of tokens is essential. is important to indirectly provide feedback on the initial direction of reasoning, keeping the teacher's
intervention minimal.
Additional experiments. We provide additional experi- ment results on: impact of in top-k token selection in indirect distillation (A.3); the ratio of token selection in entropy regularization (A.4); impact of / in (3D(3) for direct distillation (A.5); comparison of on-policy vs. off-
46.0
45.7
Aug. Performance
45.5
Performance
45.0
4.9
policy (A.6); Adaptive selection of e (A.7); and KD with PEFT (A.8).
46.0
43.7
43.5
RELATED WORK
100
Top-c9 of taken selection
Figure 3: Performance with varying
Knowledge Distillation of LLMs. Knowledge Distilla-
tion (KD) was initially proposed method for student model to imitate the output probability distribution of a teacher model (Buciluă et al., 2006; Hinton et al., 2015). This was later extended to Sequence-Level KD (Kim & Rush, 2016), which trains the student by maximizing the likelihood of high-probability sequences from the teacher. However, these approaches suffer from distribution shift, a discrepancy between the teacher-provice distribution at inference. To mitigate this, recent research has shifted towards on on-policy KD methods that use the student's own generations for training (Lin et al., 2020; Gu et al., 2023; Ko et al., 2024; Agarwal et al., 2024; Xu et al., 2025). KPOD (Feng et al., 2024) proposed progressive off- KD that transfers the teacher's CoT reasoning to the student by selective distillation of key tokens combined with curriculum learning. In contrast, we take an on-policy, dual distillation approach with token selection based on entropy. Although these methods have shown success in general tasks, their application to distilling complex reasoning remains underexplored. Building this paradigm, work proposes a student-centric framework for on-policy distillation to enhance complex reasoning capabilities.
training distribution and the student's own generated
Advanced Reasoning with LLMs. Chain-of-Though (CoT) prompting was introduced in (Kojima al., 2022; Wei et al., 2022) for solving complex reasoning tasks by generating intermediate steps in a step-by-step manner. Following the success of CoT, more sophisticated reasoning strategies have been proposed. Tree-of-Thought. (ToT) (Yao et al., 2023) proposed multi-path CoT, exploring and evaluating multiple possible reasoning paths in a tree structure to find the optimal solution. (GoT) (Besta et al., 2024) introduced the reasoning process as a graph structure, allowing for more flexible combinations of thoughts. The success of these advanced reasoning techniques attributed to the enhanced reasoning capabilities of recent LLMs (Guo et al., 2025; Yang et al., 2025). However, a common limitation of these methods is the high computational cost required to generate the reasoning process. This issue has highlighted the importance of KD, which aims to transfer the reasoning abilities of these large, computationally expensive models to more efficient ones. The proposed KD method focuses on developing a model with advanced reasoning capabilities while maintaining computational efficiency.
6 CONCLUSION
We propose Token-Selective Dual Knowledge Distillation (TSD-KD) which enhances the reasoning performance of student models by focusing distillation on important tokens. TSD-KD employs indirect distillation, where the teacher provides a preference ranking for the student's own generated output. Direct distillation is also applied selectively to a targeted set of tokens based on the relative uncertainty between teacher and student. Finally, TSD-KD maintains the student's 's confidence with entropy regularization. Experimental results various reasoning benchmarks show that TSD-KD substantially outperforms existing on on-policy KD methods and even surpasses the teacher model in several cases. Our study demonstrates the effectiveness of student-centered approaches in KD.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENT
This work was partly supported by ICT Creative Consilience Program through the Institute of Information & Communications Technology Planning & Evaluation (IIPP) grant funded by the Korea government -0LTTT--222-S--222002 819, 50%). and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2022-NR070834)
REPRODUCIBILITY STATEMENT
Training Implementation. We implement our training framework based TRL. This framework provides trainers with various loss functions, but we develop proposed loss functions.
Inference Implementation. We generate all generate datasets using vL..M.
Evaluation Datasets. We evaluate various math and code tasks in eevalaationnaareeeee which are open-access repositories.
Source Code. We uploaded the source code in the public repository.
Computational Resources, All our experiments, used eight A100 GPUs with 80GB VRAM.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation language models: Learning from self- generated mistakes. In The Twelfth International Conference Learning Representations, 2024. URL htap:::://pee wwt.thtts///ppeeeeeeeeee /wwpe/eeeeeeeeeeeeeeeeeeeeeeeeee iew. net/ forum? 1d=32KtaqxLin
Shivam Agarwal, Zimin Zhang. Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effective- of entropy minimization in Ilm reasoning. arXiv preprint arXiv:2500. 15144, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv: 100.0000000 108.07732. 2021.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Currieulum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009,
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi. Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 77682-17690 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizii Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 535-541, 2006.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14.666.... 2021.
Kaituo Feng, Changsheng Li, Xiaolu Zhang, Jun Zhou, Ye Yuan, and Guoren Wang. Keypoint-based progressive -fain-of-tthughh distillation for llms. arXiv preprint wxXi:2245.16606. 2024.
Yuxian Gu, Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language
models. arXiv preprint arx.v:2306.088.0 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint (ttl:22501.124444 2025.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.023: 2015.
Ferenc Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint www.Xiv:511.00
2015.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1317-1327, 2016.
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Young Yun. Distillm: Towards streamlined distillation for large language models. In International Conference on Machine Learning, pp.
PMLR, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22215 2022.
Tuan Anh Le. Reverse vs forward KL, December 2017. URL
tuananhle.co
www.okke//////ottt htt///eeeeeeeeeeeeeeeeeeeeeeeeee forward- k1 html. Accessed: 2025-09-22.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Ilms as mathematical problem solvers. arXiv preprint arXiv:2402.19255, 2024.
Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6121-6133, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193-20) 1975.
A Yang Qwen, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, Huang, H Wei, et al. Qwen2. 5 technical report. arXiv preprint, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-533 2023.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings. 2011.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408..0 03314, 2024.
12

---

Published as a conference paper at ICLR 2026
Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of th-huuhh with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL httpss///pperreiee net/ /aat//ooue Eorum? iininiddddynn
YQzuel.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thouggee can solve them. arXiv preprint arXiv:2210.09261, 2022.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.0355. 2020.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 2024.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.0199 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266-95290, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Ofain-of-thoouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877 2022.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions.
arXiv preprint Xiv: / 177700020999 2017.
Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen- Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher- student gap through interleaved sampling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: opw.neveeeeeeeeeeeeooeeeee net / forumm http//oommn-igiw//eeeeee Bghhddu
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 111v:22555003888 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:1 1809-11822 2023.
Lifan Yuan, Ganqu Cui. Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. In The Thirteenth International Conference on Learning Representations, 2025. URL https: petttps://opereeeee http///wppeeeeeeeeee///e 2t
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.079/11 2023.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PROOF OF PROPOSITION
Let ps ()) denote the autoregressive student model defined as follows:
exp(2=)
Ps Bel x) == Σe, exp(
where 2, the logit of the output token at position This implies that, for some given sub-response 8-st. we have that
Ts 1a(2,914 v-1)
log ps (NelWerr
Σ
2s + log
Σ
exp &o
==1
Then, given the sub-responses y 2 and (2) we have
7s(2,111) 2t
(1) log Σ
exp 2
28 + log Σ exp &
(14)
: 2(t)
and
(s ( = +/
+ log
Σ 21
Exp
28
+ Σ ==1
2e + log Σ exp &g
(15)
2(t)
Note that (t) is identical for two sub-responses (1)
y and (2) because they differ only in the last
token at position t. Then, the PL distribution of preference with the student's reward rs defined in (9) is given by
exp(rs(x,(5)))
ply --11())=
(16)
exp(rs(x, y 2)
+ exp(rs((z,))) (z, ())
2 (1)
z(t)
exp
(17)
exp
24
(1)
+ z(t)
+ exp
(2)
2(t)
exp[=t"1
(1)
(18)
expl=t (1)
exp 2
where (17) is obtained by applying (14) and (15). We observe that (18) is equivalent to model PoL given by (4) for & = 2, which proves the proposition. Similarly, we can write Lindirect as
Lindirect
log (TL x. Mee)
log.(v".. > v'aelæ)
where y and م denote the preferred and dispreferred sub-responses by the teacher.
Note that the preference relation by the teacher also determined by comparing the logits of the teacher model as follows. Let
exp(Gi
pr (yelVee, x) Ee. exp(C:)
where denotes the teacher's logit evaluated at position t. Our algorithm determines preference by comparing the teacher's logits. Specifically, let c,22) and ((2) denote the teacher's logits for subreponses (1) and y at position 1. We have that
ف > (2)
2
(T ( rr (2,
(2)
(19)
where the if and only if condition on the right can be shown using an argument similar to (14) and (15). This proves the relation between the teacher's reward and its preference in Eq. (8).
In summary, Lindiret is equivalent to the negative log- ligeliild of preference distribution of sub- responses summed over under our preference model.
14

---

Published as a conference paper at ICLR 2026
A.2 ANALYSIS OF SOFT TOKEN SELECTION VIA O, IN DIRECT DISTILLATION.
Our claim is based on the gradient rescaling property of KD (Tang et al., 2020). For mathematical tractability, only analyze forward KL and assume that only the forward KL is used in our direct distillation. Consider the teacher model p and the student model The student model is an autoregressive LLM whose output is the softmax of token logits. Specifically, let qi denote the confidence in token i given by
exp(22 244
9i Σ, exp(zi)
where 2j denotes the logit of token i. Consider the conventional (forward) KD: we minimize the loss
(xl())
If we take the gradient of L with respect to logit 2;:
dL
P: q:
(20)
dz;
This gradient is the "feedback" to logit 2,. Now suppose that token i the "ground truth" for a reasoning task. In reality, there rarely is ground truth token for LLMs. However, since consider reasoning tasks, there is highly likely token leading to the correct answer. For example, the next predicted token for "The answer to 5+3 is " is "8" " We will assume that Oracle exists and provides the one-hot label for the "best token" at each reasoning step. We will simply refer to the best token labeled by Oracle as the ground truth. (20) implies that, if the teacher pt has higher confidence than the student qi (p, > 4). then the feedback is positive. In that case, training will increase the logit 2; of the ground truth, which desirable for training. This can speed up the model convergence (Bengio et al., 2009).
Moreover, if the student's s confidence is lower (e.g., in the beginning of training), the gradient is relatively stronger. Specifically, consider two students y and q'. Suppose g has lower confidence in ground truth token than e', i.e., q q/. One can consider y as the student during the early stage of training, and q' as the same model at the later stage. We call q weaker student than g'. Consider the ratio the gradients of q and q' of the logit of i-th token:
dL
dI
P:
9i
dz;
dz,
P1
Thus, V; means that the weaker student or the student in the early training stage achieves an additional multiplicative gain on the gradient. V, is called the rescaling factor of a gradient. The higher rescaling factor can accelerate the training of the student.
Next, we consider the token-selective KD proposed in the paper. We consider "soft selection" based on entropy and sigmoid gating function as in Eq. (10) in the paper. The distillation objective has the following form (only forward KL is used):
LAs o (H)()) Ha(p))-Dx<(Plll
where H:(p) is the entropy of token evaluated by and the sigmoid function a enables the "soft-selection" based on the entropy difference. Its gradient is given by
dLis o (Hl((1)) (.))))pp 9.]
(21)
dz,
Note that (-) term is simply weight and is computed with no gradient flow p and q in (-),, i.e., we use "stop-gradin -raddeng...nt".. We can also define the rescaling factor with respect to student models g and (q is the weaker student):
(HH((g) - H.(p)) - qg) o(Hi(g') Hi(p))(p: g))
(22)
From (22), we have that
((Higg) - H((p)) V
V
a(Hi(g') H.(p))
15

---

Published as a conference paper at ICLR 2026
46.0
45.7
Avg. Performance
45.5
Performance (t)
45.0
44
46.5
43.4
43.0
20
80
100
Top-k of logits
Figure 4: Performances using different top-k of Lindirect.
Thus, the token selection has a higher rescaling factor than the conventional KD if 1. We have that
C o(H((g) H((p)) exp(-H<(p) + exp(-H:(dd) o(H.('') H.(p)) exp(-H.(p)) + exp(-11(())
(23)
the weaker student q is less confident than of in token i, has higher uncertainty. Thus, we H2(q) H,',)), which implies that C' 1. Conversely, if the confidence of the student is higher, C' <1 from the perspective that student. Thus, the student will perceive C as being greater than 1 when its confidence is low, e.g., during the early stage of training. Conversely, the student will perceive C' as being less than in the later stage when they have higher confidence. This leads to two effects the student models at different training stages.
(1) Reinforced gradient. In the early stage of training, the student model relatively weak and perceives C as C' > This means that the scaling factor under token selection stronger than the conventional KD. Thus, token selection can reinforce the gradient towards the "best" token. Such reinforced gradient helps establish logical abilities in the early stage, facilitating the convergence of training.
(2) Label smoothing. In the later stage of the training, the student is well-trained and likely to have basic reasoning capabilities. The student will perceive C as C' < 1 compared to the earlier version of itself. This means that the gradient perceived was weaker than the conventional KD. Thus, the student can diversify its reasoning process based already established knowledge. Since the gradient to "best" answer has weakened, we can liken the approach to label smoothing (Müller et al., 2019). It allows more exploration of parameter space for reasoning, which is consistent with student-centric approach.
A.3 ANALYSIS OF TOP-k IN INDIRECT DISTILLATION.
We investigate the model's performance with varying k in the top-k sub-response selection in indirect distillation. As shown in Fig. 4, we observe that performance peaks at k = 10. A smaller number of candidates (e.g., & 5) is insufficient for effective learning. In contrast, performance degrades substantially outside this optimum; for example, increasing k to just 20 causes the score to drop to 44.5.
This result suggests that focused distillation is crucial. Forcing the student to match the teacher's s distribution over a wide range of tokens appears to introduce noise in the learning signal. Therefore, we select & = 10 as the optimal setting for indirect distillation.
A.4 ANALYSTS OF THE TOKEN SELECTION RATIO FOR ENTROPY REGULARIZZTION
We analyze the effect of the token selection ratio the entropy regularization. In this section, we will denote the selection ratio by 8%. As depicted in Fig. the performance varies significantly with
16

---

Published as a conference paper at ICLR 2026
45.7
Avg. Performance
45.5
5.2
Performance (T)
45.0
43.9
44.0
43.5
43.0
42.5
0.4
Selection ratio
Figure 5: Performances with token selection ratio for entropy regularization.
8. With regularization disabled (s = 0%), the model achieves baseline performance of 42.9. Setting 10% improves the performance to the peak value of 45.7.
However, performance steadily declines 8 greater than 20%. At the maximum value of 100%. performance drops to 41.8. Our finding indicates that applying entropy minimization across all tokens can be harmful, perhaps because it causes overconfidence of the model.
The results validate choice of 10% as the optimal setting. This value is also consistent with the observation in (Wang et al., 2025) which estimated the crucial tokens for RL for LLM reasoning to be 10 -00%
A.5 ANALYSIS OF HYPERPARAMTER 3.
We vary 3 in 33) from 0,0 to 1.0 and report the average performance on our benchmarks, As illustrated in Fig. the model's performance highly sensitive the choice of
46.0
Performance
45.7
45.4
45.5
Performance (1)
45.0
44.0
43.4
43.4
43.0
0.2
1.0
Beta
Figure 6: Analysis of 3 of JSD.
When 3 set to O, which is equal forward KI. (Hinton et al., 2015), the model achieves baseline performance of 43.4, and for small values of / up to 0.5, we observe only marginal improvements. However, there is a sharp and significant increase in performance as 3 rises from 0.5 to 0.9, where the model achieves its peak performance of 45.7. This substantial gain underscores the critical role of mode-seeker seek-enekngg in enhancing the model's 's reasoning capabilities. Interestingly, increasing 3 further to 1.0 leads to a slight performance degradation to 45.4, suggesting that excessive weight this term can disrupt the balance with other training objectives. Based on this empirical evidence, we set 0.9.
17

---

Published as a conference paper at ICLR 2026
Method
GSM8K
GSM-Plus
MATH
MBPP
IFEval
MMLU-STEM
Off-Policy
56.0
37.1
24.9
39.8
55.0
49.5
On-Policy
60.1
40.5
26.1
40.1
55.2
50.0
Table 6: Performance comparison of on-policy versus off-policy in Qwen2.5 (14B 1.5B).
Method
GSM8K
GSM-Plus
MATH
MBPP
IFEval
MMLU-STEM
Qwen2.5-14B (Teacher)
80.3
59.7
21.7
78.9
85.9
70.5
Qwen2.5-1.5B (Student)
57.1
16.9
38.4
53.1
49.5
Sequence-Lovel KD
56.5
38.2
16.5
40.6
53.7
48.9
Supervised-KD
56.3
37.8
17.9
41.0
51.4
48.4
DistiLLM
57.2
38.7
18.6
42.1
52.5
47.8
Speculative KD
58.0
39.2
18.5
41.6
52.4
47.8
MiniLLM
$7.7
39.7
17.8
42.2
$4.7
48.2
GKD (3 = 0.9)
57.9
39.9
18.1
41.8
52.3
47.7
(Adaptive c%)
39.8
26.5*
55.8
50.4
Table 7: Performance comparison between distillation methods Qwen2.5 (14B - 1.5B). Best performance in Bold font. (*) mark for the student model outperforming the teacher.
Method
MMLUUProoMath
SciQ
BBH
MuSR
Qwen2.5 14B (Teacher)
77.6
86.4
60.8
60.8
Qwen2.5-1.5B (Student)
34.4
85.5
36.5
38.7
Sequence-Level KD
32.6
87.9
36.6
36.6
Supervised-KD
35.3
86.6
36.4
37.9
DistiLLM
35.2
85.7
36.5
38.7
Speculative KD
34.8
86.7
37.2
37.7
MiniLLM
36.2
84.1
36.0
38.3
GKD (8 = 0.9)
35.7
86.6
36.2
38.5
TSD-KI KD (Adaptive c%)
38.5
93.8
40.0
39.4
Table 8: Performance comparison of more advanced benchmarks in Qwen2.5 (14B - 1.5B). Best performance in Bold font. (*) mark for the student model outperforming the teacher.
A.6 ON-POLICY vs. OFF-POLICY DISTILLATION.
To examine the effects of distribution shift, compare the performance of TSD-KD when trained with on-policy versus off-policy data. The results presented in Table show the superiority of the on-policy framework. The on-policy strategy consistently outperforms its off-policy counterpart across all six benchmarks, achieving an average score of 45.7 versus 43.4, 4, significant margin of +2.3 points. The difference is particularly high complex mathematical reasoning tasks such as GSM8K (+4.1) and GSM-Plus (+3.4). These findings suggest that it is more effective to guide the student model along its own generated reasoning paths.
A.7 ADAPTIVE SELECTION OF c.
The adaptive selection of e is performed as follows.
For each sample æ, we estimate the sample "difficulty" based on the average uncertainty gap. This is consistent with our direct distillation with the uncertainty gap.
C = ( A (pr))
where Ps, Pr denote the student and teacher model, H is the average entropy of each generated sample.
18

---

Published as a conference paper at ICLR 2026
Method
GSM8K
GSM-Plus
MATH
MBPP
IFEVAL
MMLU-STEM
Supervised-Kr KD
55.7
36.3
17.6
39.4
51.9
47.2
DistiLLM
56.7
38.5
18.5
42.0
52.3
47.3
MiniLLM
56.9
38.0
18.6
41.6
52.6
47.9
GKD (3 = 0.9)
56.8
38.4
18.9
41.2
51.8
47.6
TSD-KD
57.2
38.6
18.4
42.7
54.5
49.7
Table 9: Performance comparison of Parameter Efficient Finetuning between different distillation methods Qwen2.5.
The selection ratio e of Opener is determined by the quantile interval of the estimated value of C. We take moving average of C over the window of samples for the estimation of C.
5%.
C E 00.00, 0.25)
10%,
C E 0.25, 0.50)
15%,
C E 00.50, 0.75)
20%,
C E [0.75, 1.00]
Intuitively, larger uncertainty gap C' implies that the current sample is difficult for the student. Thus, we take larger cumulative percentile (larger for supervision.
Algorithm 2 Indirect Distillation by Teacher-Guides Preference Ranking
1: Given: Teacher model pr, Student Model p.g, Prompt dataset X
2: Sample & from X, generate y ~ (-[x)
3: FIND_C(pT, ps)
4: Select the opener of up to top-c% cumulative entropy
5: for each token position in opener do
6: Select top-k logits { [1,,,,.., 2x [k]} by student (1l<cee)))))
7: Re-rank logits get the order of preference ne by teacher pr (-1
8: Compute ( m x. yee)
9: end for
10: Compute - - log (Te Ir,
Function FIND_ c(pr: ps)
12: IF not adaptive
return 10%
14: Calculate an average entropy of generated response, Hps)) ((T))
Compute C
a (ps) (Pr))
5%.
C' e [0.00, 0.25)
10%,
C' e [0.25, 0.50)
16:
15%,
C E [0.50, 0.75)
20%,
C' [0.75, 1.00]
17:
return
18:
End Function
Overall, TSD-KD with adaptive-c achieves achieve shown Table and Interestingly, the adaptive-c method performs better than the default fixed-e method on MATH and SciQ, which challenging reasoning benchmarks. Thus, the adaptive-c method is shown to good at adapting to the difficulty distillation samples. A pseudocode of Indirect Distillation with adaptive-c is shown in Algorithm 2.
performance similar to the default setting, i.e., fixed c, as
A.8 EFFECTIVENESS IN PARAMETER-EIFICEES FINETUNING (PEFT).
To assess the practicality and efficiency our method, we evaluate its performance a more resource-constrained setting using Parameter-Ffiie Efficient Finetuning (PEFT). We employ LoRA,
19

---

Published as a conference paper at ICLR 2026
popular PEFT technique, to update only small fraction of the student model's parameters during distillation.
The results, summarized in Table 9, demonstrate that TSD-KD 3D's superiority is robustly maintained even in the PEFT paradigm. TSD-KD achieves the highest average score of 43.5, clearly outperform- ing all competing distillation methods. This strong overall performance is driven by its dominant results across the majority of the benchmarks, securing the top rank out of 6 tasks, including the challenging reasoning benchmarks GSM8K and MMLU-STEM.
This experiment confirms that the selective learning signal provided by TSD-KD is effective even when updating only partial set of model parameters. This result highlights the efficiency of our paradigm, which concentrates the distillation process on a crucial subset of tokens. Consequently, TSD-KD can be more parameter-efficient fine-tuning (PEFT) approach than prior KD methods that operate on the full token set. This efficiency makes TSD-KD practical and attractive solution for developing capable, small-scale reasoning models under typical computational constraints.
A.9 HYPERPARAMETERS
Hyperparameter
Value
Hatch Size
Learning
Se-6 (Qwen2.5), le-7 (Gemmu2)
Learning rate scheduler
Cosine
Temperature of
1.0
Max Sequence Length
1024
Epochs Optimizer
Adam
Warmup ratio
0.1
top-k of Lindren Selection ratio of Landsea
10
0.1
Selection ratio CEM
Table 10: Detailed hyperparameters.
A.10 BENCHMARK DETAILS
We provide additional details of the benchmarks used in our evaluation:
GSMSK (Cobbe et al., 2021): A benchmark consisting of grade-school arithmetic word problems, widely used assess step-by-step mathematical reasoning. Following common practice, we evaluate models in the 5-shot setting as the default.
GSM-Plus (Li et al., 2024): An extension of GSM8K containing more challenging arithmetic word problems, designed to evaluate model robustness beyond simple mathematical reasoning. Following common practice, we adopt the 5-shot setting as the default.
MATH (Hendrycks et al., 2021b): A benchmark of competition-level mathematics problems spanning algebra, geometry, number theory, and combinatorics. Following common practice, we adopt the 4-shot setting as the default.
MMLU-Pro-MMr Math (Wang al., 2024): professional-level subset of MMLU-Pro focusing on advanced mathematics questions. Following common practice, we adopt the 5-shot setting as the default.
MMLU-STEM (Hendrycks et al., 2021a): A subset of the Massive Multitask Language Understanding benchmark, covering STEM-related subjects such as physics, chemistry, and biology. Following common practice, adopt the 5-shot setting as the default.
ScienceQA (SciQ) (Welbl et al., 2017): science creestion-answering dataset with multiple-choies format, requiring factual and reasoning skills. Following common practice, adopt the O-shot setting as the default.
MBPP (Austin al., 2021): A program synthesis dataset where models generate short Python functions from natural language problem descriptions. Following common practice, we adopt the O-shot setting as the default.
20

---

Published as a conference paper at ICLR 2026
BBH Big-Bench Hard) (Suzgun et al., 2022): A collection of difficult tasks from BIG-Bench that target reasoning and compositional generalization. Following common practice, we adopt the 3-shot setting as the default.
MuSR (Sprague et al., 2024): A benchmark for multi-step soft reasoning in natural language narratives. It is constructed via neurosymbolic synthetic-to-natune generation process, producing complex scenarios such as long-form murder mysteries. Following common practice, we adopt the O-shot setting as the default.
IFEval (Zhou et al., 2023): A benchmark that measures instruction-folloiing capabilities, focusing adherence task constraints. Following common practice, we adopt the O-shot setting as the default.
A.11
MODE COVERING vs. MODE SEEKING
Teacher Distribution
Mode-covering
Mode-seeking
Figure 7: Comparison of mode-covering (forward KL, D(P red) and mode-seeking (reverse KL, D()[IP), blue) behavior. The student (Q) is trained to match the bimodal teacher (P). The mode-seeking distribution fits one mode precisely, while the mode-covering distribution spreads its mass to cover both. Adapted from (Le, 2017).
Figure illustrates the difference between mode-covering (forward KL) and mode-secking (reverse KL) when distilling multi-mdal teacher distribution (P) into unimodal student distribution (Q) with limited capacity. These observations are based and are also available in the previous work (Le, c 2017; Agarwal et al., 2024; Gu et al., 2023).
Mode-secking (Reverse KL, D(Q[[P)) heavily penalizes the student (Q) for placing prob- ability mass where the teacher (P) is zero. This forces the student to focus on fitting high-probability modes of the teacher's distribution.
Mode-covering (Forward KL, D(PIIQ)), in contrast, penalizes the student for not assigning the probability mass where the teacher's distribution is nonzero. This encourages the student to generate distribution that broadly covers the teacher's modes.
A.12 QUALITATIVE ANALYSIS
We present qualitative examples in two distinct domains: mathematical reasoning and code gen- cration (see Fig. and Fig. 9). We make comparisons with strong basclines (Qwen2.5-1.5B) and competing methods (GKD).
Case Task Alignment in Code Generation. In the second example (Fig. 8), the user explicitly requests function that converts string lower case.
Baselines: Both baselines incorrectly interpret the task as verification rather than transforma- tion, generating code that checks the condition (c.g., s is 1 lowex () instead of performing the requested transformation. This results in semantically incorrect outputs.
21

---

Published as a conference paper at ICLR 2026
Ours: Despite the ambiguous context, which can often trigger check-function pattern, model accurately identifies the user's s intent to manipulate the input string. It produces the correct transformation logic (e.g.. lower demonstrating superior alignment with user intent beyond superficial pattern matching.
Case Constraint Satisfaction in Mathematical Reasoning. the first example (Fig. 9), the prompt requires multi-step calculation followed by specific output-format constraint ("in cents").
Baselines: Qwen2.5-1.5t correctly computes the arithmetic (3 2.4 = 0.6) but fails the unit-conversion constraint, outputting "0.6 dollars" instead of converting to cents. GKD, in contrast, exhibits generation collapse and fails to complete the chain of thought.
Our model maintains contextual constraints throughout the reasoning process, suc- cessfully performing the arithmetic and adhering to the formatting requirement, ultimately producing "60 cents."
Ours:
Q: Write python function to convert the given string lower
Qwen2.5-1.5B: import re
def is_lewer (s)
A Check if all characters in the string lowercase return s.islower()
GKD: import re
def is Lower Check if all characters the string are lowercase return s.islower ()
Ours:
import
def is_lower (8):
3 = str.s.rroo lower (s)
return
Figure Qualitative Analysis of Code generation (MBPP).
22

---

Published as a conference paper at ICLR 2026
Q: Trey receives 5% commission on every sale he makes. On the sale of $60 coat (before any discounts), how many more will he receive if his commission is based the original price of the coat rather than the price of the coat after 20% discount?
GT:
Qwen2.5-1.5B:
To determine how much more Trey will receive as a commission the sale based on the original price of the coat rather than the discounted price, we need to calculate the commissions for both scenarios and then find the difference.
First, let's calculate the commission based the original price of the coat: The original price of the coat $60. With 5% commission rate, the commission Trey would receive is:
60 x 0.05 = 3 dollars
Next, let's calculate the commission based the discounted price of the coat: The coat is sold at a 20% discount, so the selling price before tax is:
60 (1 0.20) 60 x 0.80 48 dollars
With commission rate, commission Trey would receive is:
48 x 0.05 2.4 dollars
Now, we find difference between two commissions:
- 2.4 = 0.6 dollars
Since the question asks for the difference
GKD:
To determine how much more Trey will receive as commission when the sale is based on the original price of the coat rather than the discounted price, let's follow these steps:
1. Calculate the original price of the coat before any discount:
The original price of the coat $60.
2. Determine the commission based on the original price:
Trey receives a 5% commission on the original price of the coat. So, the commission from the original price is calculated as follows:
Commission from original price 0.05 60 dollars
Calculate the price of the coat after 20% discount:
A 20% discount means the customer pays 80% of the original price. Thus, the discounted price is: Discounted
price 60 (1 0.20) 60 0.80 48 dollars
4. Determine the commission based on the discounted price: With the discounted price, the commission is
Not complete
Ours:
The original price the coat is 60 dollars. Trey gets a commission on sales. So he gets 60 x .05 3 dollars from the sale.
Now if the price was discounted by 20%. the new price would be 60(1 .2) = 48 dollars. He would get
48 x .05 2.4 dollars from the sale. So gets
2.4 0.6 dollars more when the price not discounted.
cents, this is 0.6 x 100 60 cents.
Therefore, he will receive 60 more cents.
Final Answer: The final answer is 60 cents.
Figure 9: Qualitative Analysis of Math solution (MATH).
23


Reviews:
[{"review_id": "6xPLLjUU9h", "reviewer": "Reviewer_jW9T", "summary": "This paper proposes Token-Selective Dual Knowledge Distillation (TSD-KD), a framework for transferring large language model reasoning abilities to smaller models. The method is designed to provide targeted supervision, focusing on high-uncertainty tokens to mitigate issues with existing KD approaches. Three key components are listed: \n\n1) Token-Selective Indirect Distillation: The teacher provides preference rankings over the student's top-k generated tokens in the initial sequence of reasoning (the \"opener\"), utilizing a Plackett-Luce model.\n2) Token-Selective Direct Distillation: A JSD-based distribution matching loss is applied only to tokens where the student's uncertainty (entropy) significantly exceeds the teacher's confidence (the \"uncertainty gap\").\n3) Token-Selective Entropy Regularization: The entropy of the student's top 10% most uncertain tokens is minimized.", "questions": "At W3", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea of Token-Selective Direct Distillation is well motivated. \n2. Authors provide comprehensive ablation study in demonstrating effects of each component and showed strong empirical results over baselines.", "weaknesses": "W1: Hyperparameter Sensitivity: The framework relies on an extremely sensitive set of hyperparameters ($c$, $k$, $s$, $\\beta$), as demonstrated by sharp performance drop-offs in the appendix analyses. This suggests the method is brittle and lacks practical generalizability. In the Table 1, authors also only report the performance from the best hyperparamter selections. I wonder how much this complex setup could transfer into new domains or tasks.\n\nW2: Conflict Between On-Policy Learning and Entropy Minimization ($\\mathcal{L}_{EM}$): The $\\mathcal{L}_{EM}$ term, which minimizes entropy on the top 10% most uncertain tokens, fundamentally conflicts with the core on-policy principle of preserving and encouraging exploration. While selectivity is claimed as a mitigation, the paper does not analyze the true impact on the student's output diversity or rigorously justify that minimizing entropy is superior to simpler confidence maintenance.\n\nW3: The paper provides insufficient analysis to attribute the performance at Token-Selective Indirect Distillation. It is unclear if the success of the indirect distillation is due to the preference ranking (teacher's subtle knowledge transfer) or simply the top-k token candidate proposal. Based on prior work, latter might be the bigger contribution. I believe authors should perform additional ablation experiments to justify that preference ranking is necessary.\n\nW4: There is a very relevant paper Speculative KD (https://arxiv.org/abs/2410.11325). Authors should consider compare to or mention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gwzRXP1gaM", "reviewer": "Reviewer_BR6w", "summary": "The paper proposes Token-Selective Dual Knowledge Distillation (TSD-KD), a framework designed to efficiently transfer the reasoning abilities of a large teacher model to a smaller student model, aiming to reduce the cost of Chain-of-Thoughts (CoT) generation. Adopting a student-centric, on-policy distillation paradigm, TSD-KD applies supervision only to the most critical or uncertain tokens during the reasoning process, thereby avoiding the distribution mismatch and overwhelming issues associated with traditional Teacher-Forcing. The method integrates three key components, all guided by token selection: Indirect Distillation (teacher acts as a preference ranker for student candidates), Direct Distillation (applying GKD loss to tokens with a large uncertainty gap—student uncertain but teacher certain), and Entropy Regularization (selectively minimizing student entropy on critical tokens).", "questions": "1) Given that the insight \"high-entropy/uncertain tokens contribute more\" is highly similar to recent RL-based works (e.g., Wang et al. (2025) and Lei et al. (2025) as mentioned), how do the core innovative mechanisms of this paper (e.g., the uncertainty gap selection, the Dual Distillation design) demonstrate a theoretical or empirical advantage over the Token Importance mechanisms in the precursor works?\n2) In the context of Knowledge Distillation, what specific advantages—such as increased data efficiency or stability—does selective supervision offer that cannot be achieved or are less efficient using traditional RL frameworks (i.e., penalizing/rewarding only critical tokens via sparse reward signals)?\n3) Indirect Distillation is only applied during the Opener phase. How does the student model maintain reasoning consistency and quality during the subsequent unsupervised phases? If the student selects a path consistent with the teacher's preference during the Opener, to what extent does this restrict its ability to develop new, non-imitative reasoning logic \"in its own words\" in the subsequent steps?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) TSD-KD achieves State-of-the-Art performance across 10 challenging reasoning benchmarks. Experimental results demonstrate its significant superiority over existing baseline methods across multiple tasks.\n2)  The student model, after training, even surpasses its teacher model on some reasoning tasks (with improvements up to 20.3%). This result strongly suggests the framework is not merely imitative but effectively promotes the student model in building its own, more generalizable reasoning logic.", "weaknesses": "1) The core insight of the paper—that \"high-entropy/uncertain tokens are critical branching points in reasoning\" and should be targeted for selective supervision—is not an original discovery. This phenomenon, which guides the model learning process, has been well-established in antecedent works (such as the RL-based methods by Wang et al. (2025) and Lei et al. (2025)). Therefore, the paper's contribution lies primarily in the engineering application and integration of this existing principle into the knowledge distillation domain for selective supervision, rather than a breakthrough in fundamental mechanism discovery or method innovation.\n2) The TSD-KD methodology lacks deep theoretical innovation in distillation, being an effective combination of existing techniques and intuitive heuristic rules. Specifically, the Indirect Distillation employs the established Plackett-Luce (PL) model from RLHF, and Direct Distillation uses the known Generalized JSD (GKD) loss. While the \"uncertainty gap\" token selection mechanism is novel, it functions as an intuitive heuristic rule. Consequently, the paper's main contribution is the effective integration of these existing components, rather than the proposal of a new foundational distillation mechanism or a novel loss function.\n3) The crucial length of the \"Opener\" for selective supervision is defined by an empirical hyperparameter: the c% accumulated entropy threshold (set to c=10% based on ablation studies). This fixed-ratio approach is a heuristic inherited from similar suggestions in other reinforcement learning works. The absence of a dynamic or adaptive mechanism that adjusts this threshold based on the specific complexity and depth of the reasoning task limits the theoretical generalizability of the method, as the optimal empirical value may vary significantly across different domains (e.g., mathematical vs. common-sense reasoning) and model architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GF6hIu3MKC", "reviewer": "Reviewer_9rZk", "summary": "This paper introduces Token-Selective Dual Knowledge Distillation (TSD-KD), a framework for distilling reasoning ability from large language models to smaller student models by focusing on important tokens in the reasoning chain. TSD-KD combines two main innovations: (1) indirect, preference-based distillation, where the teacher re-ranks student-generated token candidates without forcing its output distribution, and (2) direct, gated distillation that selectively applies distribution-matching to tokens where the student is uncertain and the teacher is confident. The approach is regularized with selective entropy minimization on the most uncertain tokens. Empirical evaluation across 10 challenging reasoning benchmarks demonstrates TSD-KD’s strong performance, including cases where the compressed student surpasses its teacher.", "questions": "1. How robust is the preference-based indirect distillation if the student’s top-$k$ candidates are mostly incorrect? Does the framework degrade gracefully if initial reasoning is off-policy, or does performance collapse? Are there analyses on very weak students or pathological candidate proposals?\n2. Do any ablation results suggest redundancy between direct distillation with entropy gating and selective entropy minimization? Are there tasks where one suffices without the other, and can the gains be attributed to only one component in certain domains?\n3. What steps are in place to detect or mitigate biases propagated from teacher to student, considering that only a subset of tokens is distilled but on the student’s own output?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The use of student-generated candidates (preference-based indirect distillation) and selective, entropy-based token gating in direct distillation is thoughtfully motivated and distinguishes the framework from prior “teacher-forcing” approaches. The focus on letting the student “explain in its own words” resonates with cognitive insights and supports the central claim.\n2.The explicit combination of indirect and direct knowledge distillation, each carefully limited to critical tokens, is well-positioned to address known weaknesses of pure distribution-matching or of-point-wise imitation.\n3. The mathematical formulation is transparent, the underlying assumptions are stated, and the algorithmic components are described with appropriate rigor. \n4. TSD-KD consistently outperforms strong baselines, with substantial absolute gains. Importantly, in multiple cases, the student model trained via TSD-KD surpasses the teacher.", "weaknesses": "1. The preference-based indirect distillation encourages the student to align with the teacher’s ranking on top-$k$ student candidates. However, this assumes that the student's beam search is likely to generate candidates close to the correct reasoning trace, which may not hold for weaker students or for highly ambiguous problems. \n2. While tables and figures provide extensive quantitative results, the paper lacks qualitative or error analysis on the types of reasoning improvements the student makes with TSD-KD (beyond aggregate accuracy).\n3. Even though performance improvements are noticeable, the paper does not report any statistical significance tests.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CiCefLSbTf", "reviewer": "Reviewer_DKho", "summary": "This paper proposes Token-Selective Dual Knowledge Distillation (TSD-KD), a student-centric framework for improving reasoning through selective and dual-mode knowledge transfer. The method integrates: Indirect distillation, the student proposes candidate tokens and receives preference ranking feedback from the teacher (similar to DPO-like weak supervision); Direct distillation, selective distribution matching on tokens with large student–teacher uncertainty gaps using JSD, Entropy regularization — confidence enhancement through minimizing entropy of the most uncertain tokens. The authors conduct comprehensive experiments on reasoning benchmarksusing Qwen2.5 and Gemma2 model families. Results show consistent improvements, with the student model occasionally outperforming its teacher.", "questions": "refer to Weaknesses, I will adjust my score according to the responses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Complete and sound framework combining distillation with entropy regularization forms a coherent pipeline.\n\n2. Writing is good.\n\n3. Use of token entropy for identifying important tokens aligns with recent research trends in reasoning-focused LLMs (e.g., 80/20 entropy rule, ARPO).\n\n4. Comprehensive experiments across two model families demonstrate generalizability.", "weaknesses": "1. The core idea of TSD-KD is highly similar to <Keypoint-based Progressive Chain-of-Thought Distillation (icml 2024)> in motivation. Both approaches emphasize selective token weighting and distillation; TSD-KD replaces KPOD’s mask-learning with entropy-based selection but retains the same underlying philosophy. However, this previous work is completely neglected. \n\n\n2. At this time, it is unclear why the authors did not conduct experiments on the Qwen3 family, such as Qwen3-8B, which has become the de-facto principle for reasoning evaluation (like in <Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning>.). The paper only reports results on Qwen2.5 and Gemma2, both of which are now relatively outdated and substantially weaker in reasoning capability. Since the proposed method explicitly targets reasoning enhancement, it is essential to verify its effectiveness on more competitive and up-to-date models.\n\n3. Limited theoretical insight are provided in this paper. For example, the paper lacks a clear theoretical justification for why entropy-based token selection truly improves reasoning robustness beyond serving as a heuristic importance measure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)