Knowledge Distillation for Large Language Models through Residual Learning
Authors: Thinh On
Hengzhi Pei
Leonard Lausen
George Karypis
Venue: ICLR 2026 Poster (2026)
Keywords: knowledge distillation; large language models; residual learning; mixture-of-experts; cross-tokenizer knowledge distillation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: The paper introduces a two-stage LLM distillation method that leverages a novel residual learning approach, enabling students to learn from teacher mistakes, with strong results even when teacher and student use different tokenizers.

Abstract: Knowledge distillation has become a crucial technique to transfer the capacities of large language models (LLMs) to smaller, more efficient models for practical deployment. While recent work exploits rich information from intermediate states of the teacher model for more effective knowledge transfer, imperfect knowledge from the teacher can also mislead student learning, restricting the student’s generalization capacity. In this work, we propose a two-stage distillation framework that is effective for diverse knowledge distillation scenarios. In the first stage, we pretrain projectors to extract and compress teacher knowledge into a low-dimensional vector space via self-reconstruction. In the second stage, we perform distillation with a hybrid objective that combines learning from the compressed teacher representations with standard supervised fine-tuning on ground-truth data. Our key innovation is residual learning for LLM distillation, where the student learns to make predictions based on the differential between its representations and projected states from the teacher. This approach encourages the student to further improve its representations beyond potentially erroneous teacher knowledge. For Mixture-of-Experts (MoE) teacher models, we further fuse the experts’ outputs using a self-attention mechanism for better utilizing the teacher knowledge. Moreover, to support the cross-tokenizer distillation setting, where the teacher and student models have different vocabularies, we adopt a cross-model attention mechanism that eliminates the need for explicit token alignment rules. Experimental results show the superior performance of our proposed framework under both same- and cross-tokenizer settings, demonstrating the effectiveness in preserving teacher knowledge and improving student generalization capability.

Published as a conference paper at ICLR 2026
KNOWLEDGE DISTILLATION FOR LARGE LANGUAGE MODELS THROUGH RESIDUAL LEARNING
Thinh On Hengzhi Pei', Leonard Lausen George Karypis'
Amazon Web Services, "New Jersey Institute of Technology
ABSTRACT
Knowledge distillation has become a crucial technique to transfer the capacities of large language models (LLMs) smaller, more efficient models for practical deployment. While recent work exploits rich information from intermediate states of the teacher model for more effective knowledge transfer, imperfect knowledge from the teacher can also mislead student learning, restricting the student's 's gen- eralization capacity. this work, we propose a two-stage distillation framework that is effective for diverse knowledge distillation scenarios. In the first stage, we pretrain projectors to extract and compress teacher knowledge into a low- dimensional vector space via reff-reconstruction. In the second stage, we perform distillation with hybrid objective that combines learning from the compressed teacher representations with standard supervised fine-tuning ground-truth data. Our key innovation residual learning for LLM distillation, where the student learns to make predictions based on the differential between its representations and projected states from the teacher. This approach encourages the student to fur- ther improve its representations beyond potentially erroneous teacher knowledge. For Mixture-of-Expee puts using a self-attention mechanism for better utilizing the teacher knowledge. Moreover, to support the cross-tokenizer distillation setting, where the teacher and student models have different vocabularies, we adopt cross-model attention mechanism that eliminates the need for explicit token alignment rules. Experi- mental results show the superior performance of our proposed framework under both same- and cross-tokenizer settings, demonstrating the effectiveness in pre- serving teacher knowledge and improving student generalization capability.
(MoE) teacher models, we further fuse the experts' out-
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable capabilities across diverse natural language tasks, from complex reasoning and mathematical problem solving creative writing and code generation (Chowdhery et al.) 2023 Grattafiori et al.) 2024 Comanici et al. 2025 Yang al.) 2025). However, deploying large models with billions of parameters requires substantial computa- tional power, which limits their adoption in resource-constrained environments, To this end, knowl- edge distillation (KD) (Buciluà et al. 2006 Hinton et al., 2015 Sanh et al., 2019) has emerged as a promising paradigm address this challenge by transferring knowledge from large teacher models to smaller, more efficient student models that could preserve most of the teacher capability.
Knowledge distillation methods can be broadly categorized into black-box and white-box ap- proaches. Under black-box KD, the student model learns from training data generated by the teacher model through standard supervised fine-tuning (Kim & Rush 2016 Taori et al. 2023 Chiang al.) 2023 Xu et al. (003). Despite its simplicity and broad applicability, black-box KD fails to utilize the rich knowledge embedded in the intermediate representations of the teacher model (Gu al.)
2023 Wen et al. 2023 Ko et al. (2024). Therefore. white-box KD methods aims to further improve student model training by leveraging more information from the teacher, including logit distributions
(Sanh et al. 2019
Liang et al. 2023) and internal hidden states (Sun et al. 2019 Jiao et al. 2019
Work done Amazon Weh Services. Corresponding author: to58@n edu

---

Published as a conference paper at ICLR 2026
Hou et al. 2020 Zuo et al. 2022 Shen et al. 2025 However, applying white-box KD becomes particularly challenging when the teacher and student models use different tokenizers, as token mis- alignment makes direct knowledge transfer between sequence representations infeasible. To address these challenges, recent work has explored white-box KD for cross-tokenizer scenarios, where not only the model structures but also the vocabularies are different between the teacher and student
models (Boizard et al. 2024 Zhang et al.. 2025b al. 2025).
Chen et al. 2025:
Cui et al. 2025 Minixhofer
Despite their success, recent work has mainly focused on divergence-based methods that optimize output distribution matching (Xu al. (024) without considering scenarios where teacher predic- tions are incorrect. In this case, the teacher can transfer detrimental bias to student, resulting in poor performance and generalization capacity of the student model (Zhang et al. 2025a). Moreover, teacher hacking (Tiapkin et al., 2025) could be another concerning issue, where the student model learns to mimic superficial patterns rather than acquiring meaningful knowledge. These limitations are further compounded by more challenging knowledge distillation settings when teacher and stu- dent models are significantly different in the model architectures, e.g. knowledge distillation from a Mixture-of of offeurrrtt-- (MoE) model to dense model.
To this end, we propose a novel white-box KD framework to address the above challenges. Our approach consists of two stages: pretraining and distillation. In the pretraining stage, we com- press teacher hidden states into a low-dimensional space and train the teacher projectors via a self-reconstruccioon mechanism, which extracts task-relevant information from the teacher model. In the distillation stage, we introduce a novel mechanism called residual learning, in which the student learns predict next tokens using the residual hidden states computed by subtracting pro- jected teacher hidden states from those of student whenever the teacher's predictions are inaccu- rate. This mechanism encourages the student to learn complementary knowledge that captures what it understands differently from the teacher and avoid replicating teacher errors. To support the cross-tokenizer distillation, we adopt a cross-model attention mechanism, inspired by /Zhang et al. (2025)), that automatically establishes the alignments between teacher and student token representa- tions through hidden state similarity. Finally, to better exploit the rich knowledge distributed across experts in MoE teacher model, we introduce lightweight scaled dot-production self-attention mechanism that enriches expert outputs before top-k aggregation. These components form a unified framework which is effective for distillation across diverse architectures and tokenization schemes.
We evaluate our framework on standard cnntuuction-oollwiinn benchmarks following recent works (Gu et al., 2023 Zhang et al. 20255 Chen et al. 2025 under different distillation settings including cross-tokenizer distillation and distillation from MoE to dense models. Experimental results demon- strate that method consistently outperforms the existing white-box KD approaches average Rouge-L scores for different scenarios.
In summary, our contributions are as follows:
We introduce two-stage knowledge distillation framework that effectively extracts teacher knowledge for fine-tuning a student model.
We introduce residual learning approach that effectively leverages teacher's 's hidden states and errors as learning signals, enabling student model to generalize better.
We adopt cross-model attention mechanism that aligns teacher and student tokens based on their hidden state similarity to handle token mismatches in cross-tokenizer distillation.
We enhance knowledge extraction for MoE teachers by using scaled dot dot-product attention mechanism, fully leveraging all experts' knowledge.
Experiments show that our method significantly outperforms the existing white-box meth- ods under diverse distillation settings.
2
RELATED WORK
Knowledge Distillation for LLMs. A straightforward method for knowledge distillation is black- box KD (Kim& & Rush 2016). Let T and S represent the teacher and student models, respectively. Black-box KI fine-tunes the student model using the standard causal language modeling objective
2

---

Published as a conference paper at ICLR 2026
on teacher-generated texts:
LD = Exxx yy |- 1ss (y<()
(I)
where y is the output sequenee generated by teacher model, Pr and Ps are the output probability distributions from the teacher and student, respectively. Although simple, black-box KD has proven
to be an effective method in diverse tasks (Wang et al. 2023; Taori et al. 2023 Chiang et al. 2023).
Recent research has shifted to white-box KD to leverage the intermediate information from LLMs beyond their final outputs. Specifically, these methods attempt to align output distributions or max-
imize the similarity between the hidden states of the teacher and student models (Sanh et al. 2019
Liang et al. 2023 Sun et al., 2019 Zuo et al. 2022 Shen et al. 2025). One common approach is minimizing the divergence between the output probability distributions of the teacher and student:
Ldivergence B(x,y)xxxx/ [DKI. (PP((l)) [y(())))]
(2)
where the data point (x.y) are sampled from the data space X' x 3 and the closed-form expres-
sion of DKL is determined based on the specific divergence metric adopted. Popular choices for
DKL include Dallbackk-eibbee (KL) divergence, reverse KL (Gu et al. 2023). Jensen- -ensen-Shannon di-
vergence (Agarwal et al. 2024). and their skew variants (Lee) 2001 Ko al. 2024 2025).
While proven to be effective, divergence-basse based methods can potentially limit the student's learning capacity. The information from the teacher model not perfect as can still make wrong predictions and exhibit certain bias (Zhang et al. 2025a). As a result, forcing the student to mimic the teacher's distributions may propagate such imperfection to the student model and prevent the student from learning beyond the teacher's knowledge boundaries.
Cross-Tokes Tokenizee Knowledge Distillation. One restriction white-box KD that it requires the same vocabulary between the student and the teacher for distribution alignment. Recent work has further explored white-box KD for cross-tokenizer distillation settings, where the student and teacher have different vocabularies. Cross-tokenizer KD setting poses more challenges than conventional same-tokenizer settings, including mismatches in sequence length and output dimensionality. To tackle these challenges, ULD (Boizard et al.] (024) applies simple truncation strategies align sequence lengths and logit dimensions, enabling direct comparison of output distributions. Based on ULD, MultiLevelOT (Cui et al. 2025) integrates
approximate optimal transport objective to reduce the divergence at both token and sequence levels. Dual-space KD (DSKD) (Zhang et al.] 20256) projects the student's 's hidden states into the teacher space and vice versa for calculating the divergence between output distributions. It further introduces cross-model attention mechanism to align tokens across disparate tokenizers using token embeddings similarity. ALM (Minixhofer et al.) 2025) further refines token alignment by exhaustively enumerating decoded chunks and identifying equivalent segments between teacher and student outputs.
Despite these advances, existing cross-tokenizer KD methods exhibit several inherent limitations. Fruncation-based strategies such as ULD may discard valuable information and suffer from align- ment errors. MultiLevelOT improves performance but incurs computational overhead due to optimal transport, limiting its scalability. Projection- based approaches like DSKD rely on KL divergence on the output distributions, which could suffer from the limitations discussed above. Brute-force meth- ods such as ALM achieve precise alignment but scale poorly with large vocabularies or extended sequences. Moreover, none of these state-of-the-ee 2025) or mitigate teacher bias (Zhang et al. 2025a),, which could restrict the generalization capacity of the student when the teacher itsell imperfect.
methods address teacher hacking (Tiapkin al.,
Distillation from Citture-ofEEpperte Despite the superior capacity of Mixture-of-Expert (MoE) models on various natural language tasks, few works have specifically explored how to fully distill their inherent power (Salinas et al.) 2022; Kim et al. 2025). While Kim et al. (2025) highlighted the importance of incorporating knowledge from all experts during distillation and proposed two mechanisms for expert knowledge extraction, their approaches face practical limitations. The first approach relies on stochastic expert sampling, which requires multiple forward passes to access different expert combinations and increases computational overhead during training. The second approach attempts to adjust router probabilities based on student preferences, but this may result in suboptimal expert selection it prioritizes the student's current state over the teacher's 's expertise. These limitations motivate the need for a more efficient and effective method to distill knowledge from MoE teachers while preserving the specialized capabilities of individual experts.
3

---

Published as a conference paper at ICLR 2026
play basketball
token
22
Teacher
h
p7 -
h
pA-T
h?
P( softmax (WTh?')
(a) Stage I: Pretraining teacher projectors. The projectors pT and pA-+ are learned by optimizing next-
token prediction objective using the reconstructed hidden states.
Teacher a
h?
A PT g
h T-AA
PA-S
h 17 A)-A8
play basket ball
token 2j
Student
h
h h -88h (T-4)-+8
mmak
P 22)) softmax (w-S.."
(b) Stage 2: Residual learning objective. We first compute the residual hidden states hf then this quantity to obtain the adjusted output prohability distribution TP (x:+1/xr.) for next-token prediction.
Figure I: Overview of our proposed two-stage framework. Modules marked with fire icons have learnable parameters that are updated during training, while those marked with snow icons have frozen weights.
3 METHODOLOGY
In this section, we introduce two-stage white-box KD framework that is applicable various KD settings to address the existing limitations. Figure [ illustrates the overall workflow our framework. In the first stage, employ learnable projectors to compress teacher hidden states into a low-dimensional space that agnostic to any student architecture and the projectors are optimized using relfsreconstructiion mechanism shown in Figure ((a) desserbs the details of the first stage. Figure [(b) illustrates the workflow the second stage which performs knowledge distillation from the teacher the model. We introduce novel mechanism called residual learning in Section 3.2 that leverages the residual hidden states calculated from the student hidden states and projected teacher hidden states effectively guide student learning. Furthermore, we present the expert knowledge fusion mechanism in Section 3.3) which utilizes the knowledge from all experts to enhance distillation from MoE teachers. Finally, we introduce a cross-model attention mechanism in Section 3.4 aiming at handling sequence mismatch in cross-tokenizer KD settings. The total objective function for distillation is given in Section 3.5
describes
3.1 PRETRAINING PROJECTORS WITH SELF-RECCOSTTRUCTION
The pretraining stage learns compress teacher representations while preserving their semantics and task-relevant information. We denote A as low-dimensional space whose dimension is d and the hidden state dimensions of the student and teacher ds and dr, respectively. Let PT-A and PA-T denote the projection functions that map hidden states between T and A. As shown in Figure I(a) these projectors form an autoencoder in the self-reconstruccion process, where teacher hidden states are compressed into A, and then reconstructed. The reconstructed hidden states h subsequently used for next-token prediction, and the projectors are optimized via the cross- entropy objective:
LCE -
-
Σ log (()141)0x) with P ((041/tt)) = sof tmax (W'h?')
(3)
4

---

Published as a conference paper at ICLR 2026
where W is the teacher's prediction head. We denote h h as the compressed teacher hidden states, Le., h = p7 PT+Ah? In the subsequent stage, PT-+A remains frozen to stabilize teacher information during distillation.
3.2 RESIDUAL LEARNING
Recent work highlights two key challenges in distillation: teacher hacking, where the student ploits superficial patterns in the teacher's outputs rather than learning meaningful knowledge Tiap- kin al. 2025), and teacher bias, where prediction errors directly transferred the student. To address these challenges, we propose residual learning for L.M distillation, which leverages not only the projected hidden states from the teacher but also the teacher errors informative signals to guide student learning.
Let h² (T-A)+ denote the projected teacher hidden states of token that mapped the student space S, i... h! (TT-A)-S = pA-Sh?7A We compute the residual hidden states by taking the difference between the student hidden states and the projected teacher representations in the student space at the positions where the teacher makes incorrect predictions:
h 3hg (T (T 1 [arg max 7 (1 xi)
(4)
where h represents the residual hidden states, 8 is scaling factor, and the indicator function equals only when the teacher's top-1 prediction is different from the ground-truth token. The scaling factor 3 is crucial for balancing the contribution of teacher and student in the residual term. If 33 is too large, student representations are overwhelmed by those of teachers, making the residuals ineffective. In contrast, if 8 is too small, the residual terms approximate the student hidden states, and the benefits of residual learning would diminish. Therefore, we adaptively compute proper B values according to the dimensions and magnitudes of the hidden states below:
ds
ns
(5)
dA
rescale dimension
magnitude
where ng stands for the student's sequence length. The first component in normalizes for di- mensional differences between spaces, while the second term aligns the magnitudes of teacher and student representations to ensure balanced residual computation. These ratios prevent either the stu- dent or teacher representations from dominating the residual calculation. We empirically observe that using 33 at sequence level (average of token-wise norm ratios) can stabilize training and yield better results.
Then, we train the student model to make accurate predictions based on the residual hidden states using the student's LM head W with the cross-entropy loss:
Lis Σ log (#1+1)=,), where P (24+110)) softmax (w-Si²)
This objective fundamentally changes the learning dynamics: instead of passively replicating the teacher's distributions, the student learns actively identify the differences between its current understanding and the teacher's knowledge. This prevents the superficial pattern matching that characterizes teacher imperfection, ultimately leading better generalization of the student model.
3.3 EXPERT KNOWLEDGE FUSION
For teacher models with MoE structures, propose efficient expert knowledge fusion mecha- nism to effectively utilize the knowledge of all experts for better knowledge distillation. Given É
experts in the teacher model, we denote h (m) as the outputs of the m-th expert the token position
for the last MoE layer. We enrich the expert outputs through scaled dot-product attention:
h (m)
(h. (1)
h mm)
Σ
Omy Sm)h
where
amj sof
Vdr

---

Published as a conference paper at ICLR 2026
where Omj captures the relevance between the outputs of the m-th and j-th experts. This self- attention mechanism allows each expert to incorporate complementary information from similar experts, resulting in informative representations. Finally, the enriched expert outputs ag- gregated to form the final states following the original MoE workflow:
h
Σ jEtop-k
9
where 9j denotes the router probability assigned to the j-th expert, and top-k denotes the k experts selected by the router. Unlike previous methods that rely on stochastic sampling or router probability adjustment (Kim et al., 2025). our attention-basee fusion can directly leverage the complementary knowledge of all experts in a single forward pass.
3.4
CROSS-MODEL ATTENTION MECHANISM
In Section 3.2 we introduce our residual learning design in the same-tokenizer setting where the input sequences for the teacher and student are exactly the same. In the cross-tokenizer setting, since teacher and student have different vocabularies, their tokenizers will output sequences of different lengths for the same text input. This hinders the application of residual learning.
To address this challenge, we propose cross-model attention mechanism align the compressed teacher hidden states h for the input tokens of the student. We first map the student hidden states h to the same low-dimensional space A (denoted as hş-A) via a trainable projector: Then, for each pair of student token and teacher token j. we compute their semantic similarity via dot- products using the normalized hidden states, followed by row-wise sof tmax normalization:
h T std hh3 44)
A Vda
Aij std ( AA)
then A softmax
By computing similarity scores between all token pairs, construct cross-model attention matrix A E _"cx*7,, where ny and are the sequence lengths for student and teacher respectively, Each element Aij represents the semantic similarity between student token i and teacher token j. Using the cross- model attention matrix A, we calculate the aligned teacher hidden states hi? h for cach student token by taking weighted sum of the compressed teacher hidden states:
T-A h
Ar,h-4
(10)
j=1
This mechanism allows each student token attend to the most semantically relevant teacher tokens, effectively bridging the tokenization gap and preserving semantic information. Then, we can use 6? 67-A instead to perform residual learning under the cross-tokenizer settings.
3.5 TRAINING OBJECTIVE
We incorporate the residual learning objective from Eqn. 6 with a standard supervised fine-tuning (SFT) objective where the student model learns to predict the next token from the ground-truth data. The final loss function is:
L XLres (1 L
(11)
where À is hyperparameter that balances the contribution of each loss component.
4 EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Models. We evaluate our framework under both the same-tokenizer and cross-tokenizer settings. For the same-tokenizer setting, conduct experiments knowledge distillation from Mixtral-8x2 xB Instruct (Jiang et al.) 2024) to Mistral-7B and distillation from LLaMA2-7B (Touvron et al.) 2023 to
L /Zhang al. 2024). To optimize I.LMMA-7B'ss permMA-- performance teacher model,

---

Published as a conference paper at ICLR 2026
further fine-tune it on the training dataset before knowledge distillation. For the cross-tokenizer setting, we use Mixtral-8 x 77B-Instruct as the teacher model and use TinyLLaMA- 1.1MMA-BBBBBBB and GPT2-
120M (Radford et al. 2019) the student models. We perform full fine-tuning GPT2-120M, and
LoRA fine-tuning on TinyLLaMA-I. Ma---.1B and Mistral-7B. In this way, our evaluation covers diverse knowledge distillation settings in practice.
Datasets. For training, we use the Dolly dataset (Ouyang et al. 2022). processed by Gu et al. (2023)
which comprises approximately 11,000 training samples and 1,000 validation samples. We evalu-
ate our method on five standard instruction-following benchmarks: Dolly (500 test samples), Self- Instruction (Wang al., 2023), Vicuna-Eval (Chiang et al.
2023). Super-Natural Instructions (Wang
et al., 2022), and Unnatural Instruction (Honovich et al. the primary metric across all datasets.
2023). We adopt Rouge-L scores (%) as
Baselines. We select the following approaches baselines: supervised fine-tuning (SFT), Univer- sal Logits Distillation (ULD, Boizard al. (2024)), Multi-Leves Optimal Transport (MultiLevelOOT, Cui et al. ),025)) Dual-space KD (DSKD, (Zhang et al. 2025bb), and Approximate Likelihood Matching (ALM, (Minixhofer et al. 2025)). Although these baselines were specifically designed for cross-tokenizer KD, they are also capable of achieving state-of-the-ar performance in the same- tokenizer setting, except for ALM which can only be applied for cross-torn tokenizer KD. We addition- ally include ABKD (Wang et al., 2025), which leverages -----divereence to improve distillation, as a same-tokenizer baseline.
Implementation Details. We conduct experiments using PyTorch Distributed on NVIDIA A100 GPUs (40GB). We use bfloat16 precision and gradient accumulations to manage memory con- straints. The dimension of the space A is set as d 64 across all experiments, and all projector modules are linear weights without bias. Teacher projectors are pretrained for 10 epochs as de- scribed in Section [.1] and we use the checkpoint at the final step. For final evaluation, we select the checkpoint with the best validation Rouge-L score during distillation. The detailed hyperparameters can be found in Appendix
4.2 RESULTS
Same-tokenizer KD results. Table I reports the Rouge-L scores for same-tokenizer KD across multiple benchmarks. For Siitrral-877--nssttre + Mistral-7B, our method achieves the highest average score (30.68), which matches the teacher zero-shot performance (30.67) and significantly outperforms the best baseline MultiLevelOT (29.16). For LLaMA2-7B - TinLLLMM-II1B our method surpasses all baselines. reaching an average score of 25.17. 17, which is +0.62 above the best baseline (DSKD). These results demonstrate the effectiveness of our proposed framework in im- proving student's s generalization, potentially matching its teacher's performance.
Crossetokenizer KD results. Table 2 presents the results for cross-tokenizer KD settings. For Mixtral-8x x 7B-Instruct - TinyLLaMA-I, our method again achieves the best average perfor- mance (25.09), outperforming the best baseline DSKD (23.89) by +1.20. For Mixtral-8 7B- Instruct - GGGPP-12M0 our approach reaches 20.01 average Rouge-L, significantly outperform- ing all baselines. We also note that the performance gap between the teacher and the fine-ture tine-tuned student is larger in the cross-tokenizer settings, which is due to the the compounded difficulty of cross-tokenizer alignment and smaller student model size.
In summary, for both same- and cross- toos--toeenizer KD, our method outperforms the existing baselines. The improvements are robust across datasets and teacher-student pairs, highlighting the effectiveness of our approach in preserving and transferring knowledge.
4.3 ABLATION STUDY
To understand the effectiveness of each design choice in our framework, we conduct comprehensive ablation studies under the cross-tokenizer setting where use Mixtral-8 7 the teacher and GPT2- 120M as the student.
Effectiveness of each technique. We consider the following variants of our method: 1) "w/o S", in which we remove 3 for computing the residual hidden states; 2) "w/o accuracy mask", in which we remove the indicator function for computing the residual hidden states; 3) "w/o pretrained pT-AA
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison using Rouge-L, scores (%) for same-tokenizer KD. We report the average scores over 3 random seeds. "Avg." is the average score across five instruction- foolowiinn benchmarks.
Methods
Dolly
SelfInst
VicunaEval
S-NI
UnNI
Avg.
Mixtral-8 IB-Instruct + Mistral-7B
Teacher zero-shot
25.55
24.12
26.62
40.84
36.21
30.67
Student SFT
25.85
22.17
17.18
33.41
30.26
25.77
ULD MultiLevelOT
28.64
21.99
21.31
35.71
34.42
28.41
30.55
23.07
20.62
34.79
36.78
29.16
DSKD
25.35
20.43
18.27
35.39
31.45
26.18
ABKD (a = 8 0.5)
29.53
24.11
21.80
37.31
36.56
29.86
Ours
31.39
24.27
20.48
38.82
38.42
30.68
LLaMA2-70 7B -> TinyLLLMAA-La
Teacher SFT
29.82
20.65
19.18
30.66
33.11
26.68
Student SFT
24.26
16.41
16.13
25.70
27.41
21.98
ULD
26.13
17.52
16.82
28.56
29.34
23.67
MultiLeveIOT
24.51
15.75
16.25
25.72
26.00
21.65
DSKD
26.66
18.05
17.61
30.11
30.30
24.55
ABKD (a 3 0.5)
26.57
17.18
16.90
30.25
30.93
24.37
Ours
26.10
18.11
17.97
31.13
32.54
25.17
Table 2: Performance comparison using Rouge-L scores (%) for cross-tokenizer KD. We report the average scores over 3 random seeds.
Methods
Dolly
SelfInst
VicunaEval
S-NI
UnNI
Avg.
Teacher zero-shot
25.55
24.12
26.62
40.84
36.21
30.67
Mixtral-8 x In-Inttrcct TinyLLaMA-I.II
Student SFT
24.26
16.41
16.13
25.70
27.41
21.98
ULD
23.61
15.45
17.06
29.01
28.45
22.71
MultiLevelOT
22.52
14.48
14.97
27.06
25.77
20.96
ALM
21.93
14.11
15.37
25.75
25.49
20.53
DSKD
26.16
18.02
18.15
27.06
30.08
23.89
Ours
25.84
19.38
17.35
30.86
32.04
25.09
Mixtral-8 8 IB-Instruct -> GPT2- 120M
Student SFT
21.45
9.45
14.89
16.86
19.14
16.36
ULD
22.12
10.54
15.03
18.16
20.12
17.19
MultiLevelOT
21.40
9.57
15.56
15.88
18.05
16.09
ALM
21.16
9.77
15.46
16.09
18.26
16.15
DSKD
23.11
11.20
15.81
20.33
21.56
18.40
Ours
22.62
11.66
15.12
24.51
26.15
20.01
in which we remove the pretraining stage and train PT-A directly during distillation; 4) "w/o MoE expert fusion"", in which we remove our expert knowledge fusion design. We further consider another variant to leverage the knowledge all experts which simply activates and aggregates all the experts, i.e. using top-k as top-8 for Mixtral-8 IB-Intruct. Table compares the results of our method with these variants. We can find that all these variants perform significantly worse than the original method, which demonstrates the effectiveness of each design. Among them, removing the coefficient 3 leads to the most significant performance drop, highlighting its role balancing the teacher-student contributions within residual learning.
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study results for different variants under the knowledge distillation from Mixtral-
8 8 x -BBItrrttuct (teacher) to GPT2-120M (student).
Description
Dolly
SelfInst
VicunaEval
S-NI
UnNI
Avg.
Mixtral-8x 7B- Instruct - GPT2-120M
Ours
22.62
11.66
15.12
24.51
26.15
20.01
w/o |
22.31
9.59
14.43
16.01
18.93
16.25
w/o accuracy mask
22.63
11.04
14.79
24.02
25,45
19.59
w/o pretraining PT-A
21.52
10.07
15.24
18.81
21.31
17.39
w/o freezing pretrained pT-A
22.33
10.46
15.39
23.21
23.98
19.07
w/o MoE expert fusion
sparse activation (2/8 experts)
22.68
11.84
14.82
23.36
25.61
19.66
dense activation (8/8 experts)
23.09
10.52
16.10
23.88
24.82
19.68
average pooling (8/8 experts)
22.90
11.17
13.52
21.11
23.43
18.43
20.01
Avg. Rouge-L, scure
20
19.70
20
19.52
19.60
19.46
19
18.74
19
18.40
18-
18
18:02
32
64
256
$12
768
1024
DSKD
DSKD + Cres
(a) Varying d
(b) DSKD Residual learning
Figure 2: Cross-tokenizer distillation results on (a) varying the dimensionality of the space A (b)
incorporating residual learning into DSKD.
Impact of the low-dimensional space. Under the cross-tokenizer settings, the low-dimensional space A is used to compress teacher hidden states and perform cross-model attention for alignment. Here, we further study the impact of the dimension d, on the distillation performance. Figure 2(a) shows the distillation performance when varying d'A. We find that increasing d does not provide additional benefit and a very large dimension (e.g. d = 768 or 1024) can even significantly harm the performance. It proves that compressing teacher hidden states into a low dimension is not only beneficial but essential for effective knowledge transfer. In our case, d = 64 yields the best result (20.01), indicating a balance between compression and expressivity.
Generality of residual learning. To demonstrate the effectiveness residual learning, we further apply our design into other white-box KD methods. Figure 2(b) compares the performance of in- corporating the residual loss Lyes with the baseline DSKD. We see an improvement of +1.20 on average after incorporating residual learning. demonstrates the generality of our residual learning mechanism, which has the potential to improve the performance of other white-bo box KD methods.
5 CONCLUSION
In this work, we introduce novel two-stage KD framework that addresses key challenges arising from the imperfect teacher knowledge white-box KD settings. We compress the teacher hid- den states into low-dimension space prior to distillation and guide student training through resid- ual learning, which ensures robust knowledge transfer and effectively mitigates overfitting to the teacher's imperfection. Our framework further provides targeted designs for diverse KD settings in-

---

Published as a conference paper at ICLR 2026
cluding cross-tokenizer distillation and distillation from an MoE model. Extensive experiments multiple instruction-followee benchmarks demonstrate that our method consistently outperforms the existing baselines and significantly improves the capacity of the student model under different settings. In the future, we will further investigate if our method can generalize to other tasks such as ceasoning-intensive tasks and code generation.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. The twelfth international conference on learning representations, 2024.
Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. Towards cross-tokenizer distillation: the universal logit distillation loss for llms, arXiv preprint 1rXXi:2402.120300 2024.
Cristian Buciluä, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1355-41,770 2006.
Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. Enhancing cross-tokenizer knowledge distillation with contextual dynamical mapping. arXiv preprint arXiv:2502. 1004, 2025.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, and Houqiang Li. Multi-level optimal transport for universal cross-toke cross-tokenizer knowledge distillation on language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 23724-23732, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. arXiv preprint arXiv:2306.00 0854000003 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 150iv 2015.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tun- ing language models with (almost) no human labor. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2023. Association for Computational Linguistics. doi: 10.18653/ a/2002.acllooog..20 URL https: ac1 antho ogy wrw.yorgg/// 2023. ac -lllllloogee 806/
Toronto, Canada, July
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
10

---

Published as a conference paper at ICLR 2026
Albert Q Jiaag,, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:240.004088 2024.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint urXiv:1909 10351, 2019.
Gyeongman Kim, Gyouk Chu, and Eunho Yang. Every expert matters: Towards effective knowledge distillation for mixture-of cottuur--ff-eppertt expeeeeets language models. arXiv preprint arXiv:2502. 1294,,
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1317-1327, 2016.
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Ye-Yo Yun. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898 2024.
Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se- Young Yun. Distillm-2: A contrastive approach boosts the distillation of llms. arXiv preprint SEIV::2003.0066,, 2025.
Lillian Lee. On the effectiveness of the skew divergence for statistical language analysis. In Inter- national workshop on artificial intelligence and statistics, pp. 176-183. PMLR, 2001.
Chen Liang. Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference Machine Learning, pp. 20852-2222222 20867. PMLR, 2023.
Benjamin Minixhofer, Ivan Vulié, and Edoardo Maria Ponti. Universal cross-tokenizer distillation via approximate likelihood matching, arXiv preprint arXiv:2503. 20088, 2025.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. CtLp:ttss://eett http://aaxvvvvv. org/ 2253 02155
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Felipe Cruz Salinas, Kenichi Kumatani, Robert Gmyr, Linquan Liu, and Yu Shi. Knowledge dis- tillation for mixture experts models in speech recognition. Technical report, Technical Report Microsoft Research, May 2022. https://www. 2022.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 19XX::1100001 108, 2019.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing ohain-of-though into continuous space via self-distillation. arXiv preprint 2rX1v:2502.21074, 2025.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 1908005500 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followng Ilama model, 2023.
Daniil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Ramé, and Mathieu Blondel. On teacher hacking in language model distillation. arXiv preprint 2wXiv:2502.02671 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint urXiv:2307.09288. 2023.

---

Published as a conference paper at ICLR 2026
Guanghui Wang, Zhiyong Yang. Zitai Wang, Shi Wang, Qianqian Xu, and Qingming Huang. ABKD: Pursuing a proper allocation of the probability mass in knowledge distillation via a-3- divergence. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 65167-65212. PMLR, 13-19 Jul 2025. URL https: http::///xrccecings..o .ml.. press/ v267 www.wwangg....... html
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2:2, 2022.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484- 13508, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/ 122202..ccl-onng.755 URL /ttppp///eellancoooooooo.o/////2 www.://////.......aoaa........oo 754/
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. F-divergence minimization for sequence-level knowledge distillation. arXiv preprint 1rXiv:2307.15190, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12444 2023.
Xiaohan Xu, Ming Li. Chongyang Tao, Tao Shen, Reynold Cheng. Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. survey on knowledge distillation of large language models. arXiv 2024.
preprint 200.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 20X0::25055093888, 2025.
Jianhua Zhang, Yi Gao, Ruyu Liu, Xu Cheng, Houxiang Zhang, and Shengyong Chen. Can stu- dents beyond the teacher? distilling knowledge from teacher's bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 22434-22442. 2025a.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang. and Wei Lu. Tinyllama: An open-source small language model. arXiv preprini arXiv:2201.0P 002385, 2024.
Xue Zhang, Songming Zhang, Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. A dual-space framework for general knowledge distillation of large language models. arXiv preprint arXiv:2504.11420 2025b.
Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, and Weizhu Chen. Moe- bert: from bert to mixture-of-experts via importance-guided adaptation. arXiv preprint arXiv:2204. 07675, 2022.
12

---

Published as a conference paper at ICLR 2026
A
CONFIGURATIONS AND HYPERPARAMETERS
In 4 we provide details of the configurations and hyperparameters for training and evaluation.
Table 4: Training configuration and hyperparameters
Parameter
Value
GPU
8x NVIDIA A100 (40 GB)
Global batch size
128
Hardware & Training
Max sequence length
512
Precision
bfloat16
Optimizer
Adam
Learning rate (student)
10-3
Learning rate (projectors)
10-3
Optimization
Learning rate scheduler
Cosine decay
Weight decay
10-1
Epochs
10
Loss coefficient (A)
0.5
Alignment space dim (d.a)
64
Architecture
Projector layers Teacher projector pretraining
Linear (without bias)
10 epochs
Modules
Q, O, Bupe/U projections
Rank
256
LoRA Parameters
a Dropout rate
8
0.1
Temperature
1.0
Sampling Generation Top P
1.0
B THEORETICAL FRAMEWORK FOR RESIDUAL LEARNING
In this section, we provide theoretical results explaining why the proposed residual learning mech- anism, which subtracts projected teacher state during training, is effective. Theorems | formalizes the key principle: the residual term induces an implicit regularizer that pushes the student away from teacher directions teacher-error tokens. We then analyze the convergence of residual learning by gradient descent Theorem
For readability, we denote u, h (T as the mapped teacher hidden states. The residual hidden states defined Eq. can be rewritten as follows:
h h fu, [arg max Py Py # æz]
(12)
We also denote the student's model parameters as and its LM head as W. Hence, the student logits and the residual logits can be defined as z; := Wh and 2; = Whţ, respectively. Finally, the cross-entropy loss function is represented by E(-).
Theorem (Implicit Regularization Against Teacher-Error Directions). For any token i on which the teacher prediction is incorrect, the gradient of Cres with respect to 0 satisfies
VoLres VoLser ((J HHu,
(13)
where:
CSFT is the standard cross-entropy loss using logits z; Wh.,
Sh
S
is the student Jacobian,
00
8 (zz))
H.
is the Hessian matrix of the cross-entropy loss, and H, = W WT W.
02,02.
13

---

Published as a conference paper at ICLR 2026
Thus residual learning adds an extra term that regularizes the gradient updates of student model parameters, pushing the gradients against the teacher's 's hidden states direction u, on tokens where the teacher predicts incorrectly.
Proof. For any position i-th where teacher exhibits error and the corresponding ground truth token Ty the residual loss can be expressed as:
Lrs 1B (((wi, 2..] E, [8(2; 2;)]
By chain rule:
Oz,
Oz:
0k(2;, xi)
h Lres
Th
Oz:
O2:
=W
W Vz, l(z,, x4i)
Perform first-order Taylor expansion of V {(;) around Wh.:
Vz, (Wh, WW,, V ((Wh)) HH,Wu, (B)
Multiplying both sides by WT, we have:
Vn Lres Vn CSFT S Wu,
V nn CSFT HHHu, where H. W H.W
H.
Finally, chaining derivatives through parameters 0s yields Eq. 13 The equation shows that the residual term induces regularization ( H,u,) during gradient update, pushing the gradients against the teacher's 's hidden states direction on teacher-error tokens.
Convergence Analysis. We now show that the residual objective used in our method guarantees convergence under mild smoothness assumptions. To simplify notations, we remove the subscript in previous proofs, denote fe (x) E Rds as the student representation for input token x, hy & as the compressed teacher hidden states, and Iø IRdA + Rdd as the projector that maps compressed teacher states to student space parameterized by d. The logits and losses are:
fo(x), ž == w [z) Syo (hT)), LSFT [[[(z,))], = = 1[(((x))
and the combined distillation objective is:
L(,, Φ) (SFT( + XCres (0,
Note that the combined objective defined here is different from Eq. 11 w.l.o.g. We consider the update of both the student parameters and the projector parameters o by gradient descent:
0 0+
0. 1/ VoL(Or, db),, Φ++1 = Φ. no V C0,, Φ.)
We make the following standard assumptions that control the step of gradient descent:
(A1) The cross-entropy ((z, x)) is Le-Lisschitz and L.-smooth in the logits
(A2) The gradient of w.r.t. logits is bounded: [|Vz(zz,)) S Bg.
(A3) For each fixed Φ. L(O,0) Φb) is smoohh in 0; for each fixed L(O, Φ) is La-smooth in Φ.
(A4) For each @ in the region interest, L(O, (b) is ug--trongly convex in 0.
(A5) The mapping I6 is Ly -ipsschit in Φ: [lg (hT) Iw, (hr) || < Lg |1+1 Øell-
(A6) The student LM head has bounded norm: ||]] a, || || a for simplicity.
Lemma (Bounded Drift From Projector Update.). We define A. L(O,, Φ--1) L(O,, Φ.)) the "drift" in total loss function caused by updating & at iteration t. If L(O, +) is smooth in s. the drift is bounded as follows:
A. ABaBBB L(
14

---

Published as a conference paper at ICLR 2026
Proof. Let de 01+1 Φ and ((s) Φ: + sd, for [0, 1]. Then:
A = / (V (. o(s)), de)ds
||dell / (()) ds (by Cauchy-Schwarz)
-Iddell / ((),
S ||dell sup
VV (+)))
sE/0,11
= =10++1 - - sup V (0,, x)
To IV Φ.)1 sup Vo 5)
From the residual logits 2 Wfe (z) X go (T)) the chain rule gives:
(.. = EE [J 00 h (- (->3w) (2)())))
where Jøp (hT) is the Jacobian of G hy. Taking norms and applying Jensen's inequality, the Lipschitz property of IΦ. the bound [W] = a, and Assumption A2,
V C(,, Φ) _ A3 E V x) || (hr))
S ByLg
V C(,, أ) 13 Lng Substituting this supremum term bound [A,] will
Therefore, sup
complete the proof.
Theorem (Convergence of Residual Learning). Suppose Assumptions Al-A6 hold. If S L/ and the projector learning rate Nh is sufficiently small, then the gradient descent update of the total loss function L( Φ) satisfies:
-
(oriaa [
(14)
where L- = info. L(O, Φ) the optimal loss value and D is bounded as Lemma 7
Specifi-
cally, for moderate X, 3 and sufficiently small na, the drift term is small and the total loss C(O, の)) decreases and converges to a neighborhood of L*
Proof. Fix Φ, and consider the update in O. By Lo -sssmoothness:
L(0 d.) L(O, 110 VoL(O,, 2.).
S L(O,, (br) - 1j0 VoL(O,, +1)11 Lo IVoL(O, &b)1
Using $ 1/Lo and -oostrong convexity in 0, the Dolyak-Eojasseen inequality gives
[VL((,, 21)1 2 20o [C(O,, Φ.) - -L*
Therefore,
C(0,11, -1) -- s (1 7o110) [[O,, Φ2) L*] Now we consider the update of Φ, which induces a "drift" term A, in total loss:
L(1 2++1) L- < (1 (elu) [C(0,, Φ.) L*] Ae)
where Ay is bounded in Lemma I With moderate À and 3, it requires that the projector learning rate 77 remains slow to maintain a reasonably low magnitude of A which is critical to guarantee convergence toward the neighborhood of the optimal value L*
15

---

Published as a conference paper at ICLR 2026
C ANALYSIS: RESIDUAL LEARNING VS. DUAL SPACE KD UNDER DIVERGENCE VARIANTS
To further evaluate the robustness of our framework, compare residual learning with the strong Dual-Space KD (DSKD) baseline under different divergence loss functions. particular, we re- place the KI. divergence in DSKD with alternatives including reverse KI., Jensen-Shannon (JS) divergence, adaptive KL, and skewed KL variants, and measure the student performance in each setting. All experiments as the student. The results in Table [ show that residual learning consistently outperforms DSKD across all divergence objectives, underscoring its generalization and effectiivnness. By leveraging the teacher's errors as learning signals, residual learning prevents the student from imitating the teacher's output distributions in rigid manner, resulting stronger generalization.
conducted with Mixtral-8 x -ettrall-x-BBB--- Instruct Inn-ttuuctt the teacher and GPT2-120M,
Table 5: Comparing residual learning vs. DSKD under divergence variants.
Description
Dolly
SelfInst
VicunaEval
S-NI
UnNI
Avg.
Mixtral-8 8x 7B Instruct - GPT2-120M
DSKD + forward KL
23.11
11.20
15.81
20.33
21.56
18.40
DSKD + reverse KI.
22.03
9.44
12.95
17.08
18.41
15.98
DSKD + adaptive KL
23.41
10.65
14.76
18.88
20.46
17.63
DSKD + JS divergence
22.60
10.59
16.21
18.44
20.37
17.64
DSKD skew forward KL.
23.63
10.89
15.57
19.42
21.21
18.14
DSKD + skew reverse KI.
23.15
11.10
15.64
20.47
21.67
18.40
Ours
22.62
11.66
15.12
24.51
26.15
20.01
D SENSITIVITY ANALYSIS: Loss COEFFICIENT X
Figure
3 illustrates the sensitivity of our method the loss coefficient À (from Eq. TII.
20.01
20
ing We evaluate the performance us- as the teacher and GPT2-120M the student. We vary the value of (0.1,0.3, 0.5, 0.7, 0.9]) and port the average Rouge-L scores over 3 ran- dom seeds. The highest score of 20.01 is achieved at À 0.5, indicating that this value provides the most effective weighting 0.9 between the SFT and residual objectives. Ex- tremely small or large values of À degrade performance, suggesting that both compo- nents of the loss are essential.
Avg. Rouge-1, score
19-
18.81
18-
18.00
16
16.3939
0.1
0.5
0.7
Figure 3: Varying
E SAMPLE PROMPTS AND ANSWERS
We investigate the outputs of distillation from LLaMA2-7B TinyILLMA--I.B and show several prompts and outputs where the teacher model and baseline methods predict incorrectly at some token position, and residual learning can learn avoid replicating the same mistakes. These examples demonstrate that residual learning has the capability to correct teacher's mistakes.
Example Teven correction
Below is instruction that describes task, paired with input that provides further context. Write a response that appropriately completes the request. Instruction: You need output the text of the passage after capitalizing all proper nouns.
Prompt:
16

---

Published as a conference paper at ICLR 2026
Input: Passage: Rami went york city business. While he was there, he met his good friend ali, who showed him around town.
Ground truth: Rami went to New York City on business. While he was there, he met his good friend Ali, who showed him around town.
Teacher output: Rami went New York City on business. While he was there, he his Good Friend Ali, who showed him around Town.
DSKD output: Passage: Rami wentto new york city onbusiness. While was there he met his good friend, ali, who showed him around Town. Passage: Rams, ali.
Ours: Rami went to New York city on business while he was there met his good friend Ali who showed him around town.
Example Sequence-level correction
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. Instruction: For this task, you will be given a question and two pieces of text. Your job is determine which piece of text answers the question. The answer should be one or more phrases in the paragraph, and it should be unambiguous.
Prompt:
Input: Question: What phrase best completes the sentence "He was pleased with himself for ??
Text I: He had defeated his opponent handily in the arm- wrestling match.
Text 2: He had finally completed his ten thousandth pushup. Ground truth: He was pleased with himself for defeating his opponent handily the arm- wrestling match.
Teacher output: The correct answer is "he finally completed his ten thousand pushup". This is because when someone says he is pleased with himself it usually refers to task or accomplishment that he is proud of. In this case, "he" in the question could refer to either "He had defeated his opponent" or "He had finally completed his thousand pushup" and we want to identify the accomplishment that he is actually pleased with. Since the question worded as "what phrase completes the sentence", the entire phrase needs fit in the sentence, therefore the answer is "he had finally completed his ten thousands pushup" (ballucination, wrong choice.)
DSKD output: In both texts, He was pleased with himself for getting handily defeated in arm-wwistling match. He had completed his ten thousandth pushing up. (hallucination,
wrong choice.)
Ours: He was pleased with himself for winning the arm- arm-wress wresleetler match. (correct answer). In this example, we can observe that the teacher and DSKI) tends to hallucinate while resid- ual learning can avoid making the same mistakes.
17


Reviews:
[{"review_id": "XNGdVysuEL", "reviewer": "Reviewer_hWgu", "summary": "This paper presents a method for knowledge distillation including several novel innovations, notably \"residual learning\", a method to guide the student in order to avoid replicating mistakes from the teacher, among other innovations around effective combination of learning signal from MoE teachers, and handling teacher-student combinations with different tokenization methods.\n\nOverall the paper was really interesting, the goals lofty, and the approach very engaging. My one gripe is I'm not sure I understood why residual learning works, and how the overall training objective can benefit positively from the student as it appears only to include negative supervision when the teacher makes mistakes. Hopefully the authors can clarify in the discussion phase.", "questions": "Presentation clarification questions, see Weaknesses.\n\nWhy does unfreezing teacher projectors has such a negative impact on accuracy?\n\nWhy is there no KD loss using the teacher in a positive manner, e.g., KL against teacher outputs, or alignment to the teacher hidden states when the teacher is correct?\n\nIn 3.3, eq 8 how is the h_i^T used? Is this a drop in replacement for a dense h_i^T in section 3.2 eqs 4-5?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper goes beyond the norm in presenting more algorithmic innovations than most. The residual part is the core contribution -- and ablations showed this to work very effectively -- but the MoE and cross-tokenizer methods were also good additions, slightly simpler in formulation, but very sensible and shown to have utility.\n\nThe experimental evaluation is solid, showing the utility of the method and its component parts on standard evals with a range of strong models (Mistral, LLama etc) at reasonable sizes 7B, and against competitive baselines. In many cases the student outperforms the teacher, despite using many fewer parameters, which is a very impressive result. I would expect these results to be interesting to the wider community.", "weaknesses": "The presentation is a bit murky at times. I was left puzzled about why the method works, and what the core motivation is around the various steps in the residual learning method. Let me elaborate on my understanding. Please correct my misconceptions in your rebuttal.\n\nStage 1: pretraining projectors, this appears to be done to as a form of regularization to smooth out meaningless variation in the teacher hidden states. It also allows for situations where the hidden dimension is different in the student, or the representations are the same size and have similar information but indices are permuted (e.g., identifiability). I'm not sure why it's trained with a softmax/categorical distribution to predict the gold next token - wouldn't matching teacher predictions, e.g., using KL, be more suitable?\n\nStage 2: during KD the model learns to project the smoothed teacher state to match the student state, and uses this only when there's a teacher mistake, and it fails to correctly predict the gold next token. In these settings, the student's hidden state is offset to subtract the effect of the teacher. Can this be understood as correcting a likely mistake in the student? And given this 'correction', how does learning via backpropagation of loss from the prediction of the gold next token affect the parameters? I see W^S may be better estimated, but I wonder about h^S_i and h^{(T -> A) -> S}_i and how they will be updated. Given this offsetting of hidden states won't be applied during inference, it's hard to see how this change in KD training will make an impact on the final model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "diyyOEGsUw", "reviewer": "Reviewer_4ahb", "summary": "Traditional knowledge distillation may forces the student model to inherit the errors of an imperfect teacher. This paper proposes a novel framework called \"Residual Learning\" to address the core issue. The key idea is to guide the student to learn only the \"difference\"(residual) between its own understanding and its teacher, but only when the teacher makes a mistake.Thereby avoiding blind imitation and holding the potential to surpass the teacher. \nCombined with specialized modules designed for Mixture-of-Experts (MoE) models and cross-tokenizer scenarios, experiments show that this framework significantly outperforms existing methods across various distillation tasks and effectively enhances the generalization ability of the student model.", "questions": "● Regarding the dependence on ground-truth labels:Do you think it is possible to extend the idea of \"residual learning\" to scenarios where no ground-truth is available? For example, could one use a discriminator model or leverage the consistency among multiple teacher models as a proxy signal for whether the teacher made a mistake?\n● Regarding the breadth of evaluation:Have you considered evaluating your method on tasks that emphasize logic rather than text matching, such as reasoning, math problems, or code generation? We are particularly interested in whether residual learning over token-level prediction errors can effectively transfer and enhance complex reasoning capabilities.\n● Regarding hyperparameter sensitivity:The final loss is a weighted combination of lambda​. In the experiments, both set to 0.5. How sensitive is the model performance to the ratio between these two coefficients? Have you conducted any sensitivity analysis regarding this?\n● Regarding the MoE fusion mechanism:In the MoE knowledge fusion, you employ self-attention to aggregate outputs from all experts. Compared to simpler alternatives like averaging all expert outputs (average pooling) or activating all experts (dense activation), how much improvement does the self-attention mechanism actually bring? This comparison does not seem to be explicitly included in the ablation study.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "● Clear Problem Definition and Motivation:This paper identifies a issue in white-box knowledge distillation: the teacher model is not perfect. Forcing the student model to align with the output distribution of an imperfect teacher essentially sets an upper bound on its capability and may propagate biases. The proposed idea of \"critical learning instead of blind imitation\" is highly persuasive and innovative.\n● Novel, Intuitive, and Easy-to-Understand Core Method:The method uses an indicator function 1[teacher is wrong] to determine when to activate \"residual learning\", shifting the focus from imitation to error correction. The idea is both intuitive and easy to understand.\n● Comprehensive Experiments and Strong Results:\n● Broad Range of Scenarios Covered:The experiments span various realistic and challenging distillation scenarios, including same/cross-tokenizer settings and distilling from Mixture-of-Experts (MoE) models to dense models. \n● Solid Ablation Studies:The ablation study is well-designed and thoroughly evaluates the importance of each component in residual learning: the accuracy mask, scaling factor β, pre-trained projector, and MoE expert fusion. This significantly strengthens the credibility of the proposed method.", "weaknesses": "1. In scenarios where there are no definitive correct answers, such as text generation or chain-of-thought (CoT) reasoning, the indicator function for determining whether the teacher is wrong may be limited in its applicability.\n2. It would greatly enhance the persuasiveness of the paper if there are concrete examples that demonstrate the core claim of this paper — for instance, showing the process where \"the teacher makes a mistake → the baseline student model follows the error → the proposed student model successfully corrects it\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2R3aKevg7C", "reviewer": "Reviewer_SYtA", "summary": "The paper presents a two-stage white-box knowledge distillation framework. In the first stage, the teacher’s hidden states are compressed into a low-dimensional latent space via self-reconstruction projectors. In the second stage, the student is trained through a residual learning mechanism, where it refines its representations by subtracting the teacher’s hidden states from its own. The paper claims that this design encourages the student to learn complementary knowledge and mitigate the propagation of teacher errors. Experimental results on instruction-following benchmarks demonstrate consistent performance gain.", "questions": "See weaknesses section", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The introduction of a feature distillation method where the student learns from the difference between its and the teacher’s hidden states.\n\n- The paper is well written and structured.\n\n- KD for LLM is a very important topic.", "weaknesses": "- Projecting the teacher’s hidden states into the student space is an established paradigm widely used in feature-based distillation to handle mismatched hidden dimensions [1,2,3].\n\n- The evaluation is restricted to instruction-following tasks and a limited set of model architectures. Experiments on more diverse tasks such as reasoning, summarization, etc are important.\n\n- The paper does not include comparisons with several recent and relevant knowledge distillation methods (e.g., DistillM, DistillM2, ABKD, CKA-based distillation), limiting the strength of its empirical claims.\n\n- The paper motivates residual learning as a means mitigate teacher errors, yet the link between this motivation and the proposed residual mechanism is not clear.\n\n\n\n[1] Jiao X, Yin Y, Shang L, Jiang X, Chen X, Li L, Wang F, Liu Q. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351. 2019 Sep 23.\n\n[2] Miles, Roy, and Krystian Mikolajczyk. \"Understanding the role of the projector in knowledge distillation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.\n\n[3] Chen, Yudong, et al. \"Improved feature distillation via projector ensemble.\" Advances in Neural Information Processing Systems 35 (2022): 12084-12095.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)