Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
Authors: Ruihan Jin
Pengpeng Shao
Zhengqi Wen
Jinyang Wu
Mingkuan Feng
Shuoyang
Chu Yuan Zhang
Jianhua Tao
Venue: ICLR 2026 Poster (2026)
Keywords: knowledge distillation
large language model
LLM routing
Primary area: foundation or frontier models, including LLMs

Abstract: Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of **Knowledge Purification**, which consolidates the rationales from multiple teacher LLMs into a single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, we further propose five purification methods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts. Moreover, router-based methods exhibit robust generalization capabilities, underscoring the potential of innovative purification techniques in optimizing multi-teacher distillation and facilitating the practical deployment of powerful yet lightweight models.

Published as a conference paper at ICLR 2026
EXPLORING KNOWLEDGE PURIFICATION IN MULTI- TEACHER KNOWLEDGE DISTILLATION FOR LLMs
Ruihan Jin'
Pengpeng Shao
Zhengqi Wen
Jinyang Wu-.* Jianhua Tao
Mingkuan Feng Shuo Yang' Chu Yuan Zhang'
'Department of Automation, Tsinghua University
*Beijing National Research Center for Information Science and Technology
jinch24 2mmalimml i 1s. tsinguua edu, on ppsbao8 inghua edu. en zywen is inghua edu on wu- jy2 Amails... testtsinggn tsinghua... edu. en jhtao singhua. edu.en.
ABSTRACT
Knowledge distillation has emerged a pivotal technique for transferring knowl- edge from stronger large language models (LLMs) to smaller, more efficient mod- els. However, traditional distillation approaches face challenges related knowl- edge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of Knowledge Purifica- tion, which consolidates the rationales from multiple teacher LLMs into single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, further propose five purification meth- ods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively al- leviate knowledge conflicts. Moreover, router-based methods exhibit robust gen- eralization capabilities, underscoring the potential of innovative purification tech- niques in optimizing multi-teacher distillation and facilitating the practical de- ployment of powerful yer lightweight models.
INTRODUCTION
The rapid advancement of LLM has revolutionized various domains, including question answering (Yue 2025) and reasoning (Plaat al. 2024). The scaling law (Kaplan et al. (020) unveils the cor- relation between the model size and generation capability, yet the practical deployment of colossal LLMs is often constrained by computational cost and resource demands, emphasizing the need for building efficient and lightweight models that preserve their power.
As the extension of model compression (Buciluá al. 2006), knowledge distillation (Hinton al. 2013) has emerged as a prominent solution to this challenge, which enables student models to inherit the capability of larger teacher models. Knowledge distillation is widely applied across various fields of machine learning (Kim & Rush 20166) Park er al.) 2019] Tang et al., 2019). To enhance knowledge diversity and specialized domain competencies, transferring knowledge from multi- teacher ensemble to the student model attracts significant academic interest. This focus leads to
the development of multi-teacher knowledge distillation approaches, such as TinyLLM (Tian al.
2025) and TwT (Xu et al. 2025).
However, existing multi-teacher knowledge distillation frameworks suffer from two significant drawbacks: (1) Knowledge Conflict: Conflicting rationales among teacher LLMs are inevitable due to hallucinations, inconsistent reasoning paths or difference expertise domains, impeding the fectiveness of knowledge transfer to the student model. This problem may become more pronounced as the number of teacher models increases. (2) High Resource Demands: Incorporating knowledge from multiple teachers inherently escalates resource requirements, necessitating complex sampling procedures and intricate training pipelines, which subsequently raises computational cost.
To investigate the adaptability of existing multi-teacher knowledge distillation frameworks toward more teacher LLMs, we perform extended experiments with TinyLLM (Tian et al.) 2025), incre-
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Arcreao we Dojertte
Decoesty Kounnoyy dunay
Avutige Aocitacy
(a) 77M student
(b) 248M student
(c) 783M student
Figure 1: Effects of increasing teacher LLMs the performance of the TinyLLM framework.
mentally introducing a series of teacher LLMs for distillation training(detailed in Appendix D.I As illustrated in Fig. [ contrary to our expectation that enlarging the teacher LLM ensemble would enhance the capabilities of student models, the distillation performance actually declines as the number of teacher models further increases. This decline indicates the detrimental impact of knowledge conflict among teacher L.LMs.
In this paper, we introduce the concept of Knowledge Purification multi-teacher knowledge distillation. The core idea is to condense the knowledge of multiple teacher models from the ratio- nale perspective. The knowledge purification integrates the rationales generated by multiple teacher LLMs into one single, consolidated rationale, which is subsequently employed during the distilla- tion. Hence, the student model is provided with the rationale that encapsulates the collective insights of the teachers, enabling more efficient and effective distillation training. By purifying the knowl- edge, we mitigate the hallucinations and divergent reasoning paths among the teacher LLMss, thereby alleviating inter-teacher knowledge conflicts.
We further propose five methods to facilitate knowledge purification from distinct perspectives in- cluding aggregation, routing, and reinforcement learning (RL)-based selection. To thoroughly evalu- ate these approaches, we conduct extensive experiments on commonsense and biomedical reasoning tasks. Our results show that knowledge purification methods significantly enhance knowledge dis- tillation performance across different student models and datasets. The effectiveness in alleviating knowledge conflicts further verified. Furthermore, methods based on L.MM routing demonstrate outstanding performance on out-of-domas datasets, underscoring the potential of utilizing knowl- edge purification to guide multi-teacher distillation across broader spectrum.
Our contributions are summarized follows:
We identify the limitations of existing multi-teacher knowledge distillation frameworks, highlighting knowledge conflicts and high resource demands that hinder effective knowl- edge transfer.
We introduce the concept of knowledge purification, which mitigates knowledge conflicts and enhances training efficiency by consolidating the rationales from multiple teachers into one coherent rationale. We propose five knowledge purification methods from different perspectives of aggregation, routing, and RI.-based selection.
Extensive experiments verify improvements of proposed methods in distillation perfor- mance and conflict mitigation. Further experiments out-of-domain datasets illustrate the potential of knowledge purification facilitating the generalization of multi-teacher knowledge distillation.
2 RELATED WORK
Multi-Teacher Knowledge Distillation Compared to utilizing single teacher, multi-teacher knowledge distillation harnesses broad knowledge diversity and rich reasoning paths, thereby en- hancing the capabilities and generalization performance of student models (Liu et al.) 2020 Zhang
al. 2024 TinyLLM (Tian et al. 2025) proposes distillation paradigm that facilitates small student LLM to learn from rationales generated by two teacher LLMs. (Xu et al. 2025) incorpo- rates rejection sampling and habitual reasoning in distillation effectively balance computational cost and performance. These methods are constrained by knowledge conflicts among teacher LLMs, underscoring effective strategies for resolving these conflicts during distillation.
2

---

Published as a conference paper at ICLR 2026
LLM Routing In alignment with Mixture-of- Expert (MoE) (Jacobs et al. 1991 Collobert al. 2003 Jiang et al., 2024). LLM routing aims to select the optimal LLM from diver candidates for given question, enabling efficient activation of LLM ensembles. HybridLLM (Ding et al. 2024) leverages a hybrid approach to optimize both cost and quality for L.MM pairs. Similarly, Router- LLM (Ong al., 2024) explores effective methods for dynamic routing between strong and weak LLM. RouterDC (Chen et al., (0224) introduces dual contrastive learning and improves the routing performance. Recent innovations further explore the structured router (Jin al. 2025) and the employment of reinforcement learning( Yue et al. 2025). Within the multi-teacher knowledge distillation framework, LLM routing presents a promising approach for effective knowledge purifi- cation.
3 FORMULATION
3.1 PRELIMINARIES
Generally, knowledge distillation uses the soft outputs/labels generated by the strong teacher model T to transfer knowledge to weak student model S. In this paper, focus primarily on multiple choice question answering problems in NLP. utilizing LLMs as the main subjects of our study.
Multiple Choice Question Answering In the k-multiple choice question answering task, given question E Q and corresponding candidate options set O = {o1, .2..... 0 the objective of LLMs is select the correct option from O that aligns with the ground truth option E O. Besides, LLMs are encouraged to generate rationales, which have been shown to significantly enhance their performance (Wei et al. 2022). The answering process of an L.M M (either the teacher or the student S) is formulated as:
0 = ((g, O,po), M(q, O O,rr),
(1)
where Po and pr- denote the prompt for predicting options and generating rationales, respectively.
Multi- Teacher Knowledge Distillation We consider the rationale generated by the teacher LLM to be an embodiment of knowledge. We sample this rationale as rr = [g. O.Pr) from the teacher T and construct the training set D {gg O o² rT)) with [D] samples. The knowledge distillation for LLM leveraging rationales (Hsieh et al. 2023) can be formulated as:
LKD
CpR + XLDL:
where À is hyper-parameter balance between the prediction loss LeR and the distillation loss LDL. The prediction loss LpR guides the student to learn directly from ground truth options and the distil- lation loss LDL supervises the student inherit knowledge from the teacher's rationale. Details of the knowledge distillation for LLM are introduced in Appendix A
Compared to the single-teacher approach, multi-teacher knowledge distillation leverages an ensem- ble of n teacher LLMs T = {Ti.T>, T2........ T, to equip the student model with broader spectrum of knowledge, leading stronger generalization capabilities. this context, we expand the train- ing set to D = {(g. O o" R)). where R (rr, (Ta: ,T,, denotes the rationales generated by each teacher I.I.M for the question g and corresponding answer options O. (Tian et al., 2025) extends the knowledge distillation framework to incorporate multiple teachers
TT::....
LMTKD = CPR +
Σ Nj Xy
j=1
where LDL is the distillation loss with respect to the j-th teacher's rationale and xy denotes the importance weight for Tj-
3.2 MOTIVATION ANALYSIS
Although most frameworks originally designed incorporate a fixed number of teacher LLMs for knowledge distillation, practically. we expect to enhance the capability and expertise of the stu- dent model by enlarging the teacher ensemble. We consider TinyL.MM as representative method
3

---

Published as a conference paper at ICLR 2026
Knowledge Aggregation
LLM Routing
RL-based Teacher Selection
Environment
Action
Agent
Teacher Selector
rafionale
quediion
Toacher
satrrdary, Agprnganor
Reward
(1 Caundir
Toatber LLM
Knowleedge Distillation
T
Saninvilyity dity Tatsed Rostee
Figure 2: An illustration of five knowledge purification methods proposed in our work.
and conduct experiments to explore its adaptability to more teacher LLMs, as detailed Appendix D.I Results in Fig. [ exhibit that, in these scenarios, the performance of TinyLLM significantly declines as the number of teacher LLMs further increases, which indicates the detrimental impact of knowledge conflicts among teacher LLMs. Furthermore, increasing the number of teachers can impose additional challenges related to computational resources and hyperparameter tuning. There- fore, there an urgent need develop a new framework address these issues.
3.3 KNOWLEDGE PURIFICATION
In this section, we introduce the concept of Knowledge Purification in multi-teacher knowledge dis- tillation. The knowledge purification process integrates the rationales generated by multiple teacher LLMS into one single, consolidated rationale, which is subsequently employed during the distilla- tion. Specifically, given the rationales generated by each teacher LLM as {rT ....... TT, the process of knowledge purification can be expressed as:
rp f(R) f PT......
where f(+) denotes the purification process and rp denotes the consolidated rationale. Through knowledge purification, we aim to mitigate knowledge conflicts among the teacher LLMs and en- hance distillation efficiency. Our study investigates the impact different purification methods on the performance of knowledge distillation.
Incorporating knowledge purification within the multi-teacher knowledge distillation framework al- ters the training objective, formulated as:
LMIKD-KP CpR ACDL-KP
(5)
where CDL-KP denotes the distillation loss calculated using the consolidated rationale rp:
CDI.KP
(6)
D
(g.ORR)
4 METHODOLOGY
As shown in Fig. we propose five methods to perform knowledge purification defined by Eq.
Knowledge Aggregation We first consider performing knowledge purification by employing aggregator, which is a global LLM that accepts all the rationales generated by teacher LLMs and combines the instructions to generate consolidated rationale. We use instruction-tuning paradigm (Wei et al., (022)) to provide instruction prompts containing in-context example as input and perform aggregation in generation fashion.
LLM Routing Build upon a pool of candidate LLMs, an LLM router is designed to allocate an input question the most appropriate LLM. Unlike aggregation, the key aspect of routing is selecting one rationale based on the probabilities predicted by the router as:
Tp arg max Pe (r(r
4

---

Published as a conference paper at ICLR 2026
In this paper, we design three representative LLM routing methods for knowledge purification:
Plackett-Luce ranking We use a Plackett-Luce (PL) model (Luce et al. 1959 Plackett 1975) for ranking multiple teacher LLMs. In the Plackett- model, the probability of selecting a candidate rationale is modeled in softmax relationship:
e
Po(rT,
I....I. T2
(8)
Σj=1 e
We learn the PL coefficients & = (E: : .....n by solving:
e
arg min
Σ Ju'' Elygu, Σj= els )
(9)
&
o' Vu
where denotes the cross-entropy loss, and Ygt denotes the ground truth label for the opti- mal selection. Inspired by (Ong et al. 2024 we use the weight w' to measure the similarity o''' between the input question and question q' in the database w' = "**+,[" where y is a typer-parameter, and and e' denote text embeddings for and q', respectively.
PLM classifier We adopt pre-trained language model (PLM) to extract textual features for standard text classification. Specifically, we employ PLM encode the input question g into semantic embedding lICLS which is derived from the final hidden state corresponding to the special classification token (CL.S). Subsequently, we use a two-layer perceptron to predict the probabilities of routing to each rationales FT, R as:
Po(rr.l)) Σ5=1 (W52((,,hhusttb))-33
I....n,
(10)
where W; and b; denote the parameters of the MLP corresponding T,.
Similarity based router We follow RouterDC (Chen et al. 2024) to perform similarity- based LLM routing. We construct trainable LLM embeddings (k; : E 1,.... n) and calculate the cosine similarities between the question embedding and LLM embeddings for routing:
(sim(E(g).ke
(o(rrll)) Σj= (
1,...?,
(11)
where denotes a language encoder, and sim(... denotes the cosine similarity. The router is trained with two contrastive losses.
Details of all LLM routing methods are introduced in Appendix B.2
adopt reinforcement learning (RL) framework to dynamically select teacher LLMs for knowledge purification. We define the state 8; to encapsulate characteristics of the question q and the rationales of the i-th teacher LLMs rr, as:
RL-based Teacher Selection Inspired by (Yuan al. 202
8; = [g),,, E(rr)) I(TT O P.) = o E Rd2d
(12)
where E is a language encoder, and I[T) (q. O P,) = 0²) indicates whether the teacher T; answers correctly. We design the policy function mg as:
(s; 02) a.o (W,s; + b,) (1 a;)(1 a (W,s, Wss b,)),
(13)
where a denotes the sigmoid function, and the action a, E {0,1) indicates whether to select the teacher T, Consider the definition of knowledge purification, we adopt the teacher LLM that ceives the highest prediction score o oWWs, s + b,) and use it to guide the distillation process.
The trainable parameter of the teacher selector 0 (W E Rnx2d b RTx1] is optimized using the standard policy gradient method:
- 0 0+3 1r Σ VoTa(si,00 u:),
(14)
9EQ
where 3 denotes the learning rate. The reward -PR LDL is computed based on the perfor- mance of the student model. During training, we alternately perform the knowledge distillation and the RL training. Detailed training algorithm is represented in Appendix B.3
5

---

Published as a conference paper at ICLR 2026
Table I: Overall performance of multi-teacher knowledge distillation on four datasets. The best results across different datasets are highlighted in bold, with the second-best results underlined. Average denotes average accuracy.
Setting
Method
OBQA
ARC
Riddle
PQA
Average
FLAN-T5 xlarge
69.20
68.24
53.73
71.50
65.67
Llama 2-chat
54.60
43.35
43.73
54.50
49.05
Teacher
BioMistral-78
51.80
51.59
23.14
73.25
49.95
Dama-3.1-8B-Intrruc
65.60
71.67
60.98
75.00
68.31
PLAN-TS small (77M)
Inference
16.60
19.31
13.33
28.00
19.31
Fine-tuning
45.60
31.76
49.22
47.50
43.52
as Student
Dastlling-Step-by
41.00
30.73
44.90
49.25
41.47
TinyLLM
40.60
30.56
44.12
54.25
42.38
Knowledge Aggregation
40.40
30.47
43.92
53.25
42.01
Plackett-Luce Ranking
42.00
31.76
45.69
50.50
42.49
PLM Classifier
44.20
30.64
49.22
53.75
44.45
Similarity-base Router
48.60
32.19
49.61
52.25
45.66
Teacher Selection
46.80
30.39
48.82
52.50
44.63
FLAN-T'S base (248M) aas Student
Inference
31.00
23.00
30.78
46.00
32.70
Fine-tuning
61.00
43.61
56.86
51.00
53.12
Distilling-Stee Sttelling-SSe--b--Step
59.00
46.01
59.41
$2.50
$4.23
TinyLLM
58.80
46.18
58.82
47.25
52.76
Knowledge Aggregation
58.00
45.75
58.43
51.50
53.42
Plackett-Lace Ranking
62.40
46.27
58.63
54.75
55.51
PLM Classifier
63.60
46.35
59.22
55.00
56.04
Similarity-bass based Router
65.60
46.35
60.78
53.50
56.56
Teacher Selection
65.00
46.70
61.76
53.25
56.68
FLAN-TS large (783M) as Student
Inference
50.40
$1.07
39.80
45.50
46.69
Fine-tuning
72.00
60.26
67.45
$3.00
63.18
SaaillnngSStep-y--SS
71.60
60.00
68.43
51.00
62.76
TinyLLM
68.80
59.83
67.25
$4.25
62.53
Knowledge Aggregation
71.40
59.74
68.63
$3.50
63.32
Plackett-Luce Ranking
72.60
59.48
69.41
$6.50
64.50
PLM Classifier
74.20
59.66
68.24
62.50
66.40
Similarity-basee Router
76.60
60.60
70.59
61.00
67.20
Teacher Selection
75.20
61.12
70.39
63.50
67.55
5 EXPERIMENTS
Models We consider multi-teacher ensemble of four LLMs: FLAN-T5 xlarge (2.85B), Llama 2-chat (Touvron et al. (7B)) BioMistral-7B (Labrak et al. 2024). and ---ma-.118B--Intme (Dubey al. 2024). We conduct experiments using FLAN-T5 (Chung et 2024 small (77M), base (248M), and large (783M) as student models, respectively.
Datasets We conduct experiments on four multiple choice question answering datasets in com- monsense reasoning and biomedical reasoning. For commonsense reasoning, we consider Open- BookQA (OBQA) (Mihaylov et al.) 2018). A12 Reasoning Challenge (ARC) (Clark et al.) 2018), and RiddleSense (Riddle) (Lin al., (021). For biomedical reasoning. we consider PubMedQA (PQA) (Jin et al.] 2019). To ensure fair comparison among knowledge purification methods, randomly retain 80% of the training set samples for distillation training, and the remaining 20% serve the public set. A joint dataset, composed of the public set of each dataset, is utilized for training LLM routers. Details of dataset construction are provided in Appendix C.2
We calculate the accuracy of the distilled student model in multiple choice question an- swering tasks as the primary performance metrie:
Metries
ACC
((S(g,O.po) o²).
(15)

---

Latency Published as a conference paper at ICLR 2026
Table Comparison of knowledge purification methods from a practical perspective. Each method is analyzed in: Prior, Parameters, Training Necessity, Transferability, and Latency.
Method
Prior
Parameters
Training Necessity
Transferability
Knowledge Aggregation
4, R
>10
X
8
Plackett-Luce Ranking
227M
X
8
PLM Classifier
~278
ms
Similarity- based Router
2278
V
ms
Teacher Selection
g, R.
277M
V
X
min
Furthermore, we assess the effectiveness of knowledge purification methods in mitigating knowl- edge conflicts among teacher LLMs by calculating the Conflict Mitigation Value (CMV). We define CMV as the average accuracy improvement achieved through knowledge purification distillation training with series of incremental teacher LLMs, compared to the baseline TinyLLM framework:
1
CMV
Σ 1=2
LACCKP.ITI= ACC ACCILLLLM./77E)
(16)
Implementation details For knowledge distillation, we use the AdamW (Loshchilov & Hutter 2019 optimizer and set the learning rate to 5 x 10-5, 5, the batch size to 8, and the maximum input length 512. We find that À 4 works best for balanced training. We employ GPT-4 (Achiam al.) 2023 as the knowledge aggregator. mDeBERTaVe --abaee (He et al. 202T) is adopted as the language encoder for the PLM classifier, the similarity-base based router and the RL-based teacher selector. Details of applying knowledge purification approaches are included in Appendix B All experiments are conducted on four NVIDIA A100 80GB GPUs.
5.1 RESULTS
We conduct comprehensive experiments to evaluate the performance of multi-teacher knowledge distillation employing knowledge purification methods. Our comparison includes the proposed methods against the original teacher LLMs and four baseline approaches. These baselines consist of direct inference, fully fine-tuning the student model, Distilling- Step-by-Step (Hsieh et 2023) which leverages teacher's s rationale as additional supervision, and TinyLLM (Tian et al. (025) which integrates all rationales generated by all teacher LLMs. Details of baseline approaches are included The overall experimental results are presented in Tab. [
in Appendix C.3
From a performance perspective, no single method demonstrates a significant advantage over the others. Overall, the similarity-basee router and the RL-based teacher selection consistently achieve the highest average accuracies, ranking either first or second across the distillation experiments involving all three student models. For distilling the FLAN-T5 small model, the similarity- based router attains the highest average accuracy of 45.66%, exceeding baselines by at least 4.9%. For distilling the FLAN-T5 base and FLAN-T5 large models, the RL-based teacher selection exhibits superior performance, surpassing the best baseline by 4.5% and 6.9%, respectively.
Generally, the implementation of the knowledge purification method demonstrates advantages over the baseline, underscoring its positive impact on knowledge distillation. The performance of LLM routing exemplary, while the PL ranking shows slightly weaker results compared the other two LLM routers. In contrast, the overall performance of the knowledge aggregation method remains relatively inferior, with no significant improvement observed. This suggests that, despite the strong capacity of the aggregator (e.g., GPT-4). the enhancing effect of the consolidated rationale through aggregation on knowledge distillation remains uncertain.
Compared to the teacher LLMs, the distilled 783M student model exceeds the average accuracy of three teachers and ranks second only to Llama-3. I-8B-Instruct, illustrating the effectiveness of knowledge purification. Additionally, we observe that knowledge purification yields more substan- tial improvements in larger student models. This can be attributed to the stronger capacity of larger models to learn from the generated rationales, leading the notable outcomes achieved through our targeted purification approaches. In contrast, smaller models tend to focus primarily on fitting the final option, which limits the enhancements gained from knowledge purification.

---

Published as a conference paper at ICLR 2026
Average Accuray
Average Accuracy
Average Accuracy
Settings
Settings
Settings
(a) 77M student
(b) 248M student
(c) 783M student
Figure 3: Evaluation of knowledge purification methods with an increasing number of teacher LLMs. We visualize the CMV of knowledge aggregation example, represented by the signed R if & Tin aow.aaseaiiiffiieeeee in Conflict Mitigation Value (CMV).
evaluated
Method
CMV TM student
CMV 248M snudeur
CMV 783M studest
Knowledge Aggregation
-0.003
-0.007
-0.044
Plackett-Luce Ranking
+0.001
+0.012
+0,010
PLM Classifier
+0.018
+00.14
+0,021
Similarity-base based Router
+0.025
+-0.200
+0.032
Teacher Selection
+0.020
+-0.119
+0.029
5.2 QUALITATIVE ANALYSIS ON KNOWLEDGE PURIFICATION METHODS
In this section, perform systematic analysis the proposed knowledge purification methods from practical perspective. We consider the following metrics to evaluate each methods: Prior refers to the imput of each model; Parameters refers to additional amount of parameters introduced for purification; Training Necessity refers to whether training is required; Transferability refers to whether the method Latency refers to the time scale required for processing a single instance of knowledge purification.
be applied to new dataset without requiring additional training; and
We present analysis in Tab. 2 For priors, LLM routing methods require only the question as input and do not necessitate pre- sampling rationales from the teacher LLM. This enables us to lever- age pretrained LLM routers to guide rationale sampling in multi-teacher knowledge distillation as further demonstrated in Section [5.4] Aside from knowledge aggregation, which employs a strong LLM (c.g., GPT-4) for synthesis, other methods only introduce additional parameters on the size of the PLM. The training process of the teacher selector is closely coupled with rewards from knowl- edge distillation, necessitating retraining when applied to new datasets. This training also introduces significant delay in the implementation of RL-based teacher selection, considerably exceeding the millisecond-leww delay of the PLM classifier and the similarity-bazd based router.
5.3 ADAPTABILITY TOWARD MULTIPLE TEACHER LLMS
We further evaluate the adaptability of knowledge purification methods in distillation training with series of incremental teacher LLMs. We adopt the same experimental setup we used when vealing the knowledge conflict of TinyLLM. The evaluation result visualized in Fig. 3 (detailed in Appendix [D22]. Tab. demonstrates the Conflict Miligation Value of each method in mitigating knowledge conflicts.
We observe that applying knowledge aggregation reports negative CMV for all three student models, suggesting that it fails to effectively mitigate knowledge conflicts. In contrast, all LLM routing methods and RL-based teacher selection report positive CMV, indicating their potential in alleviating knowledge conflicts. Notably, the similarity- based router achieves the highest CMV across all three student models, demonstrating its superior adaptability to an incremental multi-teacher ensemble.
5.4 ROUTER-GURDE OUT-OF-DOMANL KNOWLEDGE DISTILLATION
Grounded in knowledge purification, we perform oft-of-domaan knowledge distillation. We exclude the knowledge aggregation which does not involve training and RL-based teacher selection with
8

---

Published as a conference paper at ICLR 2026
Table 4: Experimental results on out-of-domain datasets. We verify the potential of utilizing LLM routing methods to guide multi-teacher knowledge distillation oft-of-domain data. The best results are highlighted in bold, with the second-best results are underlined.
Setting
Method
PIQA
BioASQ
FLAN-TS xlarge
58.43
65.85
Llama 2-chat
60.50
69.92
Teacher
BioMistral-7B
67.46
90.24
-.Strruttttttt
70.84
88.62
Inference
20.78
47.97
Fine-tuning
42.33
78.86
49.29
81.30
FLAN-TS small (77M)
TinyLLM
49.84
75.61
Student
Plackett-Luce Ranking
52.77
80.47
PLM Classifier
50.05
82.11
Similarity- based Router
53.97
82.11
Inference
30.47
57.72
Fine-tuning
47.55
89.43
Destlling-Step-hy-e TinyLLM
55.93
86.18
FLAN-TS base (248M)
56.80
78.05
Student
Plackett-l Luce Ranking
63.98
86.99
PLM Classifier
59.63
85.37
Similarity based Router
63.33
90.24
Inference
$1.90
63.41
Fine-tuning
58.43
90.24
Saailing-Stee-by-SSee
60.72
86.99
FLAN-TS large (783M)
TinLLMM
68.88
82.93
as Student
Plackett-Luce Ranking
68.77
87.80
PLM Classifier
67.90
83.74
Similarity- Router
69.53
91.87
limited transferabillity. Instead, we focus the proposed LLM routing approaches, as the gener- alization ability for out-of-domain data is a crucial metric for assessing the effectiveness of LLM routers. Physical Interaction Question Answering (BikA)(Bisk et al., 2020) and BioASQ (Tsatsa- ronis et al.] 2015) serve as two out-of-domain datasets, which represent commonsense reasoning and biomedical reasoning, respectively. As illustrated in Tab. 4 utilizing LLM routers guide knowledge distillation yields strong generalization ability. The similarity-based router achieves the highest accuracy across
settings, significantly exceeding baselines. Besides, the PL ranking outperforms the PLM classifier in overall performance and demonstrates robust generalization.
worthy noting that LLM routing approaches only require the question input, eliminating the need for pre-sampling responses from teacher LLMs. When applied to broader spectrum of out-of-domain data, high- perforrinn LLM routers can effectively direct the sampling process of multi-teacher ensemble, thereby facilitating subsequent knowledge distillation. This approach sig- nificantly reduces computational costs and resource consumption during the sampling phase while effectively alleviating knowledge conflicts and enhancing the performance of the distilled model. This presents a promising framework for the rapid and flexible implementation of multi-teacher knowledge distillation and the deployment of powerful yet lightweight models.
5.5 EFFICIENCY KNOWLEDGE DISTILLATION APPROACHES
In addition to performance evaluation, our experiments aim to evaluate the improvement offered by the knowledge purification methods concerning the efficiency of multi-teacher knowledge dis- tillation. To ensure comparability, we fix the distillation epoch at 4000 and evaluate the training efficiency of different methods for distilling the FLAN-TS large model on the ARC dataset, utilizing GPU hours the quantitative metric. For knowledge purification methods, consider the com- putational consumption of both the purification stage (e.g., aggregation, training the router) and the distillation stage. The results exhibited in Tab.

---

Published as a conference paper at ICLR 2026
Table 5: Efficiency of different methods for distilling the FLAN-TS large model on the ARC dataset.
We consider both purification and distillation stages and utilize GPU hours as the metric.
Method
T
Purification Stage
Distillation Stage
Total
Fine-tuning
0.7
0.7
DistillnggStee-by--e Step-by-Step
1.1
1.1
TinyLJLM
4
2.6
2.6
Knowledge Aggregation
5.2
1.5
6.7
Plackett-Luc' Ranking
0.1
1.3
1.4
PLMClassifier
4
0.5
1.2
1.7
Similarity- Router 4
0.6
1.2
1.8
Teacher Selection
3.5
3.5
Among the proposed knowledge purification methods, knowledge aggregation demonstrates the highest GPU comsumption due to the extensive call the open-source LLM as the aggregator. When proprietary LLM, such GPT-4, is employed the aggregator, its inefficiency reflected in the equally significant latency and elevated costs. The RL-based teacher selection uniformly opti- mizes the purification and distillation, while its iterative training entails considerable computational consumption. Conversely, routing-base based methods significantly enhance the efficiency of knowledge distillation compared to TinyL.LM, which employs all rationales. Notably, training the L.LM router demands fewer computational resources than the distillation process. Furthermore, given the gener- alization capabilities of routers, the impact of knowledge purification on efficiency enhancement expected to be more pronounced when applied to out-of-domain data.
LIMITATIONS
Due to the limited computational resources, we only construct teacher ensemble of four LLMs. While we strategically select teacher L.MMs such as BioMistral-7B - to supply domain-specific knowledge in biomedical reasoning. it remains challenging to guarantee that the knowledge rep- resented by the teacher ensemble is comprehensive. In the practical application of multi-teacher knowledge distillation, larger number of teacher models would be more conducive to enhancing the specialized domain capabilities of student models. The restricted number of teacher LLMs cur- rently limits our ability to thoroughly assess the effectiveness of knowledge purification methods. Although we conduct small-scale evaluation involving six teachers, as presented in Appendix D.3 further evaluations will be necessary to fully explore and validate the adaptability.
Knowledge distillation is universal framework for transferring knowledge from powerful models to weak ones. In this paper, we primarily focus on the NLP field and consider LLMs as the main subjects of our study. The proposed knowledge purification methods are tailored the characteris- ties of L.M. Approaches such as LLM routing and teacher selection have the potential to generalize to broader machine learning tasks, but specific implementation and evaluation still require further investigation. We leave the investigation such scenarios to future work.
7 CONCLUSION
In this paper, we tackle the challenges inherent in multi-teacher knowledge distillation frameworks, specifically addressing knowledge conflicts and high resource demands that hinder effective knowl- edge transfer to student models. We introduce the concept of Knowledge Purification, aimed at reducing divergent reasoning paths among teachers and enhancing distillation efficiency by consoli- dating the rationales from multiple teacher LLMs. We propose five methods to facilitate knowledge purification from distinct perspectives. Extensive experiments across commonsense and biomed- ical reasoning tasks demonstrate that proposed methods significantly enhance the performance of distilled models while effectively mitigating knowledge conflicts. Notably, the approach based on LLM routers showed exceptional performance out-of-domain datasets, underscoring its broad applicability and practical value. In summary, our findings contribute to the advancement of multi- teacher knowledge distillation frameworks, paving the way for the practical deployment of efficient and powerful lightweight models.
The GPU hour accessed when using Llama-3. 3.1 as the aggregator.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China under Grants U2436210.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303, 08774, 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. Routerde: Query-based router by dual contrastive learning for assembling large language models. Advances in Neural Information Processing Systems, 37:66305-66322 66328, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang. Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Iims by human preference. In Forty-first International Conference Machine Learning, 2024.
Hyung Won Chung, Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instuccionnfineturae lan- guage models. Journal of Machine Learning Research, 25(7)):1-53, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 3rXiv:1803.05455) 2018.
Ronan Collobert, Yoshua Bengio, and Samy Bengio. Scaling large learning problems with hard parallel mixtures. International Journal of pattern recognition and artificial intelligence, 17(03): 349-365, 2003.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid 1lm: Cost-efficient and quality- query routing. arXiv preprint arXiv:2404.14612 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with gradient-disentangee embedding sharing, In The Eleventh International Conference on Learning Representations, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.000 033000 2020.
11

---

Published as a conference paper at ICLR 2026
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2015.
Cheng-Yu Yu Hsieh, Chun-Liang Li, CHIH-KUAN YEH, Hootan Nakhost, Yasuhisa Fujii, Alex Jason Ratner, Ranjay Krishna, Chen-Yu Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In The 61st Annual Meeting Of The Association For Computational Linguistics. 2023.
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403.120333 2024.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b. ArXiv, abs/2310.0682. 2023. 263830494
URL https://apt: www.ntisscoolaar.....oooooocc www.aarrrrggooooooorlia
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP) pp. 2567-2577 Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/D199 1259. URL https::/aclanthologgs
119-1
Ruihan Jin, Pengpeng Shao, Zhengqi Wen, Jinyang Wu, Mingkuan Feng. Shuai Zhang, and Jian- hua Tao. Radialrouter: Structured representation for efficient and robust large language models routing. arXiv preprint (tti::25060088880 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836) 2020.
Yoon Kim and Alexander M Rush. Sequence-levé knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1317-1327, 2016.
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pitrre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains. arXiv preprint arXiv:2402. 1030: 2024.
Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. Riddlesense: Reason- ing about riddle questions featuring linguistic creativity and commonsense knowledge. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Associa- tion for Computational Linguistics: ACL-IJCNLP 2021, pp. 1504-1515, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022
I... URL
2021, https: /ettps:/aaclantee /ac///cccc.c antaanthholloo ogy. wtt.://2........................ findiniggggsse acl. 131/
Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing, 415:106- 113, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Duncan Luce al. Individual choice behavior, volume 4. Wiley New York, 1959.
12

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff. David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference Empir- ical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 118653//1/D18-12666 URL https://aclanthologg org/D18 12607
Isaac Ong. Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and lon Stoica. Routellm: Learning to route llms from preference data. In The Thirteenth International Conference on Learning Representations, 2024.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3967-3976, 2019.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Bäck. Rea- soning with large language models, a survey. CoRR, 2024.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193-200 292- 1975.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv: 1903. 12 136,, 2019.
Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and Nitesh V Chawla. Beyond answers: Trans- ferring reasoning capabilities smaller umss using multi-teacher knowledge distillation. In Pro- ceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pp- 251-260, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16(1):138, 2015.
Guanghui Wang, Zhiyong Yang, Zitai Wang, Shi Wang, Qianqian Xu, and Qingming Huang. Abkd: Pursuing a proper allocation of the probability mass in knowledge distillation via a-B-divvrgence. In Forty-second International Conference on Machine Learning, 2025.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24822 2022.
Jingxian Xu, Mengyu Zhou, Weichang Liu, Hanbing Liu, Shi Han, and Dongmei Zhang. Twt: Thinking without tokens by habitual reasoning distillation with multi-teachers' guidance. arXiv preprint arXiv:2503.24111 2025.
Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, and Daxin Jiang. Reinforced multi-teacher selection for knowledge distillation. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 14284-14291, 2021.
Murong Yue. A survey of large language model agents for question answering. arXiv preprint arXiv:2503.1 19213, 2025.
13

---

Published as a conference paper at ICLR 2026
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route llms for multi-agent systems. arXiv preprint arXiv:2502.11133 2025.
Yuzhe Zhang, Huan Liu, Yang Xiao, Mohammed Amoon, Dalin Zhang, Di Wang. Shusen Yang, and Chai Quek. LIm-enhanced multi-teacher mowleede distillation for modality-incomplete emotion recognition in daily healthcare. IEEE Journal of Biomedical and Health Informatics. 2024.
14

---

Published as a conference paper at ICLR 2026
A DETAILS OF KNOWLEDGE DISTILLATION FOR LLM
Knowledge distillation (Hinton et al. 2015) is designed to facilitate the transfer of knowledge from the teacher model to the student model. Unlike traditional deep learning. where soft labels can be obtained from teacher models, LLMs are often treated as black-box models. In this context, typ- ically regard the rationales generated by the teacher LLM as the embodiment of knowledge, (Hsieh) et al. 2023) We sample the rationale generated by the teacher LLM T Pr = T(q. O. Pr and construct the training set D = {(g. 0, with [D] samples. We expect the student model to inherit knowledge from the teacher's 's rationale and supervise this process using distillation loss, which is defined the form of the cross-entropy loss as:
rrr
DDL
Σ
Σ
log 9.
(17)
D
(v,O.TT)ED
where |rr| denotes the number tokens in the teacher's rationale, and p(rrylrei, q, O Pr) denotes the probability of generating token rry, given the inputs and the already generated tokens, Besides, a prediction loss introduced in training the student model as:
CpR
Σ JED
Σ
log p(o 1l O.pr)
(18)
DDI
where |o"| denotes the number of tokens in the ground truth option, and prr q, O O.p,) denotes the probability of generating token of given the input and the already generated tokens.
The global knowledge distillation process is formulated as:
LKD CpR + ACDL.
(19)
where À is hyper-parameter balance between the prediction loss LpR and the distillation loss CDL.
B DETAILS OF KNOWLEDGE PURIFICATION APPROACHES
B. KNOWLEDGE AGGREGATION
We employ the powerful proprietary LLM GPT-4 (Achiam et al. 2023) the knowledge aggre- gator and use an instruction-tuiing paradigm (Wei et al.] (021) to guide GPT-4 to perform aggrega- tion in generation fashion. Fig. shows the prompt used for knowledge aggregation. In practice, we pre-label 10 knowledge aggregation samples and randomly select one as the in-context example during inference.
In addition to the default GPT-4, we also consider the comparatively weaker Llama-3. 1-00 (Dubey al. 2024 the aggregator to compare the impact of different aggregator choices. For both aggregators, we adopt the same prompt format and examine the performance of applying knowledge aggregation distilling the FLAN-TS large student model. The results, presented in Tab, 6 indicate that utilizing more powerful model as the aggregator does not lead to significant enhancement in performance.
B.2 LLM ROUTING
Plackett-Luce ranking The Plackett-Luce (PL) model was initially introduced by Plackett (Plack- eti 1975) to rank the horses in gambling. It has since been applied to describe the processes of rank- ing and selecting multiple candidate items in various domains. Notably, when the number of items limited to two, the Plackett-Luce model simplifies to the Bradley- -erry (BT) model (Bradley &
Terry 1952). Chatbot Arena Platform (Chiang et al., 2024) is built based on the ranking of LLMS by the Bradley-Terry model. The ranking is formulated estimating the Bradley-Terry coefficient
E:
1
arg min E(Amn
+ clAs - )}
(20)
15

---

Published as a conference paper at ICLR 2026
Instruction
As expert aynthesizer of inforxation, you pesseas ability to distill complex ideas into clear concise imeights. Your taak ix to analyze and sumacize a veries of rexxpning rationsles into kingle oshexive rationale. please ensuze that the tinel aummary captures the exential eleranta and nuances of the uriginal cessoning, shile maintaining clarity and osherence,
In-Context Example
Heere 15 an example:
Inpuz: Question: 3o aocronsser observes that planet zetates faster after netaexite inpact. Which in the nost libely offect of this increase ln rstation Chelocarintal Flanetary dondity vill decrease, la tbl Planctary will becons longer, Antel Planstary kili becore stacter, An Plazetazy graviby will becore strunger, wil becone shorter, After neteorite iepact, the rotation Anblac happena
1. 1o) Slanetary
Earth increased.
plasetary dtep happane
asserrita ixpact.
2. tot's
planetary rotatior after the ingaet of the planet inpentilnlays
asortez, planetary gravity after the inpattlattavity will becaza atrongez. Anpption [a] ia incorsact, Aepptin (b) incorsets tcpption (d) 3. acive this question, ict's break it down stop by stapt Natel. Mtattial Conditicare; A planct has ccstain angalan ncrectin dhe to its rotation, lez. rosteorite Irpact":; lhen neteorita inpacts the planes, it tranetere sone of its nomentur to the And. strransfer of Nonentune This transferred screctin is in the forn of linear nomentur, which also includea an anqular component dae to the insact'a lccation and angle. An4. "*Cosservation of Anqular Koneatur*:; The cotel anquiar zomataa of & alosed systen remsind constant. in this case, the planet's anpuiar renentur incresses due to the transteree norentun, lnb. eretect ot incrased Angular Korentae*) An increase in angular nonentun reacits in faster rotation pericd, as the planet's is rotation speed incresses, Ans. sasulett sharter, shartor./nlanenn AntaCeanideziig the anaver cholecselate- (al Planetary donalty will dicreane: this is unlikely, as the ixpact does not recesarily attect the planet's overall deneity. Ae- (bi Planetary is Incosrect, shorter rotation poriod would result shorter years, the
planstary daya
happenx
planetary years after ingect Fl yeare
becone longer,
happens
If the plaset'a cotation period deczoases, its days will bacone
will becone longer: this longer, An- to! Plasetary days will
increasa rotation that [el planatazy days will becms shorter. the The Hiely effact of impsct reasoning
energy the planet, csuxing it zotate faster. increase in cotation would result sborter planetary days.</o>
Create ccheaive zationals that capturea the exzential points fron each nodel while ensuring clarity and loqical fiev. Octput: The coserved incroase in plaset's 's rotation folloving a netaorite iseact leads the conciuaion that planetary days will becure shorter, This occurs doe to the transter of nonentun fron the neteorice to the planet, which eahances its angalaz nosentun. according to the principle of conservation of anqulaz norentun, an increase in rotation spead sesslta in a sborter cotation pariod. clarity the implications of this phenorenon setnitial Seaters; The planet has a defired angular verectin based on its ourrent retation, (nd. extegast Eventer; The neteorite impacte the planet, isparting additionsl rerentin, And.. *Angular ar Morenton Conservation" The total angular asxectio renalas constant, but the planet's rotation speed incrcases dus to the adoed pomentun, \ns. *Bariiting Ettectr:; Rith incrensed rotation apeed, the ouration et planstazy reduced, Antntherefore, such changes dessity, lengthert la that plaretary wil becone shorter toption dt, Other onel yeare, strengthening of gravity, do not acpurate's y reftee reffest the conseqaenees the Increased rotation frod the impaet. Qoery
correct
Input: Questions
1. (25
2. (118842 NATIONALE)
3. (MAAAAY RAPTONATE)
4. [SIONISTRAL, PATISAALE)
Create a cohesive zationale that oaptures the essential points fron each nodel while ensuzing clarity and logtea1
fiew. Outpus:
Figure 4: Prompt used for GPT-4 to perform knowledge aggregation, consisted of global instruction, in-context example, and query.
where E denotes the binary cross-entropy loss. A and H denote the LLM pair and the human re- sponse. In our work, consider the Plackett-Luce ranking and extend the framework to multi- teacher routing. We transform the problem into a cross-entropy loss optimization problem weighted by text similarity. We learn the PL coefficients & = {E+ = 1,...n) by solving Eq. 9 where Ye: denotes the ground truth label for the optimal selection of the rationales and is represented in an one-hot encoding format. We consider the description generated by the teacher model that pro- duces the minimum number of tokens while making correct selections as the optimal choice, taking into account the preference for computational efficiency and cost reduction in practical applications.
w' = 7 1+ কলनन measures the similarity between the input question and a question g' in the database. Following (Ong et al. 2024), we adopt the exponential scale and choose 7 = 10.
important to note that no training is necessary for the ranking, and all computations are per- formed during inference. We will elaborate the specific inference.
Proposition 1. Let & satisfy Eq. 9 Then, it satisfies the following condition:
Enerow w' Σ E Ey w
eb
i= 11....
(21)
s.t. Q = {gl : T, is optimal for o
Proof. Note:
y(1) eli
i 1 n.
Σ E
16

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of different aggregators for Knowledge Aggregation. The FLAN-T5 large model serves as the student model.
Aggregator
OBQA
ARC
Riddle
PQA
Average
Llama-3.1-70b
71.20
60.77 59.74
67.65
52.50 53.50
63.03
GPT-4
71.40
68.63
63.32
e
g(y) Σ lao'
Elygts gts
Σ
We x)y,,,)),
Σÿ=1 ess
8'
8'80
h(y) ② y(i)) -
1=1
To maximize g(y) subject to h(y) = 0, we consider the Lagrangian:
C(y, n) = g + ry))
where 7) is the Lagrange multiplier. We take the partial derivatives of L with respect to y and and them zero:
Oc = 0, 0c 0y(1) On
0.
This yields the system of equations:
w y(i) 0, i= 1,...n,
dy(i)
r'EQ.
n
y(i) =
1=1
From this, we derive:
Egeo, w E.EnEe. Egeer I...n.
Eyee,
Σ y(i)
Thus, it satisfies the condition in Eq. 21
In practical inference, we perform Plackett-Luce ranking based on Eq. 21
PLM classifier For training the PLM classifier, the definition of the ground truth label Væ we use consistent with that in the Plackett-Luce ranking. We utilize mDeBERTaV3-bern
(He al.) 202T) as the language encoder and extract the semantic embedding hcLs for the input question. We use two-layer MI.P with hidden layer dimension of 128 to predict the routing probabilities based on heLs. We perform mall-parameter training and train the classifier for 5000 epochs, using the AdamW
Loshchilov & Hutter 2019) optimizer with batch size 16 and learning rate 1
Similarity-based router We follow RouterDC (Chen et al. 2024 to perform similarity- based LLM routing and adopt two contrastive losses to train the router. For each question in training, we assign a binary score in {0, 1} to the LLM based on correctness, and sample the LLM with the highest and lowest scores respectively (randomly selecting one in the case of ties) to caleulate the sample-LLM contrastive loss as:
(8))
Crample-LLM Σ
log (s)g))()) + esim(E()).k)))
(22)
where kt and k denote the LLM embedding of the LLM with the highest and lowest scores, respectively. The sample-sample contrastive loss is introduced enhance the robustness of the
17

---

Published as a conference paper at ICLR 2026
Algorithm Training the RL-based Teacher Selector
Input: Training dataset D, student model initialized O θ. and teacher selector initialized 0 0o; ayper-parameters: epoch number L,
size and learning rate B.
I: for epoch I 1 I. do
Shuffle D to obtain new training sequence. 3: for each mini-batch E D do
4:
Samples actions for each E B with the teacher selector to determine the selected teacher T:- by:
5:
Compute the state for each teacher 7; by Eq. 12
6:
1" arg max, (WWs
b,);
7:
Allocate actions a, based on the selection;
8:
Stored (ss, a:) to the episode history Hi;
9:
Update the parameter è of the student model under the guidance of T by Eq.
10:
for each (81, 04) E H do
11:
Compute reward
12:
y x - LDL;
Compute the policy function th (= a:) by Eq.
14:
Update the parameter of the teacher selector by:
15:
0 <- 0 + B E:" r (st,
vector representation, thereby promoting more stable training. We simplify its computation by di- rectly classifying questions based on their respective datasets, as opposed to the semantic clustering approach employed in RouterDC:
(())((()).E((*))
Csample-sampll Σ
log csim(E(g)EE(g))) + Eq exim(E(q),E(q }
(23)
where at and Q denote an in-group question and an out-group set of the question respectively.
The training objective of the router consists of sample-LLM and sample-sample losses
Sim Leamlle LLM + Sample sample-
(24)
We adopt mDDeBRTTa (He et 202T) the language encoder to encode the input question. The dimension of the question embedding and the LLM embedding is 768. We train the model for 5000 epochs, using the AdamW (Loshchilov & Hutter 2019) optimizer with a batch size of 16 and a learning rate of 2 x 10-5
B.3 RL-BASED TEACHER SELECTION
Alg. [ shows the training procedures of the RL-based teacher selector. During training, we alter- nately perform knowledge distillation and train the teacher selector. We regard the teacher selector as a broadly defined LLM router: it must simultaneously receive the question and the rationale, and it can be optimized within unified framework together with the knowledge distillation process.
For training the teacher selector, we set the the epoch number 2, the mini-batee size b 8, and the learning rate 3 to 5 x 10-5
C MORE EXPERIMENTAL DETAILS
C. DETAILS MODEL SELECTION
In our experiments, we choose the small (77M), base (248M) and large (783M) model of the FLAN-
T5 (Chung et al.) 2024) series as student models. Due to computational resource constraint, we
consider teacher ensemble comprising four LLMs: FLAN-T5 xlarge (2.85B), Llama 2-chat Toovron et al. 2023 (7B), BioMistral-77B (Labrak et al. 2024), and 1emm--.11-B-Inssee (Dubey
al. 2024)
We construct the teacher LLM ensemble purposefully: For the student models, FLAN-T5 xlarge serves homogeneous model with larger number of parameters, while Llama 2-chat operates
18

---

Published as a conference paper at ICLR 2026
Table 7: Statistics the datasets we used in our experiments. The numbers represent the sample size of each partitions for each dataset.
Domain
Dataset
Train
Test
Valid
Public
OBQA
3965
500
500
992
ARC
893
1165
295
224
ID
Riddle
2808
510
511
702
PQA
400
400
100
100
PIQA
16113
919
919
OOD
BioASQ
976
123
109
Table 8: Performance of ABKD for distilling the FLAN-T5 large model using different teacher LLMs.
Teacher Model
OBQA
ARC
Riddle
PQA
Average
FL.NN-T5 xlarge
72.40
59.83
68.63
50.50
62.84
Llama 2-chat
70.00
58.45
67.84
51.25
61.89
BioMistral-7B
71.60
58.80
64.12
53.75
62.07
Llama-3. 1.0--...---B-tIttuu
72.40
60.77
69.22
52.75
63.79
as a heterogeneous model, also with a higher parameter size. BioMistral-7B contributes domain- updated version of
specific knowledge in biomedical reasoning, and Llama 2-chat, offers enhanced performance and establishes a selection tendency that facilitates the training of knowledge purification methods.
C.2 DETAILS OF DATASET CONSTRUCTION
Our experiments involve six multiple choice question answering datasets, comprising four in- domain (ID) datasets: OpenBookQA (OBQA) (Mihaylov et al. 2018). A12 Reasoning Challenge (ARC) (Clark et al. 2018), RiddleSense (Riddle), PubMedQA (PQA) (Jin et al. (019) along with two out-of-domain (OOD) datasets: Physical Interaction Question Answering (PIQA) (Bisk al.,
2020) and BioASQ (Tsttsaronis et al. 2015).
For fair comparison, divide each in-domain dataset into four subsets: training, testing, eval- uation, and public sets. The testing and the evaluation set inherit from the original dataset. For in-domain dataset, the training and the public the original dataset in ratio of 4:1. The public sets from each in-domain dataset collectively form joint dataset comprising 2018 samples, used for training LLM routers. The remaining three subsets (training, testing and evaluation) are employed for knowledge distillation. Tab. 7 summarizes the division of both in-domain and of-mmin datasets.
are randomly partitioned from the training set of
C.3 DETAILS OF BASELINES
In our experiments, we compare the proposed methods against four baseline approaches, including Inference, which directly employs student model for evaluation; Fine-tuning, which fine-tunes the student model using the ground truth options as labels: Distilling-Step-by--e
(Hsieh et al.) 2023), defined by Eq. 2 which leverages teacher's rationale as additional supervision; and TinyLLM
Tiia al. 2025 defined by Eq. 3 which integrates all rationales generated by teachers to train
the student model. For the Distilling-Step Step--y---SSt--e bt-p-by-Step. we train with four teacher LLMs individually and report the best results for each dataset.
The standard knowledge distillation (Hinton et al. 2015) is not included in the basclines because, when applied to distilling LLMs, it merely replaces the ground truth labels with those generated by the teacher LLM. This process functionally similar to fine-tuning, and theoretical performance upper limit is lower than that of fine-tuning. Therefore, it is excluded from consideration.
We do not prioritize a comparative analysis between knowledge purification methods and single- teacher distillation approaches; instead, emphasize the comparison with multi-teacher distilla-
19

---

Published as a conference paper at ICLR 2026
tion methods. This preference stems from prior research (Hsich al. 2023) demonstrating the superior cross-task adaptability of multi-teacher distillation techniques. We additionally conduct an evaluation of the sinlletteccher method ABKD (Wang al. 2025), which represents the state- of-the-art knowledge distillation method for L.M.. ABKD uses a pair of a-8 parameters to weight the divergence loss during the distillation stage to achieve a balance between Forward Kullback- Leibler Divergence (FKLD) and Reverse Kullback-Leibler Divergence (RKLD). In our experiment, utilize ABKD distill the FLAN-15 large model using different teacher LLMs. The results are presented in Tab. 8 Notably, ABKD achieves an average accuracy up to 63.79%, outperforming the performance of Distilling-Seep.. Step-by- yet remaining inferior the optimal knowledge purification method (RL-based Teacher Selection), which attains an average accuracy of 67.55%.
Xu al. 2025) proposes distillation framework TWT consists of two stages: Dual-Criteria Re- jection Sampling and Habitual Reasoning Distillation. In the first stage, rationales produced by multi-teacher L.LMs are screened. However, TwT not regarded as baseline method or as a knowledge-purificee purifficuuiiinnnnnnnn method in our paper, for three principal considerations: (I) The data screen- ing in TwT can be interpreted as a combined approach that integrates a quality assessment process using LLMs with resampling process based similarity. This approach contrasts with incli- nation to develop atomic methods within knowledge purification. In fact, the concept of introducing additional models for evaluation and performing selection based on similarity in TwT echoes the knowledge purification methods we propose. (2) TwT retains a pair of rationales from all rationales generated by multiple teacher LLMs, introducing new requirements for the subsequent distillation process, which does not align with the knowledge purification framework. (3) The quality evalua- tion of rationales in TwT relies on the weighting of multiple qualitative factors produced by LLM, which limits its generalization and real-time sampling capabilities. Nevertheless, the design motiva- tion underlying TwT exhibits similarities with knowledge purification, and we anticipate further exploring knowledge distillation in conjunction with the TwT framework in future work.
C.4 REASONS FOR CHOOSING CMV AS METRIC
We quantitatively evaluate the effectiveness of knowledge purification methods in mitigating knowl- edge conflicts among teacher LLM by computing the Conflict Mitigating Value (CMV). The CMV serves as a performance-baat rationale level. We have opted for the performance-based CMV for two principal reasons.
metric rather than relying on information-theores measures at the
First, rationale-level metrics lack universality. Information- Shannon Divergence, effectively quantify knowledge aggregation that produces new rationales. However, they are insufficient for methods like LLM routing, which require the selection of a sin- gle rationale from multiple options. Additionally, although considered assessing the router's accuracy in choosing the rationale that aligns with the correct answer, this metric proves inadequate for measuring knowledge aggregation. Consequently, the pursuit of unified rationale-level metric becomes particularly challenging.
metrics, such as Jensen-
Second, the rationales generated by teacher LLMs represent merely intermediate states within the knowledge distillation process and do not exert a direct influence on the performance of the final student model. Accordingly, we designed the CMV concentrate performance, enabling more direct and meaningful evaluation.
D SUPPLEMENTARY RESULTS
D.I EXTENDED EXPERIMENTS OF THE TINYLLM FRAMEWORK
TinyLLM (Tan et 2025
proposes a distillation paradigm that facilitates student model to learn from rationales generated by two teacher LLMs. Specifically, it adopts the following loss function when conduction multi-teached knowledge distillation:
n1
LTimLLLM LPR
+
X AyL.
(25)
1=1
where Xy is the importance weight for Tj.
20

---

Published as a conference paper at ICLR 2026
Extended experimental results of TinyLLM as the number of teacher LLMs increases from to 4, where "+" denotes the addition of the specified LLM as teacher, and [T] denotes the number of teacher LLMs participating the distillation. The best results across different datasets are highlighted in bold.
Table 9:
Student
Teacher Setting
[T]
OBQA
ARC
Riddle
PQA
Average
FLANN-S5 xlarge
40.00
30.56
42.75
48.50
40.45
FLAN-TS small 77M
+ Llama 2-chat
2
46,80
30.21
43.92
49.50
42.61
BioMistral-7B
3
40.80
30.47
44.31
46.75
40.58
Llama-3.1-8B
40.60
30.56
44.12
50.50
41.45
FLAN-T5 xlarge
1
59.00
44.89
58.24
52.25
53.59
FLAN-TS base 248M
Llama 2-chat
2
63.40
46.44
60.98
50,50
55.33
BioMistral-7B
59.00
45.75
58.04
49.75
53.13
Llama-3.1-8B
58.80
46,18
58.82
47.25
52.76
PLAN-TS xlarge
71.60
59.48
68.43
50.50
62.50
FLAN-TS large
+ Llama 2-chat
2
73.20
59.91
68.82
52.50
63.61
783M
BioMistral-7B
3
69.20
60.26
66.08
54.50
62.51
Llama-3.1-8B
4
68.80
59.83
67.25
54.25
62.53
The basic TinyLLM framework relies merely two teacher LLMs, constraining the practical ap- plication of knowledge distillation to improve the performance and domain-specific competencies of lightweight models. To explore the adaptability of TinyLLM to more teacher LLMs, perform a series of extended experiments. We begin by using only the FLAN-T5 xlarge model as the teacher LLM and progressively incorporate additional teacher LLMs. In total, conduct four groups of distillation training experiments with varying numbers of participating teachers. For a fair compari- son, we set each Ay as 4, 1.33, and for the cases of 1, 2, and 4 teachers, respectively. Tab. presents the detailed results of these extended experiments.
Our observations indicate that as the number of teacher models further increases, the performance of the distilled student models actually declines. For all three student models, the best overall perfor- mance achieved when two teacher LLMs participate the distillation process. When the number of teacher LLMs reaches four, the performance of the distilled FLAN-TS base model is even inferior to that achieved with a single teacher. These findings contradict our initial expectation that increas- ing the number of teachers would enhance knowledge diversity and generalization capabilities, We attribute this performance degradation to the emergence of knowledge conflicts among the teachers, emphasizing the critical need for knowledge purification.
D.2 SUPPLEMENTARY RESULTS OF ADAPTABILITY TOWARD MULTIPLE TEACHER LLMS
We adopt the same experimental setup in Appendix D. to evaluate the adaptability of knowledge purification methods in distillation training with a series of incremental teacher LLMs. Tab. 10 exhibit the complete results of the experiment.
D.3 GENERALITY TOWARD BROADER TASK DOMAINS AND MORE TEACHER LLMS
The current assessment utilizes four teacher LLMs, focusing mainly the task domains of com- monsense and biomedical reasoning. Our objective is to evaluate the generalization ability of knowl- edge purification methods across broader range of task domains and more teacher LLMs. To this end, we extend our experiments using the MMLU dataset (Hendrycks et al. 2020), which encom- passes 57 tasks from various branches of knowledge. Additionally, we introduce two supplementary teacher LLMs, llemma_7b (Azerbayev al. 2003) and Mistral-7B-ool (Jiag et al. (202)), bringing a total of six teacher LLMs. In these experiments, we employ FLAN-T5 large as the student model for knowledge distillation and consider the iimilarity-based LLM routing for effective knowledge purification.
Tab. 15 demonstrates the results of evaluation. The similarity- based router achieves the highest average accuracy of 65. 19%, surpassing the baselines by at least 7.3%. On the MMLU dataset, routing-based method also attains the highest accuracy of 55.26%. The results verifies the gener-
21

---

Published as a conference paper at ICLR 2026
Table 10: Knowledge distillation results the number of teacher LLMs increases from 4, applying Knowledge Aggregation for knowledge purification. The best results across different datasets are highlighted in bold.
Student
Teacher Setting
[T]
OBQA
ARC
Riddle
PQA
Average
FI.AN-S5 xlarge
40.00
30.56
42.75
48.50
40.45
FLAN-TS small
Llama 2-chat
2
39.80
30.21
43.73
51.25
41.25
77M
BioMistral-7B
40.20
30.47
43.14
49.50
40.83
+ Llama-3.1-8B
40.40
30.47
43,92
53.25
42.01
FLAN-TS xlarge
59.00
44.89
58.24
52.25
53,59
FLAN-TS base 248M
+ Llama 2-chat
2
55.60
45.92
57.06
50.50
52.27
BioMistral-7B
3
58.20
46.35
58.63
50.75
53,48
Llama-3.1-8
58.00
45.75
58.43
51.50
53.42
FLAN-TS xlarge
71.60
59.48
68.43
50.50
62.50
FLAN-TS large 783M
Llama 2-chat
2
69.60
59.06
66.47
49.50
61.16
BioMistral-7B Llama-3.1-88
72.00
69.02
52.50
63.08
71.40
59.74
68.63
53.50
63.32
Table 11: Knowledge distillation results as the number of teacher LLMs increases from to 4, ap- plying Plackett-Luce Ranking for knowledge purification. The results across different datasets are highlighted in bold.
Student
Teacher Setting
OBQA
ARC
Riddle
PQA
Average
FLAN-T5 xlarge
40.00
30.56
42.75
48.50
40.45
FLAN-TS small 77M
Llama 2-chat
40.20
30.64
43.33
49.75
40.98
BioMistral-7B
42.20
30.73
44.31
50.25
41.87
Llama-3.1-88
4
42.00
31.76
45.69
50.50
42.49
FLAN-TS xlarge
59.00
44.89
58.24
52.25
53.59
FLAN-TS base 248M
Llama 2-chat
2
60.20
45.41
58.82
52.75
54.30
BioMistral-7B
63.60
45.58
58.63
$2.25
55.02
Llama-3.1-8B
62.40
46.27
58.63
54.75
55.51
FLAN-TS xlarge
71.60
59.48
68.43
50.50
62.50
FLAN-TS large 783M
Llama 2-chat
72.00
59.83
68.82
50.50
62.79
+ BioMistral-7B
71.80
59.66
69.02
56.50
64.25
Llama-3.1-88
4
72.60
59.48
69.41
56.50
64.50
alization ability of the LLM routing method toward broader task domains and large number of teacher LLMs, highlighting the effectiveness of the knowledge purification framework.
Despite current inability conduct experiments involving greater number of teacher LLMs (8, 10, or more) due to computational resource limitations, we are encouraged by the success of existing LLM routing methods when applied more than 10 candidate LLMs (Chen et al.) 2024 Hu et al.) 2024). It is evident that while knowledge purification methods improve performance, the effectiveness knowledge distillation does not increase indefinitely with the addition of teacher LLMs. A more practical goal is to leverage an appropriate number of teacher models to enhance the overall capabilities and specific expertise of the student model.
Furthermore, We also look forward to validating the performance of knowledge purification across a wider range NLP applications. This may necessitate stronger student models, as well more sophisticated processes for distillation data sampling. We intend to explore these scenarios in our future research to further advance the field.
D.4 CASE STUDY
We present a detailed case study and visualization of the knowledge purification process on the OBQA dataset, illustrated in Fig.
22

---

Published as a conference paper at ICLR 2026
Table 12: Knowledge distillation results the number of teacher LLMs increases from 4, applying PLM Classifier for knowledge purification. The best results across different datasets are highlighted in bold.
Student
Teacher Setting
[T]
OBQA
ARC
Riddle
PQA
Average
FI.NN-S5 xlarge
40.00
30.56
42.75
48.50
40.45
FLAN-TS small 77M
Llama 2-chat
2
40.80
30.21
45.69
51.75
42.11
BioMistral-7B
43.60
30.39
48.63
52.50
43.78
+ Llama-3.1-8B
44.20
30.64
49.22
53.75
44.45
FLAN-TS xlarge
59.00
44.89
58.24
52.25
53.59
FLAN-TS base 248M
+ Llama 2-chat
2
61.20
46.27
58.63
52.25
54.59
BioMistral-7B
3
61.00
46,44
59.41
52.00
54.71
Llama-3.1-8
63.60
46.35
59.22
55.00
56.04
FLAN-TS xlarge
71.60
59.48
68.43
50.50
62.50
FLAN-TS large 783M
Llama 2-chat
2
72.40
59.83
68.82
51.75
63.20
BioMistral-7B Llama-3.1-8B
72.80
59.66
69.02
60.00
65.37
74.20
59.48
69.41
62.50
66.40
Table 13: Knowledge distillation results the number of teacher LLMs increases from 4, applying Similarity-base Router for knowledge purification. The best results across different datasets are highlighted in bold.
Student
Teacher Setting
|T|
OBQA
ARC
Riddle
PQA
Average
LL---TS xlarge
40.00
30.56
42.75
48.50
40.45
FLAN-TS small 77M
Llama 2-chat
2
42.00
30.47
46.08
49.75
42,08
BioMistral-7B
46.00
32.02
49.22
51.25
44.62
Llama-3.1-8B
48.60
32.19
49.61
52.25
45.66
FLAN-TS xlarge Llama 2-chat
$9.00
44.89
58.24
$2.25
53.59
FLAN-TS base 248M
60.60
45.75
60.00
52.50
54.71
+ BioMistral 7B
63.80
46.35
60,20
53.00
55.84
Llama-3.1 3.1-8B 8B
65.60
46.35
60.78
53.50
56.56
FLAN-TV T5 xlarge
71.60
59.48
68.43
50.50
62.50
FLAN-TS large
Llama 2-chat
2
74.00
59.83
69.41
54.25
64.37
783M
BioMistral-7B
75.20
59,91
69.80
62.25
66.79
1lama-3.1--8
76.60
60.60
70.59
61.00
67.20
E THE USE OF LARGE LANGUAGE MODEL.S
In this section, we clarify that no LLMs were employed during the original conceptualization, or drafting phases of this paper. All core content and findings the result of original research and critical evaluation by the authors. The use of LLMs was strictly limited to post-drafting linguistic refinements, such as identifying grammatical errors, correcting typos, and improving sentences.
23

---

Published as a conference paper at ICLR 2026
Table 14: Knowledge distillation results the number of teacher LLMs increases from to 4, applying RI-bbased Teacher Selection for knowledge purification. The best results across different datasets are highlighted in bold.
Student
Teacher Setting
[7]
OBQA
ARC
Riddle
PQA
Average
FLAN-TS xlarge
40.00
30.56
42.75
48.50
40.45
FLAN-TS small
Llama 2-chat
2
41.40
30.73
47.84
48.75
42.18
77M
BioMistral-7B
44.40
31.59
48.04
52.25
44.07
Llama-3.1-88
46.80
30.39
48.82
52.50
44.63
FLAN-TS xlarge
1
59.00
44.89
58.24
52.25
53.59
FLAN-TS base 248M
Ilama 2-chat
60.80
46.44
60.78
50.50
54.63
BioMistral-7B
63.60
60.20
51.75
55.54
Ilama-3.
65.00
46.70
61.76
53.25
56.68
FLAN-T5 xlarge
71.60
59.48
68.43
50.50
62.50
FLAN-T5 large
+ Llama 2-chat
2
73.60
60.60
69.02
51.25
63.62
783M
BioMistral-7B
3
72.00
60.77
70.00
61.75
66.13
Llama-3.1-8B
75.20
61.12
70.39
63.50
67.55
Table 15: Supplementary results on commonsense reasoning, biomedical reasoning, and multitask language understanding with six teacher LLMs. The FLAN-T5 large model serves the student model. The best results across different datasets are highlighted in bold.
Method
OBQA
ARC
Riddle
PQA
MMLU
Average
Inference
50.40
51.07
39.80
45.50
45.10
46.37
71.60
60.77
68.43
$1.00
51.84
60.73
TinyLLM
70.40
54.25
67.25
52.75
49.28
58.79
Similarity baaslaiity-based Router
77.00
60.17
70.78
62.75
55.26
65.19
24

---

Published as a conference paper at ICLR 2026
Question: An animal that only eats plants is a Answer Choices:(a) rat (b) moth (c) chimpanzee (d) pig
Question
FLAN-T5 xlarge:7]
Let's think step step. plant eater animal that only eats plants An animal that only eats
Rationales Generated by Teacher LLMS
plants is plant saler. X
Llama 2-chat:722 :72 Let's think step by step Depp.inin@rosttioeenn
the following is venebrate?tn Optonsinlai Fishin(b)
Reptle X
The animal that eals plants moth. The ressoning that chimpanzees, and pigs amnicores, meaning they udl both plants and animals. Mothe, on the other hand, are bettivones, misning they unly ust plants. Therefore, the corret answer's (b) moth. o
EloMistral-7e
L1 .lama- -.. 3-1-8 3.1-8B:7, -B,7,, To fizd the woreet enswer, let's andlyze each option skey sleplaian.. **aa (al**; Rats are omnivures, cormet unswwriitp2.. Moth [b)**, Most muths are burtivures, which monas they unly est plants, They lood un leavess slems, and other matorials, This option omnivores, which meens they eat both plants and animals. They eat tnits, leares, and seeds, but they also hust and eat Insects, small animals, and even mead. This not the corrett answerteind. Psg (d]**, Pigs are umnivures, which miaass Uuy ust both insects, small
they
plants animals. They out insacts, grains, other small ammals.
like good canchlaee for the
answentrins. ** Chimpansne (a) Chimpanzons
plants and animals. conrect answer mowerriinBeee on the analy sis, the cornect unswer is (bb motht* as it the only option that only cals plants. /
grains fruits, vegetabless they
and
This
Question
To Actusue elect uplis rondaat lis lenk dovs sil sky stuutallk eaxieate
Aggregation
Knowledge
viica of the grim estier wemeess We pauyee oach optir io aind the ovrnet unsvrr solle aee baot Ouz dabidall eeguee hai
FLAN-TS rationale 13
bo Lida
Llana 2 rationale
iim ae-ottiis basase this hoeds tands are mosetinnie Chappatio are caassoos wact mase tay aeh pask and satidl They crevias buts laave and cons ven as wi'll natin varkey plnes cad winkls Thyy gotn. hhth and imppaaln,, dli lnacs snall utauua Pips asing sunpals, an clentted venabrdis thome andyata
Lay cumnas is paas They fnioe,
ynd ergiegen
BioMistral rationale 73 Llama-3.1 rationale
GPT-4 as A ggregator
righ
Instruction
Plackett- Luce
p
Ranking
Unining sel
0,253
Question q
Calculate p.
Eatra of
0.220)
Ege
0.123
L1ama-3.1 rationale f4
PLM Classifier
Encoder
PLMM I
Question
Llama-3.1 rationale
mDDBERTTW'3 base as PL.M
Similarity based Routra
p
Question q
Lians 2-chet
L1ama-3.1 rationale n,
Router
Teacher Selection
Epuch 1 Question y & 1/ T33
Epoch Question 9 & r, 72 1/4
RL basedd
Teacher Selector
Policyy Gradient
Teacher Selector
KD
T4 - KD
Figure 5: An example on the OBQA dataset. Five proposed methods used for knowledge purifi- cation.
25


Reviews:
[{"review_id": "ZUBwD4QylW", "reviewer": "Reviewer_EKzP", "summary": "The paper proposes knowledge purification, which consolidates rationales from multiple teacher LLMs into a single rationale to address knowledge conflicts in distillation. Five methods are introduced: knowledge aggregation, LLM routing (Plackett-Luce ranking, PLM classifier, similarity-based router), and RL-based teacher selection. Experiments on commonsense and biomedical reasoning tasks show some performance gains, but the improvements are modest and lack groundbreaking insights.", "questions": "1. How does knowledge purification fundamentally differ from applying ensemble methods (e.g., weighted averaging, majority voting or averaging student model) to the teacher rationales? \n\n2. Why CMV was preferred over the information-theoretic metrics like Jensen-Shannon Divergence (JSD), which could quantify the divergence between teacher rationales?\n\n3. Based on the experiments, which methods should we choose when facing the situation of multiple teacher distillation (under different scenarios)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(1) The focus on knowledge conflicts in multi-teacher distillation is attractive.\n\n(2) Experiments cover multiple datasets and student models, providing a broad assessment. \n\n(3) The five purification approaches offer varied perspectives, from simple aggregation to learned routing. Experimental results directly show the performance of the methods.", "weaknesses": "(1) The core idea of rationale consolidation resembles prior work on knowledge fusion and ensemble distillation. Methods like Plackett-Luce ranking and similarity-based routing are direct adaptations from the existing literature, showing limited substantial innovation. The paper only provide an overall view of the different methods.\n\n(2) Methods like RL-based selection and aggregation involve significant complexity and does not present an obvious theoretical advantage through classical methods.\n\n(3) While the experimental results show some improvements, the gains are often marginal (e.g., ~1–3% accuracy boosts in Table 1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iCV9j5WABI", "reviewer": "Reviewer_Bt3j", "summary": "This paper addresses a critical challenge in multi-teacher knowledge distillation (MTKD) for Large Language Models (LLMs): the performance degradation caused by knowledge conflicts among the rationales provided by multiple teacher models. The authors identify that simply increasing the number of teachers in frameworks like TinyLLM does not monotonically improve student performance, often harming it due to conflicting or hallucinated reasoning paths. To solve this, the authors introduce the concept of \"Knowledge Purification\" (KP), which aims to consolidate the rationales from multiple teachers into a single, coherent rationale before distillation. This process mitigates conflicts and provides the student model with a unified source of knowledge. Through extensive experiments on commonsense and biomedical reasoning tasks, the paper demonstrates that KP methods, particularly the Similarity-based Router and RL-based Teacher Selection, consistently outperform strong baselines like TinyLLM and Distilling-Step-by-Step.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of \"Knowledge Purification\" is a direct, intuitive, and novel solution to the clearly identified problem of knowledge conflict in MTKD.\n2. The proposal of five methods from different families (aggregation, routing, RL) provides a thorough exploration of the solution space. This allows for a nuanced comparison of trade-offs between performance, computational cost, and transferability.", "weaknesses": "1. As acknowledged in the limitations, the study is constrained to a ensemble of only four teacher LLMs. A critical question remains: how do these methods scale to 10, 20, or even more teachers? While the results with 4 teachers are promising, the effectiveness and computational overhead of, for example, the RL-based method or the PL ranking with a much larger pool of teachers is unexplored.\n2. The study is exclusively validated on multiple-choice question answering tasks. While this is a standard benchmark for reasoning, the generality of Knowledge Purification to other NLP tasks like open-ended generation, summarization, or translation is not established.\n3. While the routers offer excellent inference-time efficiency, the cost of training them is non-trivial (requiring a \"public set\" and 5000 training epochs). A discussion on the trade-off between the cost of training a router versus the cost of repeatedly sampling from all teachers for distillation across multiple tasks would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9TJXmxXERu", "reviewer": "Reviewer_Wb1Y", "summary": "This work addresses multi-teacher knowledge distillation (KD), focusing on the challenges of knowledge conflict (i.e.divergent rationales among teachers) and computational cost when aggregating many teacher models. The authors propose Knowledge Purification, a concept for consolidating the rationales from multiple teacher LLMs into a single rationale to use for distillation. They design five purification methods based on aggregation, routing, and RL-based teacher selection styles.", "questions": "See weaknesses section", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The introduction of Knowledge Purification reframes multi-teacher distillation from the perspective of rationale integration rather than mere logit or feature averaging, addressing an important problem of multi-teacher KD.\n\n- The paper proposes five distinct purification strategies (aggregation, routing, and RL-based).\n\n- The paper is well written and structured.", "weaknesses": "- The experimental evaluation is limited to commonsense and biomedical reasoning datasets. To establish the generality of the proposed approach, it should be extended to a broader range of tasks such as mathematical reasoning, coding, and instruction following.\n\n- Although the paper emphasizes improved efficiency, it does not provide quantitative evidence (e.g., training time, GPU hours) compared to existing multi-teacher methods like TinyLLM or TwT.\n\n- The baselines are limited to step-by-step distillation and TinyLLM, omitting state-of-the-art knowledge distillation approaches such as ABKD, MiniLLM, DistillM, or CKA-KD, which could challenge the claimed improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Ta2ljwzdT", "reviewer": "Reviewer_Tdsi", "summary": "This paper addresses a critical problem in multi-teacher knowledge distillation (KD) for Large Language Models (LLMs): the performance degradation that occurs as the number of teachers increases, attributed to knowledge conflicts. The authors introduce the concept of Knowledge Purification, which aims to consolidate the rationales from multiple teachers into a single and coherent rationale. They propose five purification methods in total, Knowledge Aggregation using an LLM to synthesize rationales, three LLM routing approaches (Plackett-Luce Ranking, PLM Classifier, and Similarity-based Router) which select the best single teacher's rationale based on the input question, and RL-based Teacher Selection that uses the student's performance as a reward to choose a teacher. Extensive experiments on commonsense and biomedical reasoning tasks show that routing-based and RL-based methods significantly outperform baselines like TinyLLM, and good generalization to out-of-domain datasets.", "questions": "1. The performance of knowledge aggregation is relatively weak. Have you investigated why a powerful LLM fails to synthesize a good consolidated rationale? Is the issue the aggregation prompt or the task is inherently difficult?\n2. Since the router-based methods perform well and they only need the input question, could it be used to select the teacher per query during the data generation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposal of the concept of knowledge purification\n* Five methods are proposed for a comparative analysis across multiple dimensions (performance, CMV, out-of-domain generalization, etc.), and the analysis (e.g., Table 2) shows the trade-offs of each method\n* The proposed methods consistently outperform baselines, as well as the out-of-domain generalization", "weaknesses": "* The experiments are limited to four selected teacher models\n* Though the overall performance gain is clear, why certain methods work better needs more explanation. It would be helpful if the authors could provide more analyses of pros and cons of each method\n* Excluding the TwT baseline (in Appendix C.3) is reasonable but not fully convincing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)