Correctness-Aware Knowledge Distillation for Enhanced Student Learning
Authors: Ishan Mishra
Deepak Mishra
Jinjun Xiong
Venue: Accepted by TMLR ()

Abstract: In real-world learning, students rely on their mentors for guidance but must also develop the ability to recognize and learn from their mentors' mistakes. Inspired by this mentor-critic dynamic, we propose Mentor-Critic Distillation (MCD), a novel framework for knowledge distillation in machine learning. Traditional distillation methods risk transferring both correct insights and errors from the mentor (teacher model) to the student model, which can hinder student performance. Notably, previous state-of-the-art approaches fail to account for scenarios where the teacher is incorrect, often leaving the student model vulnerable to inheriting these errors. To address this limitation, MCD introduces a weighted knowledge transfer mechanism that decouples the learning process based on the mentor's correctness. When the mentor model is correct, the student model follows the mentor's guidance with a large weight on knowledge transfer. However, when the mentor is incorrect, the student relies more on the ground truth but still learns inter-class relationships from the mentor, adjusting the weight toward task-specific losses such as cross-entropy. This mentor-critic approach ensures that the student model benefits from the mentor's expertise without inheriting its mistakes. We provide theoretical analysis proving that MCD strictly generalizes vanilla KD and guarantees reduced negative transfer. We evaluate our Mentor-Critic Distillation across diverse teacher-student configurations on benchmark datasets, including CIFAR-100, ImageNet, and MedMNIST. Notably, MCD requires no architectural modifications or additional parameters, making it a practical drop-in replacement for standard knowledge distillation. These results highlight MCD's effectiveness in optimizing knowledge transfer and its robustness across diverse domains and data regimes, particularly in data-scarce scenarios typical of specialized domains such as medical imaging.

Published in Transactions on Machine Learning Research (02/2026)
Correctness-Aware Knowledge Distillation for Enhanced Stu- dent Learning
Ishan Mishra
m0@hir.10@iitj.. oc..n
Deportment of Computer Science Indian Institute of Technology Jodhpur
Deepak Mishra
dmishru@hitj..om ac.
Deportment Computer Science
Indian Institute of Technology Jodhpur
Jinjun Xiong
jinjun@buuffllo. edu
Department of Computer Science and Engineering University at Buffalo
Reviewed on OpenReview: htips: // openreview. net/ forum? id- XpRAmzd2sF
Abstract
In real-world learning, students rely on their mentors for guidance but must also develop the ability to recognize and learn from their mentors' mistakes. Inspired by this mentor-critic dynamic, propose Mentor-Critic Distillation (MCD), novel framework for knowledge distillation in machine learning. Traditional distillation methods risk transferring both cor- rect insights and errors from the mentor (teacher model) to the student model, which can hinder student performance. Notably, previous state-of-the-arl approaches fail to account for scenarios where the teacher is incorrect, often leaving the student model vulnerable to inheriting these errors. To address this limitation, MCD introduces weighted knowledge transfer mechanism that decouples the learning process based on the mentor's correctness. When the mentor model is correct, the student model follows the mentor's guidance with large weight knowledge transfer. However, when the mentor is incorrect, the student relies more on the ground truth but still learns inter-class relationships from the mentor, adjust- ing the weight toward task-specific losses such as cross-entropy. This mentor- mentor-critic approach ensures that the student model benefits from the mentor's expertise without inheriting its mistakes. We provide theoretical analysis proving that MCD strictly generalizes vanilla KD and guarantees reduced negative transfer. We evaluate our Mentor- Critic Distillation across diverse teacher-student configurations on benchmark datasets, including CIFAR- 100, Ima- geNet, and McdMNIST. Notably, MCD requires architectural modifications or additional parameters, making it practical drop-in replacement for standard knowledge distillation These results highlight MCD's effectiveness in optimizing knowledge transfer and its robust- ness across diverse domains and data regimes, particularly in data-scarce scenarios typical of specialized domains such as medical imaging.
1
Introduction
The rapid advancements in deep neural networks (DNNs) (Krihevsky el al. 2012 He et al. 2015 Vaswani
et al. 2017 Liu et al. 2021) have revolutionized machine learning applications, powering tasks from image recognition natural language processing with unprocedented accuracy. Yet, this progress comes with substantial challenges, state-of-the-art models [Tan & Le 20099 Dosovitskiy al. 2020 Liu al. 2021 often require considerable computational resources, making them impractical for deployment

---

Published in Transactions on Machine Learning Research (02/2026)
Teacher (ft)
P: + y²
P y
y
Input
77. samples
CRITIC
772 samples
Error- correction loss C + B f
Accuracy- aligned loss a x LD + C
Student(fs)
Figure 1: Overview of the Mentor-Critic Distillation (MCD) Framework. The input images are processed by both the teacher and student models to generate corresponding predictions. The Critic module then categorizes these predictions based on the correctness of the teacher's output. Correctly classified samples (green) are assigned higher weight for KL divergence loss, leveraging the teacher's reliable guidance to regularize the student. Incorrectly classified samples (red) emphasize cross-entropy loss, relying more on the ground truth to correct errors. This dynamic, correctness-basee weighting mechanism enhances the regularization effect, improving the student model's generalization.
limited devices such as mobile phones and edge devices. Knowledge Distillation (KD) Buciluă et al. 2006; Hinton et al. 2015) addresses this challenge by compressing large, high-performing models (known as teacher models) into more compact, efficient student models without substantial losses in predictive accuracy, KD transfers the dark knowledge from pre-trained teacher models compact and lightweight student model, making it an effective tool for deploying sophisticated models in real-world, constrained environments.
The classical KD approach (Hinton et al., 2015 transfers knowledge by minimizing the Kullback-Leibler (KL) divergence between the prediction logits of the teacher and student models. Although effective in its simplicity, this oogit-based method has notable limitations (Tian et al. 2022 Yuan et al.) 2019), as it
considers only output logits, thereby missing out on valuable structural knowledge from the teacher model.
In recent years, various KD techniques (Romero et al., 2015 Tian al. 2022: Tung & Mori 2019 Park
et al., 2019 Zhao et al.} [2022 Jin et al.] (2023) have been developed to enhance the transfer of knowledge from teacher to student, broadly categorized into logit-based, feature-based, and relation-based methods.
Logit-based KD methods remain computationally efficient; however, they often fall short of capturing the complete depth of the teacher's knowledge, particularly in terms of inter-class relationships (Yuan et al. 2019 To address this, feature-based (Zagoruyko & Komodakis 2017 Romero et al., 2015 Heo et al.
Chen et al.] 2021 and relation-based (Park et al. 2019 Tung Mori 2019) KD methods were introduced, leveraging intermediate feature representations and relationships among outputs, respectively,
2019
provide richer information for the student model. Despite their performance benefits, these methods impose considerable compulational and storage demands, which can limit their practicality in real-world applications (Heo et al. 2019). Recent efforts to improve KD have introduced diverse approaches, including Class Attention Transfer-Basod Knowledge Distillation (CAT-KD) (Guo et al. 2023), an attention-based method however, it introduces additional computational overhead through changes in architecture, CAM Selvaraju et al. 2017) extraction, and normalization.
Despite these advancements, a fundamental limitation persists: all prior methods apply uniform weight- ing regardless of teacher reliability. Consider teacher model with 75% validation accuracy. Traditional KD applies the same distillation weight (A 0.9) all samples both the 75% where the teacher is re-
2

---

Published in Transactions on Machine Learning Research (02/2026)
liable and the 25% where it errs. This creates two failure modes: (1) negative knowledge transfer from teacher mistakes, and (2) underutilized regularization from correct predictions. Recent methods like Decou- pled Knowledge Distillation (DKD) (Zhao et al. 2022) decouple srrgetnoontargee components but remain pereetuess-agnosticc failing to address when the teacher itself is wrong. We hypothesize that dynamic, poreetness-awarg weighting can resolve both issues simultaneously.
Inspired by real-world learning where students trust mentors when correct but consult external resources (like answer keys) when mentors err-we propose Mentor-Criti Distillation (MCD). MCD introduces correctness-aware weighting that dynamically adjusts knowledge transfer based on teacher reliability. Specif- ically, MCD employs two complementary mechanisms (Figure
When the teacher is correct, amplify distillation loss (a x LKD) to maximize regularization from reliable guidance.
Accuracy-Aligned Knowledge Distillation (AAKD):
Error-Corroction Knowledge Distillation (ECKD): When the teacher is incorrect, increase cross-entropy weight (3 x LCF) to prioritize ground truth learning while still capturing inter-class relationships from the teacher's soft probabilities.
This simple modification requires architectural changes additional parameters only per-sample check of arg max(p:) to route training signals appropriately. Critically, this adaptive weighting achieves teacher influence configurations that no single fixed X can realize (Lemma 1), with formal guarantees that MCD attennates negative transfer when the teacher errs (Corollary 1).
More importantly, our reformulation reveals that the classical KD loss is inherently highly coupled formula- tion (as illustrated in Figure 1), which may explain the limitations of logit-based distillation. In -eritic Distillation (MCD), we identify two key issues stemming from this coupling. First, the cross-entropy (CE) loss term in traditional KD is insignificant and does not adapt based on the reliability of the teacher's pre- dictions. This inflexibility Icads to an inadequate emphasis error correction when the teacher is incorrect, resulting in negative knowledge transfer. The inability to adjust the CE loss weight dynamically restricts the student model's capacity to learn effectively from the ground truth, especially in cases where the teacher model's guidance leads to an incorrect prediction. Second, the traditional formulation does not decouple the significance of knowledge transfer from correctly-predictte and mncorrectly-preicceer samples. In MCD, we propose treating the AAKD and ECKD losses independently. By decoupling these components, MCD allows for a more targeted and adaptive approach, where the AAKD loss benefits from the teacher's reliable and correct predictions, maximizing regularization and generalization. Conversely, ECKD emphasizes error correction when the teacher is incorrect, without suppressing the benefits of inter-class relationships pro- vided by the teacher. This decoupling ensures that the contributions of both losses are optimized separately, leading to a more robust and effective distillation process.
Overall, our contributions are summarized as follows:
Correctness-aware distillation framework: We introduce simple yet effective approach that dynamically adjusts knowledge transfer based on teacher correctness. When the teacher is correct, we amplify distillation loss (a x LKD) to maximize regularization; when incorrect, we emphasize cross- entropy (3 x LCE) for error correction. This requires no architectural modifications or additional parameters.
Theoretical foundations: We prove that MCD strictly generalizes vanilla KD (Lemma 1) and provide formal negative-transfer attennation guarantees (Corollary 1).
Strong empirical validation: Extensive experiments demonstrate consistent improvements over statc-of-the-art methods: +3.6% over vanilla KD on CIFAR-100 (heterogencous pairs), +1.6% on ImageNet, and +1.3% AUC on medical imaging tasks, with zero computational overhead. Ablation studies confirm that both AAKD and ECKD components are necessary for optimal performance.
3

---

Published in Transactions on Machine Learning Research (02/2026)
2 Related Work
Knowledge Distillation (KD), introduced by Hinton et al (2015), has become a cornerstone technique for model compression. Traditional KD methods align the output logits of teacher and student models using Kullback-Leibler (KL) divergence. Despite their simplicity, these methods often struggle to capture the full range of structural knowledge from the teacher, leading to limited student performance (Yuan et al.
2019
Over the years, numerous KD techniques have been proposed, broadly categorized into logit-based Cho &
Hariharan 2019 Furlanello et al., 2018 Mirzadeh et al. 2020 Zhao et al.)
2022 Jin et al.) 2023). feature-
based [Romero et al. 2015 Tian et al., 2022 Chen al. 2021 Heo et al.
2019],
and salaadoo-bused (Park
et al. 2019 Peng et al. 2019) methods. feature-based approaches transfer intermediate representations but
introduce significant computational overhead and storage costs (Heo et al.
Staatin-bbsee methods distill inter-sample relationships but similarly increase memory requirements, limiting practical deployment.
2019
Recent advancements have improved KD through decoupling strategies. Decoupled Knowledge Distillation (DKD) (Zhao et al. )022) separales the KD loss into larget. and non-larget components, allowing flexible weighting schemes. Multi-Level Logit Distillation (MLLD) (Jin et al. 2023) aligns predictions at instance, batch, and class levels but requires double the training epochs and heavy augmentation. Attention-based methods like CAT-KD (Guo et al., 2023) guide student focus using class activation maps but introduce architectural modifications. Despite these advances, all methods apply uniform weighting regardless of teacher reliability none distinguishes samples where the teacher is correct from those where it errs.
Several works address teacher-student logit mismatch through normalization. Logit Standardization (LS- KD) {Sun et al., 2024) applies per-sample 7-score transformation relax implicit magnitudo--matching constraints imposed by shared temperature. LumiNet (Hossain et al.] (025) performs column-wise (per- class) standardization across the batch to capture intra-class dynamics. Both transformations applied uniformly to all samples- they preserve relative logit structure but do not differentiate reliable from unreli- able prodictions. While they are orthogonal and combinable with MCD (sce MCD+LS results), they do not address leacher misguidance, which our framework directly largets.
CRLD Zhang et al. 2024 employs confidence- based binary filtering: predictions with maximum softmax probability below a threshold are excluded via Soft Label Selection. However, confidence serves only as prozy for reliability overconfident wrong predictions pass through while uncertain correct predictions are filtered out. CRLD does not verify whether the teacher's top prediction matches the ground truth. Wasserstein Knowledge Distillation (WKD) {Lv et al.) 2024) replaces KI. with Wasserstein distance using transport costs from Centered Kernel Alignment, while Scaled Decoupled Distillation (SDD) (Wei et al. 2024) decomposes logits into multi-scale local predictions. Both apply uniformly across samples without correctness-awaee routing, and WKD incurs additional computational overhead from optimal transport.
Recent works explore automated distiller design. NORM (Liu et al. 2023 introduces N-to-One representa- tion matching via expanded student features. DOsWOT (Dong et al. 2023 enables training-free architecture
search using zero-cost proxies. Auto-KD (Li et al. 2023b) and KD-Zero Li et al. 2023a) employ Monte Carlo Tree Search and evolutionary algorithms to discover optimal configurations. These methods oplimize what knowledge to transfer or which architecture to use, producing fixed recipes applied uniformly to all samples-ther do not address how much to trust teacher predictions on a per-sample basis.
In contrastt, MCD operates at a fundamentally different. level: rather than modifying logits (calibration), discarding signal (thresholding), or searching configurations (AutoML), we dynamically reweight loss compo- nents based on an erplicit correctness signal-checking whether the teacher's prediction matchos the ground truth label. This preserves all teacher information, including valuable inter-class relationships from incorrect predictions, while controlling influence through instance-level gating. Our framework requires no architec- tural modifications, no additional parameters, and introduces negligible computational overhead, making it a practical drop-in replacement for standard KD.
3 Knowledge Distillation: A revisit
KD is a widely recognized model compression technique that leverages the knowledge of a large, pre-trained teacher model to train smaller, more efficient student model. From regularization standpoint, KD has
4

---

Published in Transactions on Machine Learning Research (02/2026)
been shown to provide form of implicit regularization Yuan al. 2020
Stanton et al. 2021 Ojha et al,
2023) that can improve the generalization capabilities (Zhou et al. 2021 of the student model However, this regularization effect is often underexploited in traditional logit- based distillation frameworks.
We will explore this by revisiting the core mechanisms KD from a regularization perspective, examining how the interaction between the distillation and ground truth losses influences the learning dynamics of the student model. Specifically, we investigate how the fixed weighting scheme between the distillation loss and cross-entropy loss limits the flexibility needed to adapt to variations in the teacher's reliability. By failing to independently adjust these weights based on the correctness of the teacher model, traditional KD approaches risk either overfitting to the teacher's incorrect knowledge or underutilizing the valuable regularization effect provided by the teacher's outputs.
Notation: Let fa be the student model and ft be the teacher model. This teacher model is pre-trained on the data D. Let 2/ and 2, represent the output (logits) of the teacher and student network, respectively. Let pt and ps represent the softmax outputs of teacher and student, respectively, and represents the temperature. Let X represent the original data and y represent its corresponding label. In general, refers to any deep learning model, refers to logits and p refers to softmax probabilities.
In the traditional KD framework proposed by Hinton et al., the distillation loss is defined as: LKD = KL (p' (ular, (v)l,,,))
(1)
where KL(- .) denotes the Kullback-Leibe (KL) divergence. Additionally, the student model is trained using the standard cross-cntropy (CE) loss with the ground truth labels:
LEe - Σ us (ogp"((l)r))
(2)
The lotal loss in traditional KD framework is a weighted combination of these two losses:
Liotal = XLKD + (1 - A)CCE,
(3)
where À is a hyperparameter that controls the balance between the distillation loss and the CE loss.
Despite its widespread use, this setup has two limitations. First, the hyperparameter À is often set close 0.9, ensuring that the distillation loss has a strong influence. However, this limits the potential to fully exploit the regularization benefits that come from the KD loss. Increasing the weight of the KD loss beyond 1.0 could provide substantial improvements in generalization due to smooth decision boundaries. In such cases, the amplisied regularization effect aligns the student model more closely with the teacher's 's knowledge. Conversely, if the teacher is incorrect, a large weight on the distillation loss can result in negative knowledge transfer, reinforcing errors rather than correcting them. Traditional KD does not account for this variability in teacher reliability, thereby constraining the student model's overall effectiveness. Second, the relatively low weight, assigned to the cross-entropy loss (typically around 0.1) limits the student's ability to learn directly from the ground truth, reducing its error-correction capability.
To understand this, categorize the samples into two groups:
Correctly Classified Samples by the Teacher: These are samples for which the teacher's prediction aligns with the ground truth yi-
Incorrectly Classified Samples by the Teacher: These are samples where the teacher's predic- tion does not match yi-
The traditional KD can be reformulated as below:
Lrotal 1 N N
Σ ALCD + (1- A)Cce) [eorredt (2x) + Uineorecca( (1:)]
(4)
5

---

Published in Transactions on Machine Learning Research (02/2026)
where,
Teorrect (2t) = {1 arg max('' (y]=:)) yi
otherwise,
Incorrect (x:) ceoreet (x4))
After reformulating the traditional KD loss, we can analyze its behavior under two key scenarios: when the teacher model is incorrect and when it is correct. These scenarios highlight the potential pitfalls of the wiigh- approach in traditional KD frameworks.
When the teacher model makes an incorrect prediction, the term À LKD still exerts a significant influence on the student model's learning process. Since is typically set to high value, such as 0.9, the distillation loss drives the student to mimic the teacher's erroneous putputs, resulting in negative knowledge transfer. In this case, the cross-entropy loss (1 X) x LcE, which should help correct the student's learning by reinforcing the ground truth, has minimal impact because of its small weight (0.1). As a result, the student model is inadequately equipped to override the teacher's 's incorrect guidance, limiting its ability to learn the true data distribution effectively.
Case 1: Teacher Model is Incorrect:
Case 2: Teacher Model is Correct: When the leacher model provides correct predictions, the distillation term λxLKE does facilitate beneficial knowledge transfer from the teacher to the student. However, because À is capped at 0.9, the student model is restricted in lcarning from the tcacher's knowledge. This cap constrains the potential for the student to fully leverage the teacher's correct and informative outputs. Can We Simply Scale LKD to Higher Value? One might wonder whether we can address this limitation by simply increasing the weight of the distillation loss À beyond 0.9 to enhance knowledge transfer from the teacher. However, scaling LKD to a higher value introduces a risk: it amplifies the negative transfer of knowledge when the teacher model incorrect. This trade-off highlights the inherent challenge in traditional KD's fixed weighting mechanism limiting to suboptimal performance.
These observations emphasize the necessity of more balanced and adaptive approach to knowledge distil- lation.
4 Mentor Critic Distillation
To address the limitations of traditional KD, propose the Mentor-Critie Distillation (MCD) framework. MCD introduces an adaptive and dynamic approach to balancing the distillation and cross-entropy losses based on the teacher model's reliability. The traditional KD loss formulation can be expressed as:
N
Liotal
Noorrect(s;)
[ACkD + (1 Co + Iimoorrcct(s)) * [ALiD + (1 A)
(5)
N
We redefine this loss in the MCD framework to incorporate dynamic weighting:
LMCD
Σ
Iconrect (x:1) (a x LkD + C(E) Incorrect (2,) x (CkD + CCr)
(6)
Aeruracy-Aligner Loss
Brror-Cornetion Loss
LMCD LAARD + LECKD
(7)
where a and B are hyper-parameters that adjust the influence of the distillation and cross-cntropy losses based on the teacher's correctness. LAAKD is the loss component aligned with samples where the teacher is correct and LECKD is the loss component that handles samples where the teacher is incorrect. Implementation- oriented pseudocode shown in Appendix A.1.
A student relies on a mentor for guidance but also consults reliable resources when the mentor provides incorrect information. Similarly, the student model in MCD dynamically adjusts its learning strategy. When

---

Published in Transactions on Machine Learning Research (02/2026)
the teacher is correct, MCD emphasizes knowledge transfer from the teacher, using higher weight to maximize learning. Conversely, when the teacher is incorrect, MCD reduces the influence of the teacher and increases the weight 3 on the cross-entropy loss, allowing the student to rely more heavily on the ground truth.
Can we discard LCE in LAAKD and LKD LECKD?
The LCE term CAAKD ensures that the student model continues to learn directly from the ground truth, even when the teacher model provides over-generalized, though correct, predictions. On the other hand, the LKD term in LECKD remains valuable even when the teacher makes mistakes. The softened probability distribution from the teacher still contains critical information about inter-class relationships and similarities. By including LKD in LECKD, the student model can leverage the teacher's overall understanding of the problem space, resulting in smoother decision boundaries and improved generalization.
5 Theoretical Analysis
Setup and notation. Let pa(- 2) and pe(- | x)) denole the student and leacher softmax distributions, and let y be the one-hot label. We omit the temperature parameter for clarity. The per-logit gradients used below are the standard identitics: V,CE(y,P.) P. y and V V.KL(p.ll..) Ps Pt-
Proposition 1 (Mixture-target view of vanilla KD). Consider vanilla KD with fixed X E (0,
LKD A (KT(lp)lps)) + +11 x) CF(,,p.))
Then
V&LKD = X(p. pc) + (1) =
((I-
which shows that vanilla KD is equivalent (up to the usual positive scalar factors) to ERM against the sanple-independent "soft target" " ÿ = (1 A) Xp: for every training point.
Proposition 2 (Instanco-dependen) mixture target in MCD). Let a, > 0 be the MCD weights. For sample with correct teacher prediction, MCD uses Leorr (KL(p CE(U,Po), giving
Velcor alps- - pt) + (p. - v) 3 -aa+1) P. a.p p
a+11
For an incorrect teacher prediction, MCD uses Line = KL(pellpb) + +BCE(y. ps), giving
Pt + By
LgLino = (p. -p) + B(p. - y) = (B+ 11 ps
3 ++1
Hence MCD performs ERM against an instance-dependent target
a.: +
if the teacher is correct,
a +
((x) =
Pe + By
if the teacher is incorrect.
3 ++11
Lemma 1 (No-single-A lemma (stricl expressivily gap)). Lei woorr = a1/aa+1 and Wine 1/(3++ the effective teacher weights on correct and incorrect samples under MCD. For any Åo (0,1), there exist (a, B) such that Weorr > An and Wine Ao simultaneously. No scalar X in vanálla KD can realize both inequalities at once, because varvilaa KD fiaes single leacher weight X for all somples (correct and incorrecl).
Proof. Choose > >o/(1- so that Weorr = al Ao, and 8 > 1/Ao - so that Wine 1/(B+11 < Ag. Vanilla KD has Weorr = Wine = X by construction, hence cannot satisfy Weorr > do and Wine < Xo simultaneously.
Corollary 1 (Negative-transfer allemuation guarantee). Consider a sample where the leacher's lop class is wrong. Under vanilla KD, the teacher distribution enters the target with weight X. Under MCD, the teacher weight on such samples is Wine = 11(3+11). For any fized X E (0, 1), choosing 3 > 1 guarantees Wio < X, i.e., the wrong-class mass carried from Pt into the training signal strictly smaller under MCD than under vanilla KD.
7

---

Published in Transactions on Machine Learning Research (02/2026)
Remark (Recoverability and strict generalization). Setting a A/(1 A) and 3 1/X yields Weorr X; if we ignore correctness gating, MCD reduces to vanilla KD. Conversely, vanilla KD cannot realize Weorr * Wines so MCD strictly generalizes vanilla KD in the class of per-sample targets. Moreover, in contrast to DKD-which decouples target/non-taree target components but correchees--gnosiic-CCe gating by Leacher correctness yields the atlenuation guarantee above.
Interpretation MCD selectively increases the regularizing KL term when the teacher is trustworthy (large a on correct samples) and down-weights it when the teacher is unreliable (small 1/(3 + 1) on incorrect samples), while compensating with CE. This correctness-aware control is unavailable to correct formulations (vanilla KD, DKD) and explains the empirical gains observed in our experiments.
6 Experiments
Our experimental study is designed to test the central premise of MCD- MCCD-trust the teacher when it is correct, defer to ground truth when is not across diverse architectures and data regimes. We first establish accuracy gains on CIFAR-100 and ImageNet with both homogeneous and heterogeneous pairs, then examine data-scarce medical imaging lasks. Finally, we analyze sensitivity (0, and verily that MCD introduces no training-ting time overhead.
Datasets and protocols are summarized in Appendix B (with MedMNIST in Appendix B.1 pscudocode
and training selup appear in Appendix A.1 for clinical datasets.
A.2
We report Top-1 for classification benchmarks and AUC
Table 1: Experiment results on CIFAR100 dataset for heterogeneous teacher-student pairs. The A represents the performance gain over the traditional KD. All the results are averaged over 3 trials. The best results are highlighted in bold. Eff. Iter: effective training iterations relative to standard 240-epoch training (MLLD
uses 480 epochs and 2x forward passes). Here vKD+LS vanilla KD with L. (Sun et al. 2024
ReaNet32x1
WRN40-2
VGG13
ResNet50
ResNet322d
Teacher
Distillation
E.f.
79.42
75.61
74.64
79.34
79.42
Avg
type
Iter
Student
SV1
SV1
MV2
MV2
SV2
70.50
70.50
64.60
64.60
71.82
FitNet
Romero
al.
2015
73.59
73.73
64.14
63.16
73.54
69.63
RKD
Park et a.. 2019
72.28
72.21
64.52
64.43
73.21
69.33
feature
1x
CRD Jan et ml. 2022
75.11
76.05
69.73
69.11
75.65
73.13
OFD Heo et al. 2019
75.98
75.85
69.48
69.04
76.82
73.43
ReviewKD Chen
a..
2021
77.45
77.14
70.37
69.89
77.78
74.53
KD [Hinton et al 2015
74.07
74.83
67.37
67.35
74.45
71.60
CTKD [Li et al. 2023c
74.48
75.78
68.46
68.47
75.31
72.5
logit
1x
DKD
Zhao et al.
2022
76.45
76.70
69.71
70.35
77.07
74.05
MCD
77.41
77.38
70.11
71.08
77.35
74.66
+3.34
+-2.55
+2.77
+3.773
+2.9
+3.6
MLLD Jin et al.
2023
77.18
77.44
70.57
71.04
78.44
74.93
logit
4x
MCD+MLL)
77.52
77.68
70.71
71.19
79.03
75.22
vDD+LS
74.27
75.38
68.45
68.75
75.34
72.43
logit+LLS
1x
KD+LS Sun et al. MCD+LS
2024
74.44
75.64
68.61
69.02
75.56
72.65
77.45
77.42
70.41
71.21
77.39
74.77
6.1 CIFAR-100
CIFAR-100 serves as deliberate stress test: teacher models exhibit severe overfitting (>99% training vs. 75% validation accuracy, Table minimizing ECKD activation and creating conditions least favorable to MCD. We retain teachers training mode to introduce stochasticity via dropout and batchnorm. Despite this least-favorable setting, MCD achieves +3.48% improvement over vanilla KD (Table 2), demonstrating that correctness-aree aware gating provides consistent benefits even when teacher errors are rare. Appendix D.3 provides detailed component analysis isolating the contribution of each branch.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Experiment results on CIFAR100 dataset for homogeneous teacher-student pairs. The A represents the performance gain over the traditional KD. All the results are averaged over 3 trials. The best results are highlighted in bold. Eff. Iter: effective training iterations relative to standard 240-epoch training (MLLD
uses 480 epochs and 2x forward passes).
Here vKD+TS is vanilla KD with LS (Sun et al.
2024).
ResNet56
ResNet110
ResNet32xd
WRN40-2
WRR40-2
VGG13
Teacher
Distillation lype
Eff.
72.34
74.31
79.42
75.61
74.64
Avg
Iter
ResNet20
ResNet32
ResNet8x4
WRN1G-2
WRN40-1
VGG8
Student
69.06
71.14
72.50
73.26
71.98
70.36
PitNet
2015
69.21
71.06
73.50
T3.58
71.02
RKD et
69.61
71.82
T3.35
72.22
71.48
71.73
feature
CRD
71.16
73.48
75.51
75.48
74.14
73.94
73.95
OFD
2019
70.98
73.23
74.95
75.24
74.33
73.95
73.78
ReviewK
Chen
2021
71.89
73.89
75.63
76.12
75.09
7184
74.58
KD
73.08
T3.33
74.92
72.98
CTKD
2023e
71.19
73.52
73.39
75.45
73.93
73.52
73.5
logit
DKD
2022
71.97
74.11
76.32
76.24
74.81
74.68
74.68
MCDD
71.83
74.10
76.81
76.07
75.11
74.97
74.81
+1.11
+-1.02
+3348
+11.15
+1.57
4-1.99
+.1.72
MLLD
2023
72.19
71.11
77.08
76.63
75.35
75.18
75.09
logit
>4
MCD+MLLD
72.25
74.47
77.45
76.71
75.52
75.24
75.27
*KDLSS
70.15
72.56
74.04
74.91
73.58
73.84
73.18
logit+LS
KD+LS Sun 2024
71.43
74.17
76.62
76.11
74.37
74.36
74.51
MCD+LS
71.98
74.27
77.11
76.22
75.28
75.11
74.99
Table 3: Cross-architeduure distillation on CTFAR-100 configurations. A: MCD improvement over vanilla KD.
We evaluate MCD on ViT-CNN and CNN-
Training accuracy are reported in Table g
feature
logit
Teacher Student
T. Acc
S. Acc
FitNet
RKD
CRD
KD
DKD
MCD
A
Swin-T ResNet18
89.26
74.01
78.87
74.11
77.63
78.74
80.26
81.74
+3.00
ConvNeXt-T DeiT-T
88.41
68.00
60.78
69.79
65.94
72.99
74.60
77.21
+4.22
Despite these limitations, MCD achieves superior performance across various teacher-student pairs compared to existing baselines. The previous state-of-the-art method (under the standard augmentation and 240- epoch training configuration), DKD, decouples distillation loss into Target Class (TCKD) and Non-Target Class Knowledge Distillation (NCKD), with NCKD proving particularly effective. DKD implicitly increases distillation influence by raising NCKD weights up to 8.0 for CIFAR-100 (where teachers overfit) but drops 0.5 for ImageNet (where teachers are distillation weights should adapt to teacher reliability. Similarly, MLLD employs multi-level alignment (batch, class, instance) with prediction augmentation across multiple temperatures, effectively amplifying distillation loss through implicit weight scaling. MLLD has overall best CIFAR-100 performance but uses extended training configuration with 2 forward passes (weak + strong data augmentation).
supporting our hypothesis that
While LS-KD is orthogonal and combinable with MCD (see MCD+LS results), it does not address teacher misguidance our framework's core focus. Tables [ and 2 demonstrate that MCD+LS consistently out- performs KD+LS, confirming correctness- aware supervision's pivotal role over global logit normalization. Our approach achieves comparable or superior results through explicit, dynamic weighting based on teacher correctness, without requiring complex multi-level alignment, architectural modifications, or heavy data augmentation stralegies employed by MLLD. These results underscore that more critical than alignment complexity for effective student training. Our approach shows that such complex alignment mechanisms are not always necessary. Dynamically increasing the weight of the KL divergence loss based on the teacher correctness efficiently boosts the student performance. Runtime comparison supporting 'no overhead' in Appendix (Figure
guidance is
Abbss-Archhtccture Distillation with Vision Transformers: To evaluate MCD's generalization be- yond CNNs, we test two challenging arcsttrrcie settings: VIT-CNN (Swin-Timm) (eny=)ResNee18) and CNN-VVT CConNNeet-Tinn--DeTTee Table shows MCD achieves +3.00% and +4.22% over vanilla
9

---

Published in Transactions on Machine Learning Research (02/2026)
KD prrrectivly--uubstantte larger gains than CNN-only experiments. These teachers exhibit natural error rates (5-7%, Table 8 compared to overfitted CNN teachers, providing more ECKD activation.
Impact of Image Augmentation with Temperature Regularization: To investigate the effects of incorporating multiple temperatures as a regularization strategy in our MCD framework, we experimented with MI.LD-similar framework by applying multiple temperatures, a combination of strong and weak aug- mentation. This is because MLLD has an advantage of 2x augmented datapoints, 2 to 4x epochs, and multiple temperatures compared to other baselines such as MLLD Sin-Ying). We observe that varying temperatures introduced diverse levels of regularization, which enhanced the model's performance. The re- sults are shown in Table 1 and 2 as MCD+MLLD. The core concept revolves around effectively transferring both the quality and magnitude of regularization to the student model, ensuring that the student benefits from a balanced and robust learning process.
In Table [ and 2 vKD+LS, represents vanilla KD with LS and KD+LS is weighted KD as described Sun ct al. (2024). All SOTA distillation methods work by scaling up the knowledge transfer from teacher to student either implicitly (MLLD adds 3 losses each with 4 different lemperatures and puts weight on them) or explicitly (e.g., DKD puts 8.0 weight on the loss component, LS (KD+LS) puts weight of 9.0 on KL divergence loss); MCD's novelty lies in its simplicity and decoupling strategy for maximum performance. Further, every distillation method uses its own temperature that suits their strategy.
Impact of Homogeneous vs. Heterogeneous Teacher-Student Pairs. The effectiveness of MCD varies based on structural similarity between teacher and student. In both settings, CIFAR-100 teachers achieve very high training accuracy (>97%, Table {). meaning the AAKD branch dominates while ECKD receives limited activation. However, heterogeneous pairs show larger gains (+3.6% average) compared to homogeneous pairs (+1.72% average). We attribute this to the larger representation gap in heterogeneous settings: when teacher and student architectures differ substantially (e.g., ResNet32x4 + ShuffleNet-Vll the student benefits more from AAKD's distillation (a x LKD), which provides stronger regularization to bridge architectural differences. In homogencous scttings, where teacher and student share similar inductive biases, vanilla KD already transfers knowledge effectively, leaving less room for amplification. Additionally, heterogeneous students often start with lower baseline accuracy (e.g., MobileNet V2 64.60% vs. ResNet8x4 at 72.50%), providing more room for improvement.
We note that MLLD performs strongly due to its use of heavy data augmentation (2x input views, 22x training). However, MCD+MLLD exceeds MLLD alone, indicating complementarity. Across both configu- rations, MCD consistently outperforms KD+LS, confirming its strength as a base distillation strategy.
6.2 ImageNet
The teacher models trained on ImageNett are typically well-calibrated (Guo et al. 2017). with minimal differ- ences between training and validation accuracies Table & indicating strong generalization capabilities. In our experiments, we evaluate two distinct teacher-student pairs: a homogeneous pair (Res.eet34 ResNet18), and a heterogeneous pair (ResNet50 - MobileNet The results are shown in Table 4 Our approach shows significant performance improvement over traditional KD while also outperforming existing baselines, achieving these results without relying on prediction augmentation introducing additional parameters. Notably, MCD outperforms recent methods including LumiNet (Hossain et al. 2025) WKD-L {Lv et al., 2024), and CRLD (Zhang el al.) 2024) the helerogeneous pair, despite these methods requiring additional computational overhead (offline cost matrices or 2x forward passes). It is noteworthy that, unlike MLLD (Jin ct al. 2023). which rolics on a combination of weak and strong data augmentations to enhance perfor- mance, our approach does not require such augmentations. Despite this, our method still achieves better results compared to ML..D. To disentangle the roles of AAKD and F.CDD, we include an isolation study Appendix D.2 (Table 9)
6.3 Impact of a and 3
MCD requires balancing two competing objectives, which we term the Knowledge Transfer Trade-off:: avoid- ing both overfitting to ground truth labels and over-reliance on teacher guidance. Table reveals this critical equilibrium.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Results on ImageNet dataset. We report Top-1/Top-5 accuracy for one homogeneous (RoNNo334-ReNNe118) and one heterogencous (ResNat50-MobleNaty)) pair. All results averaged over trials. Overhead: training time relative to vanilla KD. 'Requires offline cost matrix computation. UUses 2x forward passes (aassssRandAAgmeett + Cutout). Method
Top-1
Top-5
Top-1
Top-5
Distillation type
ResNet34
ResNee11
Overhead
Teacher Student
73.31
91.42
76.16
92.86
69.75
89.07
68.87
88.76
AT
70.69
90.01
69.56
89.33
OFD
Heo et al.
2019
70.81
89.98
71.25
90.34
feature
1.2-1.5x
CRD lan a 2022
71.17
90.13
71.37
90.41
ReviewKD
[Chen et al. 20021
71.61
90.51
72.56
91.00
attention
1.1x
CAT-KD
(uo et al. 2023
71.26
90.45
72.24
91.13
KD (Hinton et a.. 2015
70.66
89.88
68.58
88.98
TAKD Mizzadch ct al 2020
70.78
90.16
70.82
90.01
1.0x
DKD Zhao et al. 2022
71.70
90.41
72.05
91.05
KD+LS Sun et al. 2024
71.42
90.29
72.18
90.80
SD-KD
Wei et al. 2024
71.44
90.05
72.24
90.71
logit
LumiNet
Hossain et al." 2025
72.16
90.60
72.55
91.12
MCD (Ours)
72.08
90.59
73.65
91.88
1.3xl
WKD-L LLy et al. 2024
72.49
90.75
73.17
91.32
1.75xl
CRLD [hhang et al.
2024
72.05
90.74
73.15
91.54
2 2.0x
MLLD
(Jin al.
2023
71.90
90.41
73.01
91.42
Table 5: Impact of byperparameters and 8 on CIFAR-100. Left: varying a with B = 2 fixed. Right: varying 3 with a = 12 fixed. Best results in bold.
B = 2, varying a
a = 12, varying B
Teacher
Student
8
12
16
6
ResNet50
MobileNet- V2
67.57
71.22
71.07
70.13
70.61
71.07
70.81
70.64
ResNet.32x4
ShuffleNet-V V2
74.52
77.50
77.71
77.44
77.23
77.71
77.44
77.34
WRN-10-2
WRN-16-2
74.97
76.15
76.08
75.71
75.80
76.08
75.75
75.65
Excessive CE weighting (high 3): When the CE loss dominates, the student overfits training labels, achieving high training accuracy but poor generalization on validation data. The CE Joss encourages sharp, high-confidence predictions aligned with ground truth, causing the student to memorize training examples while missing the inter-class relationships encoded in the teacher's soft probabilities. This is analogous to rote memorization without conceptual understanding.
Excessive KL weighting (high a): Conversely, overemphasizing the KL divergence loss causes the student blindly mimic the teacher's outputs, including its biases and limitations. While KT-based regularization benefits gencralization, excessive weight prevents the student from learning task-specific details from ground truth labels, degrading performance on precise discrinination tasks. The student becomes proficient in the teacher's generalized concepts but lacks accuracy specific cases.
Optimal balance: MCD addresses this trade-off through corroctness-awarc weighting. When the teacher is correct (a x LKD), we amplily regularization to leverage reliable guidance. When incorrect (8 x LCF), we increase supervision from ground truth while retaining inter-class relationships from teacher probabil- itics. Our default values (a 12,3 2) achieve robust performance across diverse teacher-student con- figurations, demonstrating that dynamic, correctness-baset weighting effectively navigates the overfitting- overregularizatio spectrum.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: Diagnostic performance on medical imaging tasks.
Results shown as AUC/ACC. Best. student
performance bold.
DermaMNIST
BreastMNIST
OrganMNNIST-S
Method
Teacher
Student
AUC
ACC
AUC
ACC
AUC
ACC
Teacher
ResNet-50
0.9484
82.59
0.8726
82.69
0.9794
82.60
Student-only
SV2
0.8755
72.72
0.8099
79.49
0.9734
76.79
Student-only
MV2
0.9040
75.31
0.8181
82.05
0.9776
79.76
KD
Hinton et al.
2015
ResNet-50
SV2
0.9012
73.82
0.8348
83.33
0.9778
80.71
KD
Hinton el al.
2015
ResNet-50
MV2
0.9075
75.51
0.8617
84.62
0.9796
81.96
DKD
Zhao al.
2022
ResNet-50
SV2
0.9089
74.66
0.8823
84.62
0.9789
81.70
DKD
Zhao et al.
2022
ResNet-50
MV2
0.9121
76.41
0.8742
85.90
0.9808
82.80
MCD
RosNet-50
SV2
0.9119
76.51
0.8995
85.26
0.9797
82.58
MCD
ResNet-50
MV2
0.9205
78.70
0.8839
86.54
0.9822
83.55
6.4
Evaluation on Medical Imaging Tasks
To assess real-world applicability, we evaluate MCD on medical imaging tasks spanning dermatoscopic, ultrasound, and CT modalities using MedMNIST (Yang et al. 2023 (dataset specs and training details are in Appendix B.1]. These settings are data-scarce, exhibit high inter-class similarity, and demand stringent accuracy for decision support.
Results: Table T presents the diagnostic performance across all methods. We report Area Under the ROC Curve (AUC) and Accuracy (ACC) following medical imaging evaluation standards. MCD consistently out- performs traditional KD and state-of-tho-art DKD across all three medical imaging datasets. MobileNet V2 with MCD achieves improvements of +1.30% AUC Derma +2.22% AUC on BreastMNIST, and +0.26% AUC on OrganMNIST-S over baseline KD, demonstrating effective knowledge transfer across diverse medical imaging modalities. These results demonstrate that MCD effectively addresses a key challenge medical AI: compressing accurate diagnostic models for resource-constrained clinical devices while maintain- ing diagnostic performance. The correctness-awar aware distillation mechanism is particularly valuable in medical imaging where inheriting teacher errors could have serious clinical consequences.
7 Limitations and Future Directions
MCD has two primary limitations. First, it uses binary correctness signal (arg max(p:) y), treating all correct predictions equally regardless of confidence confident prediction (p: 0.95) receives the same weight as barely correct one (p: 0.35). Second, hyperparameters a and / require manual tuning, though our defaults (a = 12, 3 2) generalize well across diverse teacher-student configurations.
Two natural extensions address these limitations. Confidenco-aware soft weighting could modulate a and 3 based on prediction confidence, enabling smoother adaptation that leverages teacher calibration beyond binary correctness. Automatic hyperparameter selection via neta-learning could infer optimal values from teacher properties (validation accuracy, calibration error), eliminating manual tuning and enhancing practical deployability across domains.
8 Conclusion
We introduced Mentor-Critic Distillation (MCD), a correctness-aware framework that dynamically adjusts knowledge transfer based on teacher reliability through two complementary mechanisms: Accuracy-Aligned KD (a LKD) for correct predictions and Error- Corre-Coreetion KD (3 Lex) for incorrect ones. We prove that MCD strictly generalizes vanilla KD (Lemma 1) with formal negative transfer attenuation guaran-
12

---

Published in Transactions on Machine Learning Research (02/2026)
tees (Corollary 1), while requiring zero architectural modifications or additional parameters. Extensive ex- periments demonstrate consistent improvements, with ablations confirming both components are essential. MCD provides a theoretically grounded, empirically validated solution that explicitly accounts for teacher imperfection-a critical considcration for real-world deployment. Future work includes confidence-awaro soft weighting and automatic hyperparameter selection to further enhance practical deployability.
References
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535 541, 2006.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVP conference on compuler vision and pallern recognilion, pp- 5008 5017, 2021.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4794-4802, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 255. IEEE, 2009.
Poijic Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 11898 11908, June 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Neil Houlsby, and et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.110 11929, 2020.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference machine learning, pp. 1607 1616. PMLR, 2018.
Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11868-11877 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770 778, 2015.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1930, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Md Ismail Hossain, MM Lutfe Elahi, Sameera Ramasinghe, Ali Cheraghian, Fuad Rahman, Nabeel Mo- hammed, and Shafin Rahman. Luminct: Porception-driven knowledge distillation via statistical logit calibration. Transactions on Machine Learning Research, 2025.
Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24276-24285, 2023.
Jin- Ying.
Mult Multi-level logit distillation, issue #2.
attps://github. hom.huub.comm/ae www.o/////niiin. Ying/
Accessed: March 2025.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical
report, University of Toronto, 2009.
Alex Krizhevsky, Πla Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In Advonces in neural information processing systems (NewrIPS), pp. 1097 1105, 2012.
Lujun Li, Peijic Dong, Anggeng Li, Zimian Wei, and Ya Yang. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. In Advances in Newral Information Processing Systems, 2023a.
Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Automated knowledge distillation via monte carlo tree search. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023b.
Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculun temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1504 1512, 2023c.
Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In The Eleventh International Conference on Learning Representations, 2023. URL https:
//openreview hwtprreviiwwnee/ddfffee
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 10012 10022, 2021.
Jiaming Lv, Haoyuan Yang, and Peihua Li. Wasserstein distance rivals kullback-leibler divergence for knowl- edge distillation. Advances in Neural Information Processing Systems, 37:65445 65475, 2024.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet. v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116 131, 2018.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? Advances in Neural Information Processing Systems, 36:11037-11048, 2023.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3967-3976, 2019.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 5007-5015 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations (ICLR), 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 2018
Ramprasaath R Selvaraju, Michael Cogswell, Abhishck Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localization. In Proceedings of the IEFE international conference on compuler vision, pp. 618-626, 626, 2017.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xrxi::149. 1556, 2014.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advonces in Newral Information Processing Systems, 34:6906-660 6919,
2021.
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15731-15740, 2024.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6114, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2022. URL https:://iithub.com// hhw.obbtooooo///pssssslleet
Frederick Tung and Greg Mori. Similarity-preeerving knowledge distillation. In TEEE International Confer- ence on Computer Vision (ICCV), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS), pp. 5998 6008, 2017.
Shicai Wei, Chunbo Luo, and Yang Luo. Scaled decoupled distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation: inheritance and exploration perspective. arXie preprinl 0rXi::1910.01348, 2019.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognition, pp. 3311, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Repre- sentations (ICLR), 2017.
Weijia Zhang, Dongnan Liu, Weidong Cai, and Chao Ma. Cross-view consistency regularisation for knowledge distillation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 2011-2020 2024.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufllenet: An extremely elficient convolutional neural network for mobile devices. Tn Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6848 6856, 2018.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11953 11962, 2022.
Helong Zhou, Liangehen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Re- thinking soft. labels for knowledge distillation: A bias-variance tradeoff perspective. or Xin preprinl arXiv:2102.000 00650, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2026)
A Pseudocode and Implementation Details
A.1 PyTorch-style Pseudocode
Algorithm 1: PyTorch-style pseudocode for Mentor- Critic Distillation
Inputs:
# z_s: Logits from student
# z_t: Logits from teacher
# Ground truth
# T: Temperature
# alpha, beta: papar-parsmeters for MCD
Compute correct/incorrect masks for teacher's predictions mask_c c aargax((_tt dim=1) y) Correct predictions
mask_ ic not mask_c # Incorrect predictions
P_s F. F.softmax P_t F. (2_tt Т, dim=1)
т, dim=1)
# Compute Accuracy Aligned loss
aa kd alpha KLD [mask_ c], p_t[mask c])
CE(z_s [maskk_]]_ y[mask_c])
Compute Error Correction loss
ec kd KLD [mask_ick ic], p_t p_(mmkkiic]) _i]]
beta * CE(z_s [mask_ ic], [mask_iaa _c]]
# Compute total MCD loss
acd_loss- = aa_kd ec_kd
# Return the MCD loss return mcd_loss
A.2 Experiment Setup and details
We adopt the same experimental settings in previous studies (Zhao et al. 2022 Jin et al. 2023; Sun al, 2024) for CIFAR-100 and ImageNet. All results are averaged over three independent trials. The optimizer is SGD and the lcarning rate is multiplied by a factor of 0.1 after [150, 180, 210] epochs. The total number of epochs is 240, except for MLLD and MCD+MLLD, where the number of epochs is 480. By default, temperature set to 20, set to 2.0 and a is set to 12. For CNN-VIT and VIT-CNN experiments, we follow the same sctup as Lv et al. (2024).
We conduct experiments using both homogeneous and heterogeneous leacher-student pairs. In homoge- neous pairs, the teacher and student models belong to the same architectural family, whereas in heterogeneous pairs, the teacher and student come from different architectural families. We evaluate our approach across range of architecture families, including ResNet (He et al., 2015), Wide ResNet (WRN) (Zagoruyko & Komodakis 2016). VGG (Simonyan & Zisserman, 2014), NhufleNet-V1/VV (Zhang et al. 2018 Ma et al. 2018), and MobileNetV2 [Sandler et al. 2018).
We evaluate the effectiveness of our approach against various feature based and logit based approaches including KD (Hinton et al.) 2015
FitNet (Romero et al.) 2015), RKD (Park et al.) 2019). CRD Tian
et al. 2022 OFD (Heo et al.]
2019
ReviewKD (Chen et al. 2021), TAKD (Mirzadeh et al. 2020
DKD
(Zhao et. al. 2022
MLLD (Jin et al. 2023) and CAT-KD (Guo et. al.
2023
B Datasets
CIFAR-100 (Krizhessky & Hinton 2009 is a benchmark dataset composed of 60,000 RGB images, each of size 32 32 pixcls, evenly distributed across 100 classes. The dataset split into 50,000 training images and 10,000 test. images, with each class containing 600 images. ImageNet (Deng et al. 2009 is large-scale
16

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Medical imaging datasets for clinical evaluation.
Dataset
Modality
Task
# Classes
# Samples
Train Val/Test
DNSST
Dermatoscope
Multi-class
7
10,015
7,007/1 1,003/2,005
Breast Ultrasound
Binary
2
780
546/78/155 156
OrganMNIST--
CT (Sagittal)
Multi-class
11
25,221
13,940/2,452/8.522
dataset widely used for image classification tasks. The dataset consists of over 1.2 million high-resolution training images and 50,000 validation images, categorized into 1,000 distinct classes.
B.1 MedMNIST Subsets and Protocol
We select three representative medical imaging datasets from MedMNIST
v2 YYng et al. 2023 DermaM-
NIST (dermatoscopic images, 7-class skin lesion diagnosis), BreastMNIST (breast ultrasound, binary tumor classification), and OrganMNNSTT-S (sagittal CT, 11-organ recognition). Table 7 summarizes the dataset charactcristics.
We follow the standard MedMNIST training protocol with images resized to 224 224 pixels. ResNet-50 serves as the teacher model, while ShuffleNe (SV2) and MobileNet V2 (MV2) are used as lightweight student models suitable for mobile deployment. We compare MCD against student-only training, traditional KD (Hinton et al. 2015), and DKD (Zhao et al. 2022 For MCD, we set a = and B = 2. All experiments are averaged over 3 runs.
C Teacher Training Statistics
The training accuracy for all teacher models across datascts are reported in Table These statistics clarify the activation frequency of the ECKD branch: higher training accuracy means fewer teacher errors and thus less ECKD activation.
Table 8: Training accuracy of teacher models across all datasets. CIFAR-100 CNN teachers exhibit severe overfitting (>99%), limiting ECKD activation. ImageNet, MedMNIST, and CIFAR-100 ViT teachers show mcaningful error rates (3-15%) that activate the error-correction mechanism.
Datasct
Teacher
Training Acc (%)
Error Rate (%)
ResNet56
97.91
2.09
ResNet1 10
99.54
0.46
ResNet3224
99.97
0.03
CIFAR-100 (CNN)
ResNet50
99.96
0.04
WRN40-2
99.88
0.12
VGG13
99.96
0.04
Conw-eXtt-Tnyy
93.08
6.92
CIFAR-100 (VIT)
Swin-Tiny
94.47
5.53
ResNet34
84.69
15.31
ImageNet
ResNet50
87.46
12.54
ResNet50 (DermaMNIST)
95.01
4.99
MedMNIST
ResNet50 (NrgaNMNIST-S)
97.23
2.77
ResNet50 MNIST)
92.12
7.88
CIFAR-100 CNN teachers are severely overfitted, making gains primarily attributable to the AAKD branch. To enable some ECKD activation under such conditions, we retain teachers in training mode (Section 6.1). ImageNet teachers exhibit 12 12-15% error rates, providing substantial ECKD activation. VIT teacher
17

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Accuract
Teeer
200
Figure 2: Performance different KD techniques, with accuracy plotted on the y-axis and the corresponding computation time per batch on the x-axis on CIFAR-100 with teacher-student pair as ResNet32x4 ResN-teNN.. ResNett2xx4ReeNeesss..
CIFAR-100 show moderate error rates (5%), and notably ConvNeXt-Tiny's higher error rate (6.92%) corre- lates with larger MCD improvement (+2.61% vs +1.48%). MedMNIST's data-scarce setting yields higher teacher orror rates and corrospondingly larger MCD gains.
D Additional Results
D.1 Training Efficiency (No Param Overhead)
MCD retains the same time complexity as traditional Knowledge Distillation (KD), making a highly efficient approach. Moreover, MCD achieves its performance improvements without introducing any ad- ditional parameters as shown in Figure 2 ensuring that the training process remains straightforward and resource-effectve..
D.2 Isolating AAKD vs. ECKD on ImageNet
Our earlier ablations (Table examined the interplay between Accuracy-Aligned Knowledge Distillation (AAKD) and Error-Correction Knowledge Distillation (ECKD) by varying the scnsitivity of the hyperpa- rameters and B. While those results highlighted the trade-off between the two components, they did not directly disentangle their individual contributions. To provide clearer picture, now conduct additional study ImageNet with heterogeneous teacher reacher-student student pair (Res.et50 - MobileNetV1)..
Ablation Design. Table g shows the resulting loss functions. Crucially, both ablation configurations apply identical loss to correct and incorrect samples, effectively removing the gating mechanism and isolating each branch's contribution.
Table 9: Summary of ablation loss functions and calibrated top-1 accuracy on ImageNet with ResNet50 Accr: accuracy when leacher correct; Acc-r: accuracy when teacher incorrect.
Configuration Correct Samples
Incorrect Samples
Acc.
Accr
Acc-T
MCD (full)
12. 12. LKD +1 +1.
1 LKD + LCE
73.65
90.8
18.01
ECKD-all
1 LKD +2. Lee
1 LKD +2. LCE
68.09
83.8
17.9
AAKD-all
12 LKD +1. -CF
12 LKD + LoE
70.7
88.4
14.1
Table reports the calibrated top-1 accuracy under three configurations: (i) the full Mentor- Distillation (MCD), (ii) ECKD-all and (iii) AAKD-all. The results yield several important obscrvations. First, both AAKD and ECKD in isolation underperform the full MCD framework, indicating that neither component alone sufficient to capture the full benefit of correctness-wware distillation. Second, AAKD-all improves
18

---

Published in Transactions on Machine Learning Research (02/2026)
accuracy when the teacher is correct (Acer), but struggles significantly when the teacher is wrong (Acc-r = 14.1), underscoring the risk of blindly following the teacher's knowledge. Conversely, F.CDD-all provides stronger correction when the teacher is incorrect (Acc-r = 17.9), but suffers from limited regularization when the teacher correct, leading to reduced overall accuracy. Finally, MCD effectively combines these strengths, achieving the best overall performance (73.65%), the highest Accr (90.8), and a competitive Ace-T (18.01).
These findings clearly demonstrate the complementary roles of AAKD and ECKD: AAKD leverages the teacher's 's reliability for regularization, while ECKD safeguards against negative transfer when the teacher errs. Their synergy within MCD enables robustness across both scenarios, validating the necessity of correctness- aware decoupling in knowledge distillation.
D.3 Component Ablation on CIFAR-100
As described in Section 6.1, we relain leachers in training mode during distillation. For ResNet32x4, the teacher's prediction accuracy on the training set changes from 99.97% (eval mode) to 96.96% (train mode) due to dropout /BN stochasticity, so ECKD activates on approximately 3% of samples.
Table 10: Component ablation CIFAR-100 (ResNet32x1 ResNet8x4). All methods use identical train- ing protocol: SGD with LR=0.05, momentum=0.9, weight decay=5e-4, following standard KD evaluation practice.
Configuration
Loss Function
Accuracy
Vanilla KD (A = 0.9)
0.9 LKD + 0.1 LoE
73.33%
AAKD-all (no gating)
12 LKD 1-CCE Eq. 6-7
75.65%
MCD (with gating)
76.81%
Gating contribution (MCD AAKD-all)
+1.16%
Crucially, both AAKD-all and MCD use identical train-mode teachers, ensuring the +1.16% improvement is allributable solely to correctness-aware gating.
Although the KD/CE mixture can be renormalized to an equivalent form in principle, in practice chang- ing KD/CE coofficients changes the gradient scale and interacts with the fixed optimizer/schedule (SGD momentum, weight decay, LR schedule). Following standard KD literature and benchmarks, we keep the optimizer configuration fixed across methods rather than renormalizing the objective and jointly retuning optimizer hyperparameters for all basclines.
19


Reviews:
[{"review_id": "V4uzPefM5W", "reviewer": "Reviewer_wPDz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a simple modification to the most popular (vanilla) knowledge distillation (KD) for image classification to make the algorithm more effective. The authors note that a teacher is not always accurate for training images, and in those cases when it is not accurate, it might not make sense to do knowledge distillation. So, at each step of the training, they check if teacher's prediction for an image is correct; if it is, the weight for the distillation objective is increased, and if it is not, the weight for the default cross-entropy loss is increased. This modification requires very minimal overhead compared to vanilla KD. The proposed method achieves better performance than the vanilla KD + some other baselines on multiple datasets - CIFAR-100 and ImageNet - and also on a medical imaging dataset.\n\n### **Strengths**\n- The proposed technique is very simple, addresses a very intuitive limitation of the vanilla KD setup.\n- There is little to none computational overhead compared to vanilla KD. Consequently, compared to many other baselines, the proposed method needs relatively less compute to train a student model.\n- The proposed method achieves better performance compared to many baselines on multiple datasets.\n\n### **Weaknesses**\n- It is not how often the scenario mentioned by the authors - teacher predicting incorrect labels - happens for CIFAR100. Consequently, it is doubtful if the proposed method, while being principally sound, has any need practically speaking.\n- The proposed method does not perform better than one of the baselines.\n- The improvements, while being somewhat consistent, lean more towards marginal than a clearly convincing improvement.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims made in the submission are accurate, but not convincing and/or with clear evidence.\n\n- The authors' main idea is that when the teacher gives an incorrect prediction for an image, its (incorrect) knowledge should not be used during distillation. The problem seems to be that, in practice, for CIFAR100 at least, all kinds of teachers are almost always accurate on the training set, all having accuracy of >99% (Table 8; appendix). Therefore, the *critic* block, which engages the error correction loss (Eq. 9) will rarely be used. Furthermore, CIFAR100 results cover a decent section of the overall experiments section.\n\n- Following on from the previous point, the discussion in the section **Homogenous vs heterogenous settings for CIFAR 100** has some confusing points:  \n    - “*CIFAR-100 teacher models achieve very high training accuracy*” —  it is not clear what this has to do with whether it is a homogenous or heterogenous setting. In either case, the teacher does have close to 99% training accuracy.\n    - “*On the other hand, in heterogeneous pairs …. the critic branch has more room to operate*” — again, this is incorrect. All the teachers used for heterogenous settings (Table 2) have more than 99% training accuracy, as mentioned above. This means that similar to the homogenous setting, the heterogenous setting also doesn’t activate the error-correction loss much. The critic branch’s operation (checking whether the teacher is accurate for a particular image) has nothing to do with the discrepancy in performance between the teacher and (non-distilled) student.\n\n- While the proposed method shows a somewhat clear improvement in performance compared to vanilla KD, the way the authors have presented the results in Tables 1 and 2 seems to be misleading. “The previous state-of-the-art method, DKD” — this statement is not true. The previous state of the art is MLLD. It matters little if it is not “logit-alone”. This method achieves consistently better performance than MCD (proposed method). I do not quite understand why MLLD has to be considered a separate category of method. The authors can discuss the computational overhead, i.e., a potential disadvantage of MLLD separately. On that note, the figure 2 in the appendix doesn’t have a discussion for MLLD.\n    - On that point, it is again misleading that the authors have not mentioned anywhere in the text that MLLD performs better than the proposed method.\n    - Generally speaking, when one is comparing to a baseline (e.g., MLLD), you have to treat it as a complete method that needs to be compared to. I sense that the authors are saying that because MLLD does augmentation in the logit space, a direct comparison with it is unfair (hence; separate sections in Tables 1 and 2). If that is indeed how the authors are thinking, then that will be wrong.", "audience_yn": "Yes", "audience_explanation": "- If people are looking for a simple trick to help improve the KD performance, especially in the black box scenarios where one does not have intermediate feature maps for the teacher/student network (e.g., FitNet or CRD), then the proposed method will be a nice option to use. \n\n- If people are looking to further look into the hood of why KD works the way it works, when the teacher is useful, when it is not useful, this work sheds some light on those topics.", "requested_changes": "- The main idea of the proposed method is applicable only in those cases where the teacher is even making some mistakes. If the teacher never made a mistake during training (predict incorrect label), this paper's approach will reduce to vanilla KD. Right now, the main dataset focused, i.e., CIFAR 100, does seem to be in that domain. Ideally, the authors should include those settings in which the supposed scenario of teacher being sometimes incorrect is actually true. While ImageNet might be an example already, the authors need to give more details about its training statistics. Plus, ideally consider some replacement to CIFAR 100. \n    - This will for sure strengthen the work, but to get a clear sign for acceptance, the work will ideally also need to do a bit more study about what is going on in the process, as I explain below.\n\n- The core idea of not distilling when the teacher is inaccurate, while being intuitive, does not quite explain some rather unintuitive results in the domain of knowledge distillation. Specifically, the idea of self-distillation (e.g., born again neural networks [1]) should be noted, where a student trained in the previous iteration acts as the teacher for training the same student architecture in the next iteration, and often times ends up improving the performance. What that means is that distilling even on training samples where the prior student (teacher) was inaccurate can lead to a more accurate student. This might suggest that even in the case of a traditional bigger teacher, teaching on incorrectly classified samples might still impart some useful knowledge. The authors should think about the possibility of this, and explain why this might/might not be true in their case.\n    - This analysis can make the paper very strong. \n\n- While it will be nice if these investigations can lead to some improvement in performance, especially compared to MLLD baseline,", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qZYjyfGoEE", "reviewer": "Reviewer_pJSC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Mentor-Critic Distillation (MCD), a novel knowledge distillation framework that dynamically adjusts knowledge transfer based on teacher correctness. Its core contribution is a correctness-aware weighting mechanism that amplifies distillation loss (α·ℒ_KD) when the teacher is correct (Accuracy-Aligned KD) and emphasizes cross-entropy loss (β·ℒ_CE) when the teacher is wrong (Error-Correction KD). The method is theoretically grounded, with proofs showing it generalizes vanilla KD and attenuates negative transfer, and empirically validated across CIFAR-100, ImageNet, and MedMNIST, where it consistently outperforms state-of-the-art alternatives without adding parameters or computational overhead.", "claims_and_evidence_yn": "Yes", "claims_explanation": "claims made in the submission are supported by accurate, convincing and clear evidence", "audience_yn": "Yes", "audience_explanation": "TMLR's audience is interested in knowing the findings of this paper", "requested_changes": "1. Evaluations on image classification tasks and CNNs are outdated. Lack of recent evaluations on ViT, multimodal models, etc.\n\n2. Lack of application and evaluation of the proposed method for tasks other than classification such as detection, segmentation, etc. Expanding the evaluation to include a broader range of tasks, such as object detection, would provide a more comprehensive understanding effectiveness. For instance, recent works like \"DetKDS: Knowledge Distillation Search for Object Detectors\" (ICML-2024) highlight the importance of testing KD methods across various domains.\n\n3. The proposed method requires additional modules and is cumbersome to design, limiting the generalizability of the method.\n\n4. The performance improvements reported over existing methods appear to be marginal in some cases. Given the rapid advancements in the field, as seen in works like \"Automated Knowledge Distillation via Monte Carlo Tree Search\" (ICCV2023) and \"KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs\" (NeurIPS-2023), it is crucial for the paper to demonstrate more substantial performance enhancements to justify its adoption.\n\n5. The paper lacks a comprehensive discussion of relevant prior studies (e.g., NORM (ICLR23), and \"DisWOT (CVPR23)) in the field of knowledge distillation.", "broader_impact_concerns": "The work presents a methodological improvement in knowledge distillation with broadly positive implications for efficient AI deployment, particularly in resource-constrained settings like mobile devices and medical imaging. The paper does not raise unique ethical concerns beyond those inherent to model compression application.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y2KlhzK6hN", "reviewer": "Reviewer_SbwG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Mentor-Critic Distillation (MCD), a knowledge distillation framework that dynamically adjusts knowledge transfer based on teacher model correctness. The core insight is that when the teacher predicts correctly, the distillation loss should be amplified; when the prediction is incorrect, the cross-entropy loss should be emphasized. This method requires no architectural modifications or additional parameters.", "claims_and_evidence_yn": "No", "claims_explanation": "This is a very straightforward method, easy to implement. \n\n\nBut I see a few major **weaknesses** here. \n\n1. The core insight is obvious and not novel. Adjusting weights based on teacher correctness is the most natural idea to try. The paper does not explain why this has not been explored before, or what non-obvious insight makes this method work. DKD already adapts weights based on teacher reliability as mentioned in the paper, and MCD simply makes this explicit.\n\n2. Correcting teacher errors is not new, either. Earlier works attempted this through different approaches such as logit calibration [1][2]. In the literature, this issue has been described as overconfidence and confirmation bias. You should make it explicit why your method is better than these directions.\n\n3. The main claim has not been convincingly verified. For example, in Table 1, the CIFAR-100 benchmark uses a teacher model with around 99% training accuracy, meaning the teacher makes almost no mistakes. From this setup, it is unclear whether the gains come from ECKD or simply AAKD. The MedMNIST training accuracy is also not reported. Additionally, you should test on a more complex dataset to properly evaluate the relative effectiveness of the two techniques.\n4. Comparisons need to be up-to-date. Several papers from 2024 and 2025 should be included for comparison with this method. Examples include [1],[2],[3] and [4].\n\n**References.** \n1. LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration, TMLR 2025\n2. Cross-View Consistency Regularisation for Knowledge Distillation, MM, 2024\n3. Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation, NeurIPS 2024\n4. Scale Decoupled Distillation, CVPR 2024", "audience_yn": "Yes", "audience_explanation": "The method is a true drop-in replacement for vanilla KD, requiring minimal implementation effort, which makes it highly valuable for practitioners.", "requested_changes": "See weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)