LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration
Authors: Md. Ismail Hossain
M M Lutfe Elahi
Sameera Ramasinghe
Ali Cheraghian
Fuad Rahman
Nabeel Mohammed
Shafin Rahman
Venue: Accepted by TMLR ()

Abstract: In the knowledge distillation literature, feature-based methods have dominated due to their ability to effectively tap into extensive teacher models. In contrast, logit-based approaches, which aim to distill `dark knowledge' from teachers, typically exhibit inferior performance compared to feature-based methods. To bridge this gap, we present LumiNet, a novel knowledge distillation algorithm designed to enhance logit-based distillation. We introduce the concept of `perception', aiming to calibrate logits based on the model's representation capability. This concept addresses overconfidence issues in the logit-based distillation method while also introducing a novel method to distill knowledge from the teacher. It reconstructs the logits of a sample/instances by considering relationships with other samples in the batch. LumiNet excels on benchmarks like CIFAR-100, ImageNet, and MSCOCO, outperforming the leading feature-based methods, e.g., compared to KD with ResNet18 and MobileNetV2 on ImageNet, it shows improvements of 1.5\% and 2.05\%, respectively.

Published in Transactions on Machine Learning Research (08/2025)
LumiNet: Perception-Driven Knowledge Distillation via Sta- tistical Logit Calibration
Md. Ismail Hossain
Apurba-NSU R&D Lab, North South University, Bangladesh
M M Lutfe Elahi
huffe.clahic@hhorthhsookkk
Apurba-NSU RED Lab, North South University, Bangladesh
Sameera Ramasinghe
Gameera@phuraatee
Pluralis Research
Ali Cheraghian
ali Wiiccherghhin@@laalll..cci
Data61-CSTRO, Australia Australian National Universityy Australia
Fuad Rahman
fuod@epurbatech.com com
Apurba Technologies, Sunnzvale, CA 94085, USA
Nabeel Mohammed
wuberr.mohhnnmmddhhhrttttuu....
Apurba-NSU R&D Lab, North South University, Bangladesh
Shafin Rahman
shaii..ruimmn@@@nrtsssut..dddd
ASurba-NSU R&D Lab, North South University, Bangladesh
Reviewed OpenReview: https: /ttp:r///ppereveiee net/ pormm? id- 3rU1 1p9u2l
Abstract
In the knowledge distillation literature, deature-based methods have dominated due to their ability to effectively tap into extensive teacher models. In contrast, logit-based approaches, which aim to distill 'dark knowledge' from teachers, typically exhibit inferior performance compared to feature-based methods. To bridge this gap, we present LumiNet, novel knowl- edge distillation algorithm designed to enhance logit-based distillation. We introduce the concept of 'perception', aiming to calibrate logits based the model's representation pability. This concept addresses overconfidence issues in the logit- based distillation method while also introducing a novel method to distill knowledge from the teacher. It reconstructs the logits of a sample/instances finstances by considering relationships with other samples in the batch. LumiNet excels on benchmarks like CIFAR-100, ImageNet, and MSCOCO, outperforming the leading feature- based methods, e.g., compared to KD with ResNet18 and MobileNet on ImageNet, shows improvements of 1.5% and 2.03%, respectively.
1 Introduction
The advancement in deep learning models has undergone significant increases in both complexity and per- formance. However, this progress brings challenges associated with computational demands and model scalability. To mitigate this, knowledge distillation (KD) has been proposed as an efficient strategy (Hinton et al., 2015) to transfer knowledge from a larger, intricate model (Leacher) to a more compact, simpler model (student). The primary objective to trade off performance and computational efficiency. There are two broad categorics of KD: logit and featuro-based stratogics (Romero et al., 2014; Tian al., 2020; Tung & Mori, 2019; Yim et al., 2017). In loset-based methods, a student model aims to match the probability distributions of teacher model by mimicking the raw logits(Zhang al., 2018; Mirzadeh et al., 2020; Zhao

---

Published in Transactions on Machine Learning Research (08/2025)
et al., 2022). In contrast, ceature-based methods are centered on aligning the intermediate layer representa- tions between the two models (Romero et al., 2014). In general, it has been observed that feature-based KD outperforms logit-based KD (Zhao et al., 2022). However, feature-based KD suffers from layer misalignment (Romero et al., 2014) (reducing sample density in this space), privacy concerns (Chakraborty et al., 2021) (intermediate model layers accessible for adversarial attacks revealing training data and posing significant threats); and escalating computational requirements (Yang et al., 2023; Zhao et al., 2022) (sce Figure. 1). These issues raise questions about its effectiveness, partieularly in industrial applications. While logit-based KD avoids the computational and privacy pitfalls of feature alignment, its performance gap relative to feature-based methods has limited its adoption in resource-constrained industrial settings where efficiency and robustness are paramount. Similarly, these issues underscore the potential merits of logit-based KD over feature-based KD. This paper aims to enhance the ivennss of logil-based knowledge distillation by leveraging its underlying strengths.
Several reasons underpin the disparity between logit- and feature-basod KD. Firstly, a significant issue in logit-based knowledge distillation, also observed in data distillation (Zhu el al., 2023), is the lendency of the teacher model towards overconfidence (Zhang et al., 2024), which assigns disproportionately high probabilities to certain classes and sometimes misclassifics instances with unfounded certainty. Overconfidence refers to the teacher's overly confident predictions that can obscure useful information and destabilize student learning, This overconfidence poses a challenge to the optimization of the student model, as steep gradients (see Section 3) arise from the teacher's high probability output. Although temperature scaling (Hinton et al., 2015) is commonly used to soften these probabilities and better reveal the "dark knowledge" (Furlanello et al., 2018; Zhao et al., 2022) in non-target classes, identifying an optimal temperature remains non-trivial (Kim et al., 2021; Chen et al., 2021a; Wang & Yoon, 2021). Secondly, in relying solcly on logit matching, any considently wrong predictions from the teacher can be quickly inherited by the student, exacerbating confirmation bias (Zhang et al., 2024). Confirmation bias refers to the student's tendency to replicate the teacher's incorrect predictions without correction, reinforcing errors. Previously, the authors (Zhang et al., 2024) attempted to address this issue by sacrificing the teacher's knowledge, as they disregarded outputs that exceed a certain threshold. Moreover, raw logit matching can introduce additional challenges. (Cho & Hariharan, 2019) hypothesized that while student model can imitate the teacher, it fails to enhance accuracy. When the student struggles to replicate the teacher, it indicates mismatch in their capacitios. In both cases, issues arise, particularly with regard to high-capacily teachers. These challenges suggest that the fundamental problem lies within the raw logits themselves, highlighting the need for calibration to resolve these issues.
To address the challenges mentioned earlier, we propose LumiNet, a perception-driven knowledge distillation (KD) method that leverages logit calibration to mitigate overconfidence and confirmation bias by normalizing predictions using batch-level statistics, as illustrated in Figure. 1(e). Unlike conventional methods, which treat logits independently, LumiNet utilizes batch-wind wide contextual relationships to recalibrate logits into balanced, uncertainty-aware distributions, termed perception logits. Specifically, for each class j, we compute the mcan U, and variance V, of the model's logits across the batch. Each logit is then normalized relative to these iatistics, inherently suppressing extreme confidence values without explicit lemperal ure scaling. This normalization helps to adjust overly confident predictions, effectively reducing the teacher's mistakenly high- confidence outputs. Due to dass-wise normalization, the inter-class relationship (dark knowledge) alterod, as normalization introduces dependencies between logits across different classes in the batch. We call this new representation of instances 'perception' logits. The student model learns through two complementary objectives: (I) a cross-entropy loss with ground-truth labels to maintain fidelity to correct class boundaries, and (2) a KL divergence loss aligning student and teacher perception logits to transfer contextualized decision- making patterns. Fapirical evaluations confirm LumiNet's 's ivenes, demonstrati ing substantial accuracy improvements for instance, boosting ResNet8x4 4 performance on CIFAR-106 from 73.3% to 77.5% while ensuring practical efficiency and applicability for real-world scenarios.
LumiNet is grounded in Kurt Lewin's field theory of Gestalt psychology (Lindorfer, 2021), which posits that the behavior of an cntity is shaped by the interaction of forces within its environment. In LumiNet, when teacher model generates overly confident prediction for a sample, correction mechanism inspired by Kurt Lewin's field theory comes into play. The batch of data serves as a contextual environment where each
2

---

Published in Transactions on Machine Learning Research (08/2025)
75
VaTeechherVGGL3 Student:Moblenn Moblent:MobleNee--
1b/Fechhe:Reseet:3444 StudentRReNee-18
(c]TeacherRRssMe Student: Res
74
73.5
Teacher
4 4
Teacher
Accuracy (%)
73
73.0
72
71
72.5
40
70
010
72.0
38
71.5
68
67
71.0
36
66
65
Logit based
Feature based methods
70.5
30 60 90 120
10
12
14
18
20
15 20 25 30 35
45
Parameters (in millions)
Parameters (in millions)
Parameters (in millions)
Instance Representation
Instance Representaaiionperrcetiin))
Logit
Logit
80
40
3:00
Class Index
Class Index
Softmax Values
Softmax Values
Softmax
Softmax
0.1
0.00
20
40
80
100
20
40
80
100
Class Index
Class Index
Inter-Class Relationships (Top 10 Probabilities)
Inter-Class Relationships (Top 10 Probabilities)
13wh 21.
sweet peppers
seeet peppert
40p
(2)
0.08
0.015
0.020
Softmax Value
Softmax Value
(d) CIFAR-100 - Before Perception
(e) CIFAR-100 After Perception
Figure Performance comparison of feature-based and logit-based methods on (a) CIFAR-100, (b) ImageNet, and (c) MS COCO datasets. Our proposed LumiNet, logit-based method, achieves high accuracy without using extra parameters. (d-e): An example of (Left) before and (Right) after applying our proposed concept perception' on teacher's predicted prohabilities. The first set of plots in the top row shows spikes in raw logits for the targeted class (Sweet Peppers, represented in red). This representation changes after the application of perception. Notably, various classes exhibit similar magnitudes to the targeted class, indicating reduced specificity. In the second set, despite conventional knowledge distillation softening, as seen in the left figure, there's 's persistent overconfidence in the target class. Perception minimizes the difference in softmax values between targeted and non-targeted classes, as depicted in the right figure. The third set illustrates inter-class relat colstioss amoug the top 10 classes. While conventional scenarios (left) maintain these relationships, the perception method siguificantly alters them (right).
sample's logits are influenced by surrounding instances, creating a 'field' of interactive forces. Through nor- malization, LumiNet scales down unusually high confidence values by comparing them to the batch statistics for the same class, implementing what Lewin would describe as balancing "positive forces' (pulling samples toward consensus) and "negative forces' (restraining outliers). This dynamic adjustment process termed "perception" i treeats each sample's logits within the context of surrounding instances, mirroring how hu- man perception adapts to environmental stimuli. By viewing the batch as a Gestalt environment where each sample's representation is influenced by the whole, LumiNet creates a more robust system that pre- serves valuable teacher model information while preventing individual outliers from having disproportionate influence, ultimately improving model performance through environmentally contextualized learning.
The performance of LumiNet was evaluated across multiple computer vision tasks, including image recogni- tion, object detection, and transfer learning. Results demonstrate strong effectiveness, particularly when us-
3

---

Published in Transactions on Machine Learning Research (08/2025)
ing ResNet8x4 student model, which achieved 77.5% accuracy. LumiNet established benchmark-leadng performance on key datasets including CI0AR100, ImageNet, MS-COCO, and TinyemageNet. Beyond vision applications, we also adapted this method to the GLUE Benchmark and small language models.
Our contributions are as follows:
We propose a novel knowledge distillation framework that recalibrates logits using batch-level class statistics, dynamically suppressing overconfidence and confirmation bias.
We establish LumiNet's superiority across multiple teacher-student pairs (e.g., ResNet- 50-MoobieNNet-W V2) and diverse tasks (classification detection, transfer learning), achieving 70.97% accuracy CTFAR-100 (+3.622% over standard KD).
2 Related Works
Logit-based KD: In the domain of KD, logit-based techniques have traditionally emphasized the distil- lation process utilizing solely the output logits. Historically, the primary focus of research within logit distillation has been developing and refining regularization and optimization strategies rather than exploring novel methodologies. Noteworthy extensions to this conventional framework include the mutual-learning paradigm, frequently referenced as DML (Zhang et al., 2018), and incorporating the teacher assistant mod- ule, colloquially termed TAKD (Mirzadch et al., 2020). Nonetheless, a considerable portion of the cxisting methodologies remain anchored to the foundational principles of the classical KD paradigm, seldom prob- ing the intricate behaviors and subtleties associated with logits (Zhao et al., 2022). A novel approach to object detection distillation, combining feature-based and logit-based methods with a closod-loop knowledge distillation framework, has demonstrated improved accuracy and robustness compared to existing state- of-the-art techniques (Song et al., 2024). Recently, TTM (Zheng & Yang, 2024) has been proposed method that introduces Rényi entropy regularization into the KD objective by removing temperature scal- ing on the student side and reinterpreting it as a power transformation of probability distributions. This regularization implicitly enhances generalization performance. However, TTM does not model inter-instance dependencies or use batch-wise statistical structures for knowledge transfer and also learns from raw logits like other methods. While the versatility of these logit-based methods facilitates their applicability across diverse scenarios, empirical observations suggest that their efficacy often falls short when juxtaposed against feature-level distillation techniques.
bastere-based KD: Feature distillation, knowledge transfer strategy, focuses utilizing intermediate features to relay knowledge from a teacher model to a student model. State-of-the-art methods have com- monly employed this technique, with some working to minimize the divergence between features of the teacher and student models (Heo et al., 2019b;a; Romero et al., 2014). A richer knowledge transfer is facili- tated by forcing the student to mimic the teacher al the feature level. Others have exlended this approach by distilling input correlations, further enhancing the depth of knowledge transfer (Park et al., 2019; Tian et al., 2020; Tung & Mori, 2019; Chen et al., 2021b). DiffKD (Huang et al., 2024), a novel knowledge dis- tillation method utilizing diffusion models to denoise and align student features with teacher features, has demonstrated state-of-the-art performance across image classification, object detection, and semantic seg- mentation tasks These methods, though high-performing, struggle with substantial computational demands and potential privacy issues, especially with complex models and large datasets. These challenges not only amplify processing time and costs but can also limit their practical applicability in real-world scenarios. Recognizing these challenges, we turn our attention to logit-bascd distillation techniques.
Applications with KD: Rooted in foundational work by (Hinton et al., 2015) and further enriched by ad- vanced strategies like Attention Transfer (Zagoruyko & Komodakis, 2017), ReviewKd (Chen et al., 2021b), Decoupled KD (Zhao et al., 2022) and other methods (Park et al., 2019; Tian et al., 2020), KD has signifi- cantly improved performance in core vision tasks, spanning recoguition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016), segmentation (Qin et al., 2021; Liu et al., 2019), and detection (Li et al., 2022a; Yang et al., 2022; Zheng et al., 2023; Xu et al., 2022). Beyond vision, KD has also made notable strides in NLP tasks like machine translation and sentiment analysis (Kim & Rush, 2016; Zhang et al., 2022). KD has proven valuable in addressing broader AI challenges, such reducing model biases (Hossain et al.,
4

---

Published in Transactions on Machine Learning Research (08/2025)
2022; Chai et al., 2022; Zhou et al., 2021; Jung al., 2021) and strengthening common-senss reasoning (West et al., 2022). We assess our method in the contexts of image classification and object detection.
3 Methodology
3.1 Knowledge Distillation Revisited
KD aims to transfer knowledge from high-capacity teacher model fr to compact student fs by minimizing the divergence between their outputs. Let X' {x denote a dataset where x; E R" and let z = fx(x)) and z = {x,) represent the logits (pre-softmax outputs) of the teacher and student for sample x,- Traditional KD minimizes the Kullback-Leible (KL) divergence between the softened output. distributions of fr and fs:
LKD
Σ (L 11 (T), X.EX
where a denotes the softmax function and T > 0 is temperalure parameter that smooths the distributions to amplify "dark knowledge" in non-target classes (Hinton et al., 2015; Zhao et al., 2022).
Limitations of Logit-Based KD: Despite its simplicity, logit-based KD faces a few critical challenges:
1. Overeonfidence: Neural-network models often exhibit overconfident predictions, where the softmax prob- ability for the target class approaches (o((z))) ) + 1), suppressing non-target class probabilities. This diminishes the dark knowledge crucial for student learning (Zhao et al., 2022; Zhang et al., 2024).
2. Confirmation Bias: Students trained on overconfident teachers inherit errors through confirmation bias (Zhang et al., 2024). Once the student aligns with teacher's incorrect prediction, subsequent training struggles to correct this due to steep loss gradients around overconfident logits (see Section 3.3).
3. Temperature Sensitivity: While temperature scaling (r) mitigates overconfidence by softening outputs, tuning 7 is non-trivial. Small т preserves overconfidence, while large over-flattens distributions, crasing inter-class relationships (Kim et al., 2021; Chen et al., 2021a). Prior work (Wang & Yoon, 2021) notes that optimal varies across tasks and architectures, necessitating costly per-dataset tuning.
4. Mismatched Capacity and Multi- Obffe-Obbeetivee Complexities: Leading logit-based distillation methods (e.g.. DKD (Zhao et al., 2022) and MLLD (Jin et al., 2023)) often rely on multiple objective functions to cap- ture both target and non-target class information. While this can yield richer student supervision, it also introduces additional hyperparameters and computations ional overhead, complicating large-scale or industrial deployment. Moreover, when the teacher is substantially larger than the student, capacity mismatches can arise, making difficult for the student to effectively replicate the teacher's output space (Cho & Hariharan, 2019). This mismatch may undermine the advantages of having more accurate teacher, as the student cannot fully utilize the additional knowledge.
Why Logit-Basce KD Still Matters: Although featuro-based KD techniques (Romcro et al., 2014; Tian et al., 2020) often yield higher accuracy by aligning student and teacher representations, they require internal teacher parameters and thus introduce privacy concerns (e.g,, vulnerabilities to adversarial attacks on intermediate layers (Chakraborty et al., 2021)) as well as significant computational cost (Yang et al,, 2023)). Logit-based KD, which relies outputs alone, addresses these challenges by minimizing the need for teacher internals, making it particularly advantageous for privacy-sensilive setups (such as federated learning) and resource-constrained environments (like edge devices). Thus, improving logit-based KD is a technical challenge as well as a practical necessity for real-world scalability.
3.2 Introducing LumiNet
Based on previous discussion, while point-wins wise knowledge distillation remains fundamental to transferring teacher knowlodge, the key limitation lies in treating each instance's logits in isolation. We observe that a teacher's prediction for any instance carries implicit relationships with its predictions for other instances, particularly those sharing similar features or decision boundaries.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Image batch
teacher
Student
(# 4aa siss
X4
Xs
X6
(4 Alg
hi
h
h2
U
* hg
hg
hi
h (NE. Mig, Ma. M()
by (hi Ma, (4s
Figure 2: Given a batch of samples B {x1, X2, X3, X4, X5, xa), both the teacher and student, models generate logit for each sample in the batch, denoted as z (2 2/2- z/3- 244) and z (=f 2/2 2/3 244). In the subsequent stage, (Uan((U) Uf)) and variance Vt Vj²) for each class in the batch are computed for both teacher and student logit. These values are then used normalize the logit of both models, resulting in new logit representation referred to as the Perception logit: h = (ht M2, haa Maa) and h = (h hr hig h(a). Finally, a loss function l is calculated between the teacher and student to complete the knowledge distillation process.
Hypothesis: Traditional knowledge distillation treats logits independent outputs, transferring knowledge per-instance basis. However, model's prediction for any instance is inherently shaped by the contextual relationships within batch, particularly among instances sharing similar feature distributions decision boundaries. This aligns with Kurt Lewin's Field Theory from Gestalt Psychology (Lindorfer, 2021), which posits that an entity's s behavior is influenced by the surrounding forces in its environment.
In the context machine learning, we hypothesize that a model's logits should not be interpreted in isolation but rather as influenced by the batch as a whole. Specifically, each logit is affected by positive forces (alignment with batch-wide consensus) and negative forces (suppression of outliers). This interplay ensures that distillation captures not only instance-level knowledge but also the structural dependencies across the batch. To formalize this, we introduce perception function, denoted as:
E = Flzig ZB),
where: - 24y is the original logit for instance i and class j. Zg represents the set of logits across the batch, FF.) is a contextual transformation that adjusts each logit. based on batch-level interactions.
The function F encodes batch-aware knowledge by modulating each logit in proportion to the surrounding batch distribution. This ensures that extreme logits are tempered, reducing overconfidence, while preserving the mutual information across related instances.
Constructing the perception: To support our hypothesis, we present our approach as follows:
Considering a batch of data samples = { which is randomly selected from the original dataset X. Consoquently, the logits generated by a model for an instance x; E B across classes are represented as:
2, (2,1, Z.,...,, 2ic); where 24 symbolizes the logit for the jth class for instance 2,. We adjust the logits based the mean U, and variance Vy across each class j of batch. This transformed logit is given by:
6

---

Published in Transactions on Machine Learning Research (08/2025)
by Riy 1 U, Here, hy represents the augmented logit for the jth class for instance x,. Consequently, the V augmented logits for instance, x; are obtained as:
Zi1 U1
22 U2
Zie Up
h
(1)
VV1
VV2
VVe
In this conlext, the reconstructed logits h; serve as a normalized "perception" " of each instance, since every class's logits are shifted and scaled by their batch-wise mean Uj and variance Vy. By enforcing his -ak-Ui Un. any overconfident 2sk > 25 is automatically dampened when V& is large, so that by Fiy
VV. has Eik. This re-
scaling redistributes information across the batch via intra-class dependencics and yields more balancod softmax outpuls. Consequently, when computing LtamiNes, the KI. divergence is applied to these calibrated distributions, making distillation less sensitive to temperature scaling.
The perception transformation establishes a principled optimization-theoree framework for knowledge dis- tillation through variance stabilization and gradient preconditioning. For teacher logits with hederoscedastic class-wise variances Vj where max, Vi
min, = > 1, the perception mapping haj Ey U. transforms the gra-
3z inducing effective preconditioning A = D D where D = SiaggvvV.......... vVc). This improves conditioning and accelerates convergence while ensuring uniform gradient variance Var @CLumiNet
dient operator as @CLumiNet = =V
0CKD
Sh
84 = O(1) across classos. The approach complements (Menon et al., 2021)'s per-class nor-
statistical analysis showing that soft label distillation reduces empirical risk variance, malization similarly enforces homoscedasticity through direct gradient preconditioning. This mechanism also relates to Neural Collapse (Papyan et al., 2020), where diminishing within-class variability during terminal training results in improved conditioning and geometric regularity. The transformation preserves cross-class relationships while achieving hit ~ N0 1) normalization.
The LumiNet Loss: Traditional logit-based KD methods align raw logits or softened probabilities (Hinton et al., 2015; Zhao et al., 2022; Jin et al., 2023; Zheng & Yang, 2024; Zhang et al., 2024). In contrast, LumiNet aligns the perceplion logils of teacher and student, ensuring the student learns the teacher's s contextualized decision patterns. The LumiNet loss LaminNet is defined as the KL divergence between the softened perception logits of the teacher (h²)) and student (h$):
LlamiNett = EK KL (o(() [[o(h///))() X,EB
(2)
where T is temperature parameter. LumiNet derives its name from its ability to illuminate ('Lumi') the dark knowledge in logits via batch-level statistical perception. 'Net' refers to the implicit network of renational connections formed among samples during distillation, capturing structural dependencies that guide more calibrated learning.
The complete training objective for our knowledge distillation framework com- bines the traditional cross-entropy loss LoE with our proposed LumiNet loss LLumiNet- While Leg operates on the raw logits z; and ground truth labels ys to ensure correct classification:
Total Loss Formulation:
N
C
LCB = N
1
Σ =1
Vie log(
(3)
where (Die) is the softmax probability, (yie) is the ground truth label, (N) is the batch size, and (C) is the number of classes. The NemiiNt term works with the perceived logits h, to transfer the teacher's perceptual knowledge to the student. The total loss is thus formulated as:
Liotal Loe + ALLumiNet
where is a balancing scalar that controls the contribution of the LumiNet loss. This dual-objective op- timization ensures that the student model learns to correctly classify instances through Loe and acquires the teacher's rich perceptual understanding through LLumsNer. A theoretical validation can be found in A Appendix A.1-A.6.
7

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Recognition results on the C1FAR-100 validation, averaged over five trials with standard deviation.
(a) Same architecture
(b) Different architecture
WRN40-2
WRN40-2
WRN40-2
RN110
RN32x4
VGG13
RN32X4
VGG18
RN52X4
RNN66
HNN5
Teacher
72.34
74.31
79.42
75.61
76.61
74.64
79.42
75.61
74.04
79.34
79.42
WRN16-2
WRN40-1
RN20
RN32
RN8x4
VGGS
SN-VI
SN-VI
MN-V2
MN-V2
SN- V2
Student
69.06
71.14
72.50
73.26
71.98
70.36
70.30
70.50
64.60
64.60
71.82
Feature-Bass Baaaee Methods
Featuro- Based
FitNet [Romerr al, 2014)
69.21
71.05
73.50
73.58
72.24
71.02
13.59
73.73
64.14
63.16
73.54
RKD [Park 2019)
69.61
71.82
73.35
71.48
72.28
72.21
64.52
64.43
73.21
CRD (Tian ot ol., 2020]
71.16
73.48
75.51
75.48
74.14
73.94
75.11
76.05
69.73
69.11
75.65
OFD (Heo ct al., 2019k)
70.98
73:23
74.95
75.24
74.33
73.95
75.98
60.04
ReviewKD (Chen al,, 2021b)
71.89
73.89
75.63
76.12
17.45
77.11
70.37
69.89
77.78
FCFD(Liu et al. 2023)
71.68
76.80
76.34
75.43
74.86
78. 12
77.81
70.67
71.07
78.20
BDCKD(Lo et al., 2025)
77.25
71.98
7:1.73
77.20
70.72
Logit- Baste Methods
Logit-Basee Baatt Methods
[iinoo al., 2015)
70.66
13.05
T3.33
74.92
72.98
71.07
71.83
67.37
67.35
71,15
(2hang al., 2018)
69.52
72.03
72.12
73.58
72.68
71.79
72.89
72.76
66.63
65.71
73.45
TAKD (Mizzude et al., 2020)
70.83
73.37
73.81
75.12
73.78
73.23
74.53
75.34
67.91
08.02
74.82
DKD(o et al,, 2022)
71.97
74.11
76.32
76.24
71.81
74.68
76.45
76.70
69.71
70.35
77.07
[Thenn & Yang, 2024)
71.83
73.97
76.17
76.23
74.32
74.33
74.18
75.39
68.98
69.24
76.57
Ours
72.29
74.2
77.50
76.38
75.12
74.94
76.66
76.95
70.50
70.97
77.55
(0.11)
(0.23)
(0.19)
(0.32)
(0.13)
(0.16)
(0.08)
(0.15)
(0:10)
[0.12]
(0.21)
+1.12
-1.7
+1.58
+1.06
+2.59
+2.12
+3.13
+3.62
+3.11
4 Experiments
4.1 Setup
Using benchmark datasets, we conducted experiments on three vision tasks: image classification, object detection, and transfer learning. Our experiments leveraged four widely acknowledged benchmark datasets. First, CIFAR-100 (Krizhevsky et al., 2009), encapsulating a compact yet comprehensive represen- tation of images, comprises 60,000 32x32 resolution images, segregated into 100 classes with 600 images per class. ImageNet (Russakovsky al., 2015), a more extensive dataset, provides a rigorous testing ground with its collection of over million images distributed across 1,000 diverse classes, often utilized to probe models for robustness and generalization. Concurrently, the MS COCO dataset (Lin et al., 2014), renowned for its rich annotations, is pivotal for intricate tasks, facilitating both object detection and segmentation as- sessments with 330K images, 1.5 million object instances, and 80 object categories. We strietly adhered to standard dataset splits for reproducibility and benchmarking compatibility for training, validation, and testing. The TinyImageNet! dataset, although more compact, acts as an invaluable resource for transfer learning experiments due its wide variely across its 200 classes.
Dataset:
Network architectures: Various architectures are employed depending on the context. For CIFAR-100, homogeneous configurations use teacher models such as ResNet56, ResNet110 (He et al., 2016), and WRN- 40-2, paired with corresponding students such as ResNet20 and WRN-16-2 (Table 1a). In heterogeneous settings, architectures such as ResNet32x4 and VGGI3 (Simonyan & Zisserman, 2014) for teachers are paired with lightweight models like Shu ShuffleNet-VV V2 (Ma al., 2018) and MobileNet- V2 (Sandler et al., 2018) as students (Table 1b). For ImageNet classification, ResNet34 was employed as the teacher and 18 the student. Additionally, for object detection MS-COCO, Faster RCNN with FPN (Zhang et al., 2022) was utilized as feature extractor, with predominant teacher models being ResNet variants, while the latter served as a student. A pre-trained WRN 16 model is further harnessed for transfer learning. We also performed tests on ViT modols (Dosovitskiy et al., 2021). DeiT-Ti (Touvron et al., 2021),
http://wwwwkkgeeecoo.ooo imagenet
8

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Reported are the Top-1 1 and Top-5 accuracy (%) on ImageNet validation.
Feature- Fexture-Based Methods
Logit-Based Methods
ResNet34
(Teacher) and ResNet18 (Student)
Teacher
Student
AT
OFD
CRD
ReviewKD
KD
DML
TAKD
DKD
Ours
Top-1 Top-5
73.31
69.75
70.69
70.69
70.81
71.17
70.66
70.82
70.78
71.70
72.16
+1.5
91.42
89.07
90.01
90.01
89.98
90.13
89.88
90,02
90.16
90.41
90.60
+0.72
ResNet50
(Teacher) and MobileNet-
(Student)
Top-1
76.16
68.87
69.56
71.25
71.37
72.56
70.50
71.35
70.82
72.05
72.55
+2.05
Top-5
92.86
88.76
89.33
90.34
90.41
91.00
89.80
90.31
90.01
91.05
91.12
+1.32
Table 3: Comparison of training time per batch (Lateney), number of extra parameters (0), and accuracy CIFAR-100, and delection results on MS-COCO using Faster-RCNN--PN (Lin et al., 2017).
(a) Elficiency Comparison CIFAR-100
(b) Delection results on MS-COCO using Faster- ReIl----CNN-FNN
Method
Lastas)
Acc (%)
Festure Based Methods
Logis Based Methode
Rex
Net101
(Teacher)
and ReaNe118 (Student)
KD
11
0
73.53
Teather
Sindent
Fit
PGF1
RevtewDD
KD
TAKD
DKD
Ouns
FitNet
RKD
25
1d
16.8K
71,90
AP
42.04
33.35
53.61
34.13
55.51
35.44
36.75
33.97
55.35
34.50
56.60
55.05
35.34
+1.37
+2.16
CRD
OFD
19
41
12.3M
56.9K
74.95
T5.51
APps
46.88
35.20
ReaNet50 (Tuncher) and MobileNet-V2 V2 (Studeent
36.71
38.17
34.00
35.02
37.12
37.54
37.56
+11.5
RevioWKd
26
Ours
D&D
11
11
77.50
76.52
Af%
APho
61.02
45.88
48.87
30.90
29.47
31.00
40.80
32.92
50.65
31.16
83.71
36.13
53.16
51.35
59.26
20.13
33.46
31.25
51.05
53.77
31.01
58.84
33.57
32.35
+2.22
+3356
12.25
PiT-Ti(Heo al., 2021), PVT-Ti(Wang et al., 2021), and PVIv2-BO(Wang et al., 2022) served as student models, with ResNet50 acting the teacher model.
Evaluation metric: We assess methods' performance using Top-1 and Top-5 accuracy for classification tasks. We employ Average Precision (AP, AP50, and AP70) to gauge precision levels in object detection tasks. We calculate a A that denotes the performance improvement of LumiNet over the classical KD method, underlining the enhancements of our approach.
We investigate knowledge distillation with two configurations: homogeneous architecture (ResNet56 as teacher and ResNet20 as student) and a heterogeneous architecture (ResNet32x4 teacher and ShuffleNe-VV V1 as student). The study includes various neural networks like ResNet, WRN, VGG, ShuffeeNet, and MobileNet Training parameters are: for CIFAR-100, batch size 64 and learning rate 0.05; for ImageNet, batch size 128 and learning rate 0.1; for MS-COCO, batch size 8 and learning rate 0.01. We followed the implementation settings of Zhao et al. (2022). To implement distillation in the ViT variant, we adopted the implementation settings detailed by Li et al. (2022b). All models are trained on a single GPU. Detailed implementation for each task can be found Appendix A.9.
Implementation details:
4.2 Main Results
Comparison methods: We compare our method with well-established feature- and logit-based distillation methods, underscoring its potential and advantages in the knowledge distillation domain. Notable meth- ods in the Saature-Based Methods category include FitNet (Romero et al., 2014), which aligns features at certain intermediary layers, and CRD (Tian et al., 2020), which minimizes contrastive loss between teacher and student representations. RKD (Park et al., 2019), and our proposed LumiNet both aim to transfer relational knowledge by modeling the relationships between samples. However, categorize RKD within feature-based method group since it utilizes pooled embeddings from both teacher and student networks. It calculates pairwise distances and triplet-wise angles in the feature space, aligning these structures using smooth L1 loss. In contrast, LumiNet simply operates in the logil space. Other feature-based methods in- clude OFD (Cho & Hariharan, 2019) and ReviewKD (Chen et al., 2021b), each bringing unique strategies to leverage intermediary network features. Logit-Based Based Methods include KD (Hinton et al., 2015), DML (Zhang et al., 2018), TAKD (Mirzadeh et al., 2020), and DKD (Zhao et al., 2022), which ensure that the student's logits closely match the teacher's output.
9

---

Published in Transactions on Machine Learning Research (08/2025)
Table GLUE dev set results of BERT based student models with various KD methods.
CoLA (Mec)
MNLI-mm/mmm) [Acc]
SST-2
QNLI (Acc)
MRPC
QQP (Acc)
RTT
STS-B
Model
#Params
(Acc)
(F1)
(Acc)
(Spear)
Avg
BEITT (Teather)
110M
80.3
84.9/8488
B3.7
91.7
91.4
91.5
60.7
89.4
84.1
BERT, (Student)
66M
81.7/82.6
90.4
KDiHinton et al, 2015)
B6M
53.6
12.7/83.1
91.1
B0.1
89.4
90.5
66.8
88.7
81.6
PDCTure et al., 2019)
66M
82.5/8314
91.1
89.4
89.4
90.7
66.7
PKD(Sun al. 2019)
6631
15.3
81.3/ 8/-
91.3
58.4
85.7
58.4
66.3
86.2
79.2
TinyBERT(Jiao
2020)
06M
63.8
83.1/83.4
86.8
90.5
06.9
88.3
81.7
CKD(Park ct ol., 2021)
B6M
55.1
83.6/844 (8.1
93.0
90.5
89.6
91.2
67.3
80.0
82.4
MGSKD(Lin et al., 2022)
66M
49.1
83.3/831 83.9
91.7
90.3
80.8
91.2
67.9
88.5
81.5
06M
55.8
83.72/84.2 28.
91.3
90.7
89.9
91.6
69.7
80.3
83.0
A
+-102/41.38
+4.6
40.5
+1.1
Table Accuracy and ECE on CIFAR-100 for different teacher-student distillation sctups and methods.
Teacher
Student
KD
KD+FL
KD+DFL
FCUC
Ours+DFL
Ours
ACC t BCE
RN32x4
RN8x4
73.33
0.11
74.41
/ 0.10
74.51 / 0.09
76.37 / 0.09
76.50 / 0.07
77.50
0.06
VGG13
VGGS
72.98
0.12
73.88
/ 0.11
73.39 / 0.10
74.28 / 0.06
74.66 / 0.07
74.94
0.06
WRN40-2
SNVI
74.83
0.13
75.26 / 0.12
75.11 / 0.11
75.02 / 0.09
76.98
0.04
76.95 / 0.07
We perform image recognition tasks on CIFAR-100 and ImageNet. On CIFAR-100, when teacher and student models shared identical architectures, shown in Table 1a, LumiNet presented improvements of 2-3%. And when the architectures were from different scries, shown in Table 1b, the improvements between 3-4,, consistently outperforming the baseline, classical KD, and other methods rooted in KD's principles. Similarly, the intricate ImageNet dataset, LumiNet outshined all logit- based distillation techniques and beat state-of-the-art feature-based distillation methods, shown in Table 2. These results consistently demonstrate that, regardless of variations in the dataset or architectural differences, LumiNet performs exceptionally well. In particular, it highlights the distinetive ability of LumiNet. to learn based on the concept of "perception. In Table 3(a), LumiNet shows a superior trade-off between extra parameters/rumnt time and precision. achieves 11 ms latency, matching the best-performing models in speed, and operates efficiently 77.50% accuracy without extra parameters.
Recognition tasks:
Detection task: The quality of deep features is crucial for accurate object detection. One persistent challenge is elfective knowledge transfer between established teacher models and student detectors (Li et al., 2017). Generally, logits cannot provide knowledge for object localization (Wang et al., 2019). Although logit- based techniques have traditionally been used for this, they often do not meet stato-of-thc-art standards. On MS COCO dataset, LumiNet delivered noticeably better results (Table 3()) compared to logit- methods, which are comparable to feature-based methods. Also, it is possible to enhance accuracy through hyperparameter tuning. Additionally, we enhance our approach by integrating feature- based technique. The combination of these two methods yields ttato-of-the-art results (Table 12), as detailed in appendix A.10.
GLUE Benchmark: Our proposed method demonstrates consistent and robust improvements across di- verse NLP tasks on the GLUE benchmark. As shown Table 4, our BERT -oTTg-bbsed student model achieves an average score of 83.0, outperforming traditional KD (Hinton et al., 2015) by +1.4 points and surpassing other competitive basclines such as CKD (Park et al., 2021) and TinyBERT (Jiao et al., 2020). The method shows strong generalization across various task lypes including classification (SST-2, QNLI), similarity (STS-B), and entailment (RTE). It shows noticeable improvements on challenging tasks like CoLA (+2.2) and RTE (+2.9), suggesting that it can effectively transfer relational patterns and linguistic details from the teacher model.
Transfer learning task: To assess the transferability of deep features, we carry out experiments to ver- ify the superior generalization capabilities of our algorithm LumiNet. In this context, we used the Wide Residual Network (WRN-16-22) distilled from WRN-40-2, as our principal feature extraction apparatus. Subsequently, sequential linear probing tasks were performed the benchmark downstream dataset,
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Results after applying Auto Aug-
Table 7: Top-1 mean accuracy (%) comparison on CIFAR-
mentation.
100
Pracher
WRNNO 2
VGG
RN
WRN40-2
Arcanaey
75.61
75.61
74.64
79-42
75.61
Student
Vanilla
AT
SP
T.G.
AutoKD)
Qurs
Student
WRN16_2 2
WRN40
VGG 8
SN-V2
SN-VI
DeiT-Ti
65.08
73.25
73.51
67.36
78.15
78.68
79.05
+6.8
Accaracy
73.25
71.38
70.36
7L82
70.50
P:T-Ti
73.58
76.47
76.03
74.97
78.48
78.51
79.80
+4.333
MLLD
76.53
76.35
78.18
78.44
77.44
PVT-TI
69.22
73.60
74.86
70.48
17.07
77.48
78.12
+4.52
Ours
76.91
76.01
76.07
79.12
77.97
PVTv2-B0
77.44
78.81
78.64
78.33
79.30
79.37
79.94
+1.13
+0.86
+0.53
tably Tiny-ImageNer Our empirical results, delineated in Figure. 2(a), manifestly underscore the exemplary transferability of features cultivated through LumiNet.
Table 6, we report performance after using auto-angmentttion by increasing the complexity of training samples (Cubuk al., 2019). LumiNet outperforms auto augmentation- based method (Jin et al., 2023) in heterogeneous and homogeneous settings on the CIFAR-100 dataset. The rosults show effectiveness in distilling knowledge from challenging samples.
Effect of Strong Augmentation:
Table 5 presents a comparison of varions tion methods on CIFAR-100. While traditional KD (Hinton et al., 2015) serves as the baseline, all other methods-Fooce Focal Loss (FL), Dual Focal Loss (DFL) (Mukhoti ct al., 2020; Tao et al., 2023), Feature Clipping for Uncertainty Calibration (FCUC) (Tao al., 2025), and our proposed method (LumiNet)-iintrolluee introduce calibration either through loss functions or post-logit transformations. FL and DFL replace the standard cross-entropy with calibrated loss terms during training, while FCUC and LumiNet operate directly the logits, making them modular and compatible with other loss functions. Notably, all calibrated methods out- perform vanilla. KD in both accuracy and ECF, indicating that improving the model's confidence alignment is critical in student training. Our approach (Ours), which uses batch-level relational calibration of logits, achieves the best overall performance, reinforcing the necessity of integrating calibration mechanisms into the distillation pipeline for better generalization and reliability.
Effect of Calibration:
ceabraation-nhaneee knowledge distilla-
Vision Transformer: To explore the capabilities of LumiNet beyond conventional ConvNet models, we performed experiments using different variants of vision transformers (ViT) in the CIFAR-100 dataset. We trained VIT with the optimal distiller obtained using ResNet-56 results of experiments that involve both vanilla and distillation models in variety of distillation methods. The results indicate a notable improvement in the performance of vision transformers with the application of LumiNet, showcasing improvements ranging from 2% to 14% compared to vanilla. In particular, Lu- miNet consistently outperforms other methods, demoustrating improvements of 1 to 6% compared to KD, particularly. It is essential to emphasize that our approach, despite being a straightforward logit-basodi soft logits) method in this context, proves to be more effective in Transformer-based architectures compared to feature- based distillation methods.
CNN teacher. Table 7 presents the
4.3 Confirmation Bias & Calibration Analysis
Confirmation Bias Analysis: To empirically validate our hypothesis about confirmation bias in knowledge distillation, we analyze the error rates across different classes, particularly focusing on the classes where the teacher model performs poorly. Table 8 presents the error rates for the top 10 most challenging classes for the teacher model (ResNet32x4), comparing them with both traditional KD and our proposed method using ResNet8x4 as the student architecture. The results provide strong evidence of confirmation bias in traditional KD. For most difficult classes where the teacher exhibits high error rates (ranging from 34% to 45%), the KD student model not only inherits these mistakes but often amplifies them. For instance, in Class 72, while the teacher model shows a 41% error rate, the KD student's performance deteriorates to 55%, indicating strong propagation of teacher's misconceplions. This pallern is consistent across multiple classes (Class 11, 46, 74, 10, 55), where KD consistently shows higher error rates than the teacher. In contrast, our proposed method demonstrates remarkable resilience to confirmation bias. In 6 out of 10 challenging classes (marked with asterisks), our approach achieves lower error rates than the teacher model, effectively breaking the cycle of error propagation. Most notably, in Class 46 and Class 74, our method reduces the error rates
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: Error rates by class and method for the top 10 most inaccurate classes of the Teacher models. The table shows the error rates of different Saadeer-Suudet architectures, along with KD and our proposed method. An asterisk (*) indicates an error rate lower than the Teacher model's error rate. Our proposed method consistently outperforms KD across different architectures.
Tescher (RenNet32cl)) (lunent,BesNetBeel
Method
C35
C11
C46
C72
C74
C52
C64 (Skunk)
C10 (Orchids)
C5S (Camel)
C50
(Bee)
(Poppies)
(Castle)
(Girl)
(Woman)
(Monntain)
(Cloud)
Teacher
45.0
43.0
42.0
41.0
40.0
39.0
38.0
37.0
36.0
34.0
KD
43.0"
19.0
17.0
55.0
47.0
40.0
43.0
12.0
19.0
38.0
Our
46.0
47.0
38.0"
52.0
37.0"
37.0"
38.0"
36.0"
44.0
38.0
Teacher (WideRenNet-S0o) (0-2)) Soudent Soudent WideResNNet (0-1))
Method
C72
Ca5
C55
C10
C50
C46
C64
CG7
CT1
C74
(Girl)
(Bee)
(Camel)
(Orchids)
(Cloud)
(Custle)
(8kunk)
(Small)
(Poppies)
(Woman)
Teacher
51.0
50.0
47.0
46.0
45.0
11.0
11.0
13.0
13.0
KD
52.0
55.0
45.0
42.0"
48.0
39.0"
44.0
47.0
52.0
47.0
53.0
54.0
45.0*
45.0
31.0"
41.0"
42.0"
44.0
Teacher (VGG13) (VGGen(VVGs))
Mechod
CS5
C72
C55
C41
C46
C10
C25
C11
C74
C64
(Bee)
(Girl)
(Camel)
(Wolf)
(Castle)
(Orchids)
(Clock)
(Poppies)
(Woman)
(Skunk)
Teacher
54.0
53.0
50.0
48.0
47.0
46.0
46.0
45.0
45.0
43.0
KD
63.0*
54.0
67.0
54.0
46.0
33.0
56.0
53.0
50.0
Oure
$3.0*
49.0"
45.0"
51.0
44.0"
41.0"
33.0"
45.0"
45.0"
47.0
from 42% and 40% to 38% and 37% respectively, showing that the student can actually outperform the teacher in challenging cases. Even in cases where our method doesn't. surpass the teacher, it consistently outperforms traditional KD, suggesting more robust learning of class features.
Calibration Analysis: To evaluate the calibration of our models, we employ three widely accepted metrics: False Positive Rale al 95% True Posilive Rate (FPR95) (Wei et al., 2022), Expected Calibration Frror (ECE) (Nacini et al., 2015), and Marimum Calibration Error (MCE) (Nacini et al., 2015). FPR95 measures the false positive rate when the true positive rate is fixed at 95%. It assesses the reliability of high-confidence predictions, with lower values reflecting fewer false positives at high recall. In multi-class settings, FPR95 is computed per class and averaged. ECE measures the average discrepancy between model confidence and accuracy, calculated as the weighted average of differences between bin accuracy and confidence. MCE identifies the maximum discrepancy across all bins, indicating the worst-case calibration error. Lower FCE and MCE values signify better calibration and reduced extreme miscalibration. Implementation details are in Appendix A.9.
We compare three methods: (1) baseline trained with standard cross-entropy loss (CE), (2) model trained using KD with cross-entropy and KL divergence applied to raw logits, and (3) our proposed method, which uses cross-cntropy and KL divergence on perceived logits. As shown in Table 10, our method achieves superior calibration performance across all metrics. Specifically, it reduces FPR95 by 23.46% (from 3.58% to 2.74%) compared to the CE bascline and by 33.98% (from 4.15% to 2.74%) compared to the KD baseline for ResNet8x x4, 4, indicating fewer false high-confidence predictions. Similarly, for VGG8, our method lowers FPR95 by 25.13% (from 5.61% to 4.20%) compared to CE and 27.13% (from 5.75% to 4.20%) compared to KD. Additionally, our method lowers FCF. by 33.33% (from 0.09 to 0.06) and MCE by 14.29% (from 0.21 to 0.18) for ResNet8x4 compared to the CE bascline, demonstrating improved alignment between confidence and accuracy. For MobileNet-V2 V2, the reductions in ECE and MCE are 47.06% (from 0.17 to 0.09) and 44.74% (from 0.38 to 0.21), respectively.
4.4 Discussion
Achievments of LumiNet (1)Novel Conceptual Framework: LumiNet introduces 'perception' novel approach to knowledge distillation that reconstructs logits by considering their relationships within a batch, rather than treating them in isolation. Grounded in Kurt Lewin's Field Theory, this method captures richer knowledge transfer patterns without needing intermodiate features, bridging the gap between featuro-basod and bagit-based methods. Techencla Advantages: Our method tackles challenges in knowledge distillation using statistical calibration. It reduces overconfidence and confirmation bias through normalized logits with
12

---

Published in Transactions on Machine Learning Research (08/2025)
batch-level context. Empirical results indicate reduction of confirmation bias in error rates by up to 14% in difficult classes compared to traditional KD. (3) Practical Benefits: LumiNet combines sophistication with practical efficiency, maintaining the same latency and computational overhead as vanilla KD (11ms). Its versatility across architectures (CNNs, Vision Transformers) and tasks (classification, detection, transfer learning) makes it ideal for industrial applications.
Limiation: LumiNet has some drawbacks in spite of its advantages. In complex or multi-modal tasks where intermediate feature representations are essential, it might not perform likewise. The performance of the model with small or less diverse batches may be limited by its dependence batch-level relationships. Although LumiNet has demonstrated impressive performance in computer vision tasks, it is still unclear if it can be applied to non-visual fields like natural language processing.
Future work: Future research could examine LumiNet's approach to KD outside of computer vision, as it is thought have significant potential in other disciplines. The perception-based logit calibration tochnique could be used improve the deployment and compression of large language models in resource-onsstrainee environments. Furthermore, LumiNet could be used for continual learning settings to investigate how the method can aid in successful knowledge acquisition while avoiding catastrophic forgetting.
5 Conclusion
We propose LumiNet, novel knowledge distillation method, which introduces unique representation for instances through a concept we term "perception.1 II'n this novel reprosentation, we depart from the fundamen- tal philosophy of classical KD, which centers around extracting relative information from the teacher model. Within this framework, our main focus lies on addressing overconfidence issues to achieve improved opti- mization. also tackles the capacity gap issue, where the student model struggles to learn due to the high variance in the teacher model's logit distribution. In addition, we integrate statistical knowledge from other instances into instance, resulting in substantial improvement in accuracy compared to leading meth- ods, which mitigates the problem of overconfidence and confirmation biases. Also, LumiNet demonstrates efficiency on par with traditional KD, solidifying its suitability for industry adoption. Our comprehensive empirical experiments, spanning recognition using both convnets and vision transformers, detection, and transfer learning, consistently highlight the superior performance of LumiNet.
6 Acknowledgement
The authors gratefully acknowledge the Machine Intelligence Lab (MTLab), North South University, for their valuable support and resources that contributed to this research.
References
Junyi Chai, Taeuk Jang, and Xiaoqian Wang. Fairness without demographics through knowledge distillation. Advances in Neural Information Processing Systems, 35:19152-1916: 2022.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technology, 6(1):25 45, 2021. 2, 5
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp.
2021a. 2,
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 5008-5017, 2021b. 4, 8, 9, 27, 28
13

---

Published in Transactions on Machine Learning Research (08/2025)
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEFE/CVP international conference on computer vision, pp. 4794-4802, 2019. 2, 5, 9, 28
Thomas M. Cover and Joy A. Thomas. Flements of Information Theory (Wiley Series in Tedecommunicaaions and Signal Processing). Wiley-lnterscience, USA, 2006. ISBN 0471241954. 21
Ekin D Cubuk, Barret Zoph, Dandclion Mane, Vijay Vasudovan, and Quoc V Le. Autoaugment: Learning augmeenattion strategies from data. In Proceediings of the IEEE/CVE conference on compuler vision and pattern recognition, pp. 113 123, 2019. 11
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference Machine Learning, pp. 1607-1616. PMLR, 2018. 2
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, 770 778, 2016. 4, 8
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. comprehensive overhaul of feature distillation. In Proceedings of the TEFE/ /CFF International Conference on Computer Vision, pp. 1921 1930, 2019a. 4,
Byeongho Hco, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 33, pp.
2019b. 4
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions vision transformers. In Proceedings of the IEEE/C/ International Conference on Computer Vision, pp. 11945,, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. ar preprint arXiv:1503. 02531, 2015. 1, 2, 4, 5, 7, 8, 9, 10, 11, 20, 26
Md Ismail Hossain, Mohammed Rakib, Sabbir Mollah, Fuad Rahman, and Nabeel Mohammed. Lila- Leveraging isolated letter accumulations by ordering teacher insights for bangla handwriting recognition. In 2022 26th International Conference on Pullern Recognition (ICPR), pp. 1770 1776. IEEE, 2022.
Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge diffusion for distillation. Advances in Neural Information Processing Systems, 36, 2024. 4
Sergey Toffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. pmlr, 2015. 23, 24
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tiny- BERT: Distilling BERT for natural language understanding. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163 4174, Online, November 2020. Association for Computational Linguistics. doi: /.118663/1112220.fmmm
URL https: / /ttp//acaaaattooyy..oooa. org/ findings- emnlp.376 10
Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023. 5, 7, 11
Lie Ju, Xin Wang, Lin Wang, Dwarikanath Mahapatra, Xin Zhao, Quan Zhou, Tongliang Liu, and Zongyuan Ge. Improving medical images classification with label noise using dual-uncertainty estimation. IEEE Transactions on Medical Imaging, 41(6):1533-115 1546, 2022. doi: 30. 28
14

---

Published in Transactions on Machine Learning Research (08/2025)
Sangwon Jung, Donggyu Lee, Taccon Park, and Taesup Moon. Fair feature distillation for visual recognition. In Proceedings of the IEEE/CVE conference computer vision and paltern recognition, pp. 121224 2021. 5
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-lojasiowizz condition. In Joint European conference on machine learning and knowledge discovery in dalabases, pp. 795 811. Springer, 2016a. 23
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and
methods under the polyak-tojasiewier condition. In Joint European conference on machine learning and knowledge discovery in databases, pp. 795-811. Springer, 2016b. 23
Taehyeon Kim, Jaehoon Oh, Nak Yil Kim, Sangwook Cho, and Se- Young Yun. Comparing Hallback-leible divergence and mean squared error loss knowledge distillation. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2628 2635. International
Joint Conferences Artificial Intelligence Organization, 8 2021. doi: 10.24963 /tt//aaaaaii./ 2021 /362. URL https://doi... org/ 10. 20222/2944ieee wtt.iiiiii////e...... 2021/362. Main Track. 2,5
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing, pp. 1317-1327, 2016. 4
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's
thesis, University of Tront, 2009. 8
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. 4
Gang Li, Xiang Li, Yujie Wang. Shanshan Zhang, Yichao Wu, and Ding Liang. Knowledge distillation for object detection via rank mimicking and rredction-guiied feature imitation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 1306-1313, 2022a. 4
Kehan Li, Runyi Yu, Zhennan Wang, Li Yuan, Guoli Song, and Jie Chen. Locality guidance for improving vision transformers on tiny datasots. In European Conference on Computer Vision, pp. 110-127. Springer, 2022b. 9, 26
Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings of the ieee conference on computer vision and pottern recognition, pp. 1356,6364, 2017. 10
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
2014: 13th
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming Hc, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017. 9, 27
Bernadette Lindorfer. Personality theory in gestalt theoretical psychotherapy: Kurt lewin?s field theory and his theory of systems in tension revisited. Gestalt Theory, 43(1):29 46, 2021. doi: 10.2478/gth-2022
2, 6
Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao Multi-granularity structural knowledge distillation for language model compression. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Lánguistics (Volume 1: Long Papers), pp. 1001 1011, Dublin, Ireland, May 2022. Association for Computational Linguistics. http:///aarrrrlooooooo/r////....
doi: 10.18653/v1. 100.8.5//1//2/222.a.cal /202 URL
org/2022. acl- aall-logg.... 71/. 10
Dongyang Liu, Meina Kan, Shiguang Shan, and Xilin CHEN. conssten- feature distillation. In
The Fleventh International Conference on Learning Represenlations 2023. 8
15

---

Published in Transactions on Machine Learning Research (08/2025)
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the TEEE/CW conference on computer vision and pattern recognition, pp. 2604 2613, 2019. 4
Guomnig Lu, Heng Yin, Zhiyong Shu, Jielei Wang, and Guangehun Luo. Bdckd: Unlocking the power of brownian distance covariance in knowledge distillation. In ICASSP 2025 2025 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (TCASSP), pp. 1-5, 2025. doi:
10.1109/ICASSPP49660 222119666002.600000 1085.998..
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018. 8
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyoon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- PMLR, 18 24 Jul 2021. URL httpss//proceeddin samiriiinnnmm........mmmmm. www...ressvviee htt///11///mmm/// 3tt/////////.aa.m/////.////a/aam html. 7
Seyed Iman Mirzadch, Mchrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 34, pp. 3191-5198 2020. 1, 4, 8,
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, and Puneet K. Dokania. Calibrating deep neural networks using focal loss. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. 11
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrocht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. 12
Vardan Papyan, X. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, (47640))2665-22466) 2020. doi: 10. (0073/pass.220 2015509117. URL https:/ htops:///wwww./ http//wwwppaa...p..//.a wrgg/oi//bs///a//ooooogggg w0w....//.a.......m 2015509117. 7
Geondo Park, Gyeongman Kim, and Eunho Yang. Distilling linguistic context for language model compres- sion. In Marie Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 364 378, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.1 (0.18653/v1 1220/111/221.// .mmnl--main.30. URL httpp://aclaanthooggy wwwpo/olooooo.og//222//// org/20211 emnlp-main 30/. 10
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 3967 3976, 2019. 4, 8,
Dian Qin, Jia-Jun Bu, Zhe Liu, Xin Shen, Sheng Zhou, Jing-Jun Gu, Zhi-Hua Wang, Lei Wu, and Hui-Fen Dai. Efficient medical image segmentation based on knowledge distillation. IEEE Transactions on Medical Imaging, 10(12):3800-333) 3831, 2021. 4
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXio preprinl arXiv:1412.6650 41500550, 2014. 1, 2, 4, 5, 8, 9
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015. 8
Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4)e 2018. 20
16

---

Published in Transactions on Machine Learning Research (08/2025)
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Inverted residuals and linear bottlenecks. In Proceedings of the IEFE conference on compuler vision and pattern recognition, pp. 4510 4520, 2018. 8
Chen. Mobilenetv2:
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems, 31, 2018. 23, 24
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X99 1556, 2014. 4, 8
Yaoye Song, Peng Zhang, Wei Huang, Yufei Zha, Tao You, and Yanning Zhang, Closed-loop unified knowl- edge distillation for dense object detection. Pattern Recognition, 149:110235, 2024. 4
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression.
In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNL.-IJCNLPP), pp. 4323 44322, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D194 1441, URL https://aclanthooogy org/D19- 1441/. 10
Linwei Tao, Minjing Dong, and Chang Xu. Dual focal loss for calibration. In Andreas Krause, Emna Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learn- ing Research, pp. 33833 33849. PMLR, 23 29 Jul 2023. URL bttpa://proceediie mlr. www..l..e//...... press/v202/ tao23a. html. 11
Linwei Tao, Minjing Dong, and Chang Xu. Feature clipping for uncertainty calibration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 20841 20849, 2025. 11
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020. 1, 4, 5, 8, 9, 28
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021. 8
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1374, 2019. 1, 4
Tulia Ture, Ming-Wee Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn botter: The impact of student initialization on knowledge distillation. ArXiv, hts/1908.08999 2019. 10, 26
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and cundent-teacher learning for visual intelligence: A review and outlooks. IEFE transactions on pattern analysis and machine intelligence, 44(6):3048- 3068, 2021. 2, 5
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4933 4942, 2019. 10
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CV ECFF international conference on computer vision, pp. (68-578, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang. Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3): 415-424, 2022. 9
17

---

Published in Transactions on Machine Learning Research (08/2025)
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International conference on machine learning, pp. 23631- 23644. PMLR, 2022. 12
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolie knowledge distillation: from general language mod- els to commonsense models. In Marine Carpuat, Catrei-et de Marnelle, and Tvan Vladimir Meza. Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 4602-4625, Scattle, United States, July 2022. Association for Compulational Linguistics. doi: 10.18653/v1 00553//11/222.aam// maaac-maiin344 URL https:: httpa////ccaaathooogo. org/2022 naacl-main. 341.
Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, and Rui Wang. AD-KD: Attribution-driven knowl- edge distillation for language model compression. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61sl Annual Meeting of the Associalion for Compulational Linguistics (Volume 1: Long Papers), pp. 8449 8465, Toronto, Canada, July 2023. Association for Computational Lin- URL https: /tttps://aclase aal anthology. org/2023. acl- acl-long.477
guistics. doi: 10.18653/v1 22200/222222/22o2g/ 26
Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation, Advances in Neural Information Processing Systems, 35:38571-38584, 2022. 4
Guoliang Yang, Shuaiying Yu, Yangyang Sheng, and Hao Yang. Attention and feature transfer based knowledge distillation. Scientific Reports, 13(1):18369, 2023. 2, 5
Jihan Yang, Shaoshuai Shi, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Towards efficient 3d object detection with knowledge distillation. Advances in Neural Information Processing Systems, 55:21300-21313, 2022.
4
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimiza- tion, network minimization and transfer learning. In Proceedings of the IEFF conference on computer vision and pattern recognition, pp. 4133 4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Repre- sentations, 2017.
Bowen Zhang, Xu Huang, Zhichao Huang, Hu Huang, Baoquan Zhang, Xianghua Fu, and Liwen Jing, Sentiment interpretable logic tensor network for aspect-term sentiment analysis. In Proceedinggs of the 29th International Conference Computational Linguistics, pp. 6705-6714, 2022. 4,
Weijia Zhang, Dongnan Liu, Weidong Cai, and Chao Ma. Cross-view consistency regularisation for knowledge distillation. In ACM Multimedia 2024, 2024. 2, 5, 7
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp.
2018. 1, 4, 8, 9
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. Tn Proceedings of the IEEE/CC /CVF Conference on computer vision and pattern recognition, pp. 11953-11962, 2022. 1, 2, 4, 5, 7, 8, 9, 25, 26
Kaixiang Zheng and En-Hui Yang. Knowledge distillation based on transformed teacher matching. In The Twelfth International Conference on Learning Representations (ICLR 2024), 2024. URL https: //openreview.net wwttpdffiddd/Saeee
4, 8
Zhaohui Zheng, Rongguang Ye, Qibin Hou, Dongwei Ren, Ping Wang, Wangmeng Zuo, and Ming-Ming Cheng. Localization distillation for object detection. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2023.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang, Rethink- ing soft labels for knowledge distillation: A bias bias-variane variance tradeoff perspective. In International Conference
on Learning Representations, 2021. 5
Dongyao Zhu, Bowen Lei, Jie Zhang, Yanbo Fang, Yiqun Xie, Ruqi Zhang, and Dongkuan Xu. Rethinking data distillation: Do not overlook calibration. In Proceedings of the IEEE/CW International Conference on Computer Vision, pp. 4935-4945 2023. 2
A Appendix
A.1 Preliminaries and Notation
Let X' Rd be an input space, and let (1,2....,C) be the set of C classes. We have:
teacher model fr :2: + RC, producing logits
z xx(x) (x] ..... =E(x)).
A student model fs X R producing logits
z 2 (x) = (xf(x), 25 (x), (x))
For mini-batch B mm C X of size m, the teacher produces logits
{x
{z (x.) y
for each class j E { {I..... C}.
Analogous notation applies for the student. Denote
m
1 m
(x)).
of
2 (x,) uz] tE,
m
m
where O is small constant (e.g., 10-5) to avoid division by zero. Likewise, let 1s and of be the analogous means and (biased) standard deviations of the student logits.
We write ozz) E 10 for the softmax distribution:
exp(zz)
al));
Ek= exp(zz)
A.2 Ablation Study
Varying batch sizes: Figure. 3(b) showeases an ablation study that compares the performance of the LumiNet method with both a basic student model and the KD method in various batch sizes. Batch sizes range from 16 to 256. The student model, which serves as a standard baseline, demonstrates a slight decline in performance as the batch size increases. In comparison, LumiNet consistently outperforms both the student and the KD methods in all batch sizes tested.
Varying The logits within our perception framework are reconstructed with clear statistical under- standing of intra-class logits. For this, both the teacher and the student models exhibit "softened' values, achieved through normalization by variance and maintaining an intra-class mean of zero. Consequently, the dependency on temperature 7 is minimal. Empirical evaluations in Figure. 3(c) suggest minimal performance fluctuations across т (ranging between and 8) yield optimal results.
Ensemble of teachers: We employ an ensemble of two teacher models: ResNet 8x4 and WRN-40-2 (labeled in the figure as "8x4" and "40-2"). This ensemble technique, which term *Logit Averaging Ensemble,"
19

---

Published in Transactions on Machine Learning Research (08/2025)
(b) Methods Test by batch sizes(C) Effect of Temperature (d) Ensemble LumiNe'
T-ResNet32x4
T-ResNet32x4
T-ResNet32x4, wrn wrn_400,
(a) Transfer Learning
S-ResNet8xx x4
80.0
S-ResNet8x4
Log L-aaed
Student
Sazed Mathoda
77.5
79
Teacher
75
Accuracy
70
72.5
77
70.0
65
KD
FitNet CRD
ReviewKD DKD
Ours
32
Batch Size
64
128
256
65.0
Temperature
2
4
8
32x4
40
2__
16
Z(En)
16
Figure (a) Transfer learning experiments from CIFAR-100 Tiny-ImageNet. (b) Ablation study on
different batch sizes. (c) Impact of different. values. (d) Performance on ensemble learning.
involves averaging the logits produced by the two teacher models (Sagi & Rokach, 2018). When training the student model, WRN-16-2 (labeled as "16-2" for the regular student and "16-2(en)" for the student learned by ensemble techmique), we observed a notable improvement in accuracy using this ensemble-derived guidance. As shown in Figure. 3(d), when conventionally training with our LumiNet approach with only the WRN-40-2 teacher, we achieve an accuracy of 76. 38%. However, the results improve slightly 76. 52% when training is augmented with insights from the ensemble technique. This suggests that the ensemble's aggregated information potentially enables the student model to capture more intricate patterns and nuances from the teachers.
A.3 Definition of Perception Logits
Definition 1 (Teacher's Perception Logits). For each sample x; E B, define the perception logits of the teacher by:
-13 ur
h (xx)
(j = 1, I,....C.. C).
of
We denote h² (x;)):) = (xT(x))..... (x()))
Similarly, the student produces perception logits
(x,, 1
hx:))
0g
giving hx4))) = (x))),), (x,)).
A.3.1 Comparison to Classical KD
Classical Knowledge Distillation (KD) (Hinton et al., 2015) aligns the teacher's and student's raw logits (after temperature-scalingg). Concretely,
Σ (L (x:1) o (x=)))))
LKD
X.EB
where 7 > 0 is temperature parameter. LumiNet instead applies this KL-divergence over the normalized (perception) logits, i.c.,
Σ KL(((( (x.)) o (مxم)))))
LtmmiNet.
x.EB
The training objective combines it with cross-entropy:
Liotal = Lor(zS, 4) + A LtamiNet:
for balancing coefficient. À 0.
20

---

Published in Transactions on Machine Learning Research (08/2025)
A.4 CooommttionTThorettie Perspective
We study why normalizing logits by batch-level means and variances can preserve or increase the mutual information (Cover & Thomas, 2006) with class labels, effectively mitigating overconfidence.
Let (x) and (x) be random vectors denoting the teacher's raw logits and its perception logits for x.
Let (x) be the true class label.
Theorem (Mutual Information Redistribution under Class-wise Normalization). Suppose H is derived from z" via class-wise mean-variance normalization over a batch of size m, and assume the logit distribution has finite second moments. Then there exists a constant a > 0 such that the mutual information at the batch level satisfies
I(H,; ;Y) 2 1(Z ; Y) + ;)
Σ Var(Z)))
1=1
where the increase is allribuled to the batch-wise alignment of logits. However, due to class-wise normal- ization: The inter-class mutual information structure is altered, as normalization introduces dependencies between logits across different classes. - The sample-wise mutual information ;(H; for an individual sample i may increase, decrease, or be redissbributed, depending on the batch-wide logit distribution.
Thus, while the total mutual information across the batch can increase, the information content ovailable to individual samples is reshaped by batch statistics, making the perception logits more representative of intra-class and inter-class relalionships.
Sketch of Proof. Recall that the mutual information defined as:
I(H:Y) HHH) - H( |H
where H-) denotes the Shannon entropy and H(-1) represents conditional entropy.
Step 1: Entropy Expansion due to Normalization
Batch normalization modifies logits by standardizing each logit dimension within its class distribution:
Zj 1 Jj
H,
(j,
where My and jj are the batch mean and standard deviation for class j. This transformation affects entropy in two key ways: De-mean and variance scaling increase entropy: From entropy scaling properties (Cover & Thomas, 2006), normalization generally increases entropy when the original logits have high intra-class variance:
H 2 H(Z) +
Var(Zz).
1=1
where depends on the scaling factor of batch statistics.
2. Effect on differential entropy: The entropy of a standardized Gaussian variable (logits after normalization) is given by:
H(H) H (Z) + EE [log /det s/]
where J is the Jacobian of the transformation. Since normalization whitens the logit space, the determinant of J is linked to variance reduction, leading to a net increase in entropy.
Step 2: Conditional Entropy and Information Redistribution
The conditional entropy term H H(H Y) is affected as follows: Since normalization is class-wise, knowledge of Y still provides information about logits, ensuring that conditional entropy does not increase arbitrarily. However, due to dependencies introduced across logits within the batch, the sample- wise mutual information T(H;;) Y) can either increase decrease.
21

---

Published in Transactions on Machine Learning Research (08/2025)
Using the data processing inequality, obtain:
HH Y) s (Z Y),
which follows since normalization does not remove label-relevant information but redistributes it within the batch.
Step 3: Bounding Mutual Information Increase
Combining the results from Steps 1 and 2, we obtain:
I(H;Y) ((H) - H(H) Y),
[Z) + Σ Var Var(2,)) ( Y).
Thus, at the batch level, mutual information is lower-bounded by:
I(: 2 I(
Var Var(Z,) Zz).
Convergence Rate (Moving Average)
(a)
(c)
(d)
0.0010
KD
0.30
KD
Gradient Variance Trend
1.2
Stability Scores: KD; 0.808
0.0005
Ours
Log Gradient Variance
0.25
Ours
1.0
Ours: 0.899
10
Loss Trend
0.0000
0.20
0.8
-0.0005
0.15
0.6
-0.0110
10-4
0.10
0.4
0.0015
KD
0.05
0.2
-0.0020
10-5
Ours
0.00
0.0
200
200
0
200
KD
Ours
Epoch
Epoch
Epoch
Figure Comparison of various metrics between the proposed method and standard KD: (a) Loss Con- vergence Trend over epochs, showing the progression of training loss; (b) Gradient Variance Evaluation on
logarithmic scale, highlighting the stability of gradient updates; (c) Convergence Rate Analysis using
moving average, illustrating the rate of model convergence; (d) Gradient Stability Distribution represented by boxplot, summarizing the distribution of gradient variance trends.
A.5 Gradient Analysis and Convergence
A.5.1
Gradient Form of LumiNet
We now examine how the perception logits yield more stable gradient flow when applying gradient- baaeed methods (e.g., SGD). The stability arises due the influence of batch statistics, which dynamically rescale logits, mitigating the effects of overconfident predictions.
The LumiNet distillation loss is given by
LLamiNet = Σ DKL 1
o0( (o
1=1
22

---

Published in Transactions on Machine Learning Research (08/2025)
where h = (x:) and h (x;) denote the perception logits. Recall that perception logits computed
h
where 2 and of are the batch-wise mean and standard deviation of logits for class j. Since og is dynamically computed per batch, it acts adaptive scaling factor.
Differentiating w.r.l. the raw logils, obtain
ahe
ahe Oz
for k + j.
025
3,6
Thus, the gradient of LumiNet loss w.r.l. zf becomes
-ea(")--0' o 12 1 05
LumiiNet
8eg.
This leads to variance-baec based adaptive gradient scaling, where classes with high variance (i.e., greater uncer- tainty in logits) have dampened gradients, preventing the dominance of outlier logits. Since batch statistics are dynamically computed, this scaling effect adapts throughout training, ensuring stable convergence.
A.5.2 Convergence Under the
Condition
To analyze the convergence properties of LumiNet training, consider the total loss
Liotal (0$) LCE(0²) + A LeamiNer (0
We assume that Ltotal satisfies the following conditions:
Crotal is L-Lipschitz.
2. The function satisfies the Polyak Rojasiewicz (PI.) condition (Karimi al., 2016b):
Liotaal (05)12 2 μ (Ctotal(OS) - -iotal).
3. The batch variance remains bounded, i.e., of 2 > 0, preventing degenerate class dimensions (loffe & Szegedy, 2015; Santurkar et al., 2018).
Under these assumptions, establish the following convergence result: Theorem 2 (Linear Convergence Rate). Let 1) > O be a learning rale S 1/L. Then gradient descent on Ciotal satisfies;
(rota((0(11) Liotal S (1 (p) [Lunta fiotal
That is, the sequence of iterates {0 Je20 converges linearly to a global optimum 6".
Sketch of Proof. Since Crotal is L-Lipschitz, for any gradient update we have
L
(1) & (0)) + Liooaa ((S)) - -0,5) + 100 - -,5112.
2
A standard analysis (Karimi et al,, 2016a) for gradient descent 0 0² -nooaaa implies
(tutal < (1.)
") | (rotaa((0))1 Lrotwl (0²) nu (rotal(0)) Liotal
where we used the PL condition 1 V 2 p (Erotal - Liota)) Thus,
Liotal Listal $ (1 - 17 pu) (rotai(08) Liotal
A straight.frrward induction completes the proof.
23

---

Published in Transactions on Machine Learning Research (08/2025)
This result confirms that under mild assumptions, the additional normalization and KL-distillation in Lu- miNet do not degrade convergence speed. Instead, by adaptively scaling gradients through batch-dependent variance normalization, LumiNet prevents overconfident logits from destabilizing training (loffe & Szegedy, 2015; Santurkar et al., 2018). Consoquently, the variance of gradients remains well-controlled, promoting robust optimization.
Table 9: Entropy Analysis
Table 10: Calibration Analysis
Toacher
KD
KD*
Qurs
Model
FPR35 (%)
ECE J
MCE
Temp
Entropy
0.03
0.42
0.40
1.26
CIFAR- 100
CE KD / Ours
Instance Variance
4.4
2.3
0.91
Mutual Information
3.64
3.60
3.56
3.65
ResNetöx4
3.58
/ 4.15
2.74
0.00 0 0.06
0.21 / 0.23 0.18
Arg. Gradient L2 Norm
VGGS
5.61
5.75
4.20
0.13 / 0.12 0.06
0.25 / 0.30 0.20
Gradient Variance
0.013
0.013
0.015
MobileNet- V2 WRN-40-1
10.7
11,71
6.14
0.17 0.21 / 0.09
0.38
/ 0.35 0.21
Accurary
79.42
73.08
72.91
77.50
4.13/4.39
3.51
0.09 0.15
0.07
0.17 / 0.34 0.14
A.6 Empirical Validation
To rigorously validate our theoretical claims, we conducted extensive experiments on CIFAR- 100, comparing LumiNet against standard KD. The results align closely with our theoretical analysis, as follows:
1. Gradient Stability and Convergence: LumiNet achieves gradient variance of O(10-), an order of magnitude lower than KD (O(10-3)), corroborating our analysis in Section 3.3 and Theorem 2 (Appendix
A.5). This reduction in variance reflects the adaptive scaling mechanism in perception logits (Eq. 1), which dampens outlier gradients by normalizing class-wise statistics. The improved stability is further quantified by stability score of 0.899 (vs. 0.808 for KD), where stability score is defined as the inverse of the standard deviation of training loss oscillation across epochs.
Figure 4(a-b) further demonstrates smoother convergence curves and reduced gradient volatility, directly supporting our claim that LumiNet mitigates gradient noise caused by overconfident logits. Additionally, Figure 4(d) illustrates the controlled spread of gradient variance, validating that class-wise normalization prevents unstable updates.
2. Mutual Information and Entropy: As predicted in Theorem (Appendix A.4), LumiNet's perception logits preserve richer label-relevant information. Table 9 reports an increase in mutual information I(H; Y) to 3.65 (vs. 3.60 for KD), verifying that. batch-wise normalization enhances ratrass signal retention while filtering irrelevant noise.
Furthermore, LumiNet's entropy increases to 1.26 (vs. 0.42 for KD), confirming our thcoretical claim that normalized logits mitigate overconfidence and retain knowledge about non-target classes. This supports our assertion that batch normalization amplifies class-relevant signals while suppressing extrancous variance, effectively preserving dark knowledge for improved distillation.
Figure 4(c) illustrates LumiNet's 's faster convergence, achieving a 75% loss reduction within 50 epochs, compared to KD's 60%. This aligns with Thcorem 2's lincar convergence guarantce under the Polyak-Lojasiewii (PL) condition, enabled by LumiNet's 's stabilized gradients.
3. Convergence Rate:
Additionally, curvature analysis of the Hessian spectrum estimates that LumiNet's PL constant is 2.1x larger than KD's, implying accelerated optimization. This result is particularly significant for resource- constrained deployments, where faster convergence translates to reduced compulational cost and improved training efficiency.
A.7 On the Invariance of Perception under Matching Batch Sizes
Theorem (Batch Size Dependecy). Assume the teacher fr and student j$ each form "pereeption" logits via class-wise and variance computed on the same batch size m. Then, for each batch B, the KL.
24

---

Published in Transactions on Machine Learning Research (08/2025)
0.0
0.0
00
30
0.6
60
04
66
68
30
00
Figure 5: Comparing teacher-student prediction
Figure 6: Comparing Teacher-Studer Predic- tions Similarity in Our Method
similarity in DKD method.
divergence between their perception distributions,
Σ KL KL N7 (d B
(()))
cannot systematicully degrade when m changes, aside from O vm sampling fuctuations. In particular, using the same m for teacher and student preserves the invariance of their class-wise normalization scales and does not hamper distillation performance.
Sketch of Proof. By definition, the teacher's perception logits h (xx) (z) - depend on (uf,of)) computed over the same m samples as the student's 's (n) g.)) When m changes, both (n² of and (55.a5) shift by 0( (a)))) due to sampling variance. Hence the relative studeet scaling remains consistent.
Formally, let (xx))) = 2 (x,) M and estimated from samples. Thus, scaling in hT and h aligns in expectation, keeping KL(o(h") /T), ((h//T) invariant up (.))) Consequently, matching batch sizes for teacher and student ensures that the perception- based distillation loss does not degrade simply due to changing m.
Both are divided by of of that are likewise
A.8 Mimicking Perception rather than Raw Logits.
The primary goal of traditional knowledge dist illaaiooion is to replicate the raw logits of the teacher, illustrated in Figure 5. This figure demonstrates that the predictions closely resemble the teacher's logits.In this method, we often face overconfidence issues, resulting in inferior performance compared to feature-based KD. Moreover, despite our aim to mimic the logits of the teacher, a substantial gap persists between teachers and students. However, our approach, LumáNet, depicted teacher's logits significantly lower compared to DKD. Yet, detailed in the main paper, LumiNet achieves better performance scores than DKD (Zhao et al., 2022) also, the gap between teacher and student is minimized. Importantly, teacher and student predictions are independent, diverging from similar logits. This indicates that, in logit-based distillation, we can achieve superior performance without directly mimicking the raw logits. Also, within the parameters, the student models are capable of independently learning features through their innate pattern recognition abilities without being explicitly guided to mimic the pattern learning process of the teacher model. Consequently, this empowers the student model to create new representations and inter-class relationships, for instance, capability that traditional knowledge distillation methods lack.
Figure 6, the prediction similarity to the
25

---

Published in Transactions on Machine Learning Research (08/2025)
A.9 Implementation Details
For a. fair comparison, we maintain a similar setup to previous methods(Hinton et al. 2015; Zhao et al., 2022). In traditional KD(Hinton et al., 2015), both Cross-Entropy loss and Kullback-Leibler (KL) Divergence loss are employed. Consistent with traditional methods, we utilize Cross- Pross-Entrrpy loss with the regular logits of a neural network, while the Luminet loss is applied newly generated representations of instances. Further details of the Luminet loss are provided in the main paper. In this scenario, the hyperparameter is set such that > 12, where a represents constant associated with the Luminet loss when combined with Cross-Entropy loss and t represents temperature. Specific implementation details for each task are outlined below.
Image Recognition: For training a student model on the CIFAR-100 dataset, we use a batch size of 64 and train for a total of 240 epochs. The initial learning rate (L.R) is set to 0.05, with learning rate decay applied at epochs 150, 180, and 210, where the LR is reduced by a factor of 0.1 each time. We employ a weight decay of 0.0005 and momentum of 0.9 in our stochastic gradient descent (SGD) optimizer.
When training on the ImageNet dataset, we use a batch size of 512 and train for total of 100 epochs. The initial LR is set to 0.2, with learning rate decay scheduled at epochs 30, 60, and 90, where the LR is decreased by a factor of 0.1 each time. We apply a weight decay of 0.0001 and utilize a momentum of 0.9 in the SGD optimizer.
Object Detection: For training object detection student models on the MS-COCO 2017 dataset, we use image per batch of 8. The base learning rate is set to 0.01, and the maximum number of iterations is set to 180,000. Learning rate decay is applied at specific steps during training, with decay steps set at 120,000 and 160,000 iterations.
Vision Transformer We adopt the settings described in reference (Li et al., 2022b) for training the student model of vision transformer variants. The transformer architecture includes a patch size 16, a hidden dimension of 192, 12 transformer layers, four attention heads, and a multi-layer perceptron (MLP) ratio of 4. We set the dropout rate to 0.0, the drop path rate to 0.1, and the attention dropout rate to 0.0. For optimization, we use the AdamW optimizer with a base lcarning rate of 5.0 x 10-4 and minimum learning rate of 5.0 x 10-6, The learning rate policy is cosine annealing (cos) with maximum of 300 epochs. We apply weight decay of 0.05, a warm-up factor of 0.001, and warm-up epochs of 20.
Glue Benchmark: Our experiments are conducted on the GLUE benchmark using the same overall sctup as the distillation framework presented by (Wu el al., 2023). Specifically, we utilize their training pipeline, model configurations, and hyperparaneter search space without modification. The teacher model is a fine- tuned BERT,ase, while the student model is a compact 6-layer BERT variant introduced by (Ture et al., 2019), matching the architecture used in their work. For all distillation runs, retain the original search space for learning rate, temperature, and distillation weights The only change introduced in our setup is the application of our proposed method on top of the standard knowledge distillation process. We replace the raw teacher logits with our percoption-normann normalized logits during training.
A.9.1 Calibration Analysis
To comprehensively evaluate the calibration properties of LumiNet, we compute three key metrics: Expected Calibration Error (FCF), Maximum Calibration Error (MCF), and False Positive Rate at 95% True Positive Rate (FPR95). These metrics assess how well the model's confidence scores align with actual correctness and its ability to distinguish between correct and incorrect predictions.
Expected Calibration Error (ECE): ECE quantifics the overall miscalibration by measuring the dis- crepancy between model confidence and accuracy across multiple confidence bins. Predictions are grouped into 15 equally spaced bins based on confidence scores. Within each bin, we compute the average confidence and the actual accuracy. The ECE is then calculated as a weighted sum of the absolute differences between accuracy and confidence, where the weight corresponds to the proportion of samples in that bin. A lower ECE value indicates that the model's predicted probabilities better reflect the true likelihood of correctness.
26

---

Published in Transactions on Machine Learning Research (08/2025)
Table 11: Performance of our method with the incorporation of ReviewKD Loss on CIFAR-100 dataset
TTeacher/Studee Architecture
ReviewKD
Ours
Ours#
WRN-40-2 - ShuffleNet-Vi
77.14
76.95
77.29
ResNet322x + ShuffleNet-V?
77.78
77.55
77.93
Table 12: Detection results on MS-COCO using Faster-RCNN-FPN (Lin et al,, 2017) backbone with incor-
porating ReviewKD.
Feature-Based Methods
Logit-Bane Based Methods
ResNet101
(Teacher) and ResNet18
(Student)
Teacher
Student
FitNet
FGFT
ReviewKD
KD
TAKD
DKD
Ours
Ours*
AP
42.04
33.26
34.13
35.44
36.75
33.97
34.59
35.05
35.34
36.89
AP30
62.48
53.61
54.16
55.51
56.72
54.66
55.35
56,60
56.82
57.05
APn
45.88
35.26
36.71
38.17
34.00
36.62
37.12
37.54
37.56
39.59
Net50 (Teacher) and MobileNet-V V2
(Student)
AP
40.22
29.47
30.20
31.16
33.71
30.13
31.26
32.34
32.38
34.18
AP'o
61.02
48.87
49.80
50.68
53.15
50.28
51.03
53.77
53.84
53.95
AP7s
45.88
30.90
31.69
32.92
36.13
31.35
33.46
34.01
33.57
36.44
Maximum Calibration Error (MCE): MCE identifies the worst-case miscalibration by determining the maximum absolute diffcrence between accuracy and confidence across all bins. Unlike ECE, which provides weighted average measure, MCE focuses on the most severe miscalibration present in any confidence range. This metric is particularly useful for identifying whether the model is drastically over- or under-confident in specific confidence intervals.
False Positive Rate at 95% True Positive Rate (FPR95): FPR95 is a robustness metric that eval- uates the model's ability to distinguish between true and false positives. mcasures the false positive rate when the true positive rate (TPR) fixed at 95%. The calculation is performed using the Receiver Oper- ating Characteristic (RÔC) curve, where the threshold is adjusted such that the TPR reaches 95%, and the correspouding false positive rate is recorded. Lower FPR95 values indicate improved robustness, as fewer incorrect samples are classified with high confidence.
Implementation Details: We implement these calibration metrics following standard evaluation proce- dures:
ECE and MCE Calculation: We use 15-bin equal-width binning. Predictions are grouped into bins based on confidence scores, and we compute accuracy and confidence within each bin. The ECE and MCE are derived from these statistics. FPR95 Calculation: We apply the ROC curve method to compute the false positive rate a fixed true positive rate of 95%. This involves converting softmax probabilities into binary classification labels per class and analyzing class-wise ROC curves.
A.10 Incorporating with feature-based distillation
In our experiments, we typically refrain from utilizing feature- based distillation loss, as our research primarily aims advance the domain of logit-based knowledge distillation methods. However, certain architectures, and explore its compatibility with existing feature-based KD methods, we incorporated the feature-basc based loss (ReviewKD (Chen et al., 2021b)) alongside our LumiNet loss.
This combination resulted in significant performance improvements, as demonstrated in Table 11 for the image recognition task and Table 12 for the object detection task. In the tables, the asterisk (*) denotes the utilization of the combined loss function. Overall, it highlights how integrating feat ure-baedd losses enhances overall performance and showcases compatibility with existing methodologies.
Despite the performance improvements, we also investigated certain limitations in feature- distillation methods. These methods often require longer convergence times, which deterred us from incorporating
27

---

Published in Transactions on Machine Learning Research (08/2025)
cetture-based KD. For instance, ReviewKD (Chen et al,, 2021b), despite its comprehensive approach, requires significant training time due to its multi-level distillation process and complex components like the Attention- Based Fusion module. OFD (Cho & Hariharan, 2019), while focusing on multi-layer distillation, demands cxtra convolutions for feature alignment, increasing computational needs. Similarly, CRD (Tian et al., 2020) employs a contrastive loss that requires a large memory bank, adding to computational costs.
In summary, while incorporating reature-based logits into our knowledge distillation method yields better results, it also introduces significant drawbacks in terms of privacy, computational requirements, and training time. Hence, advocate for logit-based knowledge distillation more resource-effiieen and versatile alternative for various applications.
Comparison of top-1 and top-5 accuracy and ECE of knowledge distillation and our method under noisy settings across different teacher-student architectures.
Table 13:
Teacher (Acc.)
Student (Acc.)
Method
Top-1
Top-5
ECE
KD
53.88
78.95
0.16
ResNet322x4 (55.57)
ResNet8x4 (52.21)
Ours
54.70
79.18
0.12
KD
54.78
78.05
0.27
VGG13 (53.77)
VGG8 (51.75)
Ours
55.06
79.96
0.09
A.11 Effect of Overconfidence
To evaluate performance under realistic label noise, we constructed noisy CIFAR-100 dataset using instance- dependent label noise rather than random corruption Ju et al. (2022). Specifically, we trained ResNet-18 for two epochs the clean training set and used its cross-entropy loss to identify the 30% most misclassified training examples. We then replaced their ground-truth labels with the weak model's predictions, simu- lating plausible human annotation errors. Under this challenging noisy setting, our method significantly outperforms traditional KD across two leacher-sludent. architecture pairs. For the ResNet32x4 Res pair, our approach improves Top-1 accuracy and reduces ECE from 0.16 to 0.12, demonstrating both better accuracy and confidence calibration. Similarly, for VGG13-VGGS, we achieve 1.5% gain in Top-1 accu- racy and drastically reduce ECE from 0.27 to 0.09 (shown in the table 13 ). These results show that our method is more robust under noisy supervision and mitigates the overconfidence often seen in traditional KD approaches.
ResNet32x4 + ResNet8x4
VGG13 - VGG8
Batch size 4
Batch size
30
Batch size 64
Batch size 64
Batch size 256
Batch size 256
KL Divergence Loss
20
15
10
50
100
150
200
250
50
100
150
200
250
Epoch
Epoch
Figure 7: KL divergence loss between teacher and student logits during training for varying batch sizes (4, 64, 256) across two teacher-student pairs: ResNet32x4 - ResNetSx4 and VGG13 - VGG8, All losses are scaled equally for visual comparability.
28

---

Published in Transactions on Machine Learning Research (08/2025)
A.12 Effect of Batch Size
As shown in Figure 7, examine the effect of batch size the KL divergence loss between teacher and student logits during training. For batch sizes of 64 and 256, the KI. divergence curves remain stable and nearly identical, indicating that even moderate batch sizes provide sufficient sample diversity to generate reliable batch-level statistical perception, resulting in consistent. representations between teacher and stu- dent. However, when the batch size is reduced to 4, we observe a noticeable increase and volatility in KL divergence. This is duc the limited number of samples per batch, which weakens the quality of class- level statistical signals and leads impacts optimization and highlights a limitation of method in extremely low batch size settings. All KT. divergence values were scaled equally for fair visual comparison.
greater mismatch in logit distributions. This divergence negatively
Knowledge Distillation Loss (Batch Size = 64)
0.09
ResNet32x4 ResNet884
wrn40 2 ShuffleNet_W v1
0.08
KL Divergence Loss
0.07
0.06
0.05
0.04
0.03
50
100
150
200
250
Epoch
Figure 8: KT. divergence loss comparison across different teacher-student configurations with batch size 64. Homogencous pairs (c.g., ResNet3224 - ResNet8x4) show lower divergence than heterogencous ones (c.g.. WRN40 + ShuffleNet vl).
Table 14: Top-1 Accuracy (%) on CIFAR-100 for different leacher-student model pairs. 'DD refers to standard knowledge distillation. 'Ours' uses perception-normalll logits based on local batch statistios, while + replaces local statistics with global teacher statistics aggregated across the training set.
Teacher + Student
Teacher
Student
KD
Ours
Ours
ResNet32x4 + ResNet8x4 79.42
72.50
73.33
77.50
76.82
VGG13 + VGGS
74.64
70.36
73.98
74.94
75.11
WRN-40-2 - ShuffleNetV1
75.61
70.50
74.83
76.95
76.62
A.13 Global vs Local statistics
To investigate the impact of normalization scope in distillation method, compare two variants: (1) one using batch-wise (local) statistics (Ours) and another (2) using dataset-wide (global) statistics (Ours++) for normalizing teacher logits. For the global variant, we precompute the mean and variance of teacher logits over the entire CIFAR-100 training set and apply them consistently during training, In contrast, the local variant computes these statistics dynamically within each batch. As shown in Table 14, both variants outperform the standard KD bascline across all teacher-student pairs, demonstrating the effectiveness of our logit normalization approach. Notably, the local version achieves the best performance, suggesting that adapting to the distribution of each batch introduces beneficial regularization. The global version
29

---

Published in Transactions on Machine Learning Research (08/2025)
remains competitive, indicating that both strategies enhance distillation, though local statistics offer slight advantage in accuracy.
A.14 Logit-Space Misalignment
In knowledge distillation settings where the teacher and student networks differ architecturally, we observe that KL divergence loss tends to remain higher throughout training compared to homogeneous configura- tions, as shown in Figure 8, This is because in heterogeneous setups, the representation of a sample in the logit space is typically not aligned between the teacher and student; architectural differences induce mismatched inductive biases, feature extraction hicrarchies, and semantic docompositions. As a result, the student struggles to mimic the teacher's output distribution effectively, even under the same temperature and training hyperparameters. This structural misalignment limits the efficacy of logit-based distillatonn alone, which relies on the student approximating the teacher's soft targets directly. In contrast, featuro-based distillation approaches can partially recover this gap by aligning intermediate feature representations, which are often more robust. to architectural divergence and carry richer localized information. This explains why heterogeneous distillation with only logits underperforms, despite sharing the same optimization regime.
A.15 Logit Complexity Analysis
Neural knowledge distillation faces inherent challenges due to the architectural capacity gap between teacher and student models, where students with fewer parameters struggle to directly mimic the complex distribu- tions generated by larger teachers. Two critical issues arise in traditional KD. First, there is a significant disparity between the probabilities of target and non-target classes. The teacher model tends to produce overly confident predictions for the target classes, which creates a considerable learning burden for the student model, as discussed in section 3.1 of the paper. Second, this challenge intensifies with an increasing number of classes, manifesting as multiple high-probability regions (multi-mode) across the class space. These issues become particularly pronounced in large language models, where the vocabulary size far exceeds typical image classification lasks, resulting in substantially more complex probability distributions for the student to learn. Our perception-basse approach effectively addresses these limitations by significantly reducing the class dwarfing effect and diminishing the multi-mode peaks, as demonstrated in Figure 9. Using RosNet18 on ImageNet (1000 classes), we observe that our method produces more balanced probability distributions compared to temperature-scaled KD (T=4), making the dark knowledge transfer more tractable for the student model while preserving essential class relationships.
A.16 LumiNet in Large-Language Model
KD in Large Language Models (LLMs) presents unique challenges compared to its application in computer vision tasks. In vision models, the logit distribution usually displays a single-mode pattern, making it rel- atively easy for student models to replicate the teacher's probability distribution. However, L.MMs operate with vocabulary spaces that span thousands to millions of tokens, resulting in complex 'multi- dis- tributions for a sample. This fundamental difference makes traditional KD approaches less effective for LLMs.
We used the dataset split within this space for our exporiment 2 Dolly dataset for fine-tuning, while 500 samples were reserved for testing. Additionally, 80 and 240 samples were used from Vicuna and SelfInst, respectively for evaluation. We have adapted our method to make it suitable for LLMs. Our experimental results, as shown in Table 15, demonstrate that our method consistently outperforms existing KD approaches across different model sizes. For instance, with GPT-2 340M as the student model, our method achieves 27.8, 13.8, and 17.1 R-T. scores on Dolly, SelfInst, and Vieuna test sets, respectively, surpassing both conventional KD (25.0, 12.0, 15.4) and Sequential KD. Notably, in several cases, our student models even outperform the 1.5B teacher model.
We have used 13.5k samples from the
For our experimental setup, we used a 1.5B parameter model as the teacher and tested student models of varying sizes (120M, 340M, and 760M parameters) based on the GPT-2 architecture. The training
Phtep::/hhuggingfacccco// coMMiniLI
30

---

Published in Transactions on Machine Learning Research (08/2025)
Probability Distribution Comparison
Methods
KD
Ours
0.005
Probability
0.004
0.003
0.002
0.001
200
400
600
800
1000
Class Index
Figure 9: Comparison of probability distributions between traditional KD and our proposed method ImageNet using ResNet18. The blue line represents tomperature-scaled KD (T=4), showing multiple high- confidence regions and significant disparities between target and non-target classes. The red line shows our method's distribution, which effectively reduces both the class dwarfing effect (lower peaks) and multi-modal nature the distribution, resulting in more manageable class relationships for the student model to learn. This visualization demonstrates how our approach simplifics the dark knowledge transfer while maintaining informative class relationships across the 1000 ImageNet classes.
Table 15: Evaluation results. We report the average R-L scores across random seeds. The best scores of each model size are boldfaced, and the scores where the student model outperforms the teacher are marked with *
Model
# Params
Method
Dolly
SelfInst
Vicuna
Teacher
1.5B
27.6
14.3
16.3
SFT w/o KD
23.3
10.0
14.7
KD
22.8
10.8
13.4
120M
SeqKD
22.7
10.1
14.3
Ours
23.8
11.4(0.42)
14.9(o.10)
SFT w/o
KD
25.5
13.0
16.0
KD
25.0
12.0
15.4
GPT-2
340M
SeqKD
25.3
12.6
16.9*
Ours
27.8* (0.47)
13.8(o.48)
17.1* (o.10)
SFT w/o KD
25.4
12.4
16.1
KD
25.9
13.4
16.9*
760M
SegKD
25.6
14.0
15.9
Ours
28.6* (0.49)
(8.66((0.4))
14.7*
17.5*
was conducted with a batch size of We implemented sequence-level lokenization and used the AdamW optimizer with learning rate of 5e-5. The training was performed on single 4090 GPU.
A.17 Broader Impact
LumiNet enhances logit-based knowledge distillation by avoiding the use of intermediate feature representa- tions, which improves privacy and efficiency especially in scenarios involving commercial APIs federated
31

---

Published in Transactions on Machine Learning Research (08/2025)
learning. However, caution is warranted in high-stakes applications (e.g., healtheare, surveillance), output
logits can still leak sensitive training information. We strongly encourage responsible use, including thorough calibration, bias analysis, and evaluation of potential misuse in downstream contexts. Future iterations could integrate built-in safeguards or metrics to assess and mitigate such risks.
32


Reviews:
[{"review_id": "YDVp0qoGo0", "reviewer": "Reviewer_zjPH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. Reduces overconfidence and confirmation bias using batch-level logit normalization.\n2. Adds no extra parameters and matches vanilla KD in speed.\n3. Outperforms state-of-the-art on classification, detection, and transfer tasks across models.\n\n**Weakness**\n1. Since calibration can improve KD on logits, how about other calibration methods? Even some models that originally trained by a calibration loss such as focal loss[1,2]?\n2. [3] is a recently proposed method that can be applied on top of logit, how's the performance if applied with such methods?\n3. Why batch level is better? how about the whole dataset statistic for calibration?\n4. Why normallzing can suppressing extreme confidence values ? This can also make under confidence, right? \n\n\n[1] Calibrating Deep Neural Networks using Focal Loss\n[2] Dual focal loss for calibration\n[3] Feature Clipping for Uncertainty Calibration", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces **LumiNet**, a novel logit-based knowledge distillation method that addresses the limitations of conventional logit matching—namely overconfidence and confirmation bias—by introducing a *perception-driven calibration mechanism*. Instead of treating logits independently, LumiNet normalizes them using batch-level class-wise statistics (mean and variance), forming *perception logits* that reflect contextual relationships across samples. This calibration reduces extreme predictions and better captures inter-class dependencies without requiring intermediate features, preserving privacy and efficiency. Empirical results on CIFAR-100, ImageNet, MS COCO, and TinyImageNet demonstrate that LumiNet consistently outperforms state-of-the-art logit- and feature-based distillation methods across classification, detection, and transfer learning tasks, all while maintaining low computational overhead.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Add more calibration methods for comparison, since your calibration techniques works, how about others?\n\nHow about your method applied on standard calibration task, how does it compare to others?", "broader_impact_concerns": "LumiNet may still inherit and subtly reinforce biases from the teacher model, especially when batch-level calibration is applied without safeguards, potentially affecting fairness in real-world applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G5esopl4eu", "reviewer": "Reviewer_w2Kg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The core idea of perception logits is original and intuitive, presenting a practical means to contextualize logits using batch-level information.\n\n- It delivers consistent empirical gains across various datasets (CIFAR-100, ImageNet, MS-COCO) and architectures (ResNets, MobileNet, ViTs), showing both robustness and scalability.\n\n- Unlike many recent distillation methods, it introduces no additional trainable parameters and keeps latency on par with standard KD.\n\n- The authors provide solid ablations and calibration analyses, demonstrating reduced sensitivity to temperature and improved resistance to confirmation bias.\n\n- The framework is architecture-agnostic, successfully extending to vision transformers, a domain where feature-based methods often dominate.\n\n**Weaknesses:**\n\n- The theoretical underpinnings—particularly around how batch normalization of logits improves generalization—are more heuristic than formal. The Gestalt theory analogy, while creative, lacks rigorous justification.\n\n- All experiments are within computer vision. Despite claims of broader relevance, no non-visual domain results (e.g., in NLP) are shown.\n\n- The method relies on sufficiently large and diverse batches to function optimally. Its performance in low-data or imbalanced class settings remains unclear.\n\n- Some results, though strong, could benefit from direct statistical significance tests to distinguish gains from noise across trials.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes **LumiNet**, a new approach to knowledge distillation that revisits logit-based methods with a twist. Unlike prior logit-focused strategies that treat each instance independently, LumiNet introduces the idea of “**perception**”—a form of batch-aware logit calibration. By leveraging batch-level mean and variance statistics per class, the method constructs “perception logits” that temper overconfidence and reduce confirmation bias. This is inspired by Gestalt psychology, particularly Kurt Lewin’s field theory, where each instance is influenced by the context of surrounding samples. LumiNet is shown to work well across several tasks—image classification, object detection, and transfer learning—outperforming both standard and state-of-the-art feature-based KD techniques, without incurring additional computational costs or architectural complexity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Essential:**\n\n- Clarify limitations of LumiNet more explicitly. For example, what happens when batch sizes are small, or classes are highly imbalanced?\n\n- Include a more formal justification (e.g., statistical or optimization perspective) for why batch-wise normalization of logits benefits the KD process beyond mitigating overconfidence.\n\n**Optional but Recommended:**\n\n- Include a simple NLP experiment—even if limited—to support generalization claims.\n\n- Provide more analysis on failure cases or scenarios where LumiNet underperforms compared to feature-based methods.\n\n- Consider evaluating the method in semi-supervised or noisy label settings, where overconfidence might be even more detrimental.", "broader_impact_concerns": "The paper does not raise significant ethical red flags. In fact, by avoiding the use of intermediate representations, LumiNet offers greater privacy protection in scenarios where teacher internals are sensitive or inaccessible—such as in federated learning or model distillation from commercial APIs.\n\nHowever, some implicit risks are worth noting:\n\n- Logits can still carry latent information about training data, especially in high-stakes domains (e.g., healthcare, surveillance), which may warrant caution.\n\n- The improved efficiency might encourage deployment in low-resource or under-regulated environments, where calibration and fairness checks are often overlooked.\n\nIf not already included, a brief Broader Impact Statement addressing these points—especially around calibration, privacy, and misuse—would be a helpful addition.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KfkvcBcU56", "reviewer": "Reviewer_F6Zz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Exploring logit-based knowledge distillation helps the deployment of small models.\n\n2. The proposed method is simple and effective.\n\nWeaknesses:\n1. Differences between overconfidence and confirmation bias are not clear. As described in the main body, confirmation bias is based on overconfidence and both harms the distillation performance.\n\n2. Why does the propose method being called LumiNet? This is not an abbreviation of some phrases, nor is it a network structure. \n\n3. Lack of comparison of knowledge distillation methods on 2025.\n\n4. The taxonomy of knowledge distillation needs discuss relation-based ditsillation. RKD and this paper belongs to this category.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the logit-based knowledge distillation from the perspective of statistical logit calibration. Authors argue that logit-based distillation have the challenge of model's overconfidence and confirmation bias, which leads to inferior performance\ncompared to feature-based methods. To bridge this gap, authors present LumiNet, a novel knowledge distillation algorithm designed to enhance logit-based distillation. By normalizing the batches of samples, the propose LumiNet can improve the distillation performance. Extensive experiments shows LumiNet's effectiveness and mitigate the corresponding challenges.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)