SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines
Authors: Itai Morad
Nir Shlezinger
Yonina C. Eldar
Venue: ICLR 2026 Poster (2026)
Keywords: knowledge distillation
SGD-based learning
Bayesian machine learning
Primary area: learning theory
TL;DR: We adopt a Bayesian perspective on KD to analyze student convergence under SGD and advocate using Bayesian deep learning models as teachers to improve student performance.

Abstract: Knowledge Distillation (KD) is a central paradigm for transferring knowledge from a large teacher network to a typically smaller student model, often by leveraging soft probabilistic outputs. While KD has shown strong empirical success in numerous applications, its theoretical underpinnings remain only partially understood. In this work, we adopt a Bayesian perspective on KD to rigorously analyze the convergence behavior of students trained with Stochastic Gradient Descent (SGD). We study two regimes: $(i)$ when the teacher provides the exact Bayes Class Probabilities (BCPs); and $(ii)$ supervision with noisy approximations of the BCPs. Our analysis shows that learning from BCPs yields variance reduction and removes neighborhood terms in the convergence bounds compared to one-hot supervision. We further characterize how the level of noise affects generalization and accuracy. Motivated by these insights, we advocate the use of Bayesian deep learning models, which typically provide improved estimates of the BCPs, as teachers in KD. Consistent with our analysis, we experimentally demonstrate that students distilled from Bayesian teachers not only achieve higher accuracies (up to +4.27\%), but also exhibit more stable convergence (up to 30\% less noise), compared to students distilled from deterministic teachers.

Published as a conference paper at ICLR 2026
SGD-BASED KNOWLEDGE DISTILLATION WITH BAYESIAN TEACHERS: THEORY AND GUIDELINES
Itai Morad', Nir Shlezinger' and Yonina C. Eldar
'School of ECE, Ben-Gurion University, Be'er er Sheva, Israel FFaculty of Math and CS, Weizmann Institute of Science, Rehovot, Israel itaimorri @post. bgu. ac. i 11, nirsh wl.guaac.... ac. yonina e eldar wei wmannmmmn.... ac
ABSTRACT
Knowledge Distillation (KD) is a central paradigm for transferring knowledge from a large teacher network to typically smaller student model, often by leveraging soft probabilistic outputs. While KD has shown strong empirical success in numerous applications, its theoretical underpinnings remain only partially understood. In this work, we adopt Bayesian perspective on KD to rigorously analyze the convergence behavior of students trained with Stochastic Gradient Descent (SGD). We study two regimes: (t) when the teacher provides the exact Bayes Class Probabilities (BCPs); and (fi) supervision with noisy approximations of the BCPs. Our analysis shows that learning from BCPs yields variance reduction and removes neighborhood terms in the convergence bounds compared to one-hot supervision. We further characterize how the level of noise affects generalization and accuracy. Motivated by these insights, we advocate the use of Bayesian deep learning models, which typically provide improved estimates of the BCPs, as teachers KD. Consistent with our analysis, we experimentally demonstrate that students distilled from Bayesian teachers not only achieve higher accuracies (up to +4.27%), but also exhibit more stable convergence (up to 30% less noise), compared to students distilled from deterministic teachers.
INTRODUCTION
Knowledge Distillation (KD) Hinton et al. (2015) is a fundamental technique in machine learning, widely used for model compression, transfer learning, and improving generalization Gou et al. (2021 Yim et al. 2017). A core idea in KD is to transfer knowledge from a "teacher" model to a (typically smaller) "student" model by training the student to match the teacher's output probabilities rather than categorical outputs (one-hot labels). This softened supervision has been shown to lead to improved performance across range of tasks Mansourian al. (2025). Consequently, a substantial body of research has focused on designing KD mechanisms, including strategies for dynamic temperature scaling, feature-based distillation, and task-aware teacher-student matching Zhu et al. (0024)) Waang et al. 2024). typically assuming a large teacher which is optimized to maximize its own performance.
Despite its widespread adoption, the theoretical foundations of KD remain only partially understood Phuong & Lampert (2019). In particular, the impact of the teacher's 's output probabilities on the student's s optimization trajectory and generalization has not been fully characterized. While insights have been developed for special cases such self fiff-iistillatinn Safaryan et al. (2024), a principled understanding of how KD affects common learning algorithms, such as Stochastic Gradient Descent (SGD) and its variants, is still lacking. Recent works have begun to explore KD through Bayesian lens Menon et al. 22021); Ye et al. (2024), interpreting the teacher's outputs as (possibly noisy) estimates of the true class posterior. This probabilistic perspective gives rise to the possibility of analyzing relatively unexplored aspects of KD; particularly in terms of the dynamics of SGD-based learning, as well as the statistical calibration of the teacher and its influence on the student.
Contributions Motivated by the Bayesian viewpoint of KD, our work provides rigorous analysis of the interaction between probabilistic supervision and SGD-based learning. Our analysis considers two regimes: (i) supervision with the Bayes Class Probability (BCP), i.e., the exact posterior probabilities,

---

Published as a conference paper at ICLR 2026
which correspond to perfect teacher from the Bayesian perspective KD; and (ii) supervision with noisy estimates of the BCP, ie.. a realistic imperfect teacher. Based on this modeling, we are able to show variance reduction compared to one-hot supervision in SGD-based learning. Through numerical study, we show that this variance reduction translates into improved performance of the trained student.
Based on our analysis, which indicates that the effectiveness of KD depends on how well the teacher approximates the BCP (i.e., how well-calibrated the teacher is), we advocate the use of Bayesian deep learning models teachers Gawlikowski al. (2023) in KD. Bayesian deep learning brings forth a key advantage, as it typically results in better-calibrated probabilistic predictions, thereby producing more faithful approximations of the BCPsJospii et al. (2022). We show this benefit can be effectively harnessed in KD, through two complementary strategies: (i) training the teacher directly using Bayesian learning techniques, or (fi) converting an existing deterministic (frequentist) pre-trained teacher into a Bayesian model via posterior approximation techniques. Our experimental study demonstrates the effectiveness of this approach, both empirically validating our theoretical analysis, as well as showing the usefulness of Bayesian teachers. Our results show consistent improvements in accuracy and variance reduction when distillation is performed with properly calibrated teachers. These empirical findings align with our theoretical analysis, supporting our claim that leveraging Bayesian teachers in KD improves both the convergence and generalization of SGD-based learners.
RELATED WORKS
Theoretical Justification for KD Since the introduction of KD Hinton et al. (2015), several works focused on explaining its mechanisms from different theoretical perspectives. A common approach attributes its usefulness to the soft estimates of the teacher regularizing the learning procedure et al.. (2019) y Yuan et al. (2020). Recently, Cha & Cho provided a minimal working explanation of KD through a precision-recal trade-off. Huang et al. (202T) framed KD as a trade-off between knowledge inheritance and exploration. PPhuon & Lampert 2019) provided generalization bounds and identified factors explaining the success of KD, namely data geometry and optimization bias, and Lopez- ct al. (2155) connected KD to privileged information. In contrast to these works, both our theoretical analysis and our algorithmic contributions are based on a Bayesian perspective on KD, which facilitates characterizing the learning dynamics, as well as unveil how one can enhance a teacher model for KD.
Bayesian Perspective on KD Menon et al. 2222 inspected the soft estimates of the teacher from a Bayesian perspective, aiming to provide theoretical justification for its empirical success. There, it was shown that learning from accurate BCPs can reduce the variance of the student's 's objective, and excess risk bounds were derived, depending on the l2 distance between the teacher's predictions and the true BCPs. Similarly, Dao et al. (202T) provided generalization bounds on the student including terms such as the ε distance between the true BCPs and the teacher's predictions. While our work is motivated by this Bayesian viewpoint, we differ in focus: rather than analyzing the statistical properties of the risk, we study how the probabilistic teacher outputs affect the behavior of learning algorithmms. Safaryan et al. (2024) is, to our knowledge, the first to examine the impact of distillation on SGD. However, their analysis considers specific scenarios such as eelf-distillation or having the student be compressed version of the teacher, for which dedicated gradient approximation is formulated. In contrast, our results hold for arbitrary teachers, do not rely on gradient approximations and explicitly model the teacher's 's predictions BCP estimates. Furthermore, our analysis reveals that learning from BCPs holds the desired interpolation property under mild assumptions, which allows student Neural Networks (NNs) generalize in settings where learning from one-hot labels results in overfitting.
Distilling from Calibrated Teachers Beyond theory, our work also contributes the design of teacher models for KD, advocating for the use of Bayesian teachers to enhance calibration. Recently, Kim al. (2025) and Fan al. (2024b) recognized calibration as a key driver in KD and suggested algorithms to enhance calibration of deterministic models. Related recent works Ye et al. (2024), Hamidi et al. (2024) aimed to improve teacher calibration by modifying the training objective, encouraging predictions that better approximate the true BCPs. While these approaches operate by altering loss functions or improving calibration of deterministic model, we instead adopt the Bayesian deep learning paradigm, which naturally yields calibrated predictions and quantifiable uncertainty measures. This both improves the accuracy of the distilled student and provides an indication of the calibration of the teacher. Beyond calibration, recent works have focused on how the divergence
2

---

Published as a conference paper at ICLR 2026
used in KD affects how probability mass is transferred from teacher to student. ABKD Wang et al. (2025a) develops an a-3 divergence framework that balances concenttncnnn effects to improve distribution matching, and sequence level f-divergence methods Wen et al. (2023) similarly examine how different divergences influence the way probability mass is transferred from teacher to student in sequence models. In contrast these approaches, rather than modifying the divergence, focus how the quality of the teacher's s probability estimates impacts student performance, Other works have considered the intersection of Bayesian models and distillation, but with fundamentally different goals. For instance, Bulô et al. (2016), Gurau et al. (2018), and Korattikara Balan et al. (2015) focused on distilling Monte Carlo dropout ensembles to reduce inference cost or improve uncertainty estimation in the student model. Similarly, [ee et al. (2023) studied coff-distillatinn under dropout. In contrast, our work focuses on leveraging Bayesian teachers as means to enhance student learning performance as direct consequence of our theoretical analysis.
3 ANALYSIS OF SGD-BASED LEARNING FROM PROBABILITY ESTIMATES
This section characterizes the impact of supervision with BCP estimates on bGDed learning. Specifically, we provide convergence guarantees for both cases of supervision with perfect and with noisy probability estimates, and compare the results to the standard case of learning from one-hot labels. We show that under common assumptions, learning in all cases converges in expectation to the same model, but with potentially better convergence rate and lower stochastic noise at the optimum when supervising with probability estimates. Intuitively, as the probability estimates become noisier, the convergence of SGD worsens, up point where it is longer beneficial to supervise with probability estimates. The proofs of the results presented this section are delegated to Appendix C
3.1 PRELIMINARIES
Learning Framework We consider the supervised learning of classification task, where each input X' C I&d is associated with one of K classes, represented by the one-hot label y E ). The inputs are related to the labels via data generating distribution P PSShaee-Shwaare & Ben-David (2014), which dictates true BCPs P ()) i..., the conditional distribution of label (in one-hot form) y given an input æ. The target model, referred to as the student model, a mapping Φo : X + y. typically NN, parameterized by f Rd where Y is the K -ccategr statistical manifold. The desired model is one that minimizes the risk, also referred to as the generalization error, namely,
min (r) = - [ [Z()o(x)).)) 4)] BERd
Here, e y + R is loss function, typically the Cross-Entropee (CE), defined henceforth as lCE-
The learning procedure aims approach (I) without knowing P by seeking Empirical Risk Minimization (ERM), formulated using labeled dataset D = {LIn, yn) ) i.e.,
min fo 0
(()o(a))) yn). n=1
Σ
OER
D
Rcarly,Eiffe fo for a random sample (or an i.i.d. batch) ( = (x, y) distributed via ( ~ P.
SGD-based KD
In KD, teacher model & (often pretrained) is used along with the dataset D. The teacher model usually has larger capacity and is a more complex model compared to the student. The softmax predictions of the teacher, (...) Y, used additional soft labels in training the student.
The student model is then trained with distillation parameter E [0, 1]. modifying 2 into
min 0
Σ
(1 x)e(Go(z.)) v..) + (do ((x,))
BERd
D
When is linear in the second argument CEor KL-dvverneeeeeeecce loss), the objective simplifies into into
1
min
(0)
[(o((an), (1- A)y, -x((x)))
(4)
AER
D
n=1
3

---

Published as a conference paper at ICLR 2026
The standard SGD framework then be readily applied by iterating over
0++1 a SE. (0'),
where > 0 is the learning rate, and & c D is randomly sampled data point mini-batch from D.
analyze the convergence profile of the student model in SGD-based probabilistic supervision, we define the desired student parameters as 0" arg minøeRs fp (0) and their corresponding risk fp == f0()). We make use of the following standard assumptions, commonly adopted in the analysis of SGD.
Assumptions & Definitions
ASI (Strong quasi-convexiny The risk function fr Rdd IR. is differentiable and jı-strongly quasi-convex for some constant μ > 0, i... for any O E IRd holds that
0 + 1 11100 0
fp
fr 0 + (V
weaker assumption the Volyak-Lojasicen (PL) condition.
AS2 Lojaskeeijjsiiwiizz condition). The risk function fr R2. - is differentiable and satisfies the PL condition with parameter μ 0, i.e. for any 0 R it holds that
f 2p(ff(0) f8).
Since strong quasi-convexity implies the PL condition, we use either AS7 addition to the following assumptions:
AS2 our analysis, in
AS3 (Expected smoothness). The loss fe (0) is differentiable and L-smooth in expectation (for some constant C) for & i.e. for any 0 E He it holds that
E E | f (0) V fe((0")11 < 2C(fr(0) -fp). Moreover, the risk fp is L-smooth, namely ([v))(()) V (02)ll s L| 02 for any 0,, 02 E Rd Garrigos Gower 2023).
AS3 often holds for the CE loss with standard NN architectures. This stems from the fact that the training procedure or activations of NNs often lead to its Jacobians being locally bounded, which, combined with the fact that the CE loss is smooth in the logits, ensures that the gradients vary smoothly in expectation Hardt et al. (0166).
AS4 (Student expressiveness) The student model Φ9 is sufficiently expressive such that can real- ize the true BCP, i.e., there exists some 0 E Rd such that Φe (xx) = (P(y)lat (æ),... P(yk(æ)] for all supp(P). where yı the one-hot encoding of the kth label.
The above assumptions are used to maintain tractable analysis of SGD-based probabilistic supervision. Moreover, the design guidelines that arise from our analysis are empirically shown in Section enhance <GD-based KD even in settings where AS7 A.44 do not necessarily hold.
Our analysis also uses the notions of interpolation and gradient noise, defined as follows: Definition (Interpolation). The optimization problem [ satisfies the interpolation property if there exists some Oin Rd such that foatfor all (x. y) E supp(P)), the model perfectly fits the data, i.e., Edpo: (w),y) min ((oo())y)) BERd
v(x,y) supp(P)).
By Definition I interpolation holds if there exists 0r: which minimizes the loss for every input-label pair in the support of the data generating distribution P. When this condition holds, we clearly have that i E arg mingers (0). meaning that the interpolating model is also a global minimizer of the generalization error. This formulation is valid in expectation, and naturally extends the classical finite-sum interpolation definition to continuous distributions Garrigos & Gower (2033)
Definition (Gradient noise). We define the gradient noise the optimization problem 7 as
inf 8- fr Exyy))~ [[] oo (z), 1
The gradient noise controls the variance of the gradients at the optimum and can be seen as a measure of how far one from interpolation.
4

---

Published as a conference paper at ICLR 2026
3.2 SUPERVISION WITH PERFECT BCPS
We proceed to examine the effects of distilling from a perfect teacher (from the Bayesian KD perspective) on the training dynamics of the student. In this case, the student is supervised with the Irue BCPs The resulting setup can be viewed as distilling from teacher whose mapping is given by (xx) [P lz.................. Pgy [z]]. We henceforth focus on learning with the CE loss, setting where the BCP is bounded away from zero (to avoid instability of the CE with vanishing probability mass functions) Haussler 1992). our analysis, we first show that the formulation of the generalization error using the BCPs yields the same optimal model as the one using one-hot labels in ① and then we analyze the convergence of SGD-based learning of the empirical version of the BCP-based risk.
BCP-Based Risk
To formulate the BCP-based risk function, replace the one-hot labels y in the with the true BCPs. The resulting objective is given by
stochastic optimization problem (
min (0) B(m.))~ [ (z)), P(y]==))]] ôER
The loss landscape of fo over R4 may differ from that of /p. The convergence of SGD-based learning analyzed w.r.t the optimal value and weights of the considered objective function. Therefore, before studying convergence, we compare f and its minimizer 0 fū and its minimizer 0", respectively. Proposition 1. When AS4 holds, the inference rule which minimizes both 7 and (6) is the true BCP, i.., (xx)) (P(y) xx) P(yk|()) [z)], and the minimal loss is the conditional entropy of y given a.
Proposition I also claimed in a different form in Menon et al. (021)) proves that both ( and (6) share the same minimum and the same minimizer. The key benefit of learning with true BCPs instead of one-hot labels comes from Proposition
Proposition 2. When AS4 holds, the stochastic optimization task 6 satisfies the interpolation property (Definition]
7..
Based on Definition I interpolation here means that the minimizer of (6) matches the true BCPs at each sample, as opposed to attaining zero error on the one-hot labels. In particular, matching the true BCPs implies that the CE loss is minimized at every individual sample, and therefore the gradient of each per-sample loss also vanishes, as stated in the following Lemma.
Lemma 1.
When AS4 holds, the minimizer Ô the stochastic optimization task (6) satisfies
(a)( P(y)=))) = for all x: E (supp(P).
Proposition indicates that the stochastic optimization task 6 has several benefits over (1) Particularly, when learning from data using an empirical risk measure, croosition@jipliaii that the model that minimizes the risk also minimizes the corresponding empirical risk, i.e., achieves
min (0) BERd
Σ D
1
(()g(zn), P(yn
for any data set D C supp(P). This property does not hold when learning from one-hot labels, where the empirical risk often minimized by an overfitted model which does not generalize.
BCP-Based SGD Learning The observation in pnaosittionmmtiiaattit analyzing the convergence
profile (in the sense of the risk function fo of SGD based on the empirical version of 77). We note that such learning coincides with SGD-based KD with a perfect teacher (in the BCP sense), as (3) reduces to ? when & outputs the true BCPs and 1. Specifically, using the Assumptions described in Subsection 3.1 we obtain convergence guarantees for SGD-based learning starting from an arbitrary student model 00 The resulting bounds, formulated in the sense of both the model parameters difference, as well as the resulting risk function, are stated in the following theorems:
1
Theorem 1. Let Assumptions ASI
AS3 and hold. For any step-size a S C the parameters
learned via SGD (5) with true BCPs instaad of one-hot labels converge as
[E[[[0 001 (1 1000 11
5

---

Published as a conference paper at ICLR 2026
AS3 and AS4 For any step-size a S p LL
Theorem Let Assumptions AS2 by the model learned via SGD
the risk achieved
(S) with true BCPs instead of one-hot labels converges
E[fp(0') [ S (1 (x=" -pp)
Theorems 2 showcase the usefulness of SGD-based learning from true BCPs by comparing them with the corresponding bounds in the general gradient-based stochastic optimization literature Garrigos Gower (2023). The standard SGD convergence bounds, which hold for similar constraints on the learning rate, are made up of the convergence speed term, dictating the rate of convergence, and the neighborhood term, corresponding to learning stability. Contrasting these with Theorems reveals two main differences:
Model Variance: Theorems 1 2 include only convergence speed term, which decays with o as in standard SGD Garrigos & Gower (2023). The neighborhood term, which in standard SGD is Thm. 5.8), vanishes due Lemma I indicating
proportional to # o (Garrigos & Gower 2023
stable learning.
Supported Learning Rates: The range of learning rates a for which convergence can be guaranteed twice as large compared to (Garrigos & Gower 2023 Sec. 5), allowing the usage of larger step-sizes which in turn lead to faster convergence.
Overall, the results indicate that distilling from well-trained teacher that sufficiently estimates the BCPs alters the optimization task from seeking to fit the one-hot labels to an interpolation task.
3.3 SUPERVISION WITH NOISY BCPS
While distilling from perfect BCPs yields desirable convergence guarantees, KD is typically done with an imperfect teacher. Here, we examine the effects on the training dynamics of the student when distilling from an imperfect teacher, i.e., when the student supervised with noisy BCPs. Such noise corresponds to modeling errors, limited training data, or flawed training of the teacher. We show that even with noisy BCPs, we are able achieve variance reduction and improved generalization under certain conditions.
Noisy BCP-Based Risk We model the noisy BCPs the true BCPs with additive distortion This can be viewed as having the teacher mapping be ((x) = [P (æ)) + e1,..., P(yx (2t + ex). where model Pe zero-mean noise with variance and uncorrelated entries. This representation facilitates incorporating the deviation of the teacher from the BCPs while maintaining analytical tractability. While this formulation focuses on modeling BCP deviation as additive noise, similar findings can also be obtained for alternative models for noisy probabilities which remain on the simplex, such as Dirichlet-distributee,, as shown in Appendix(D)
The noisy BCP-based risk function is formulated in similar fashion to the BCP- risk ( but with the noisy BCPs instead of the perfect ones. The resulting objective given by
min fP. (0) EE [((dg(z) (x) P Plyla) E)
(10)
BERd
Noisy BCP-Based SGD Learning We now analyze the impact of noisy BCPs on the convergence of SGD-based learning. We again employ the assumptions described in Subsection 3.1, and start from arbitrary student model 09, The resulting bounds, formulated in the sense of both the model parameters difference, as well as the resulting risk function, are stated in the following theorems:
Theorem 3.
Let Assumptions AS1
AS3 and AS4 hold. For any step-size S 2L
the parameters
learned via SGD 5 with noisy BCPs instead of one-hot labels converge as
2a
[
0- 11) s (1 o 112 +
o
(11)
μ
where of is the gradient noise (Definition 2) for the noisy BCP risk of TO. Theorem 4. Let Assumptions
AS2
AS3 and AS4 hold. For any step-size S 2LC the risk
achieved by the model via SGD
with noisy BCPs instead of one-hot labels converges
La
Elfp,p,
ft] I
- (b)
(12)

---

Published as a conference paper at ICLR 2026
0.50
Aosrape ginereliizaton eeve can
0.020
Omn lot labels
Less sely prebitiltes
0.8
Loss reley grobabriitoe
More noney probahilitos
Nare maop probebalitacs
0.6-
Tens Bupsa pesbubiitics
0.18
0.4
Cus bet labela
0.014
Leor anlcy prnbabiiitee.
0.0
Marn roley probabllizee
Tran Bagin probanditins
0.18
20000
20020 30000
40000
12000
20003 Deration
S0000
40000
0.000
0.0
0.2
0.6 0.0 0.5
Steration
Figure 1: Learning curves, test accuracies, and average generalization error gaps in the learning curves when training with one-hot labels, when supervised with the true BCPs, when supervised with true BCPs corrupted with different noise levels, and supervision by combinations of one-hot labels and noisy BCPs adjusted based on several À values. Complete experimental details are provided in Appendix E
Theorems 3 mirror the standard SGD convergence Garrigos & Gower (202)), with one main difference: the gradient noise o reflects the variance induced by distilling from noisy BCPs. To under- stand this term, compnar it with the gradient noise in the standard case of learning with one-hot labels. Proposition Under AS4 with K classes, let Jo. [x)( denote the k-th column of the Jacobian of the student model, i.e., Veo(e xx) [Je, loo ( [x))....,, Jo. <[oo(=)]]. Then, for the CE loss, the gradient nosse of the one-hot risk
oj Papp
(13)
p)
and the gradient noise of the noisy BCP risk (I0)
is
K
of = v. EPD
11 (2)]1/]
(14)
P[yn|o)]]
The expressions above demonstrate how gradient noise is affected by the form of supervision. In both cases, the resulting gradient noise is weighted average of the K columns the Jacobian matrix evaluated the optimum. In the standard case, the weights BCPs P(y) z). When distilling from noisy BCPs, the weights are the noise variance v multiplied by the inverse of the squared entries of the true BCPs ((y (z). Since is a measure of how well the teacher mates the true BCPs, as v decreases, so does the gradient noise, and convergence becomes tighter. Note that when / i... the teacher outputs the true BCPs, then (II) reduces to (8) and (12) reduces (Q).
the inverse of the K entries of the true
An alternative point of view stems from Definition Since the gradient noise is a measure of distance from interpolation, the gradient noise in the standard case can be seen as a measure of how far the one-hot labels are from the true BCPs, i.c., how noisy the data generating distribution is. Similarly, the gradient noise in (14) can be seen as a measure of how far the noisy BCPs are from the true BCPs.
Proposition 3 indicates that supervision with noisy BCPs is beneficial (under the above modeling assumptions) when o; < oj. That is, when the noise level smaller than the inherent variance from one-hot labels. The point at which this balance flips depends on the data generating distribution (through P), model smoothness (through the Jacobian), and teacher quality (through v)).
Numerical Study The effects of the vanishing neighborhood term when supervising with true BCPs. as well as the neighborhood term when supervising with noisy BCPs on student performance, are demonstrated and explored in a synthetic experiment. We compare student models trained using SGD based on (i) one-hot labels; (ii) true BCPs; and (iii) BCPs corrupted with two noise levels (denoted more noisy and less noisy). The latter are also compared with different combinations of one-hot labels and noisy BCPs adjusted based on values. Noisy probabilities were generated using the Dirichlet distribution (Appendix D). The complete details of the study, alongside additional experiments, correlation analysis, and numerical validation Proposition are provided in Appendix
The learning curves in F demonstrate that all students converge to the minimal loss the same speed (for a given learning rate). Nevertheless, the loss of the student trained with true BCPs converges smoothly and exactly to the minimal loss, while students trained with one-hot labels or noisy BCPs
7

---

Published as a conference paper at ICLR 2026
converge only up to neighborhood term, which depends on the noise level. The smaller neighborhood term demonstrated in Theorem Fand Proposition Bis reflected in the experiments as both a lower error floor and reduced variance in the generalization eπror curves. The middle plot of Figure T shows that accuracy behaves in the same way: less noisy BCPs yield higher and more stable accuracy, while noisier BCPs result in larger fluctuations. This indicates that the variance reduction predicted by our analysis not only improves convergence in terms of loss but also translates directly into improved student performance. It is also demonstrated in the right plot in Figure that the value of that yields the best student performance depends on the level of noise in the BCP, i.e.. on how calibrated the teacher is. This motivates future work of finding optimal À values, possibly based on uncertainty measures provided by the teacher. In summary, supervision with noisy BCPs reshapes SGD convergence by changing the gradient noise, in manner that depends on the data, the model structure, and the teacher calibration. Yet, KD with an imperfect teacher (providing noisy BCPs) can still outperform learning from one-hot labels, offering improved convergence and performance in SGD-based learning.
4 GUIDELINES FOR DESIGNING TEACHERS
4.1 METHOD
Rationale The characterization of supervision with BCP estimates on SGD-based learning gives rise to concrete guidelines for designing teacher models in KD. Specifically, it implies that having a more calibrated teacher (in the sense of producing accurate BCP estimates), improves the convergence of the student model in SGD-based learning, aligning with the observations made by Mennon et al.) (202T) Dao et al. (202T), which analyzed the KD-risk with the true BCPs, This motivates using teacher models based on Bayesian deep learning, as Bayesian Neural Networks (BNNs) are known to be better calibrated compared to standard (frequentist) Jospon et al. (2022): Kristiadi et al. (2020)
Gawlikowski et al. (2023).
Bayesian Deep Learning BNNs extend traditional NNs by modeling the network parameters as random variables. Unlike traditional NNs which view learning from data D as seeking a single deterministic parameter vector, BNNs learn a posterior distribution over the parameter space. After the posterior distribution over the weights is estimated, inference carried out by marginalizing the likelihood, typically by aggregating multiple realizations of the stochastic forward pass. Existing approaches to approximate such Bayesian inference by inferring the posterior distribution employ Markov Chain Monte Carlo sampling methods Neal [193);; Variational Inference (VI) approaches which approximate the posterior distribution by optimizing over a family of tractable distributions, e.g., Bernoullllll (as in Monte Carlo Dropout) or Gaussian Fortuin 2022); > Graves (201T); and Laplace Approximations (LAs), which use a quadratic (second order Taylor) expansion of the log- -ossttrrr,, followed by deriving normal distribution over the NN weights Ritter et al. (2018). A key advantage of the LA approach is that it can be applied to pre-trained deterministic NNs Gawlikowski et al. (202)).
Bayesian Teachers Our method employs KD with BNN teacher. We consider both BNNs trained from scratch using VI, and transforming pre-trained NNs into BNNs via the LA. As such, our approach can be utilized both cases where is desirable to train a teacher model, and in cases where one wishes to leverage pre-trained NN. Stochasticity can be introduced across all network weights, in specific lay- ers, or only in the final layer. This flexibility enables one to adapt the complexity of the teacher model to the computational budget. The trained Bayesian teacher is incorporated into standard KD practice using response-based based distillation. Specifically, the teacher's s soft labels are obtained via the Monte Carlo ap- proximation by averaging over 5 stochastic forward passes (averaging is done after softmax is applied).
4.2 NUMERICAL STUDY
We evaluate our approach on the CIFAR-100 dataset, which contains 60,000 color images of size 32 x 32 across 100 classes, split into training and test sets with a 5:1 ratio. We evaluate 12 teacher-student pairs: 6 with matching architectures and 6 with different ones. For each pair, we compare six teacher types: deterministic, Bayesian (BNN trained with variational inference), Laplace (BNN obtained by applying the LA to the pre- rre-trined deterministic teacher), MCM et al. (2024) (the deterministic teacher fine-tuned with a conditional mutual information loss to capture contextual structure), TTDA (the pre-trained deterministic teacher adapted to generate stochastic predictions using
8

---

Published as a conference paper at ICLR 2026
Table 1:
Test accuracy (%) of student networks on the CIFAR-100 dataset, averaged over runs. Both the teachers and the students were trained with varying random seeds between runs. Results are reported for teacher-studed pairs with both matching and different architectures. The subscript denotes changes relative to the corresponding deterministic teacher/student.
Teachers and students with matching architectures
Teacher
Res Net-11118 RosNet-18 73.23
ResNet-50
ResNet-50 ResNet-34 73,61
WRN--002 WRN-16-2
WRN-40-2 WRN-10-1 65.03
VGG-13
Student
ResNet-18 18 73,23
VGG-8
Student accuracy (mo KD)
67.70
73:52
Teacher Type
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
Deterministic
73.25
75.92
75.41
76.26
75.41
76.82
70.97
70.80
70.97
68.92
74.46
76.08
Bayesian (Ours)
74.61
76.92
75.67
77.27
75.67
77.63
74.27
72.94
74.27
70.77
74.72
77.61
+1.35
+1101
41.25
-1011
+0:22
-0.01
+3.0
+2.14
-330
41.35
40.27
+1.33
Laplace (Ours)
72.72
76.30
74.83
76.69
74.83
76.23
70.45
71.80
70.45
70.04
73.95
76.09
4130
+b38
.1.5
41.9
-2.1
40093
-.51
-0.0
MCMI Yost ol. 2024
73.46
75,86
75.60
76.44
75.60
76.86
71.28
70.87
71.28
68.66
74.61
76.35
TTDA
73.23
76,21
75.41
76.30
75.41
76.83
70.97
70.75
70.97
69.05
74.46
76.61
+0.9
-001
4001
4155
60.33
-0.5
Hamidi
2024
74.56
71.76
74.89
71.76
69.99
69.99
65.42
73.86
8990
-3.5
-1.37
-388
-13
-999
897
-350
-115
Teachers students with different architectures
Teacher
ResNet-50 550 WRN-400 40-2 70.76
ResNet-50
VGG-13
VGG-13 W WRN-40-1
ResNet-50 WRN-16-2 67.70
VGG-13
Student
VGG-8
ResNer-18
WRN-50-2
Student accuracy (nn KD)
73,52
73.23
65.03
70.76
Teacher Type
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
Dererministic
75.41
73.79
75.41
75.66
74.46
76.36
74.46
67.90
75.41
69.36
74.46
73.71
Rayesian (Ours)
75.67
75.82
75.67
77.27
74,72
77.41
74,72
71.37
75.67
73,63
74.72
75.45
40
+1.
+1.36
-160
+0:37
+1.0
+0.37
+1.07
-6.76
+133
40.77
75
Laplace (Ours)
76.11
73.95
72.52
74.39
4135
+3/77
-1.3
-0,6
-0.5
4009
-2.00
+2:97
63.38
.030
MCMI Ye all 2024
35.60
73.50
75.60
75.75
74.61
76.40
74,61
67.48
75.60
69.58
74.61
74.20
TTDA
75.41
73.76
75.41
76.00
74.46
76,80
74,46
67.53
75.41
69.24
74.46
74.05
-508
-0.35
10.45
4037
40.22
.0.34
MSE
71.76
71.09
71.76
74.09
73.31
74.87
73.31
71.76
68.55
71.39
-365
-1.00
-3.55
-1.66
-1.33
-1,40
+L.5
-2:0
-398
-0.1
-1.15
-232
test-time data augmentation SanpenttttionGaallloowkkkk. Gawlikowski et al. (2023)) and MSEHHamidi et al. (2024) (trained with mean squared error loss instead of CE). Unless stated otherwise, we will refer to students trained from each teacher type as deterministic students, Bayesian students, Laplace students, etc., even though the student is always the same deterministic model for a given seacher-studen pair.
Training follows standard response-based HD]Hinton et al. 2015).. For each teacher-student pair, we use 6 combinations of teacher temperature, student temperature, and À values, and repeat the experiment with each combination 5 times. We report the mean student accuracy achieved by the best configuration. Accuracy variances and results of all 6 combinations of temperature-X values are provided in Appendix K while AappennixFcoonteeeeec is available at ttttts: htt//gittuuu hub. com Betp://aaeeiiaaa KD As shown in Table Bayesian students consistently achieve the highest accuracies across the board. The gains over deterministic students reach up to 4.27% (ResNet-50-
full details of the training setup. The source code
even though the corresponding Bayesian teachers themselves improve on deterministic teachers by up to 0.26% in almost all cases. Laplace students improve in all cases except one, with gains of up to 3.07% despite all Laplace teachers performing worse than their deterministic counterparts.
We also examine the stability of the students' performance during training. The average noise level during the last 50 epochs of the test accuracy curves is reported in Figure measured by (i) the gap between the highest and lowest accuracies reached and (łi) the standard deviation around the mean. The results show that Bayesian students not only achieve higher accuracy than deterministic students, but also consistently converge with less variance, indicating highly stable learning. For example, in the ResNet-50--Reet RReNNet-1 18 case, Bayesian students show a 30% improvement in convergence noise compared to deterministic students.
These observations are aligned with our theoretical analysis in Subsection 3.3 There, we proved that as noise decreases in the supervisory noisy BCP estimates, the neighborhood term in the convergence

---

Published as a conference paper at ICLR 2026
Student distilee Buyesian toacher
distilled Setermitistic
with bot
Max Mon Aos [8]
0.6-
0.4
Ncc. STD [W]
0.15
0.05
0.00
Reciette RReNaltt
ResNatse Reseles3 Restintt
WIEN A0 -WN- 167
WRN-40-1
VGG-12 VGG-
Restiato voca
voG-1a Restiete
VOG- WiN WiiN-40-1
VGG WIN 40-2
RedNeto WRN
Rostatoo WIN
Restette
WRUE 40.7
Figure 2: Test accuracy stability over the last 50 training epochs with standard ooe-hot training, deterministic teachers, and Bayesian teachers.
bounds for both the weights and the generalization error becomes smaller. In synthetic experiment we confirmed this effect in the generalization error, and also observed that accuracy behaves analogously. The experiments reported here show the same: students distilled from Bayesian teachers, which provide more calibrated probability estimates (i.e., provide better approximations of the BCPs compared to deterministic teachers), converge with reduced variance and higher accuracy. The experiments performed with the Adam optimizer in order to show that our theoretical results, which hold for standard SGD, generalize to related optimizers as well. This empirical generalization is in fact supported by recent theoretical studies Xia & Massei (2023) showing that Adam admits convergence guarantees under PL-condition and smoothness assumptions similar to those used in our analysis, with convergence up to a noise-dependent neighborhood term (analogous to SGD). These findings provide theoretical backing for the empirical applicability of our conclusions beyond SGD.
5 CONCLUSION
We characterized the learning dynamics students trained from both exact and noisy approximations of the BCPs, corresponding different levels of teacher calibration. Our analysis shows that teachers that better approximate the true BCPs yield variance reduction in the learning dynamics of the student, and numerically this leads improved performance, both in terms of higher accuracy ceilings and reduced variance the accuracy curve near convergence. Moreover, we advocate the use of Bayesian deep learning models teachers, since they provide better-calibrated outputs. Aligned with our theoretical analysis, our experimental study shows that students trained from Bayesian teachers achieve higher accuracies and more stable accuracy curves compared to students trained from deterministic teachers.
REFERENCES
Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. Dropout distillation. In International Conference Machine Learning, pp. 90-1 PMLR, 2016.
Sungmin Cha and Kyunghyun Cho. Why knowledge distillation works generative models: A
minimal working explanation. arXiv preprint 1311::::500..111.....
Lei Chen, Tieyong Cao, Yunfei Zheng, Yang Wang. Bo Zhang, and Jibin Yang. A dynamic dropout self-distillation method for object segmentation. Complex & Intelligent Systems, 11(1):1-14, 2025.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. arXiv preprint arXiv: 100.:v:9904.0422 2019.
Tri Dao, Govinda Kamath, Vasilis Syrgkanis, and Lester Mackey, Knowledge distillation as semiparametric inference. arXiv preprint arXiv:2 2021.
10

---

Published as a conference paper at ICLR 2026
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network. arXiv preprint arX.v:1910.0125500 2019.
Chengyu Dong, Liyuan Liu, and Jingbo Shang. Toward student-oriented teacher network training for knowledge distillation. arXiv preprint arXiv:2206.06661, 2022.
Wen-Shu Fan, Xin-Chun Li, and De-Chuan Zhan. Exploring dark knowledge under various teacher capacities and addressing capacity mismatch. arXiv preprint arXiv:2405. 2024a.
Wen-Shu Fan, Su Lu, Xin-Chun Li, De- De-Chuan Zhan, and Le Gan. Revisit the essence of distilling knowledge through calibration. In Forty-first International Conference on Machine Learning, 2024b.
Luyang Fang, Yongkai Chen, Wenxuan Zhong, and Ping Ma. Bayesian knowledge distillation: A bayesian perspective of distillation with uncertainty quantification. In Forty-first International Conference on Machine Learning, 2024.
Vincent Fortuin. Priors in Bayesian deep learning: A review. International Statistical Review, 90 (3):563-591, 2022.
Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv: 12235, 2023.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513-1589, 2023.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6): 1789-181191111 19,8,992222 2021.
Alex Graves. Practical variational inference for neural networks. Advances neural information processing systems, 24, 2011.
Zhen Guo, Dong Wang, Qiang He, and Pengzhou Zhang. Leveraging logit uncertainty for better knowledge distillation. Scientific Reports, 14(1):31249, 2024.
Corina Gurau, Alex Bewley, and Ingmar Posner. Dropout distillation for efficiently estimating model confidence. arXiv preprint arXiv:1809 10562, 2018.
Shayan Mohajer Hamidi, Xizhen Deng, Renhao Tan, Linfeng Ye, and Ahmed Hussein Salamah. How to train the teacher model for effective knowledge distillation. In European Conference Computer Vision, pp. 1-18. Springer, 2024.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp. 1225-1234, PMLR, 2016.
Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, and Sanjiv Kumar. Supervision complexity and its role in knowledge distillation. arXiv preprint arXiv:2301. 12245, 2023.
David Haussler. Decision theoretic generalizations of the PAc model for neural net and other learning applications. Information and computation, 100(1):7--150, 1992.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 150::0233303333 2015.
Md Ismail Hossain, MM Lutfe Elahi, Sameera Ramasinghe, Ali Cheraghian, Fuad Rahman, Nabeel Mohammed, and Shafin Rahman. Luminet: Perception-driven knowledge distillation via statistical logit calibration. Transactions Machine Learning Research.
Tao Huang, Shan You, Fei Wang. Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. Advances in Neural Information Processing Systems, 35:33m,,3533371666 35:33716-3377777 2022.

---

Published as a conference paper at ICLR 2026
Zhen Huang, Xu Shen, Jun Xing, Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian- Xian-Sheng Hua. Revisiting knowledge distillation: An inheritance and exploration framework. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3579-3588, 2021.
Minoh Jeong, Martina Cardone, and Alex Dytso. Data-driven estimation the false positive rate of the bayes binary classifier via soft labels. In 2024 IEEE International Symposium on Information Theory (ISIT), pp. 368-373. IEEE, 2024.
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands-on bayesian neural networks tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2)229-48, 2022.
HM Dipu Kabir, Abbas Khosravi, Mohammad Anwar Hosen, and Saeid Nahavandi. Neural network-based uncertainty quantification: A survey of methodologies and applications. IEEE access, 6:36218-36234, 2018.
Suyoung Kim, Seonguk Park, Junhoo Lee, and Nojun Kwak. The role of teacher calibration in knowledge distillation. IEEE Access, 2025.
Anoop Korattikara Balan, Vivek Rathod, Kevin Murphy, and Max Welling. Bayesian dark knowledge. Advances in neural information processing systems, 28, 2015.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in relu networks. In International conference on machine learning, pp. 5436-5446. PMLR, 2020.
Hyoje Lee, Yeachan Park, Hyun Seo, and Myungjoo Kang. Self-knowledge distillation via dropout. Computer Vision and Image Understanding, 233:103720, 2023.
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv: 11/55111133,,,.1 2015.
Amir M Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei, Elaheh Badali Golezani, Zeynab Yasamani Ghamchi, Vida Ramezanian, Alireza Taherian, Kimia Dinashi, Amirali Miri, et al. A comprehensive survey on knowledge distillation. arXiv preprint arXiv:2503. 12067, 2025.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In International Conference on Machine Learning, pp. 7632-7642. PMLR, 2021.
Thomas Minka. Estimating a dirichlet distribution, 2000.
Radford M Neal. Probabilistic inference using markov chain monte carlo methods. 1993.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference on machine learning, pp. 5142-5151. P 2019.
Yi Ren, Shangmin Guo, and Danica J Sutherland. Better supervisory signals by observing learning paths. arXiv preprint orXiv:2203 22t.::::22002288..... 2022.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th international conference on learning representations, ICLR 2018- 10018-coneeeecee track proceedings, volume 6. International Conference on Representation Learning, 2018.
Mher Safaryan, Alexandra Peste, and Dan Alistarh. Knowledge distillation performs partial variance reduction. Advances in Neural Information Processing Systems, 36, 2024.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Gehui Shen, Xi Chen, and Zhihong Deng. Variational learning of bayesian neural networks via bayesian dark knowledge. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences Artificial Intelligence, pp. 2037-2043, 2021.
12

---

Published as a conference paper at ICLR 2026
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, et al. Variational learning is effective for large deep networks. arXiv preprint arXi::2402. 17641, 2024.
Guanghui Wang, Zhiyong Yang, Zitai Wang. Shi Wang, Qianqian Xu, and Qingming Huang. Abkd: Pursuing proper allocation of the probability mass in knowledge distillation via a- a-v-divergence. arXiv preprint arXiv:2505.c 04560, 2025a.
Lu Wang, Liuchi Xu, Xiong Yang. Zhenhua Huang, and Jun Cheng. Debiased distillation for consistency regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7799-7807, 2025b.
Shuo Wang, Chihang Wang. Jia Gao, Zhen Qi, Hongye Zheng, and Xiaoxuan Liao. Feature alignment- knowledge distillation for efficient compression of large language models. arXiv preprint arXiv:2412. 19449, 2024.
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. F-divergence minimization for sequence-level knowledge distillation. arXiv preprint 1rX1v:2307./5190, 2023.
Lu Xia and Stefano Massei. Influence of hyperparameters on the convergence of adam under the polyak-lojasiewicz inequality. In European Conference Numerical Mathematics and Advanced Applications, pp. 485-493. Springer, 2023.
Shunzhi Yang, Liuchi Xu, Mengchu Zhou, Xiong Yang, Jinfeng Yang. and Zhenhua Huang. Skill-transferring knowledge distillation method. IEEE Transactions Circuits and Systems for Video Technology, 33(1 :)6477-6522 2023.
Shunzhi Yang, Jinfeng Yang, MengChu Zhou, Zhenhua Huang, Wei-Shi Zheng, Xiong Yang, and Ren. Learning from human educational wisdom: student-centered knowledge distillation method. IEEE Transactions on Puttern Analysis and Machine Intelligence, 46(6):41 81-888845..-11111111111111110111 2024.
Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and En-Hui Yang. Bayes conditional distribution estimation for knowledge distillation based on conditional mutual information. arXiv preprint arXiv:2401.08732. 2024.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4133 4141, 2017.
Hua Yuan, Yu Shi, Ning Xu, Xu Yang, Xin Geng, and Yong Rui. Learning from biased soft labels. Advances in Neural Information Processing Systems, 36:59566-535 59584, 2023.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3903-3911, 2020.
Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Jialu Liu, Michael Bendersky, Marc Najork, and Chao Zhang. Do not blindly imitate the teacher: Using perturbed loss for knowledge distillation. arXiv preprint arXiv:2305.050/11 2023.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pp. 11953-11962, 2022.
Kaixiang Zheng and En-Hui Yang. Knowledge distillation based on transformed teacher matching. arXiv preprint arXiv:2402.11
2024.
Songling Zhu, Ronghua Shang, Bo Yuan, Weitong Zhang, Wenjie Li, Yangyang Li, and Licheng Jiao. Dynamickd: An effective knowledge distillation via dynamic entropy correction- based distillation for gap optimizing. Pattern Recognition, 153:1 10545, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
MAIN SECTIONS
Introduction
Related Works
Analysis of SGDD-Based Learning from Probability Estimates
Preliminaries
Superviston with Perfect BCPs
Supervision with Noisy BCPs
Guidelines for Designing Teachers
Method
Numerical Study
Conclusion
10
APPENDICES
Additional Related Work
15
Basic Linear Algebra, Facts, and Inequalities
15
Detai led Proofs
16
Proof Proposition
16
Proof of Proposition?
17
Proof Proposition
17
Proof of Theorem
19
Proof Theorem
20
Proof of Theorem
20
Proof Theorem
21
Dirichlet-Perturber BCPs
21
Synthetic Experiments
24
Implementation Details of the Experiments in Section 4
27
Few-shot Classification
29
Temperature Scaling
29
Bayesian Teacher Parameters
31
Additional Distillation Methods & Tiny ImageNet
32
Accuracy Variances and Full Results
34
Limitations
36
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
Probabilistic supervision The effects of supervision with probability estimates or soft labels have been investigated in several works. [aan et al. (2024a) examined the dark knowledge provided by teach- ers of different capacities, showing that larger teachers tend produce probability vectors with lower distinction among non -oon-gound--o -ruth classes, and explored multiple ways to address capacity mismatch. Zhang (2023) argued that a teacher minimizing the CE loss sub-optimal, and proposed the PT- Loss objective, which transforms the original teacher into proxy teacher whose distribution is closer to the ground truth. They further established a theoretical link between distributional closeness and student generalizability. Ren et (alll(222))) (2022) studied the learning paths of models trained with noisy BCPs, one-hot labels, and BCPs. Yuan et al. al. (023) analyzed whether biased soft labels can be effective, proposing con- ditions under which biased soft label learning remains classifir-cooniister and ERM-leamable. For bi- nary classification, Jeong et al. (2024) studied probabilisticc supervision and introduced theoretical prop- erties of their estimator, including consistency, unbiasedness, convergence rate, and variance. While also develop a theoretical analysis of probabilistic supervision, we focus on the dynamics of SGDD-based convergence and their role in improving student performance. We introduce advantages of KD in stochastic optimization, whereas prior work focused on generalization gaps and estimator properties.
Customized teachers and improved KD schemes Additional works focused on modifying teacher predictions to enhance student performance. For example. Guo et al. (2024) recognized that uncertain classes in teacher outputs hinder learning and proposed increasing the weight on confident classes in teacher predictions, capacity, identifying calibration as a key driver of KD. Dong et al. (2022) established that an ERM minimizer can approximate the true label distribution under Lipschitz continuity and robustness of the feature extractor, and introduced a teacher training method with Lipschitz and consistency regularization. Others explored more novel directions: Yang al. (2024) drew inspiration from human educational strategies, Wang et al. (2025b) facilitated knowledge sharing within the same class, Harutyunyan et al. 2023) aligned teacher predictions with the student's 's neural tangent kemnel, and Yang et al. (2023) employed meta-learning to strengthen the student's s ability to generate new knowledge. In contrast, our approach leverages Bayesian teachers. whose outputs naturally better calibrated driven by the Bayesian paradigm, which was motivated by our theoretical analysis.
et al. calibrated teacher logits based on the model's 's representational
BNNs and KD In addition the works discussed in Section 2 several additional studies considered BNNs and KD. The work of Chen et al. (2025) proposes dynamic dropout (which can be viewed as form of Bayesian modeling) self-distillation method for object segmentation, which solves this problem by discarding the knowledge that the student struggles to learn. The work of Fang et al. (2024) developed a method named Bayesian Knowledge Distillation (BKD) to provide transparent interpretation of the working mechanism of KD. In BKD, the regularization imposed by the teacher model in KD is formulated as a teacher-informed prior for the student model's parameters. The work of Shen et al. (22TT) performed KD with complex BNN MCMC teacher into a BNN VI student. These works on BNNs and KD have fundamentally different goals from our work. Our approach towards utilizing BNNs is a direct consequence of our theoretical analysis, and is from the perspective of their supervisory signal containing better information in KD compared to deterministic models, and not from the perspective of reducing the cost of Bayesian inference and uncertainty quantification.
B BASIC LINEAR ALGEBBA, FACTS, AND INEQUALITIES
We first review some standard inequalities and basic facts which are used in the proofs.
The squared norm of the sum of two terms can simply be bounded by the sum of each squared norm individually
||a b1l2 < 2 2||a|| + 2|1B/
(15)
.f(0) is called L smooth if its gradient is Lipschitz continuous with constant L 2 0 (Definition
2.24 in Garrigos & Gower (2023), ), which means that
[V,( ((())) S T. 02ll-
(16)
15

---

Published as a conference paper at ICLR 2026
Additionally, f(0) is L smmothh then
f(01) - (v.(())))))))))))))))))))))))))) -02) + 2
10 - 02l1²
(17)
for all 01, 02 Rd (Lemma 2.25 in Garrigos & Gower (2023)
Similarly to other analyses that establish linear or exponential rates, in our proofs we rely on a standard technique: converting one-step recurrence into convergence inequality. Assume the sequence {2e h22 satisfies
2++1 S (1 7) N,
t= 0 2....
for some constants (0, 1] and N 2 0. Unfolding the recursion gives
2 (1 n)'= 20 (1 -)) (1-1)==
(1-1))
By recognizing the geometric sum, this yields
20
N 7/)- 7)) + 17
2 S (1 - 17)* 20 + N "
(18)
j=0
We make use of Gibbs' incquality, which states that for the discrete probability distributions P (pi.-.. ......... Pu) and Q (m)) holds that
Σ
log S
log q:.
1=1
1=1
with equality and only p: = qi for .,... n. A direct consequence of Gibbs' inequality is that
min g.s.l.g discrete probability distribution
log = Σ Ps log
(19)
We make of the law of total expectation, which states that for random variables (X, Y) and any measurable f.
B(f(X,Y)
Ex Exyx(
|x]]
(20)
It holds that
i lo))
0o(0)
(21)
00
x(0)
00
for (0)) > 0.
For any T. and M mmxm
Ma Tr(M aa ))
(22)
where Tr(-) is the trace operator. Moreover, since both trace and expectation are linear operators, can interchange them, i.c.,
[[Tr(M)] = TF[[M)].
(23)
For the generalization error (I. the optimal inference rule in the sense of minimal cross-entropy risk is the true conditional distribution Shalev-Shwartz Ben- Ben-David (2014), i.e.,
<(x) [P(y) (..,.,, P(UK x)).
(24)
C DETAILED PROOFS
C.I PROOF OF PROPOSITION
Proof. For ease of presentation, instead of using the K K-categorical one-hot vector y, we define the label variable E (1...., K), and since db (æ: represents a K K-dimensional probability mass function,
16

---

Published as a conference paper at ICLR 2026
write p(s k//) (do(())l With these notations, the BCP-based risk (6 with the CE loss becomes
(0)
~= (x) P(y[z))]
(((
(22
log ((s|=))
log p(s|z)
(- ((s
P(s(ar))
DKL (f(s))][[((\ H(sz))
where (s)ar) { is the conditional entropy of & given æ. while (-I.- the Kullback-Leiibe divergence ShareenSsShaleeySSaattttee & David (014)) As D non-negative and equals zero only when
p(s) x) ( x i.c. (0g (x)] = plys x). it holds that the BCP-based risk (6 is minimized when
the inference output is the true conditional distribution. Under Assumption AS4 the minimizer is
attainable by some 0" R and the proof is concluded.
C.2 PROOF OF PROPOSITION
Proof. To prove that the interpolation property holds, we note that (xx represents K K-dimeesionall probability mass function. Accordingly, under the CE loss, holds that
K
min ((y)())))()))) dgRd
P(yla)) = min
Ply x)) log (x)]
k=1
Plyx x5) log a))
(
E(Øg- (z)) P (z)())
Therefore, dg- (xr) [Ply) l.),..., Plyx [z)]] brings (2)) Plyl=)) minimum for all x E supp(P)) and interpolation holds at 0", proving the proposition.
C.3 PROOF OF PROPOSITION 3
Proof. In order to express the gradient noise of both the one-hot risk (I) and the noisy BCP risk (IO). we recall the resulting expressions for the gradient noise based on Definition i.c.,
inf 0 Earg min fo
(x [ (x) v)
and
of
inf の- Earg min f pe
E EGDD)-, [ P(yl=) +)
Next, we give an expression for (ol (x), 3) for any K --dimensional probability mass function ÿ. In this case, we have that Vol(de. (x), Vel "" log Φe- (æ)], whose ith element is
dE aE(0o- (n), 9)
K
alog Φo- (x)
00;
00,
Ol (x)
[
[00- (x) ]
00,
A=
doo (a)
M
(())(x)
00,
Accordingly, (ol()o-(x) ÿ) can be written as
1
Vel(oe- (w), Ø) Jo Vol (x)) diag
3.
P
17

---

Published as a conference paper at ICLR 2026
Finally, are able to plug in the expression for Vol(de- (x), y) into the definition of the gradient noise.
oj = 0" Earg min fp
inf
((
[I Vel(o (æ), y)1l
| 00 (a:) diag(
[3/xx]
P(y x))
1 P(y)(z)
NM
En Bm-
Bayl (1 Jolcba (z=)
diag(
[y)x ||ae]]]
1
P(ys (a))
Ja (x:)][+²
P(y)lo))
k=1
Enp-
11J0 (x)]]13
P(y):))))
km 1
where (x)) is the ken column of the Jacobian of the student model.
Next, give expression for V Vá (æ), Plylz) + E). In this case, P is the true BCP which is a K-sized vector, and is the aero-mean noise with variance v and uncorrelated entries, also a K-sized vector. Therefore,
V (x), + +)
VallØg- (x), P(yla)) + (gg(og-))))) Vollog- (x), ([yl)) + (ol(dg-(x),
C
ADC
Vgl(og.(z).))
V Wol-E" log oe (x)
In the first equality we used the fact that the cross-entropy loss linear in the second argument. In the second equality we used the fact that objectives 6 and share the same minimizer, which we prove Appendix C.7 Accordingly, we have that
0V Voll(og P(y(() +
K Jelx
aloo Φg. (x)]
00;
00,
A=1
1
al g- (x)
Σ [E)K
(x)]
00;
K
a 0 (æ)]
JEL
P(yxlæ))
00,
A=
In the third equality we used the fact that objectives and share the same minimizer, which prove in Appendix C.7 and Proposition
The gradient vector (x), + c) can therefore be written as
1 + = Joloz. (x)] diag( P(y x)
18

---

Published as a conference paper at ICLR 2026
Finally, are able to plug in the expression for (2)) x) + into the definition of the gradient noise.
j inf Earg min fr. ((e. PerP. [IV (x), (x), + e)
P. o0 (z)] diag( P(y) cc|
1
E(ay)) t diag( p(y)
1 -00- (+)] Ja 10 (a:) diag(
E]
P(y)ar) ae)
1
B(a.y) [Trl E diag(
1/ (m)) 10 (a:) diagg(
Plylac)
ply x
Tr(Bepp, le e71 ppdiag
1/ 16 (z) Jzl0g. (x)]
diag(
P(y|=)
P(y
1
Trip Ix (m |dagg(
1/ 0g. (x) J 10 (z)) diag(
)]]
P P a
P P(y
= V. Tr (r diag
Jd
0 16 (a)) Jol l0g (x:) diag(
( mz)
P(y az)
1 P(y]z)
w Tr(diag(
10 (m)) Jol 10gg (a:) diag(
P(y]a))
K
V. Egp
J0.6 px
J
(z))1
k=1
K
= Eg-p
| 16g. (x)]
P Ply 2²)²
where Jg.u ldg (x)) is the kun column of the Jacobian of the student model.
We provided expressions for both the gradient noise (Definition 2) of the one-hot risk I oj. and the noisy BCP risk (10.. o7 which concludes the proof.
C.4 PROOF OF LEMMA
Proof. In order show that the gradient of each per-sample loss vanishes for the minimizer Or of the stochastic optimization task (6), we make use of Fermat's 's Theorem (proposition 8.9 Garrigos & Gower )).3)). Fermat's ' Theorem states that for f Rd - R U {+oo} and 0- E RR then 0* is minimizer of if and only 0 @f(²)). From the definition of interpolation (Definition I and from Fermat's theorem, it is implied that since interpolation holds for 0* (Proposition 2 then
V VoL(Øg (z), ()) = 0 for all supp(P), which concludes the proof.
C.5 PROOF OF THEEREM
Proof. In order to bound the error on the weights, we start by conditioning the expected error on the weights the last iteration.
We denote by F.I-] := TE[ the conditional expectation w.r.t 0', We
now have that
Fy [||0
11
E
F.,
[1|0]
a (0') 11
110-
+ 0'1 (0' 0. ( +
110 - 0-1 1|2 00' 0-, ,'V - v
A3A
aaa 011 20 20(
(i((0))
S (1 a4t)l 0-1 - 2a(f +20)) f(0)))
(1 ap)) 00 1 2a -)(((0))-(00))
< (1 ap) 0' 0-
19

---

Published as a conference paper at ICLR 2026
In the last inequality used the bound on the step-size Z. Next, apply expectation and unroll the recursion to get
E E||0
0-1131 - (1 - aa)) aa) 0-1 ||
S (1 ah) 0100 0 00
which concludes the proof.
C.6 PROOF OF THEOREM
Proof. In a similar fashion to the previous proof, we start by conditioning the expected error on the loss in the last iteration. We denote by [. == El- [ the conditional expectation w.rt f We note that
La Ty V V
E. 1 - / (f(0)) /*) (.
(D (f(0)) 1*) - al (i(0')) + La
Fy, (V(((() - -vfe(0))11)
2
(f(0") /r) 2apl /*) + LLO'( - -
= (1 - (((0)) fr) alu (f)) (r)
$ (1 (1-a f(),,
where in the last inequality used the bound on the step-size S Hhe Next, we apply expectation and unroll the recursion to get
E[f(0') - fr - (1 - (1) - f-1
IV
an) (f(00) f*),
Which concludes the proof.
C.7 PROOF OF THEOREM
Proof. Before diving into the learning dynamics of the student in the noisy case, we explore the connection between the stochastic optimization problem of the noiseless case (6) and the noisy case (O).. We now note that
((0) (( [((og(a)) + c)]
K Σ ((ynlz))0g) log (0)(1)))
K Σ 1=1
= E2
+ E
E log [dg
=((0) + E( [BENP. [-e log ( = =(0),
where in the last equality we used the fact that is independent of æ, y and is zero-mean, Note that for the current proof we change between O+ and 0*, which allowed since assumption ASZ ASS holds, which implies uniqueness of the minimizer.
Even though we proved that the generalization error in the case of perfect probabilities and noisy probabilities is equal, the learning dynamics in the two cases are different. This is because we focus on SGD-based optimization, and not full gradient-descent. Nevertheless, importantly, this implies that ff and f share the same minimum and minimizer, which means that in our proofs we can switch between 0" and 0r (assuming the minimizer is unique), as well as between f and f* interchangeably.
Again, in order to bound the error on the weights, start by conditioning the expected error on the weights in the last iteration. We denote by E. [+) TE (*) the conditional expectation w.r.t Ø'. and
20

---

Published as a conference paper at ICLR 2026
obtain
E2 [| 0 11 E, 110 (V - 0-|101
||0| 01 20 0-, vi((')) + + -'EE)IVV(
||0r 1'12 (0' 0', vi(0')) 20 - 20 [
- (1 a1)1100 0-1 2α (f(')) 20")) V 2a²03
g (1 a 20 2a( f(O')) ((0)))+42(( j(0)) 20²03
(1 (nn1 | 1 20 -)0))))((())+++)))))))))))))))))
< (1 ay)) 0 0- 12 22
Where in the last inequality we used the bound the step-size and unroll the recursion to get
1. Next, we apply expectation
Ell
0-1 S (1 -1)==111 0'
02 - 20'oj
2a (1 - 10 1²
μ o.
which concludes the proof.
C.8 PROOF OF THEOREM
Proof. Again, in order to bound the error on the loss, we start by conditioning the expected error on
the loss in the last iteration. We denote by [-] == [F[-I["]] the conditional expectation w.r.t f
Note that for the current proof we change between f and j', which is allowed even though the minimizer might be unique, since this is the minimum value and not the minimizer.
La²
E (f(++) f" I (f(0') - - -V(()
+ 2
E. E 1
(f(Ø') - (*) - az v 12 La L - vie(0") + La La
(j(0)) jr) 20p - jn) + Lo²aj
(1- (1) fr) aap aap- (() ft) La²or
S - ap) - -}) LaZoj.
where in the last inequality we used the bound on the step-size a S ttx. Next, we apply expectation and unroll the recursion get
[l((O') jo < (1 (1- 8r1 + La"oz
La
(1 (1-a))((((0))
f*) μ oj.
which concludes the proof.
D DIRICHETT-PERURRER BCPs
Our analysis of SGD supervised by noisy BCPs provided in Section used an additive noise model. In this appendix we show that the insights obtained there also hold for alternative models for noisy BCPs, using the Dirichlet distribution over probability mass functions. In this case, instead of using additive perturbations e, we perturb each true BCP Ply)a) (z) using Dirichlet distribution, which guarantees that the resulting target remains a valid probability distribution and is unbiased in expectation. Specifically, for some ε > 0, the perturbation is formulated as follows:
P(ylæ:)
Dir(e
(25)
21

---

Published as a conference paper at ICLR 2026
Similarly to the definitions of the BCP-based risk the noisy BCP-based risk 10 the resulting noisy Dirichlet -CP-based risk is
(0) == (. (z), P(y/2a))
(26)
Dirichlet characteristics
For P(yla)) Dir(=P(z) we have Minka (2000)
E P(yk P P(y (x),
(27)
Var(P(yn/<>)aa>> (1) P(ux(a))
(28)
E E+1
and
P(ys)))
Cov(P(ys(z)) P(y,|x>)a>]
(29)
E+1
Proof of Cand Dirichlet Noisy BCPs
Proof. We explore the connection between the stochastic optimization problem of the noiseless case and the noisy Dirichlet case 26
K
(0) (a - P.
loglob
P(ynlz) 10200((())))
= f0)
(30)
In the third equality in (30). we utilized the expectation characteristic of the Dirichlet distribution 27 Similarly to the proof Theorem even though we proved that the generalization error in the case of perfect probabilities and noisy Dinchlet probabilities is equal, the learning dynamics in the two cases are different. This is because we focus on SGD-based learning. Nevertheless, importantly this implies that ff and f share the same minimum and minimizer, which means that in our proofs we can switch between 0 and 0 (assuming the minimizer is unique), as well as between f and f* interchangeably.
The remainder of the proofs remain exactly the same in Appendix C.7 and C.8 but with switching between f and f, and between 0 and 0. The difference lies in the resulting expression for the gradient noise, which depends on how the noisy BCPs are modeled, as stated in the adaptation Proposition detailed below.
Adaptation of Proposition Dirichlet Noisy BCPs
Proof. In order to express the gradient noise of the noisy Dirichlet BCP risk (26). we recall the resulting expressions for the gradient noise based on Definition i.e.
inf @-Earg min Fr.#
((Ea))) P(yla))
[[[V (eVellog-ao (2r), (x)) P(y)a)))
We first give expression for Vol(dg. (m), P(yl=)). In this case, P(y (z)) is the true BCP which is a K -sieed vector, and Ply) x)) is its Dirtublee version. Since the CE loss is linear in its second argument, have
V Vol(pn- P(ylz)) = Val(án- (z), P(yla) + Vol(do- P(yla) - P(yj Val(ág- (m)) P a:) + Vol(do- P(yla) P(y (()))))
Vol(op. 6(x)),
22

---

Published as a conference paper at ICLR 2026
where we defined (()) := P p)y)a)). The second equality uses the fact that objectives 6 and (26) share the same minimizer, which we just proved above.
This means that we have,
Val(og-(z) (z),, P(yl==)) = Val(Øp- (2),, 8(x))
Σ (x: Vo loglog- (z)]
1
Jeløg- (x)) diag( a(yla)
6(x),
where Joløg- (a)] is the Jacobian of the student outputs with respect to 0.
can now plug the expression for Valldg- (z), P(y)=) into the definition the gradient noise.
inf dnEarg min fr
E (a.v))- P(y)ar)
[| @f( (x:), P (1)
1
((..)))>..8 ((x))
1ol00 (x)] diag(
8(x))1
P(y x))
1
1
((y.)) 6(w)
T
J 100
Lda (x)) (a)) P(y)(z) )6(z))
P(ylæ)
B(m.y))P,6(0m
Tr(6(=)6(x)) diag( P(yla) Jo ldg (z)] ldg- (x) diag P x))
(( ( [x]] diag(
J/ ( (a)) Já ldhg- (a:) diag(
)]}-
P(y|æ:)
P az)
We now utilize the covariance Shaaracteristcc29900 the Dirichlet distribution, which means that
se (()))((x))
[=]]
(diag(P(y)=z) P(yl)))
Therefore,
E (m (I(((_)a((((((())))a) P(yl=))((\zz) diag( P(y)æ)
E+1
5 (00- (=)) Jal (a:) diag(
P(yla:) )]|-
(31)
Finally, using the simplex identity Joldb- (a=) 0, the second term with P(y))))P(llax) T vanishes. We thus obtain
w) (Trdaaa(((PaaaaPlllaaaa diag(
P(ylx) o 1oo (x) diag( P )]]
++1
8
E ( Tr[J (6p- (x)) Ja Φ0 (x)) diag( P(y) x)) |
E
K
Eaxp
Jo (x)]]]
E +
Ply
(r)
where Ja. is the k-th column of the Jacobian. This shows that, compared to the additive uncorrelated noise model, the gradient-noise expression has: (i) scale factor (e + 1)-1 instead of v; and (is) weighting by 1/P(g) (x) instead of 1/P(yn (z=) due to the Dirichlet covariance structure.
23

---

Published as a conference paper at ICLR 2026
E SYNTHETIC EXPERIMENTS (SECTION 3
In this section, we empirically demonstrate the effects of different supervisory signals on the learning dynamics of the student SGD-based learning. Specifically, we consider the cases of supervision by one-hot labels (standard learning), perfect BCPs, BCPs corrupted by different noise levels, and supervision by combinations of one-hot labels and noisy BCPs adjusted based on several À values. Since the true BCPs are unknown for popular datasets such as MNIST or CIFAR, we generate a synthetic dataset for which we are able calculate the true BCPs.
We generate a dataset of N = x 10 samples. Each input associated with label y/. that takes a value from K potential classes. To get sample, we first select the label yr, k from a uniform distribution over the K classes. Each class is centered around M4. 30 x 1 vector where each entry is randomly selected from {-1, 0, 1]. Once the label is selected, the input Tn is drawn from Gaussian distribution centered at the selected class mean øly= ~ (u o² I). We use x² = 2.5, which is the noise level for all samples. For this dataset, we are able to calculate the true BCP p(yn s..) for each pair (I yn). The true BCP is calculated as follows.
En, n,, which a x 1 vector, i.e., = R-00, is
In order to calculate p(y a)) we use the Bayes' rule
p( (olly) p(y)
p(y/<) =
Σ plally = j) plu j)
Since the label is sampled uniformly, p(y) 1/4 for all classes. We know that plally = k) ~ W14x,0211)
so we have
|| 1x
p( = k) =
(2TO²) 30 exp
20
If we plug the above expression into Bayes' rule, the expression simplifies into
exp
| 1mm-1413 20
ply k|ar)
K
||00 14,113
exp
2n
This allows us compute the true BCP for each input.
We split our data into 50% for the train set and 50% for the test set. A total of 22 student models super- vised by different signals were trained. All models were trained in the same manner for 45,000 iterations via SGD, with a learning rate of 5 x 10 For the model architecture, we used a standard MLP with 2 hidden layers, each with 128 hidden units and ReLU activation functions. The first student model, re- ferred to as "One-hot labels" was trained with the standard labels yn- The second student model, referred as "True Bayes probabilities" was supervised with the true BCPs p (x,.). This student corresponds to subsection 3.2 in which the dynamics of student trained with the true BCPs are analyzed. The two student models, referred as "Less noisy probabilities" and "More noisy probabilities" were supervised with the true BCPs corrupted with different noise levels. These students correspond to subsection 3.3 which the dynamics of student trained with noisy BCPs are analyzed. The noise was added by per- turbing each BCP using Dirichlet distribution, which guarantees that the resulting noisy BCP remains a valid probability distribution and unbiased in expectation. Specifically, for some & > 0, we sample
Dn Dir(e p..),
so that E(p..] Pr. controls how noisy the resulting BCPs are. For the "Less noisy probabilities" student we used æ = 5 and for the "More noisy probabilities" student we used = = 0.5. The rest of the student models were supervised by combinations of one-hot labels and noisy BCPs adjusted based on different values between and 1, as in the standard KD framework. The objective in this case is described in (3).
In the left plot of Figure I we plot the generalization error of the model that is calculated using Equation (1) with the finite test set. The constant line referred to the "Bayes classifier" is the
24

---

Published as a conference paper at ICLR 2026
generalization error calculated with a perfect model that produces the true BCPs. In the middle plot of Figure we plot the accuracy of the model on the test set in each iteration.
Finally, the right plot of Figure illlstratts how the generalization enror behaves during training for different values of X. Each point in the plot represents the average L1 distance between the generalization error given model and the minimal generalization error achieved by the optimal Bayes classifier. Specifically, for model trained with a specific X, noisy BCPs, and labels,, we compute
1o+T
oj
Σ
- (pat]
where gpert denotes the generalization error achieved by the optimal Bayes classifier, and e is the generalization error at iteration of the corresponding noisy model. We use to 20,000, which is after the initial phase of rapid convergence. This metric represents the average generalization error gap between given student model and the reference optimal loss throughout the training process, capturing both the overall error level and its variability.
Next, we quantitatively assess the effect of noise in the BCPs on the convergence behavior of student models. We introduce two complementary metrics, each applied to both accuracy and generalization error during the final convergence stage of training.
The first metric captures the average performance in the last N iterations:
T
Lavg
Σ Lt N++1
ACC
Σ
A
N
TT
where L2 and A denote the generalization error and accuracy at iteration f, respectively. This metric quantifies the achievable performance while suppressing fluctuations in the curves.
The second metric captures the stability of training by measuring the noise level in the accuracy and generalization error curves. To smooth out slow transitions, we first compute a moving average X, with window size for time series X, E (Tn, A.3. The noise is then calculated by:
Σ
(x,- -
σx
which reflects how stable the model's performance is.
We computed these four metrics for student models trained with different levels of noise corrupting the true BCPs. The results shown in Figure Each experiment was performed 15 times; standard deviations of the computed values are shown in the figure. The bottom plots show that as the noise level decreases, average accuracy improves, while its variability decreases, indicating improved student per- formance. The top plots show that for the generalization eтror metrics, lower noise leads to both reduced error and reduced variability. Appendix(D we proved that the gradient noise in the convergence analysis of Theorems 3 4 the adapted case for Proposition in the case of Dirichlet noise, is:
Σ P(yk x))
|J (=)]1l
As seen in Figure 3 for all four accuracy and generalization error metrics, the performance aligns 1 with the fitted curves proportional I+e- These results provide empirical validation for Proposition 3 demonstrating that decreasing the noise added to the true BCPs reduces variance and improves convergence behavior.
Next, we examine the relationship between average performance and stability, and between accuracy and generalization error. Flgute4plots the four metrics against each other for different noise levels. Each experiment was performed 15 times; standard deviations of the computed values are shown in the figure. For comparison, we also include the student trained with the true BCPs and the one trained from one-hot labels. The plots show that higher average accuracy consistently coincides with lower variability, and the same pattern can be seen for the generalization error. Moreover, accuracy behaves similarly to the generalization error, both in terms of average values and noise. These findings empirically validate two key points: (1) the variance reduction from using less noisy BCPs (Theorem 4). translates directly into improved average performance and stability; and (2) improvements in generalization
reflected as similar improvements in accuracy.
25

---

Published as a conference paper at ICLR 2026
Average Generalize alization arror
Genterllization etror nojise (STD
0.200
0.006
0.198
0.196
1/1 5)
0.004
1/11
One-bot
Oce-bot
0.194
True BCP
Trae BCP
0.192
0.002
0.190
0.001
0.188
0.000
10
10-1
10c
Accuracy nose (SID) (%)
0.20
Average Accuracy
93.4
93.3
One hot
One-hot
True
True BCP
93.2
0.10
93.1
0.05
10
10"
10
10
10
10"
Figure 3: Effect of noise level e on student performance. Each plot shows results for students trained with noisy BCPs. Top: average generalization error (left) and variability in generalization error (right), Bottom: average test accuracy (left) and variability accuracy (right). Each plot includes
fit proportional to E and student trained with the true BCPs.
alongside the result achieved by student trained from One-hot labels,
10a
0.25
10a
93.5
Noisy BCPs (e-colored)
Accuracy (STD) (%6
Average Accuracy
True BCP
0.20
0.15
0.10
100
100
0.05
93.1
0.190
0.192
0.194 0.196 0.198 0.200
0.000 0.001 0.002 0.003 0.004 0.005 0.006
Average Generallzation error
Goneralization error noise (STD)
0.25
104
Generalization (STD)
102
Acouracy nolse (STD) (%6
0.006
0.005
101
0.003
0.10
0.002
0.05
0.001
0.000
93.1
93.2
93.3
4
93.5
0.190 0.192 0.194 0.196 0.198 0.200 Average Generalization error
Average Accuracy (%)
Figure 4: Correlations between performance and stability metrics across different noise levels e. Each point corresponds to student trained with noisy BCPs, with baselines for one-hot labels and true BCPs also included. Top left: higher accuracy is strongly associated with lower generalization error. Top right: models with noisier generalization error curves also display noisier accuracy curves. Bottom left: higher accuracy coincides with reduced variability in accuracy. Bottom right: lower generalization enoor coincides with reduced generalization error variability.
26

---

Published as a conference paper at ICLR 2026
F IMPLEMENTATION DETAILS OF THE EXPERIMENTS IN SECTION 4
We elaborate on the implementation and training procedures of all teachers and students used in the experiments reported in Subsection 4.2 Importantly, all students were trained with an identical procedure, independent of the type of teacher (deterministic, Bayesian, Laplace, MCMI, TTDA, or MSE). The base training setup for students the same that of the deterministic teacher. Modifications were performed only where required by the specific teacher type:
1. Deterministic teacher: trained identically to the student.
2. Bayesian teacher: trained identically to the student in terms of hyperparameters, but with an additional NLL loss term for variational inference.
3. Laplace teacher: obtained by applying the Laplace approximation post-hoc to a pretrained deterministic teacher.
4. MCMI teacher: obtained by fine-tuning pretrained deterministic teacher with additional MCMI loss term.
5. MSE teacher: trained identically to the student but with the MSE loss replacing the standard
CE loss.
6. TTDA teacher: predictions are obtained by applying test-time data augmentation as a drop-in method, where the pretrained deterministic teacher is evaluated multiple augmented versions of each input and the outputs are averaged.
For all teacher and student training procedures, train for 200 epochs, with ADAM the optimizer. The initial learning rate is set to 0.001 by default, which is decayed by a factor of 0. at epoch 100. The momentum was set to 0.9, alongside batch size of 100. Deterministic teachers were trained with the standard CE loss, and all students were trained with the distillation loss 3 with weighting parameter A. Additionally, the teacher and student probabilities used in were smoothed using T, and T., respectively.
For each teacher-student pair, trained five independent trials under six different (A, Tú, T.) configurations: {1. 1}. 2, (1,4, 4}, {1, 1}. {1, 1), (0.474, 4, 4}. The first combination corresponds full distillation without temperature scaling, and the last combination corresponds to the original implementation in Hinton et al. (2015) with a = 0.9 and T = 4. For each pair, reported values correspond the (X, Tú, T.)) combination in which the best average student accuracy over the five trials was achieved. The reported noise values in Figure 2 were first computed per trial and then averaged across the five runs of the best configuration.
Bayesian teacher To train Bayesian teachers with variational inference, utilized the implemen- tation provided in/pitts:::tt... /ttrttss://itutute /gtthtttt....g. com/ microsoft mnicrosoft/bayesinne turns deterministic NN into a Bayesian NN with configuration file that includes priors the weights and related settings. We used the default configuration offered in their repository. Their training procedure includes an additional NLL loss term, in which we used their default schedule of 100 epochs with the NLL term and 50 epochs of gradual annealing. For distillation, teacher logits were obtained via Monte Carlo prediction averaging with 10 stochastic forward passes.
This implementation
Laplace teacher As described earlier, Laplace teachers are obtained by applying the Laplace approx- imation post-hoc to pretrained deterministic teachers. For this purpose, we used the implementation provided in https:////// / // model parameters and applying the approximation directly. In our experiments, we approximate the posterior distribution only over the weights of the final layer, using Kronecker-factoree Hessian to capture curvature information. The prior precision is optimized via marginal likelihood maximization on the training data, without requiring validation split. For predictions, we employ the generalized linear model formulation with a closed-form probit approximation, which incorporates uncertainty from the Laplace posterior into the predictive probabilities. As a result, instead of point estimates from a deterministic teacher, the Laplace teacher produces predictive distributions that reflect uncertainty in the last-layer weights.
com www.aa/ww.neer/a/ooo.ooaaaaa.aea / which allows loading pretrained
MCMI teacher implementation from https:://gitiet thubb com/icila... r202 4mami /CCLMMMT The approach
To realize the MCMI teacher proposed by Ye et al. (2024), we used the official
27

---

Published as a conference paper at ICLR 2026
fine-tunes a pretrained deterministic teacher by adding MCMI loss term. We used the hyperpa- rameters which the authors recommend in their paper for fine-tuning. Specifically, we used a cosine annealing learning-raie rate schedule with initial value of 2 x 10 fine-tuned for 20 epochs, and set the MCMI weighting parameter to 0.2 for ResNet-1 18, ResNet-50, and VGG-13 13 teachers, and 0.15 for WRN-40-2. The MCMI weighting values correspond to the best settings reported in the original paper, which we adopted without hyperparameter searching, as they did. For the ResNet-18 teacher, no best value was reported, and we used 0.2 by default. It is worth noting that training hyperparameters (epochs, batch size, etc.) used in their training setup differ from those employed in this work. Additionally, in their work, they reported results for other values of T,, T,) and did not run several combinations while reporting the best one. This may partly explain performance differences between our results and those reported in their paper.
MSE teacher To realize the Mean Squared Error (MSE) we used the official implementation htomhttss://// g/tphbggihtbb com WWCEECCC222222 4MSE, In this setting, teacher models trained with the MSE loss instead of the standard CE loss, as suggested by the authors. It is important to note, however, that the results reported in their paper were obtained under different ex- perimental settings and hyperparameters than those used in this work. Additionally, they reported results for other values of (A, Te, T,) and did not run several combinations while reporting the best one. These differences may explain why the MSE teacher performs poorly in our experiments compared to other teacher types. In Appendix K it is shown that for the case of standard distillation setting (temperatures and are equal to I), the MSE teacher performs well and often outperforms the deterministic teacher.
teacher proposed by Hamidi et al. (2024),
TTDA teacher As described earlier, predictions obtained from the TTDA teachers obtained by applying the test-time data augmentation Garthoowkk et al. (2023) to the outputs of the pre-trained deterministic teacher. Specifically, the teacher predictions
obtained by averaging the output probabilities over N = 10 stochastic forward passes, each computed on a differently augmented version of the same input. For each pass, standard test-time data augmentation pipeline is applied, consisting of random cropping and random rotation. The resulting probability vectors are averaged to form the final teacher prediction used in the distillation process.
28

---

Published as a conference paper at ICLR 2026
G FEW-SHOT CLASSIFICATION
Few-shot learning refers to training model when only limited number of labeled data is available for each class Chen et al. (2019). In the context of KD, the few-shot setting adapted by utilizing a teacher that was trained on the full dataset, while the student is trained with limited data, specifically a B-proportion of the samples from every class Ye et al. (202)). Few-shot experiments allow to examine how well the student can benefit from distillation when the data available for its training is limited. Additionally, highlights the benefits distillation and ability to improve generalization in such scenarios.
We evaluate the effectiveness of Bayesian teachers in several few-shot settings on the CIFAR- 100 dataset. We consider VGG-13 the teacher architecture, and VGG-8, ResNet-18, WRN-40-2, and WRN-40-1 as student architectures. Students are trained either from Bayesian teachers or from deterministic teachers, and we compare their performance. For each stuudet-ttudett pair, employ the (T,.T. >) configuration that yielded the best performance in Section 4.2 Both teacher and student training follow the same implementation details and hyperparameters described in Appendix F Experiments are conducted for different values of / E {5, 10, 15, 25, 35, 50), and each setting is repeated five times. For fair comparison, methods in each few-shot level. The results are reported in Table
the same partition of the training across all
Table 2: Test accuracy (%) of students trained from both Bayesian teachers and deterministic teachers, under few-shot setting, averaged over 5 runs. Results are displayed for several teacher-studers pairs with both matching and different architectures. The subscript denotes improvement in the student trained from the Bayesian teacher relative to the corresponding student trained from the deterministic teacher.
Teacher + Student
5
10
15
25
35
50
Teacher kind
Deter.
Bayesian
Deter. Bayesian
Deter.
Bayesian
Deter.
Bayesian
Deter. Bayesian
Deter.
Bayesian
VGG13 > ResNet18
35.79
41.11
50.97
55.57
58.94
63.43
65.75
68.83
69.33
71.87
72.27
74.04
45:32
+4.11
+4.49
43.08
+2:54
-1.77
VGG13 VGGS
41.57
52.45
54,23
63.07
61.28
67.97
66.65
71.44
69.50
73.42
72.38
75.33
100.8
18.34
1669
14.9
13.02
-29
VGG13 - WRN-4002
30.80
40,96
44.10
54.17
52.04
61.44
59.76
67.33
64.45
70,20
68.08
72.53
+7.57
VGGI3 WRN-40011
24.53
32.88
35.49
45.08
43.21
52.76
50.40
59.19
55.67
63.45
60.34
66.39
+5.35
+8.99
+9.45
+8.79
+7.76
-605
As seen, students trained with Bayesian teachers consistently perform better across the board, compared students trained with deterministic teachers. Specifically, 33 decreases the improvements more substantial, with improvements of over 10% for the case of 8 = 5. These results show that Bayesian teachers are very useful in KD few-shot settings, consistently offering improved performance compared to deterministic teachers.
H TEMPERATURE SCALING
Temperature scaling was first introduced in KD Hinton et (201)) as way to soften the teacher's output probabilities. The temperature is introduced inside the "softmax" operation applied to the output of the network:
(0) exp(2(0) (T)
Σ, exp((()))T))
where = are the logits, ((*) is the softened probability for class i, and is the temperature. Choosing T = corresponds to standard softmax without temperature scaling. Increasing T produces smoother distribution, which has been shown to act as a regularizer and improve the transfer of dark knowledge. Hinton et al. (2015) applied the same temperature to the student's logits during training, but this not necessary. In fact, a recent work by Zheng & Yang (2024) studied temperature scaling, and shows that dropping temperature scaling the student side causes the student generalize better. From a calibration perspective, temperature scaling also serves as a simple yet effective method to reduce overconfidence and align predicted probabilities with the true BCPs, thereby improving the
29

---

Published as a conference paper at ICLR 2026
78
76
74
27
Student Aec. (90
75
73
72
Student Acc.
76
74
Studont Acp.
71
75
73
70
69
74
Buvesian teacher Determinicts teacher
72
Bayesian teacher
Determinitiic teacher
68
Determiniets twacher
Buvesian teacher
73
67
Teacher Temperature
Teacher Temperaturo
Toacher Temperature
Resnet50 Resnet18, T3
Resnet50 WRN 40-2, T,=1
WRN WR--40022 WRN 16-2,
78
76
74
27
72
76
Ruudent Nec.
Studont Acp.
71
Student
75
70
69
Dayestan teacher Deterministic Leacher
Bayestan teacher Determinisiic teucher
68
Deterministi Leacher
Payessin teacher
73
67
2
3
Teacher Temperature
Teacher Temperature
Teacher Temperature
Resnet50 Resnet18, =,=2
Resnet50 WRN--0-2, =.=2
WRN-40-2 WRN-166222
78
76
74
Student (96)
77
Student Apc. (%0
75
72
76
74
Student Acc.
75
70
69
74
Inyesaa teacher Deterministic teacher
Bayestan teacher
Deternenisti: teacher
68
Deterministk toacher
Boyrsstin teacher
73
67
8
3
Teacher Temperature
Teacher Temperature
Teacher Temperature
Resnet50 Resnet18, =
Resnet50 WRN 40-2,
WRN WRN-40-2 2 WRN 16-2,
Figure 5: The accuracy achieved by students (averaged over runs) trained from both deterministic and Bayesian teachers in KD with different teacher and student temperatures applied. The teacher-student pairs displayed are ResNet-50 ResNet-18 (left), ResNet-50-wWRNY (middle), and
(right).
quality of supervision for distillation Kim al. (202)). Here, investigate the effects of temperature scaling in KD settings with both Bayesian (VT) and deterministic teachers, and compare them.
We consider three teacher- seacher-student pairs. For each pair, the student is trained with teacher temperatures
ranging from T, 1 to T, = 9, with three different student temperatures T, E {1, 2, 4). Each experiment is repeated five times, and the average accuracies across and student trainings follow the implementation details and hyperparameters described in Appendix and results are displayed Figure
reported. All teacher
It can be seen that students trained with Bayesian teachers consistently achieve higher accuracy across all temperature combinations and architectures compared to students trained with deterministic teachers. In particular, under standard distillation without temperature scaling (T; = 1, T, 1). the gains are substantial: student accuracy improves by up 3.18% SoosNett50--RNN--00 though the Bayesian teacher itself is only 0.26% more accurate than the deterministic teacher.
even
Additionally, the effect of temperature scaling is consistently smaller for students trained from Bayesian teachers compared to those trained from deterministic teachers. For example, in the ResNet- 50 + ResNet- 18 case, the gap between the highest and lowest student accuracy across all temperature combinations is 3.32% when using deterministic teachers. On the other hand, for students distilled from Bayesian teachers this gap is only 1.07%, about three times smaller. This shows that Bayesian teachers produce probability estimates that are inherently better calibrated than those of deterministic teachers, making them more suitable for KD. Additionally, shows that Bayesian teachers are less sensitive to hyperparameters and are easier to tune, further strengthening them effective teachers in KD..
30

---

Published as a conference paper at ICLR 2026
1 BAYESIAN TEACHER PARAMETERS
BNNs trained with variational inference represent model parameters not as fixed values but as probability distributions Blundell et al.) (015): Jospin et (2022). This allows the model to capture uncertainty by considering different possible configurations of the weights, rather than a single point estimate Kabir al. (2018). As result, running the same input through the model may yield different predicted probabilities each time. To obtain reliable predictions, one typically performs Monte Carlo sampling at inference time, averaging outputs from several stochastic forward passes Gawlikowski et al. (2023). Increasing the number of samples generally improves the stability and accuracy of the predictive distribution of the teacher model.
We next investigate how the number of Monte Carlo samples used to compute the teacher's probability estimates for distillation affects the performance of the student model. To that aim, we trained two teacher-student pairs with varying numbers of samples used for Monte Carlo prediction averaging. For each setting, we report both the average accuracy of the teacher model and the average accuracy of the corresponding student model. Each experiment across runs are reported. For reference, we also include the performance of the deterministic teacher and its student. Distillation in this experiment = (full distillation). In addition, to assess the effect of the number of Monte Carlo samples used on gradient noise, we report the average noise in the accuracy curves, in similar fashion to the one conducted in Subsection a.2 and shown in Figure 2 Both teacher and student training follow the same implementation details and hyperparameters described in Appendix |P
repeated five times, and the average values
carried out without temperature scaling and with
The results are displayed in Figure and Figure Teacher performance is in line with prior work on BNNs trained with VI; accuracy improves substantially as the number of Monte Carlo samples increases Shen et al. (2024). We observe gains of up to 6% when moving from 1 to 12 samples, with most of the improvement being in the low-sample regime. Interestingly, the corresponding student accuracy acts differently. While adding more samples does not harm the student, its effect is modest. For VGG-13 + VGG-8, the student improves by only about 0.4% when increasing from to samples, after which accuracy saturates, compared to the teacher's 's much larger improvement of about 5.5%. For VGG-13 - WRN-40-1, 1, student accuracy roughly remains unchanged regardless of the number of samples used. This is explained by the fact that although single realization is used, the fact that it is randomized anew on each sample leads to improved calibration when used for supervision in KD as it is averaged in multiple epochs of SGD. Specifically, the student is trained using the BNN teacher outputs thouughout multiple epochs. This means that although training in each individual iteration performed with a noisy BNN teacher sample, over multiple iterations training is performed with the expected value of the BNN teacher sample, effectively capturing the Monte Carlo sample average. The noise this case refers to the noise introduced by the distributions on the weights of the Bayesian model and not the noise in the probability estimates.
VGG 13 VGG-B 8
VGG-13 13 WRN-40 URN-40-1
0.20
0.20
Accuracy nose nose (std) [%)
Accuracy noss (a04) 136)
0.16
0.16
0.14
0.14
0.12
0.10
0.10
Deterministic Student Bayestan Student
Deterministic Student Bayestan Student
10
11 12
3
10
11 12
Monte Carlo Samples
Monte Carlo Samples
Figure 7: The average noise in the accuracy curves present during training for students trained from Bayesian teachers using varying number of Monte Carlo samples to compute teacher predictions in the distillation process (averaged over 5 runs). Corresponding average noise in the accuracy curves present during training of the students trained from deterministic teachers are reported for reference. The teacher- pairs displayed are VGG-13 - VGG-8 (left) and VGG-13 13G-133-0 -WRN--0-1 (right).
31

---

Published as a conference paper at ICLR 2026
VGG-13 3 - VGG-8
VGG-13 WRN-40-1
77
76
76
74
75
73
Accuracy (96
74
71
73
70
72
68
Deterministic Teacher
Deterministic Stadent
Determinstte Student
70
Bayesian Teacter
Bayesian Teacher
Bayesian Student
65
Beyesian Student
Monto Carlo Samples
Monte Carlo Samplos
Figure 6: The accuracy achieved by Bayesian teachers, and their corresponding students (averaged over runs). Teacher accuracies correspond to the average accuracy achieved while doing inference with different number of Monte Carlo samples, and student accuracies correspond to the accuracy achieved while using that number of Monte Carlo samples to compute teacher predictions for distillation. Corresponding deterministic teacher and student accuracies are reported for reference. The teacher-student pairs displayed are VGG-13 + VGG-8 (left) and VGG-13
(right).
Moreover, for all cases, students distilled from Bayesian teachers show better performance compared to students distilled from deterministic teachers, regardless of the number of samples used. This shows that Bayesian teachers are better suited for KD compared to deterministic teachers, even when only a single sample is used. Finally, the number of samples used seems to have no noticeable effect on the noise present in the accuracy curves. Nevertheless, students trained from Bayesian teachers exhibit 10-30% less noise in the accuracy curves compared to students trained from deterministic teachers, regardless of the number of samples used.
J ADDITIONAL DISTILLATION METHODS & TINY IMAGENET
In this section, present additional experimental results obtained using several response-base based distillation methods, as well as experiments conducted on the Tiny ImageNet dataset. In this paper, we develop theoretical understanding showing that teachers that better approximate the true BCPs yield improved student models. Building on this insight, we propose leveraging BNN teachers for standard response-based KD Hinton et al. (2015) enhance student performance. While our method focuses on improving teacher predictions, many works instead improve response-based KD by modifying the distillation procedure itself. Typically, these methods manipulate teacher outputs, student outputs, or the distillation loss.
Here, we demonstrate that combining different response-basee based distillers with Bayesian NN teachers trained via VI leads to consistently better students compared to using deterministic teachers. The experiments are conducted on the CIFAR-100 dataset, and we follow the hyperparameter settings recommended in the original papers.
Huang et (202)) propose DIST, which includes a correlation-basee loss to capture the intrinsic inter-class relations from the teacher, and extend the relational match to the intra-class level. Going by their notations, we use 9 = 2 and as suggested by them. For a, the parameter balaneing between one-hot training and KD, use both 0 and 1 and report the student achieving the highest accuracy between the options.
Zhao et al. 2022) propose Decoupled KD (DKD), which decomposes the KD loss into target-class and non-target- class components to separately preserve cample-difficull information and amplify the core benefits of response- bassdd KD. Going by their notations, = for WRN and VGG teachers. For the parameter balancing between one-hot training and KD, we use both and 1 and report the student achieving the highest accuracy between the options.
1, 3 8 for Resnet teachers, and
32

---

Published as a conference paper at ICLR 2026
Zheng& & Yang (2024) propose weighted transformed teacher matching (WTTM), which augments
transformed teacher matching with a sample-adaptive weighting scheme to better align the student with the teacher's 's power- transformed distribution. Going by their notations,
use ry = 0.3, 3 = 1.5 for Resnet teachers, 7 = 3 = 2.25 for VGG teachers, 7 = 0.1, 8 = 3 for the WRN-40-2 teacher with the WRN-40-1 student, and y 0.1, 8 = 4 for the WRN-40-2 teacher with the WRN-16-2 student. We use I for the parameter balancing between one-hot training and KD.
The deterministic teacher, Bayesian VI teacher, and student training procedures match those used in the main experiments [ and are detailed in Appendix F Additionally, for each teacher-student pair we train students under multiple teacher temperatures (T, E (1,2,4]) For each pair, reported values correspond to the teacher temperature in which the best average student accuracy over five trials was achieved. The results, provided in Table 3 show that using BNN teachers consistently improves student performance across all three methods compared to using deterministic NN teachers. These findings demonstrate that the advantages of BNN teachers in KD extend beyond the standard distillation loss and generalize to additional response-based distillation methods.
Table 3: Test accuracy (%) of student networks on the CIFAR- 100 dataset, averaged over 5 runs. Both the teachers and the students were trained with varying random seeds between runs. Results are reported for teacher-student pairs with both matching and different architectures. The subscript in the Bayesian columns denotes changes relative to the corresponding deterministic distillation method. Deter. stands for Deterministic. Bayesian VI and Deterministic teacher accuracies are reported in Table
Teachers and students with matching architectures
Teacher
ResNet-18 ResNet-18 73.23
RcaNei--0 ResNet-18
ResNet-50 ResNet-34 73.61
WRN-90-2 WRN-16-2 67.70
WRN-4002 WRN-10-1 65.03
VGG-13 VGG-S 73.52
Student
Student acc. (no KD)
Teacher type
Deter:
Bayesiun
Deter.
Buyesian
Deter.
Bayesiun
Deter.
Buyesian
Deter.
Bayesiun
Deter:
Buyesāan
KD HDitioo
75.92
76.92
77.27
76.82
77.63
70.80
72.94
68.92
76.08
+1.0
+1.01
+081
-2.1
+1.88
-1.13
DIST Huang
2022
75,01
76.93
75.39
77.19
74.52
75.23
70.84
72.53
68.92
70.69
75.43
77.10
+1.02
41.30
+1.77
+1.57
DKD Zhoo al.
2022
74.86
77.14
75.75
77.21
76.10
77,29
71.04
73.32
69.77
71.68
75.41
77.85
WTTM Zheng & Yang
2024
76.05
76,89
77.11
77.28
76.54
77,23
72.16
73.54
70.78
70.82
76.81
77.64
+0.6
+0.00
Teachers and students with different architectures
Teacher
ResNet-50 WRN-10-2
ReaNoi-50 VGG-8 73.52
VGG--3 ResNet-18 73,23
VCKG-1 WRN-10-1 65.03
ResNer--50 WRN-16-2 67.70
VGG-13
Student
WRN-10-2
Student ucc. (no KD)
70.76
70.76
Teacher type
Deter:
Bayesian
Deter:
Bayesian
Deter:
Bayesiun
Deter.
Buyesian
Deter.
Bayesiun
Deter.
Buyesian
HDdintto al.
2015
73.79
75.82
75.66
77.27
76.36
77.41
67.90
71.37
69.36
73.63
73.71
75.45
+1.0
+1
+3-47
+1.27
-1.3
DIST Huang et al.
2022
73.40
76.04
75.42
76.88
75.32
77.34
68.61
71.81
70.45
73.03
73.41
76.10
12.66
+1.45
12.02
+5.29
+2.58
$2.99
DKD Zhao et al.
2022
74.11
75.90
77.79
75.68
77.61
71.86
70.82
74.02
+2.70
-241
+1.5
+2.05
43:25
+1.64
WTTM Zhong& Yang
2024
75.34
75.39
77.94
77.43
71.71
70.12
74.78
+0.0
+1.75
+0.04
Next, we present results evaluating our approach on the Tiny ImageNet dataset, which is smaller subset of the larger ImageNet dataset. Tiny ImageNet contains 1100000 images of size 64 x 64 across 200 classes, split into training and test sets with 10:1 ratio. Compared to CIFAR-I00 both datasets contain 500 examples per class, but Tiny ImageNet has double the number of classes, making the classification task more challenging. In this set of experiments, we compare student models distilled from either a deterministic NN teacher or a Bayesian NN teacher trained via VI.
The teacher and student training procedures follow those described in the main experiments (Table Ty and detailed in Appendix[H [F] with two modifications: training is performed for 120 epochs, and the learning rate is decayed by a factor of 1 at epoch 60. For each teacher-student pair, we conduct five independent trials under six different (A, T.) configurations, following the setup used in Appendix
F For each pair, reported values correspond to the (A, Tú, T.) combination in which the best average student accuracy over the five trials achieved. The results are provided in Table Consistent with the results on CIFAR-100, students distilled from Bayesian NN teachers achieve higher accuracy than those trained using deterministic NN teachers.
33

---

Published as a conference paper at ICLR 2026
Table 4: Top-1 I and Top-5 accuracy (%) on Tiny ImageNet for different teacher-student pairs, averaged over runs. Both the teachers and the students were trained with varying random seeds between runs.
Deterministic
Bayesian Teacher
Student (No KD)
Student from
Student from Bayesiun Teacher
Teacher-Student
Teacher
Deterministic Teacher
Top
Top
Top
Top
Top
Top
T
Resnet-3 34 Resnet-18 18
59.76
80.85
60.68
81.73
58.47
80,24
62.40
82.71
64.77
84.63
WRN.
60.68
81.73
80.46
K ACCURACY VARIANCES AND FULL RESULTS
Table 5: Mean and variance of the accuracies corresponding to the numerical study in Subsection 4.2 performed with distillation hyperparameters (A, Tú T.) = (1, [1,1,1] (averaged over runs).
Teacher Student
ResNet-18 ResNet-18
ResNet-50 ResNet-18
ResNet-50 ResNet-34
WRN-40-2 WRN-16-2
WRN-40-2 WRN-10-1
VGG-13 VGG-8
Deterministic
73.32 + 0.21
73.48 + 0.21
73,89 + 0.36
67.94 + 0.33
65.07
+ 0.60
73.48
0.27
Bayesian
76.28 + 0.26
76.20
+ 0.27
76.76 + 0.29
69.64 +: 0.16
67.55
+ 0.39
76.24 + 0.15
Laplace
74.88 + 0.40
75.80 + 0.20
75.56 + 0.38
70.45 +: 0.41
68.87
1: 0.42
75.79 + 0.31
MCMI
73.61 + 0.28
73.61
+ 0.27
73.85
+ 0.22
68.20
+: 0.18
65.27
0.39
73.36 = 0.41
MSE
74.84 + 0.45
74.84 + 0.28
73.41 +: 1.78
67.74
+: 0.31
54.30
+ 0.54
73:25 + 0.30
Teacher
ResNet-50 WRN-40-2
ResNet-50 VGG-8
VGG-13 ResNet-18
VGG-13 WRN-40-1
ResNet-50 WRN-16-2
VGG-13 WRN-40-2
Student
Deterministic
71.28 0.38
73:41
+ 0.30
73.44 + 0.21
65.32 + 0.55
6
71.12 0.37
Bayesian
74.47 0.17
75.56
0.100.1000 0.10
76,48
+ 0.22
68.37 + 0.40
10.84
74.26 0.17
Laplace
74.57 0.41
0.23 75.97
75.84
0
70,53.0.33
72.52
74.39 0.36
MCMI
71.17
0.22
73.28
6,233
0.29
0.17 70.64
71.14
MSE
70.71 0.24
0.32
0.11
53.56 13.6610.7
1422:00 0.18
Table 6: Mean and variance of the accuracies corresponding the numerical study in Subsection 4.2 performed with distillation hyperparameters (A, T, T.) = (0.474, 4, 4) (averaged over 5 runs)
Teacher Student
ResNet-18 R8sNet-18
ResNet-50 ResNet-18
ResNer-50 ResNet-34
WRN-40-2 WRN-16-2
WRN-10-2 WRN-10-1
VGG-13 VGG-8
Deterministic
75.92 + 0.24
76.05 + 0.40
76.76 + 0.25
70.38 + 0.27
68.06 + 0.65
75.62
0.15
Bayesian
76.84 0.17
77.27
+ 0.27
77.54
0.10
72.42
+ 0.47
70.14 +:144:00 0.27
77.51 = 0.19
Laplace
76.30
76.69
+ 0.34
76.23
0.12
71.66
+ 0.38
69.65 + 0.29
76.09 0.26
MCMI
75.83 0.35
76.35
+ 0.20
76.55
+ 0.40
70.40
+ 0.61
68.10
1:100:00 0.20
75.66 0.53
MSE
74.48 + 0.12
74.46 + 0.23
75.58 + 0.22
68.62 + 0.44
65.42 +: 0.39
73.86 0.34
Teacher
ResNet-50 WRN-10-2
Re VGG-8
VGG 13
VGG-13 WRN-40-1
ResNet-50 WRN-16-2
VGG-13
Student
ResNet-18
WRN-40-2
Deterministic
72.73 1: 0:21
75.56
0.21
0.21
76.33
4 0.27
66.98
+: 0.38
68.64 1: 0.40
73.13 1: 0.26
Bayesian
75.35 1 0.27
77.27
+ 0.46
77.41
+ 0.20
70.64
+ 0.28
73.
0.38
0.38
75.37 - 0.23
Laplace
72.84 0.05
76.11
4 0.35
76.42
+ 0.10
68.02
+ 0.26
70.69
0.36
72.95 0331
MCMI
72.44 1: 0.34
75.27
2 0.29
76.31
+ 0.14
67.10
1 0.51
68.47
0.411
73.23
- 0.10
MSE
71.09 1 0.27
74.09
+ 0.20
74.48
+ 0.26
65.61
1 0.07
68.55 1: 0.26
71.39 - 0.35
34

---

Published as a conference paper at ICLR 2026
Table 7: Mean and variance of the accuracies corresponding the numerical study in Subsection 4.2 performed with distillation hyperparameters (A, T, T.) = (1, 2, (averaged over 5 runs).
Teacher Student
ResNet-18
ResNet-50 ResNet
ResNet-50 ResNet 34
WRN-40-2 WRN 16
WRN-10-2 WRN 40.
VGG-13 VGG-8
ResNet 18
Deterministic
74.27 + 0.26
74.13 + 0.07
74.66 +: 0.21
68.26 1 0.34
65.80 + 0.49
74.50
0.21
Bayesian
76.81 + 0.26
76.92 + 0,23
77.30 + 0.04
71.89 + 0.60
69.75 + 0.26
77.33
0.15
Laplace
74.14 + 0.26
75.07 + 0.40
75.19 + 0.29
71.79 + 0.38
69,96 + 0.28
75.26
0.28
MCMI
74.22 + 0.21
74.05 + 0.08
74.78
+ 0.19
68.10 + 0.38
65.86 + 0.24
74.39
0.18
MSE
74.19 + 0.35
74.22 = 0.34
73.70
+ 2.00
67.73 + 0.33
49,99 + 0.55
72.61
1.20
Teacher
ResNat.50 WRN-40-2
ResNet-50 VGG-8
VGG-13
VGG-13 WRN-40-1
ResNet-50 WRN-16-2
VGG-13
Student
ResNer-18
WRN-40-2
Deterministic
71.64 + 0.32
74.16
0.27 0.27
74.62
+ 0.35
65.41
+ 0.39
68.02
+ 0.18
71.56 0.26
Bayesian
75.60 + 0.21
76.92
= 0.30
77.37
+ 0.18
70,75
+ 0.17
72.78
0:78::00 0.25
75.29 = 0.31
Laplace
74.24 0.22
75.53 + 0.29
75,00
+ 0.40
70.87
+ 0.44
71.89
:: 0.25
74.18 = 0.30
MCMI
71.86 + 0.38
73.95
- 0.29
74.54
0.13
65.70
1: 0.37
67.82 0.25
67.82
71.92 - 0.17
MSE
69.81 + 0.33
72.61
= 1.00
74.55
+ 0.24
49.86
+ 0.30
67.99
+ 0.67
70.06 = 0.33
Table 8: Mean and variance of the accuracies corresponding to the numerical study in Subsection 4.2 performed with distillation hyperparameters (A, Tv. T.) = (1, 4, 1) (averaged over 5 runs).
Teacher Student
ResNet-18
ResNet-50 ResNet-18
ResNet-50 ResNet-34
WRN-40-2 WRN-16-2
WRN-40-2 WRN-40-1
VGG-13 VGG-8
ResNet-18 18
Deterministic
75.69 +: 0.29
76.20 +: 0.16
76.78
+: 0.35
70.80
: 0.46
68.92
+ 0.42
76.08
: 0.25
Bayesian
76.83 +:
77.14
0.35
77.25
+ 0.16
72.94
0.19
70.77
+ 0.49
77.61
+ 0,20
Laplace
73.77 0.42
0.41 74.68
74.91
+ 0.68
71.01
+ 0.27
70.04
0.33
74,92
0.25
MCMI
75.66 0.32
0.229
76.86
+ 0.30
70.87
: 0.52
68.66
- 0.54
76.35
+ 0.14
MSE
74.03 0.13
73.88 0.14
73.98
+ 0.27
67.73
+ 0.41
49,30 - 0.21
72.88 + 0.06
Teacher
ResNet-50 WRN-40-2
ResNet-50 VGG-8
VGG-13
VGG-13 WRN-40-1
ResNet-50 WRN-16 16-2
VGG-13 WRN 40-2
Student
ResNet-18
Deterministic
13.79:00.33 0.33
75.66
è: 0.39
76.36
+ 0.22
67.90
1: 0.30
69.36 - 0.38
73.71 + 0.20
Bayesian
75.82 + 0.17
77.27
+ 0.14
77.26
+ 0.20
71.37
0.23
73.63
+ 0.36
75.45 + 0.31
Laplace
73.97 + 0.33
74.96
0.32
74.57
+ 0.22
69.33
2 0.30
71.30
0.21
73.76
+: 0.44
MCMI
73.50
75.75
0.13
76.40
+: 0.34
69.58
+ 0.27
74,20
+ 0.27
MSE
6.49
72.59
72.59
0.49
73.89
+: 0.29
49.39
+ 0.33
67.88 + 0.43
70.11
+ 0.54
Table 9: Mean and variance of the accuracies corresponding to the numerical study in Subsection 4.2 performed with distillation hyperparameters (A, Tr, T.) = 21, 2) (averaged over 5 runs).
Teacher
ResNet-18
ResNet-50 ResNet-18
ResNet-50 ResNet-34
WRN-40-2 WRN-16-2
WRN-40-2 WRN-4D.1
VGG-13 VGG-8
Student
R8
Deterministic
74.36 0.12
74.23
74.23 0.40
74.58
0.14
0.14
68.16
65.57
74.16 0.22
Bayesian
76.92 0.27
77.18
0.28
77.63
0.10
71.73
0.16
0.16
69.73
- 0.19
77.16 0.24
Laplace
74.10 0.30
75.26
15.26:0..34
75.20
0.50
69.33
- 0.21
75.51
0.34
MCMI
74.14
7..1640041
74.59
+ 0.22
68.25
0.446
65.70
0.28
74.16 0.34
MSE
74.65 + 0.21
74.89 0.24
73.34 171
67.78 0.63
50.02 + 0.50
73.09 + 0.32
Teacher
ResNet-50 WRN-40-2
ResNet-50
VGG-13
VGG-13
ResNet-50 WRN-1662
VGG-13
Student
VGG-8
ResNet-18
WRN-4001
WRN-40-2
Deterministic
71.24 0.35
73.55
4: 0.32
74.66
+ 0.32
65.45
+ 0.44
67,55 - 0.47
0.47
71.66
4: 0.22
Bayesian
75.62 = 0.19
77.01
0.25
77.1
0.10
70.58
+ 0.16
72.94
75.43
+ 0.30
Laplace
74.36 0.27
0.27
75.49
1 0.39
75.27
+ 0.43
70.63
1: 0.23
71.96
- 0.44
74.09
+ 0.38
MCMI
70.87 + 0.37
73.75
0.15
74.54
0.18
65.33
1: 0.43
67.81
0.31
71.58
1: 0.40
MSE
70.24 4 0.13
73.25
4 0.28
74.74
d 0.39
50.17
1 0.12
67,58 -: 0.44
70.22
0.51
35

---

Published as a conference paper at ICLR 2026
Table 10: Mean and variance of the accuracies corresponding the numerical study in Subsection 4.2 performed with distillation hyperparameters (A, Tú T.) = (1, 4, 4) (averaged over 5 runs).
Teacher Student
ResNet-18 ResNet-18
ResNet-50 ResNet-18
ResNet-50 ResNet-34
WRN-40-2 WRN-16-2
WRN-40-2 WRN-40-1
VGG-13 VGG-8
Deterministic
75.91 + 0.32
76.26 +: 0.22
76.82 + 0.27
70.65
= 0.14
08.70
75.74
+ 0,34
Bayesian
76.77 + 0.23
77.08
+: 0.23
77.18 + 0.14
0.27
70.29 + 0.24
77.30
: 0.13
Laplace
73.78 + 0.45
74.83
0.09
0.09
75.03 + 0.56
70.61
0.20
69.66
00666:0.11
74,90
+ 0.41
MCMI
75.86 0.33
76.44
0.21
76.68
+ 0.23
70.04
0.13
68.33
= 0.14
75.64
# 0.26
MSE
75.01 + 0.20
74.75
0.36
74.46 + 0.47
67.41
+ 0.46
49.62 + 0.24
72,97
# 0.29
Teacher
ResNet-50 WRN-40-2
ResNet-50 VGG-8
VGG-13
VGG-13
ResNet-50 WRN-16-2
VGG-13
Student
ResNet-18
WRN-40-1
WRN 40-2
Deterministic
72.94 0.31
75.43 + 0.23
76.22
+ 0.28
66.99 +: 0.35
68.71 + 0.37
73.29 + 0.33
Bayesian
7 0.09
77.06
1:06::00 0.24
76.95 + 0.21
70.92
73.14 0.23
75.18 + 0.33
Laplace
73.88 +
75.10
1:10::00 0.31
0.31
74.96
0.27
69.79
71.14
0.38
73.79
+ 0.17
MCMI
72.68 + 0.37
0.15
76.40
+ 0.23
6.214
68.45
+ 0.40
73.09
+ 0.38
MSE
10.74
73.05
0.33
74.87 0.21
49.65
0.50
67.25
+ 0.24
69.90 0.36
L
LIMITATIONS
Our theoretical results in Section 3 rely on several assumptions commonly used in optimization literature, such as strong quasi-convexity or the Lojjsk-Loassiewicz condition. We also assume model expressiveness to ensure convergence to an optimum. These assumptions, while standard, can of course be questioned.
To make our claims rigorous, we modeled noisy BCPs as either perfect BCPs with additive noise, or as Dirichlet-distributee BCPs. One can always question this type of modeling and suggested more complex modeling, such as incorporating bias or correlation between samples.
Our work advocates the use of BNNs as teachers in KD, possibly obtained by converting pre-trained deterministic teacher model into a BNN using LA. Still, one has to account for the additional computational complexity of BNNs.
36


Reviews:
[{"review_id": "6yMfpUqBq4", "reviewer": "Reviewer_afuu", "summary": "The paper analyzes SGD when the student is supervised by (i) exact Bayes class probabilities (BCPs) and (ii) noisy BCP estimates, then argues for Bayesian teachers as better BCP estimators. Core technical claims: (a) when supervising with exact BCPs, the empirical optimization interpolates and classical SGD “neighborhood” terms vanish, enabling larger admissible stepsizes (Thms. 1–2); (b) with noisy BCPs, convergence rates include a variance (gradient-noise) term whose magnitude depends on how close the teacher is to the true BCPs (Thms. 3–4 and Prop. 3).", "questions": "AS1/AS2/AS3 are standard in optimization but nontrivial for deep CE losses. Two concrete requests:\n\n   (a) Explain where **expected smoothness** (AS3) comes from for CE with typical architectures (e.g., via bounded logits/Jacobians).\n\n   (b) In Thms. 2–4 the stepsize bounds use “(\\mu/(LL))” and “(\\mu/(2LL))”. Please define “(LL)” or fix the notation—likely (L) vs. another constant (L'). As written, it’s ambiguous.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Showing that the CE risk with BCP supervision shares the same minimizer as standard supervision (the Bayes posterior; the minimum equals (H(Y|X))), then establishing interpolation for the BCP-supervised objective (Props. 1–2), is crisp and well-grounded.  \n\n2. Thms. 1–2 remove the variance neighborhood term found in standard SGD and allow a wider stepsize range, formalizing a compelling optimization advantage of distillation from *accurate* probabilities. \n\n3.  The Dirichlet perturbation appendix helps ensure the targets remain on the simplex and shows the main conclusions persist.", "weaknesses": "1. Prop. 3 weights Jacobian norms by (1/P(y_k|x)) (or (1/P(y_k|x)^2) with noisy BCPs). If any class probability can be arbitrarily small, the gradient-noise bounds can blow up. You should make explicit an assumption like (P(y_k|x)\\ge \\epsilon>0) (or work with smoothed targets) and reflect this in all statements depending on Eqs. (13)–(14). \n\n2. Additive perturbations can leave the simplex. While Appendix D covers a Dirichlet alternative, the main text should either use the Dirichlet model (preferred) or state an explicit projection/renormalization step and argue it does not break linearity in the second argument of CE used in the proofs. \n\n3. Prop. 2 proves interpolation for the *BCP-supervised* objective under AS4. It would help to make explicit that interpolation is in the sense of *matching the Bayes distribution* (not zero training error on one-hots). Also, connect more tightly to Def. 1 and spell out that interpolation implies zero gradient at every sample (Eq. (25)), which underpins the disappearance of neighborhood terms. A short lemma bridging these steps in the main text (not only App. C) would aid readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "msatH6z9bv", "reviewer": "Reviewer_Ki2F", "summary": "The given paper demonstrates, from a Bayesian theoretical perspective, that utilizing soft probabilistic outputs in Knowledge Distillation (KD) leads to the highest performance. Based on the SGD optimizer, it shows that when the teacher’s accurate Bayes Class Probabilities (BCPs) are used as target data, the performance surpasses that obtained using one-hot encoding. Furthermore, the paper analyzes how the deviation of errors and accuracy change as noise levels are added to the original BCP distribution.", "questions": "1.\tDoes the proposed SGD_Based_Knowledge_Dist property still hold if optimizers other than SGD - such as Adam or RMSprop - are used?\n2.\tIs the proposed method applicable only to classification tasks, or could a similar approach be extended to regression problems as well?\n3.\tIn Table 1, under what specific conditions were the tests conducted? (For example, were results averaged over multiple runs with different random seeds, such as Seed 0 to Seed 5?)\n4.\tIn line 272 on page 6, the paper states that “we model ϵ ∼ P_{ϵ} as zero-mean noise with variance ν and uncorrelated entries.” - How was the variance ν determined or chosen?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1.\tOriginality:\nThe paper provides a mathematical proof, from a Bayesian theoretical perspective, explaining why using soft probabilistic outputs in KD leads to better performance under the setting of an SGD optimizer. As mentioned in the Related Work section, the authors generalize this theoretical result beyond special cases such as self-distillation or model compression to more general classification settings, which represents a clear contribution compared to prior research.\n\n2.\tQuality:\nThe paper offers mathematically rigorous explanations throughout all derivations, giving the theoretical sections a strong sense of completeness and internal consistency.\n\n3.\tClarity:\nThe logical flow of the paper is well-structured, and the authors successfully connect theoretical findings with experimental results, presenting a coherent narrative from theory to practice. However, the extensive mathematical derivations make it somewhat difficult for readers to follow the core ideas and fully grasp the main contributions.\n\n4.\tSignificance:\nBy providing a mathematically complete explanation for why soft probabilistic outputs improve performance in KD—a question that has not been sufficiently addressed in previous KD research—the paper offers meaningful theoretical significance within the field of knowledge distillation.", "weaknesses": "1.\tIn Figure 1, it would be beneficial to further quantify the amount of noise and present this quantitatively. Moreover, based on the plots of generalization error and test accuracy per epoch, it seems that the results were obtained using a single random seed. If the authors were to test with multiple seeds and compute the variance of generalization error and test accuracy per epoch for the four cases, it could more clearly demonstrate that the true Bayes probabilities exhibit significantly lower variance. Additionally, instead of using abstract expressions such as “less noisy probabilities” or “more noisy probabilities”, it would be clearer to explicitly specify the exact noise levels, which would also enable a direct comparison with one-hot labels.\n2.\tAlthough the paper’s overall contribution is meaningful, it is somewhat disappointing that it does not go beyond showing that using BCPs as labels for the student model improves training stability due to lower variance.\n3.\tThe transition from Equation (10) to Equation (11) omits too many intermediate steps, making it difficult to follow the derivation. Furthermore, mathematical expressions are overly complex, and the overall explanations feel somewhat unfriendly and difficult to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AbRRUzCcmJ", "reviewer": "Reviewer_FywD", "summary": "This paper aims to provide a theoretical analysis of logit-based knowledge distillation from a Bayesian perspective. The authors provide analyses for both perfect BCPs and noisy BCPs, drawing the conclusion that knowledge distillation can lead to variance reduction and neighborhood term removal. Based on the theoretical findings, the authors further propose to utilize Bayesian deep learning models to improve effectiveness.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well organized, highly detailed, and balanced between theoretical depth and readability.\n2.\tThe theoretical analysis is supported by empirical evidence.\n3.\tThere is potential practicality, as the authors also show the benefit of converting pre-trained models into BNNs to improve the effectiveness of knowledge distillation.", "weaknesses": "1.\t(Minor) The analysis is based on SGD, but the experiments are conducted on Adam. Although this can show that the analysis also applies to other SGD-related optimizers, it would be better if there were some analyses or at least citations to show such generalizability from a theoretical perspective.\n2.\t(Minor) The experiments are based on image classification solely. Could there be more complex tasks, such as semantic segmentation or object detection?\n3.\t(Minor) Some related work is recommended to be discussed, such as [1, 2]\n\n[1] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML 2025\n\n[2] f-Divergence Minimization for Sequence-Level Knowledge Distillation. ACL 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RNMTkrw8sf", "reviewer": "Reviewer_Faxx", "summary": "The paper carries out convergence analysis for SGD-based KD with Bayesian teacher and noisy Bayesian teacher, showing faster and more stable convergence compared to standard SGD. Based on the above analysis, the paper proposes to use BNNs as Bayesian teachers, either trained from scratch or converted from deterministic pretrained models. Experimental results are provided to validate the theoretical analysis and some performance gain is demonstrated.", "questions": "1. For converting a deterministic pretrained model into a BNN, is there any way to introduce stochasticity without modifying the teacher model itself (since in many cases, it's not desirable/feasible to modify the teacher model)? For example, through the use of data augmentation or adding auxiliary probabilistic modules to the teacher model.\n2. The paper refers probability distributions that are closer to the BCPs as more \"calibrated\" probability distributions. Then, why not show some results on model calibration such as expected calibration error (ECE) and reliability diagram [1].\n\n[1] C. Guo et al. On Calibration of Modern Neural Networks. ICML 2017.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper studies the benefit of Bayesian teachers in KD from the perspective of SGD convergence, which is a solid and principled choice.\n2. The paper connects the literature on BNNs to KD through the use of Bayesian teachers.", "weaknesses": "1. The paper lacks high novelty. The Bayesian teacher perspective of KD has been well-studied in the literature and many results, both theoretical and empirical, have been presented to show that Bayesian teacher is optimal for student learning. The real important issue is how to pratically obtain a Bayesian teacher. However, the paper doesn't emphasize much on this issue, and simply resorts to some existing Bayesian DL methods.\n2. The experimental results are not comprehensive enough. (1) All results are based on the standard KD, without extending to any latest logit-based distillers. (2) No result on ImageNet is presented. (3) No result for transformer-based models, e.g., ViT, is presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)