The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model
Authors: Kaito Takanami
Takashi Takahashi
Ayaka Sakata
Venue: NeurIPS 2025 poster ()
Keywords: knowledge distillation
self distillation
statistical physics
replica method
dynamics
machine learning
Primary area: theory

Abstract: Self-distillation (SD), a technique where a model improves itself using its own predictions, has attracted attention as a simple yet powerful approach in machine learning. Despite its widespread use, the mechanisms underlying its effectiveness remain unclear. In this study, we investigate the efficacy of hyperparameter-tuned multi-stage SD with a linear classifier for binary classification on noisy Gaussian mixture data. For the analysis, we employ the replica method from statistical physics. Our findings reveal that the primary driver of SD's performance improvement is denoising through hard pseudo-labels, with the most notable gains observed in moderately sized datasets. We also identify two practical heuristics to enhance SD: early stopping that limits the number of stages, which is broadly effective, and bias parameter fixing, which helps under label imbalance. To empirically validate our theoretical findings derived from our toy model, we conduct additional experiments on CIFAR-10 classification using pretrained ResNet backbone. These results provide both theoretical and practical insights, advancing our understanding and application of SD in noisy settings.

The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model
Kaito Takanami
Department of Physics
Graduate School of Science, The University of Tokyo, Tokyo, Japan
Center for Interdisciplinary Al and Data Science, Ochanomizu University
Tokyo, Japan
takanami2550g. ecc. u a-tokyo.ac jc.
Takashi Takahashi
Institute for Physics of Intelligence, The University of Tokyo Tokyo, Japan
RIKEN center for AIP
Ayaka Sakata
Department of Information Science, Ochanomizu University
Tokyo, Japan
RIKEN center for AIP
Abstract
Self-distillation (SD), technique where model improves itself using own predictions, has attracted attention simple yet powerful approach in machine learning. Despite its widespread use, the mechanisms underlying its effectiveness remain unclear. In this study, we investigate the efficacy of hyperparameter-tume multi-stage SD with linear classifier for binary classification on noisy Gaussian mixture data. For the analysis, physics. Our findings reveal that the primary driver of SD's performance improve- ment is denoising through hard pseudo-labels, namely discrete labels generated from the model' own predictions, with the most notable gains observed in moderately sized datasets. We also identify two practical heuristics to enhance SD: carly stopping that limits the number of stages, which is broadly effective, and bias parameter fixing, which helps under label imbalance. To empirically validate theoretical findings derived from our toy model, conduct additional experiments on CIFAR- 10 classification using pretrained ResNet backbone. These results provide both theoretical and practical insights, advancing understanding and application of SD in noisy settings.
employ the replica method from statistical
1 Introduction
Knowledge distillation (KD) Hinton [2033 is a technique in machine learning that transfers the learned information from complex model (often referred to as the teacher) to simpler model (the student). This method attracted attention for achieving model compression with minimal performance loss, and has been applied across various domains, including image classification Liu et al.) [2018] Xu et al. (2020), object detection Chen et [017], and natural language processing Calderon et al. [2033] Gu et al. 2024).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Among the various forms of KD, self-distillation (SD), originally termed born again neural net- work Furklnenle et al. [2018] particularly intriguing. In SD, the teacher and student models share identical architectures. This means that SD) does not attempt the model compression; rather, it retrains the student model using the teacher's s output. SD presents a intriguing paradox: despite training an identical model on the same dataset, the student model can outperform the teacher Furlanello et al. [2018) Hahn and Choi 22119) Clark et al. (0019) Yang et al. (2024) Chen et al. [2025].
Two main hypotheses have been proposed to explain such seemingly puzzling performance gains. The first suggests that the soft labels generated by the teacher provide dark knowledge |Hinton et al. [2015] Here, dark knowledge refers to the information implicitly embedded in the prediction probability distribution of the teacher model's output, which is absent in hard labels. It provides the student with additional information that captures subtle relationships within the data. The second hypothesis
attributes the improvement to a denoising effect Das and Sanghavi| [023). [as et al. [2025) where the teacher model reduces the influence of the incorrect noisy labels in the training data, enabling the student model to learn a more reliable representation of the underlying patterns Pareek et al. (2024).
Although these hypotheses offer plausible explanations, the optimal behavior of SD, achieved through hyperparameter optimization and repeated iterations Pareek et al. [2024]. remains poorly understood. This lack of understanding makes it difficult to identity the key factors that genuinely contribute to the performance improvement of SD. One reason for this difficulty is that exhaustive exploration of the hyperparameter space is usually computationally expensive, limiting the scope of experimental studies. As result, evaluating the effectiveness of SD and identifying optimal strategies for its application remains challenge.
To address this issue, we consider a multi-stage SD procedure using linear classifier on Gaussian mixture data with label noise. This setup provides a controlled environment for analyzing both the dark knowledge and denoising hypotheses within a unified theoretical framework. In particular, we analyze this setting in the proportional asymptotic limit, where the input dimension N and the data size M diverge at the same rate, i.e., N, M + OO with M/N a E (0, o0). A salient feature of this proportional asymptotic regime is that it allows precise characterization of the trained classifier's behavior, rather than merely providing rough lower or upper bounds. This enables us to explicitly determine optimal hyperparameters and iteration procedures, at least within simplified settings. Moreover, because this precise characterization involves only a finite number of variables, an exhaustive search for the optimal hyperparameters becomes computationally feasible. In this context, Gaussian mixture classification with linear models has served as a standard setting for gaining
valuable insights into high-dimensional learning problems Dobriban and Wager [2018] Mignacco cet al. [2020], Kini and Thrampoulidis [2020), Loureiro et al. [2021). Deng et al. [2022], ). Mannelll
et al. 2024], Pesce al. [2023], J. Takahashi 2024]. Technical tools for analyzing such asymptotic
regimes include the replica method] Mezard et al. (986), CCarbonneau et al. 2023), Convex Gaussian
Min-max Theorem Thrampoulidis et al. [2015]. Approximate Message PussingDDonoho
Donoho et al. [2009),
which builds on Gordon's 's inequality Gordon 1988]. In our study of multi-stage SD, we employ the replica method, which has recently been shown to be applicable to multistage optimization problems, including self-training Takahashi [002) and alternating minimization Okajima and Takahashi 2025
Our main results are as follows:
1. The statistical properties of the trained classifiers are precisely characterized in the asymp- totic limit where the input dimension and the data size diverge at the same rate. The precise formula for the generalization error is also be derived (Section 4
2. SD using soft labels with dark knowledge can outperform hard-label training, particularly under low-noise or limited-data conditions. However, the performance gains achieved with soft labels are often quantitatively comparable to those obtained using hard labels across all settings we investigated. These findings suggest that, at least within our toy model, dark knowledge is not the key factor driving the success of SD (Section
3. Naively applying multi-stage SD over too many stages degrades performance. This can be mitigated by employing an early-stopping heuristic that terminates the SD process at an appropriate stage. The resulting performance improvement most pronounced for medium- sized datasets, where the denoising effect of SD is strongest. In addition, even though pseudo-labels contain label noise, performance comparable to training with ground-truth labels can be achieved in large-scale datasets (Section 6)
2

---

4. When ground-truth -ruth labels are imbalanced, learning solely from the teacher's is pseudo- becomes challenging. This difficulty arises because the optimal regularization strength for aligning decision boundaries differs from that required for appropriately estimating the bias term. It shown that fixing the bias term in the early stages of multi-stage SD serves as an effective heuristic to mitigate this issue (Section 7
5. Experiments on CIFAR-10 10 with pre-trained ResNet backhone qualitatively validate several theoretical predictions, extending beyond toy-model settings (Section X)
These results provide comprchensive understanding of the mechanisms underlying SD with linear classifier on noisy Gaussian mixture data, and offer insights into how to optimally apply SD,
Reproducibility:
The codes to reproduce some of our results are available at Phttps://githhi com/taka255 self disf-dddstllatioon -
Impact statement: We believe this work, which is a theoretical study of the learning behavior of simple linear model in a synthetic setting, does not have notable societal consequences.
2 Related Work
Replica method for multi-stage learning. The application of the replica method to analyzing the dynamics of fiigh-iiensional systems was originally proposed for studying discrete optimiza-
tion Krzakala and Kurchan [2007] and stochastic processes in glassy systems Franz and Parisi (2013).
In recent years, it has been extended to learning problems, particularly for analyzing sequential optimization processes, including self-training in semi-supervised learning Takahashi [2022] and alternating minimization OOkjima and Takahashi (2025). Our work builds on and advances this approach analyze modern machine learning algorithms.
This methodology can be interpreted as variant of Dynamical Mean Field Theory, fundamental
tool of statistical physics for analyzing the dynamics of high-dimensional systems, including gradient- based learning dynamics of neural networks Zou and Huang 2024]. Helias and Dahmen (2020 (see appendix A for details).
Theoretical analysis of aif-lisiillation Theoretical analyses of distillation have predominantly focused on separable datasets due to their analytical tractability (Phuoin and Lampert [2019), Das et al.
[2025]. Das and Sanghavi [2023]. separable settings, pseudo-labels generated by a teacher may be able to, in principle, match the ground-truth labels exactly. However, in realistic scenarios where data are not perfectly separable, pseudo-labels inevitably include errors. This fundamental limitation not captured in these existing theoretical analyses, which rely on the separability assumption. While some exceptions Di and Zhu [2020,, (Saglittii and Zdeborova [022] have extended the analysis to non-separable datasets, they fall short of characterizing the behavior of optimal distillation under label noise. Our study fills this gap by quantitatively analyzing the improvements through SD with hyperparameter optimization on noisy and non-separable datasets.
3 Notations and Problem Setup
For convenience, we summarize all symbols and their definitions in Table B. B.I Appendix B and provide a graphical illustration of the multi-stage SD model in Figure of the same appendix.
3.1 Gaussian Mixture Data with Noisy Labels
We consider the binary classification of Gaussian mixture data with noisy labels using a single-layer neural network. Let x E IR be the input data, where μ = ,,..., is the index of the data point and N is the dimension of the input data. Here, we define the data-to-dimension ratio The true labels y E {0, 1} are independently generated according to the Bernoulli distribution p(ytrr = p²5 me (1-p) p) -) with p (0, 0.5). We consider a noisy observation in which the observed labels y/2 E {0, }} differ from the true labels with probability 0: Priy, + = = 0 E 10,1 The are generated from the Gaussian mixture distribution:
M/N.
feature vectors
I₁ =2y 1)8/VN V Azp.
(I)
3

---

where to E R" are the mean vectors of the Gaussian mixture, ( ISwS are i..d. standard Gaussian vectors, and A > 0 controls the variance of the additive noise. Since the noise is rotation (1, ,,...,........... 1)" without loss of generality. The goal is train
invariant, in the following, we set M good classifier from Dir = pz Up j-= that can classify an unseen observation æ, generated in the same way as in ① correctly as yarue.
3.2 Multi-stage Self-Distillation Model
We define the multi-stage SD model a learning process that progresses through stages 0, 1, 2,... The loss function at the t-th stage is given by
M
X² 2 /
Cew ;') - Σ Clvfo Y(w", xx)),)) +
(2)
where f(y, :) is loss function, and the minimizer of Eq. (2) is denoted as w² and B Here, Y(w", (T) is the activation, and v/. is the target label used for (-th stage learning. For / = 0. v
represents the observed label yu. and for t > 0, it is interpreted as the pscudo-label. The activations and pseudo-labels are defined by the following rules.
Activations: The activation at the t-th stage is given by
w # B²
Y(w', xp) o
where (z) is an activation function, and the factor 1/ I/VN ensures the output remains at order ((1) in N. We use two coss-activation combinations: (i) cross-entropy loss and sigmoid activation, ((y. - y log ; - (1 - y) log(1 ú). (x)) = 1/(1 + exp(-x)); (ii) mean squared error loss and linear activation, E(v, 9) (y) 5) (zz) (ar 1)/2. See Appendix B.3 for more details. These choices ensure convexity of the loss at each stage, which is crucial for our theoretical analysis. We refer to the model based Eq. (2) the t-SD model, with the O-SD model the base model before distillation. We refer to the f-SD model with cross-entropy loss as the logistic t-SD model, and the version with mean squared error loss as the linear t-SD model.
Pseudo-Labele Labels at stage t are computed from the (t 1)-th stage as
//1 o (8 1
(
x²
t 82 1,
N
with vf. = yリ- Here, 8 > is the inverse temperature controlling the hardness of the pseudo-label. In the limit 8 - oc, becomes hard (0 or 1), whereas finite 3 yields soft labels in (0, 1). Note that this parameter is meaningful only in the logistic SD model, since the linear SD model is scale-invariant
3.3 Effect of Self-Distillation
We evaluate the effect of SD by measuring generalization error with optimal hyperparameters, tuned to minimize it. We define O' = {Xi) ( for t 2 0, with @ {2²}. and define the error metrics:
[M B²; + y )
E EDEE
8" min 8r, 8r
&Hard min
lim Et (L 1),
(6)
,"....
BA.. 1 000
where Y(O',, B) IW B':z) > 1/2], I(A) is the indicator function, { y My)) and ttr. is test input generated in the same way the training data. Hard represents the error when dark knowledge is removed by hardening soft labels. This limit is meaningful only for the logistic i-SD model. We include this as a reference to assess the role of dark knowledge. A significantly smaller ge than 800 indicates SD effectively improves generalization.
'The results remain valid if n, are replaced by i.i.d. random vectors with zero mean, unit covariance, and finite higher-order moments due to the central limit theorem.

---

Precise characterization of multi-stage self-distilattion
For new input æ generated from the Gaussian mixture in (T). the distribution of the pre-activation w x/va VNN= + B' can be characterized as
vt ax VN
D'
m (2y - 1) D' + AQ
where 4 denotes equality in distribution, Qtr || |12 /N represents the norm of the weight vector, m' r0. v/N is its alignment with the cluster center direction v. and 2 ~ No 1). The label 1) represents the signal component, while vQ" controls the uncertainty due to the variance in the classifier's weights. This result follows from the independence
ttu follows ply true) = pa " (1 (1-p) As such, the term m² m (2y
from D,r and its Gaussianity.
At the proportional limit N, + oo, M/N - a E (0, xo), these quantities are expected to converge to deterministic values, which do not fluctuate aginst the realization of Du. as
(D., + Q", m' (D,,) + m'. (*(,.,) b'.
(8)
This leads to the following expression of the average generalization error Mignacco et al.) [2020]. Proposition 4.1. Under the proportional asymptotic limit (N, M oc, constrained by M/N E (0, oo)), the average generalization error of the t-SD model is given by
m² +
m² - 60
E' = pH
+11 (1-
VAQU
VAQ
where = (1=))=-ff dt e /2T.
Proposition 4.1 indicates that the key quantities to evaluate the generalization error are the aligument m' /VQU (the cosine similarity between the direction of the decision boundary and v) and the rescaled bias B / / (the offset of the decision boundary from the origin).
Since Q", m and B' are expected to be concentrated to deterministic values as in ( they are evaluated by investigating the average values Ep Ep|""), [Q"), Ep [m*], E,[,, the large system limit. For evaluating these averaged quantities, we used the replica method and obtained the following results, which precisely characterize the values of them.
Result (Statistics of the T-SD model parameters). There exist constant matrices Q = (Q (x*) (TT+))((T+I) and constant vector m (f'²) E RT++ such that, in the proportional asymptotic limit (N, M + co, M/N + a E o0)).
T-1 2
d
1 + 10 200
m + + and of d
m ET -
(T 1),
QTT XT
(10)
where E = =(")) E X).
We deliberately use the notation Result rather than Theorem, as the derivation relies on the replica method is not yet mathematically rigorous proof The derivation is given in appendix where the explicit determination of the constant matrices Q X and the vector ri is also detailed. Furthermore, Result T is validated by numerical simulations in appendix D demonstrating good consistency.
The constants Q m and Å that appear in Theorem 1 and b' in Equation 9 can all obtained by solving at most O(T) coupled equations (see Appendix C so they can be computed efficiently. Moreover, from Equation (1O) one directly obtains Q* ET (ui))) and m = R[uif). Together with Proposition 1.1 this gives exact characterization of the generalization error in the asymptotic limit.
Beyond yielding closed-form for the asymptotic generalization error, Theorem T also makes clear that the parameters learned via SD are effectively governed by only small number of key order
Forr possible directions toward rigorous proofs and the associated technical challenges, see the remark in appendix C.10
5

---

parameters. To see how this plays out in practice, we now unpack the roles of the parameters in Theorem [ The weights are composed of three components: (i) the signal term m representing the amount of correlation with the cluster center v. (ii) the noise term S capturing the randomness inherent in the data, and (iii) the correction term - -x= Q " accounting for correlations induced by the labels generated by previous teacher models, i.c. 0-th to (t 1) SD models.
5 The role of soft labels in self-distillation
In this section, we focus on / case, where SD involves a single teacher and student.
Dark knowledge effect is marginal. We investigate the generalization error improvement of the optimal logistic 1-SD model noiseless setting (0 0), where any improvement would stem solely from the teacher's soft labels, because there is no label noise to be removed. As shown in Figure the linear 1-SD model exhibits only modest improvement across various dataset sizes and data variances. Gains are slightly more visible when the dataset is small and variance is low, but even under such conditions, the improvement remains under 0.4%. These results suggest that in linear models, the contribution of dark knowledge is limited.
Error inprovement E"1-6"
14-
12-
88-
2001
Figure 1: Heat map of the improvement error 8-0 +1 at p = 0.4 and 0 = 0 in linear 1-SD model.
Soft labels vs. hard labels. Soft labels provide dark knowledge beyond simple class predictions. However, since denoising primarily aims to identify the correct label, this additional information may not always be necessary. Motivated by the limited effect of dark knowledge observed in the noiseless case, hypothesize that hard labels, obtained by binarizing soft labels, may suffice for SD. To test this hypothesis, we compare generalization errors in logistic 1-SD using hard and soft labels.
Figure 2/ and B show the improvement in generalization errors achieved using soft labels (E*0 (E-0-E") 1 and hard labels (E-0 E respectively. Both exhibit similar qualitative trends: large improvements are observed in higher noise and larger dataset settings, consistent with the denoising interpretation.
Figure 2C shows the ratio of the improvements in A and B, revealing purple region, soft labels offer a clear ad advantage over hard labels, indicating that dark knowledge contributes meaningfully to SD. In contrast, the bright yellow region shows that hard labels are nearly as effective as soft labels, suggesting that dark knowledge is unnecessary in these conditions. However, even the region where soft labels outperform hard labels, the quantitative improvement is small (Figure 2 These findings suggest that while soft labels may be beneficial in specific conditions, their overall impact on SD performance relatively limited in our setting.
distinct regions. In the dark
This observation may refine our understanding of dark knowledge in SD. While previous studies Ma al. 2022]. Mandal al. (2025) emphasized its general importance, our results suggest that its effectiveness depends on dataset characteristics and noise level, and that it is not always necessary.
Understanding the effect of multi-stages
So far, have examined the role of dark knowledge using the logistic t-SD model in single-stage settings. Here, we shift our focus and consider the effect of repeated SD across multiple stages.
To investigate this, we consider the (-stage model (1 > 2) and study how performance evolves with repeated applications of SD. Due to computational constraints, we focus on the linear i-SD model. Although it does not distinguish between soft and hard labels, expect it still capture the essential

---

A
B
Soft Label Improvement
Hard Label Improvement
Hard Label Efficacy Hard
Significant
Significant Improvement
improvement
02-
02-
02-
6500
Soft knowledge crucial
01-
Improvement
Marginal
01
Marginal improvement
01
mar marginal improvement total)
01003
Figure 2: (A) and (B): generalization error improvements in the optimal logistic 1-SD model using soft labels (g-0 - E and hard labels (E-0 &Hard), respectively. (C): the ratio of the two: (E-0 Hard ( (8+0 ). Parameters: p 0.4, A 1.0.
behavior of multi-stage SD. We first focus the label-balanced case (p 0.5), and diseuss the label-imbalanced setting (p < 0.5) in Section 7
Denoising effects and dataset size dependence. We investigated the denoising effect of SD by comparing the optimal I-SD model with two baselines: the optimal O-SD model (E*0 with 0) and the optimal O-SD model trained on noiseless data (E*0 with 0 = 0), as shown in Figure 3
The behavior of * can be categorized into three types depending on a: large a. intermediate a. and small a. When a is sufficiently large (a Z 10) or small (a X 0.2). the decrease in edd slows down around 3, with x*3 and E-10 being almost the same, as shown in Figure 3A.
At sufficiently large a (a Z 10 in Figure 3). multi-stage SD quickly achieves performance close to that of noise-free setting. Specifically, by ł= 3, the generalization error ex becomes nearly equal to the noise-free case xD with = O, even though the error before distillation 8 with > is clearly higher than noise-free case. This finding is noteworthy, due to the overlapping data distributions. In contrast, for small o, Ext remains close to g-U even at t 10, indicating that SD's denoising effect limited. Unlike large or small a, in the intermediate range, the improvement progresses slowly with the number of iterations, but gx approaches to the noise-free case increases. As a consequence, the error improvement by multi-stage SD exhibits non-monotonic behavior with dataset size, with the most pronounced denoising effect observed for moderately-sized datasets, as indicated by the arrow Figure 3
perfect noise correction is challenging
Intuitively, this phenomenon can be explained as follows. For small datasets, the limited data makes difficult for the teacher to learn pseudo-labels that effectively correct label noise, reducing the effectiveness of SD. In contrast, for large datasets, the teacher becomes strong enough to generate pseudo-labels that allow SD correct noise effectively. However, the pre-SD classifier already performs well, leaving little room for further improvement. For intermediate dataset sizes, the available data is sufficient for the teacher produce pseudo-labele that enable noise correction. At the same time, the pre-SD classifier remains suboptimal. As a result, the most substantial performance gains are observed in this intermediate a region.
A
B
Optimal SD
Gererakzaioon Emer
Generditzaton erFOr
X yoo strasegy (Optimal)
0:077
datteee!
Baseline
00075
V
(8=
6022
10*
6
Figure 3: (A) A comparison of the optimal generalization error for the linear t-SD model, 0-SD model, and the noiseless case. (B) Dynamics of / VAQ and m' for the linear t-SD model with
0 2 - xo. (C) Comparison of generalization error between the linear i-SD model with 0 d
and optimal I-SD. Parameters for (A): p 0.5, A 0.5, 0.4. (B, C):
1.0, A 1.2.0 0 0.3, f 1/ 1/VQt-1,t-1

---

Fixed point analysis and learning dynamics. A natural question may be how the enror of t-SD behaves t + 0o. To investigate this, the limit 2 ....., X - oc considered, under which closed- form expression for the generalization error of the linear t-SD model can be derived. This provides theoretical predictions for the error as - DC. This setting is not only mathematically tractable but also interpretable: it simplifies the solution to a the averaging estimator v. x Σn= yu As shown in Dobriban and Wager [0188). Lelarge and Miolane [219], Mignacco et al. (2020). this averaging estimator is Bayes-optimal when noise-less (0 0) and balanced (p 1/2) case. Furthermore, Akhtiamov Akttiamor et al. 2024] shows that in the À - do regime, any corruption rate (0 < 0.5) in the balanced case (p 1/2) can be eliminated. We also confirmed in preliminary experiments that the optimal hyperparameter schedule obtained by black-box optimization starts with very large regularization and then switches to zero, which naturally supports early stopping sensible strategy. Based on the above considerations, we now present the following theorem for this large regularization limit.
Result 2 (The generalization error at t + oo). For arbitrary choice of the set of the temperature parameters {3" Tt20. the generalization error of the linear t-SD model with 0.5, X ... oo and - oc is given by lim4-+c0 Ed 0.5 whenever A
The proof is given in appendix
Result 2 shows that, under certain data conditions, naively continuing multi-stage SD can reduce the model s performance to the level of random guessing. In particular, when A the generaliza- tion error exhibits a phase transition that separates meaningful predictors from random ones (see Appendix I
To gain deeper insights into the learning dynamics, we analyze the time evolution of m' Elw' e//N and Q" "//""w''/NN / which quantify signal extraction and prediction uncer- tainty, respectively. The temperature is set to , 11/ 1/1QQ=I1;1- to prevent the norm of the weight vector, || from vanishing. The values of m² and Q" are plotted in Figure BB. As shown in the figure, m² peaks during the initial iterations, whereas the predictive uncertainty / /AQ" increases steadily throughout the stages, leading to performance decline. Therefore, optimal learning may be achieved by halting the training process when signal extraction is maximized. Interestingly, this early stopping strategy closely matches the results obtained through comprehensive hyperparameter optimization across the entire model (Figure 3C).
These results are consistent with experimental studies Zhang and Sabuncu 2020), where the term "diversity" specifically refers to the predictive uncertainty of teacher predictions, which has been suggested to relate to the success of SD. Our result may support that such predictive uncertainty (diversity) plays key role in effective signal extraction. However, the results also imply that the extractable signal saturates after a few iterations, highlighting an intrinsic limit to the benefit of repeated distillation.
7 The hardness of learning bias in label imbalanced cases
Next, we examine the label imbalanced case p < 0.5, where performance improvement results from the interplay between alignment and rescaled bias, and compare the م = 0.5 case.
The difference from the label-balanced case lies in the difficulty of simultaneously learning the bias and the alignment in imbalanced datasets. Figure 4 shows the evolution of the rescaled bias (b' / /T)) and alignment (m//v/ /"" over distillation stages. As the figure indicates, while aligmment improves gradually, the rescaled bias worsens as training progresses, deviating from its Bayes-optimal value.
This behavior can be attributed to the effect of ridge regularization, which acts only on the weight vector wt. Strong regularization, which may be necessary to improve alignment, shrinks the norm the weight (Q" = | (3/N) and consequently increases the rescaled bias. When the rescaled bias becomes too large compared to the alignment, the model tends classify most data into a single class (either positive or negative), resulting in poor generalization performance (see also Eq. ())
Hence, in label-inbalanced cases, loss minimization may not be suitable for jointly optimizing bias and weight. In contrast, for balanced data, the optimal bias is simply zero (b' = 0), and no such trade-off arises.
8

---

To address the challenge of balancing bias and alignment, we find that fixing the bias at an early stage a simple and effective heuristic. Similar approaches, which separate the training of alignment and bias, have also been proposed in logistic regression Mignacco et al. [0200 and self-training Takahashi [2022] in imbalanced Gaussian mixtures. The dotted lines Figure f illustrate the results with the bias fixed at its value obtained in the optimal O-SD, followed by performing the optimal t-SD. Figure further compares the generalization error t-SD with and without this heuristic. Applying bias fixing significantly improves both rescaled bias and alignment, and exhibits convergence towards the Bayes-optimal solution, as observed in the = 0.5 case.
SD eteccii with mockrstt soed stdaaut
Oytimal g+
Genortation
calenel
ling Max Improves securecy
canrel dencbe
Seaetre
8 8'(4-00
E
Optimal 1:30
bas
Figure 4: (A) Optimal generalization error of the linear t-SD model compared with the O-SD model and the noiseless case under label imbalance (p 0.4). (B) Evolution of the rescaled bias (bb1/v/ Q and alignment (m" VQ")) from / to ( for the optimal I-SD model (solid lines) and the variant with fixed bias (dotted lines). (C) Generalization error over stages for the same models as in (B). Parameters: (A) = 0.4, = 0.5, 0 = 0.4; (B, C) A 1.0, I = 0.4, a = 10.0.
Experiments on real datasets
We have analyzed the behavior of multi-stage SD using the Gaussian mixture model with label noise, which allowed precise asymptotic characterization and provided valuable insights. However, this setting is highly idealized, and there remains significant gap between this toy model and real-world datasets. To bridge the gap, we conduct sanity check to test whether our theoretical predictions hold in standard vision task. We fine-tune only the final layer of a ResNet pretrained on IMAGENETIE K_ V2 maintainers and contributors [2016) (BSD 3 3-Clause "New" License) with L2 regularization on noisy CIFAR- 10 (cat vs. dog) Krizhevsky al. (2009) (MIT License) and compare the results to our theoretical predictions. See Appendix for experimental details.
Figure 5 shows the generalization error of optimal 1-SDD, compared optimal O-SI) and optimal 1-SI) using hard labels (see Eq. (6). When using R8 for feature extraction, we observe virtually no improvement due to SD in the large-o region, while in the middle-a region SD achieves denoising effect that yields ~ 5% performance gain. According to the results in Section 6 the benefit of SD should peak moderate and decline again a becomes even smaller; however, this downturn cannot be observed here because it corresponds to an excessively small sample size in our setup. Nevertheless, Figure partially reproduce the prediction from Section 6 that SD 's effectiveness is maximized at intermediate values of a.
Next, focusing the improvement in generalization error when propagating hard labels from O-SD to 1-SD, we find that the relative importance of soft-label error reduction grows as a decreases, mirroring the results of Section Remarkably, completely discard dark knowledge, still provide a significant denoising benefit. We observe a similar trend when using ResNet-50; however, since this model has higher baseline performance, the overall magnitude of improvement is smaller. See also Appendix H for further results.
anticipated in Section [ even hard labels, which
Conclusion
We investigated optimal multi-stage SD with a linear classifier for binary classification on noisy Gaussian mixture data. The technical crux of our analysis is a precise asymptotic formula for multi-stage SD, derived using the replica method from statistical physics. As this formula involves only small, finite number of variables, it enables exhaustive hyperparameter search at a reasonable computational cost. This contrasts with experimental studies, where exhaustive exploration of the hyperparameter space is often computationally intractable. By using this, we obtained the following

---

ResNet18
ResNet50
644
+00.0
Generrltizatin Error
0.28
Generalizaton Error
0.2
Figure 5: Comparison of the optimal generalization error the logistic O-SD model, 1-SD model and 1-SD model using hard pseudo labels for CIFAR-10 dog versus cat classification using pretrained ResNet-18 (N = 512) and ResNet-50 (N = 2048) feature representations. Parameters: = 0.4. Error bars represent the standard error of the mean over 10 trials per point.
results. First, we found that dark knowledge in soft-labels plays a more limited role than previously assumed, with denoising likely being the primary driver of its success. SD's strong denoising capability is evident even with inseparable data distributions. Second, SD is most effective with moderate dataset sizes, showing weaker effects in both very small datasets (where denoising is difficult) and very large datasets (where noise has small impact). Third, fixing the bias and focusing on alignment optimization serves as a useful heuristic in SD. More broadly, this suggests a general strategy for multi-stage SD: progressively narrowing the parameters optimized each stage. These findings not only enhance our theoretical understanding of SD mechanisms but also provide foundation for developing improved algorithms.
Limitations: While our asymptotic analysis provides valuable insights, is limited to linear models under a Gaussian mixture setting. Extending the analysis to deep networks or alternative SD strategies remains an important direction for future work. See Appendix I for further discussion.
Acknowledgements
The study was conducted as part of the exploratory project "Mathematical Exploration of Universal Structures in Multicomponent and Polydisperse Systems," supported by the Toyota Konpon Research Institute, Inc. This work was also supported by JSPS KAKENHI Grant Numbers 22H051 17 and 23K 16606 JST ACT-X Grant Number JPMJAX24CG, and JST BOOST NAIS Grant Number JPMJBS2418.
References
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv [stat.Ml.), March 2015. URL http:://axxxxi www./rgg//////eeeeee//. abs/ 02531
Yongcheng Liu, Lu Sheng. Jing Shao, Junjie Yan. Shiming Xiang, and Chunhong Pan. Multi- label image classification
knowledge distillation from weakly-supervised detection. In Proceedings of the 26th ACM international conference on Multimedia, New York, NY, USA,
15 October 2018. ACM. ISBN 9781450356657. doi: 10. (000114///200088322444444 33206677. URLphttps //al. acm org doi/ abs 10. 1145/3240503 02400333 http://ccccccooooooooooooonnnnna
Kunran Xu, Lai Rui, Yishi Li, and Lin Gu. Feature normalized knowledge distillation for image
classification. In Lecture Notes Computer Science, Lecture notes computer science, pages 664- 680. Springer International Publishing, Cham, 2020. ISBN 978800055555559783306655321 doi: 10.100779778-3330-58888 978 978-3--030-58555-22222
URL httth:://ppll spr inger. com com//hhapeer//0010000
Guobin Chen, Wongun Choi, Xiang Yu, Han, and Manmohan Chandraker.
Learning ef-
ficient object detection models with knowledge distillation. Neural Inf Process Syst, 30: 742-751, 4 December 2017. Utt httpe//eeeeeoroouuuueppp///eeeee :ttptppe://prooeedinge. www.nee......appppeeepee pttprrpps.cc/ppppp//////aaaaeaep
e1 elte5ee2bbeelss5777777aa6666655. Absteaarrddddddatttttt....eaee html
10

---

Nitay Calderon, Subhabrata Mukherjee, Roi Reichart, and Amir Kantor. A systematic study of knowledge distillation for natural language generation with pseudo-target training. Anna Rogers. Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61xt Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 14632-14659 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (00118653/11/2223..o 818. URL BtRLittts///aalaathooll.gg org. acl sell-lngg.88 818/
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
URL https:///ppenrevier wtt.://www..eeeeooorr/ddddtooooo
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1607-1616. PMLR, 2018. URL https://prooeeddigg. mmll mlr press/v80/ Furlanell 111.1a. html
Sangchul Hahn and Heeyoul Choi. Self-knowledge distillation in natural language processing. In Ruslan Mitkov and Galia Angelova, editors, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 423-430 Varna, 050.. URI.ttpp
Bulgaria, September 2019. INCOMA Ltd. doi: 00226115978--954455222 //aclanthology org R19 1050/
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! born-again multi-task networks for natural language understanding. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 9931-5937, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 118833/11/11911559 URL https 7/ //aclanthology. www.oolloyyorgoogoogoo - 1595/
Zhaorui Yang. Tianyu Pang. Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. Self-distillation bridges distribution gap in language model fine-tuning. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1028-1043, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1/2653////2224.cllllbaa URL
org/ 2024.022444. acl- 58/
Yinjie Chen, Zipeng Yan. Chong Zhou, Bo Dai, and Andrew F. Luo. Vision transformers with self-distilled registers, 2025. [tttps:ss:// org/ 2505. 21501
Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7102-7140. PMLR, 23-29 Jul 2023.
http://22///aass// html
Rudrajit Das, Inderjit S Dhillon, Alessandro Epasto, Adel Javanmard, Jieming Mao, Vahab Mirrokni, Sujay Sanghavi, and Peilin Zhong. Retraining with predicted hard labels provably increases model accuracy. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 12509.12538... PMLR, 13 Jul 2025. URL https: https://pproceecingg mlr. press/v267/ das25b. html
Divyansh Pareek, Simon Shaolei Du, and Sewoong Oh. Understanding the gains from repeated self- distillation. In The Thiht Annual Conference on Neural Information Processing Systems, 2024. Rttp:ttttss::/penreeeee
Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1):247-279, 2018. ISSN 00905364, 21688966.
URL https: : //www//www//w/// jstor. mng/staalle/65427B/
11

---

Francesca Mignacco, Florent Kızakala, Yue M Lu, and Lenka Zdeborov'a a. The role of regularization in classification of digensiinii noisy gaussian mixture. JCML, 119:6874 6883, 26 February 2020. URL https:htts:////ttttm hattps::/ppoceeedingg mlr pttpss//v11//mmmmmccc/ooooo http://pwcccconn....../////////p
html
Ganesh Ramachandra Kini and Christos Thrampoulidis. Analytic study of double descent in binary classification: The impact of loss. In 2020 IEEE International Symposium on Information Theory (ISIT), pages 2527-2532. IEEE, 2020.
Bruno Loureiro, Gabriele Sicuro, Cedric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymptoties in high-dimensions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 10144-10157. Cur- Associates, Inc., 2021. URL htt.tttsssspprrceeedngg.ggeeeeee iew.eeeeeddinnnnnnnei.........pe www.ieuu...... wwtppcpppp.filia/ccpee files/ /
Paperr Paper.pdf
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high- dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11(2): 435 2022.
Stefano Sarao Mannelli, Federica Gerace, Negar Rostamzadeh, and Luca Saglietti. Bias-inducing geometries: exactly solvable data model with fairness implications. ICML 2024 Workshop Grounery-groundeed Representation Learning and Generative Modeling, 2024. URL https //openreveew //oporooreev.ne//forrrr// wwweet/forummdddouuiipeeeee odpiiizzp
Luca Pesce, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Are gaussian data all you need? The extents and limits of universality in high-dimensional generalized linear estimation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 27680-27708, PMLR, 23-29 .mw.mng...........mmm..m..m.mm.. wwwpppeessaaae wttppp/svv//ppseeeeeeea..e
Jul 2023. URL
Takashi Takahashi. replica analysis of undern Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
M Mezard, G Parisi, and M Virasoro. Spin Glass Theory and Beyond. WORLD SCIENTIFIC, 1986. doi: 10.1142/0271. URL
1142/0271
Patrick Charbonneau, Enzo Marinari, Mare Mézard, Giorgio Parisi, Federico Ricci- Mecci-Tersenghi, Gabriele Sicuro, and Francesco Zamponi. Spin Glass Theory and Far Beyond. WORLD SCIEN- TIFIC, 2023. doi: 10.1142/13341. URL httpt::t///// //www. www.wwoieewieiiiiccccoo.occcc..e wttpic//coooocooooooeee
1142/13341
Christos Thrampoulidis, Samet Oymak, and B Hassibi. Regularized linear regression: A precise analysis of the estimation error. Conf Learn Theory, 40:168::11 10011683-11 1709,, 26 June 2015. URL https
//proceedings.. .mlr. wwwpp//e/e/arrararrree hhwp://aaaaaaoooinin55555 html
David I. Donoho, Arian Maleki, and Andrea Montanari. Message- passing algorithms for compressed sensing. Proc. Natl. Acad. Sci. U. S. A., 106(45):189 1899914-888 18911999,, 10 November 2009. ISSN 0027- 8424,1091-6490, doi: 10. 111.0//pas.0000999000 00r3/pmaas.00989221
URL htt.:htttp//// http::::///www.////eeee pnas org//oi/ab///11
Y Gordon. On milman's inequality and random subspaces which escape through mesh in R
In Lecture Notes in Mathematics, Lecture notes in mathematics, pages 84-106. Springer Berlin Heidelberg, Berlin, Heidelberg, 1988. ISBN 978:540019553,9973330392223
doi: 10.1007) bfb0081737. https: sprinner....grrrrrr.mmr com/ chwpter 1007 1007000000000000707077777
Takashi Takahashi. The role of pseudo-labels in self-training linear classifiers on high-dimensionn gaussian mixture data. arXiv [stat.ML], 16 May 2022. URL http:::/aaxiv.rrg///aaaa/ 07739
Koki Okajima and Takashi Takahashi. Asymptotic dynamics of alternating minimization for bilinear regression. Journal of Statistical Mechanics: Theory and Experiment, 2023(5),05331,, jun 2025. doi: 10. 12..0888/742255668aaaa URL BtLLhttps:// :/op:::://oi... org/ 10. 1088/ 1742- 5468/ 5468/add1ce addice
12

---

Florent Krzakala and Jorge Kurchan. Landscape analysis of constraint satisfaction problems. Phys. Rev. E Stat. Nonlin. Soft Matter Phys., 76(2 1):021122, August 2007. ISSN 1539-3755.1550 2376. doi: 10.1 T0S/PhyysReeE..662222222 URL htt.httts://jornallsss aps wwwporggggg/..eeeeeeeeeeeeeeeeee 10. www.10///hyseee...... 76.021122
Silvio Franz and Giorgio Parisi. Quasi-equilibrium glassy dynamics: an algebraic view. J. Stat. Mech., 2013(02):P02003, February 2013. ISSN 1742-5468. doi: 10. 10011008/7422-5546/2211001 PO2003. URL 102003/meta.
/1t0////000000000/22222///////// 546//201//////
Wenxuan Zou and Haiping Huang. Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings. SciPost Phys. Lect. Notes, (79), 20 February 2024. ISSN 2590- doi: 10..1 10011468kiipposphyssctoessss
URL https
//scipost. aagg/saPootPPrraeectootee 79/pdf
Moritz Helias and David Dahmen. Statistical field theory for neural networks. Lecture notes in physics. Springer Nature, Cham, Switzerland, 1 edition, 21 August 2020. ISBN 0100000464411198833334444444
doi: 10. 1011077/978-3-03-4444444 URL https://
springer com http:/ooo//10// 1001://100000000900 00000/9977---00044444------
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5142-5151. PMLR, 09-15 Jum 2019. URL https:///proceedingg mlr. http://dd.nnn.......ppe////////a Sycess/vv7//hhuonee
html
Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Neural Inf Process Syst, abs/2010. 100000000000000000000000 20 October 2020. URL httpps://procceeddnngg
newrtiiinnnnnnnis....o...rr wwtppcppppppp..cccccccccceee...p http://sccccppppe.eooo/ee/ee/eee haaaf//ff0a9aa7166000022236666.3 Abstract html
Luca Saglietti and Lenka Zdeborova. Solvable model for inheriting the regularization through knowledge distillation. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova, editors, Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pages 809-846. PMLR, 2022. htttt::::ppppeeeeeeeeeeeeee mlr. www.ppeeee///eeeeeeeeee http:///1145//allett222222 html
Haoyu Ma, Yifan Huang, Hao Tang, Chenyu You, Deying Kong, and Xiaohui Xie. Sparse logits suffice to fail knowledge distillation. ICLR 2022 Workshop on, 25 March 2022. URL https //openreview net/forum?id Bew.rrmmiddddZduuuuuuuuu Bx2gduuND15
Saptarshi Mandal, Xiaojun Lin, and Rayadurgam Srikant. A theoretical analysis of soft-label vs hard-label training in neural networks. In Necmiye Ozay, Laura Balzano, Dimitra Panagou, and Alessandro Abate, editors, Proceedings of the 7th Annual Learning for Dynamics &amp; Control Conference, volume 283 of Proceedings Machine Learning Research, pages 04-06 Jun 2025. URL /ttp/attp///prrccedinng
PMLR, mlr www.sppeesseeeeee v22///22////// 202233/aanda25a httml
Marc Lelarge and Leo Miolane. Asymptotic bayes risk for gaussian mixture in semi-supervised setting. In 2019 IEEE 8th International Workshop on Computational Advances in Multi- Sensor Adaptive Processing (CAMSAP), pages 639-643, IEEE, December 2019, ISBN
doi: 10. 11w9///amaaa4666600000000000 //ieeexplore ieee. www.iieeeeeeeeeeeeooooee...e http://dccmeent/0222222223
9788128115494.77117888888
URL https
Danil Akhtiamov, Reza Ghane, and Babak Hassibi. Regularized linear regression for binary classi- fication. In 2024 IEEE International Symposium on Information Theory (ISIT), pages 202-207, 2024. doi: 10.
Zhilu Zhang and M Sabuncu. Self-distillation as instance-specific label smoothing. Neural Inf Process Syst, abs/2000
June 2020. http.httsss//pprrreennnss..ss neurips
cc/ htc/aaaee_flleee/ppper/////////p html
Torch Vision maintainers and contributors. Torchvision: Pytorch's computer vision library. https
// //gi giu.. comm/pytoorrc//iiioo 2016.
13

---

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. features- 2009 TR. påf
URL buttps://wwt cs toronto. edu/"kriz/leannnee htrar//eaanin---ftureeeess
E Gardner.
The space of interactions in neural network models. J. Phys. A
Math. Gen., 21(1):257-2700 7 January 1988. doi: 10.1088/03055-440021110900
ISSN 2305-4470.1331-00
URL https://iopsciennce 1op org/
UnP8Zv wttp:///ww.atiii.....ooo.oooooa.
LDYUkavVQQ--uccqPPPPa
Federica Gerace, Bruno Loureiro, Florent Krzakala, Mare Mezard, and Lenka Zdeborova. Generali- sation error in learning with random features and the hidden manifold model. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3452-3462. PMLR, 13-18 Jul 2020. URL httppttt////roocoodiiie
mlr.
wttp://erreeaaaaeeee html
Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with teacher-student model. J. Stat. Mech., 2022(11):114000 November 2022. ISSN 1742-5468.
doi: 10. 10.1100//772 55105442255556//982555... URL httppp//ipppccieee.....oooogg 1742 1492-55468/a3922//eee
Jean Barbier, Florent Kızakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proc. Natl. Acad. Sei. U. S. A.,
116(12)5551--540, 19 March 2019. ISSN 0027-8424,100 10914 doi: 10.100//pss.1100 18027051 16.
URL httph:t////// pnas. orw./ogg/oooo..oooooooo http//////aaa//ooa/o/aaa /10. 1073/pnas 1802705116
Songbin Liu and Junjie Ma. Unifying amp algorithms for rotationally-invariant models. arXiv
preprint arXiv:2412.01574 2024.
Vaibhav Kumar Dixit and Christopher Rackauckas. Optimization..i: A unified optimization package, March 2023. URL
7738525
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chengiong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 3722.772.. IEEE, October 2019. ISBN 9781728148033 480388.
doi: 109.ccccccccc.......... 209000011. URL http: //openaccess. theccf wwt.:/coo....oeee. ICCV Bte_____CVV_____ http:22099htmtmmmae //hang Be Your Own Teacher_ Improve the of wtt__CCovollliion otw.re/rrrrraaaaeeeooooooooooooa NeereeNeuraa__ ICCV 2019 20w1__aaeerr______________111111 pappr.html httl
Haoyu Ma. Tianlong Chen, Ting-Kuei Hu. Chenyu You, Xiaohui Xie, and Zhangyang Wang. Undis- tillable: Making a nasty teacher that [cannot] teach students. In International Conference on Learning Representations, 2021. URL http::::/pppmeerrree.eee/ooorrrr
Eda Yilmaz and Hacer Yalim Keles. Adversarial sparse teacher: Defense against distillation- based model stealing attacks using adversarial examples. IEEE Access, 01.20774-2208, 2025. doi:
John Blitzer, Ryan T McDonald, and Fernando C Pereira. Domain adaptation with structural correspondence learning. Empir Method Nat Lang Process, pages 1200028,, 22 July 2006. doi: org/ 10. 311.//1111111...11. 1610094
httl::tt:////
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, New York, doi: 10.11145/155334.155533 URL
NY. USA, 14 June 2009. ACM. ISBN 9781605585161.
http://dx..o doi. org/o0../111 1 1145/1553374 5533343334... 000153333771161389000 1553380
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural
networks: A survey. IEEE Trans. Pattern Anal. Mach. Iniell., 44(9):5 44(9):5149-511 149-5169, 5169, September 2022. ISSN 0162-8828, 1939-3539. doi: 10.1199/PAAM..2021.3799920 URL http:://p/// doi.doi..ooooooooo org//
10. 1109/TTAPAAA TTAMTPAMMM APM... 30799999
14

---

Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue Lu, Lenka Zdeborova, and Bruno Loureiro. Asymptotics of feature learning two-layer networks after one gradient-step. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 2024. URL https: / wttp:///e/occeee...//o/ mml. mlr. wtwpppp23// cui cui24d. html
PMLR, 21-27 Jul
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims do accurately reflect the paper's contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 9 and further remarks Appendix []]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

We provide the full set of assumption in Section 4 and complete proof in Appendix Fnd
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide detailed information reproduce experiments in Appendix G
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We include an anonymized GitHub repository link in the last of Section T reproduce main experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
oooliccyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctpccc/ppppo httpppulllllddeeeeeeeppep muublic/guudds//ddeebbbisss. ponddeSubmissiooPoo..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the training and test details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include error bars defined as the standard error of the mean.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of the computational resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: httppp////earrr...ooooooo////aaa
Answer: [Yes]
Justification: This paper conforms to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide a discussion of potential impacts in Impact Statement subsection at the end of Section [
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This theoretical work does not release any datasets models that pose significant risk of misuse, so no additional safeguards are required.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have credited all third-party assets and fully complied with their license
terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
We provide the full reproduction code in GitHub repository including documentation.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work did not involve any crowdsourcing or human-subjech experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work did not involve any human-subjects experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
21

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methods in this paper do not use any LLMs as important non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy for what should or should not be described.
22

---

A Further remarks on related works
Replica method for machine learning problems. As machine learning models and datasets grow increasingly complex, traditional mathematical approaches often fall short in providing rigorous analytical solutions. This complexity gap has led rising demand for alternative theoretical tools that can offer insights into model behavior and performance, even when rigorous mathematical solutions are out of reach.
In this context, the replica method, originally developed in statistical physics, has emerged as a powerful analytical technique for machine learning problems. While not yet mathematically rigorous in all aspects, this method has been widely applied to various models, from simple perceptrons Gardner 19888 modern non-i.i.d. datasets Gerace al. (2020). Loureiro al. [2022), with some of its predictions later rigorously proven Barbier et al. [011]] The replica method offers unique ad vantages, such as the ability to compute exact generalization errors rather than bounds necessary conditions.
This precision enables explicit optimization of hyperparameters in multi-stage SD, providing deeper insights into model behavior and performance.
Relationship between multi-stage replica method and DMFT. Traditional DMFT Helias and Dahmen (2020] is primarily used for analyzing learning dynamics. This approach is effective when the system's 's state at time can be expressed explicitly using the state at time / - 1, allowing for direct averaging over data. However, in more complex scenarios like SD, conventional DMFT techniques face challenges. In our model, the transition from one state to the next is not explicitly defined but implicitly determined through an optimization process. Specifically, the previous state (wb-11 B influences the output labels y (Eq. (4). which then feed into the optimization problem minimizing the loss function (Eq. (2) to determine the new state (w² B¹). This implicit dependency, mediated by an optimization step, makes the dynamics more complex than those typically handled by traditional DMFT approaches. To overcome these challenges and extend DMFT's applicability to such complex scenarios, we employ the replica method, which allows us to analyze these implicit optimization-bal based state transitions effectively.
23

---

B Further remarks on model and notation
In this appendix, we compile and summarize all notation used throughout the paper, provide detailed commentary on the model's structure, and include illustrative figures aid the reader's understanding.
B.1 Notation
Table B.1
lists each symbol and its definition.
Table Summary of Notations
Category
Symbol
Definition
Data Generation
N
Dimension of input features
M
Number of training samples
a = M/N
Data-to-dimensie
ratio, O(1)
Bp
Input vector (μ 1, 1,..., M)
Class prior (p E (0, 0.5)
true
True label, {0, 1} with p(y = pD p)
Noisy observed label; Pr(y F y!
Label noise rate (0 (0, 0.5))
Gaussian mean vector, set (1, 1.... 1)T w.l.o.g.
v
A
Feature noise variance in 11 (())
Dtr
Training set { F/ )p M
Distillation Process
Stage index base model) Weights and bias at stage / (minimize Target label at stage t: y/ Yu for > see In-regularization strength in (2)
(w", B*)
X²
ge
Inverse temperature for soft pseudo-label hardness in (4)
Loss & Prediction
((y,0)
Loss function: (cross-entropy, MSEJ
o(x)
Activation: sigmoid 1/(1 + t or linear (z + 1)/2
Y(uo, B²;a)
Activation output, see Eq.
3
Li(w B²)
Stage-t objective, see Eq.
Performance Metrics
gt
Generalization error after stage t, def. in Sec. 3
800
Optimal O-SD error (minimized over 29)
Optimal I-SD error (minimized over X (9.1)
Hard
Optimal t-SD error (3)...
and minimized over 0..1)
Asymptotic Quantities
Qat
Weight-weight overlap, limit of w ú0 / Weight-signal overlap, úf -uN Rescaled bias, 1"
m²
b²
B.2 Illustration of the model
We present graphical illustration of the model. odgepctss the overall architecture, highlighting its key components and the flow of information.
B.3 Motivation for the choice of activation function
We chose the activation function (xx) = (a + 1)/2 instead of the simpler (xx = x because ensures that the decision boundary remains unchanged when adjusting the temperature parameter. This choice isolates the effects of soft labels, avoiding confounding influences from shifting decision boundaries.
24

---

Data Generation
Self Distillation
0-SD
Zp (= (
16 8" argmin L (10", 13")
optimized (Hyperparameters)
Distilition
Bi))
1-SD
t B' argmin C² (ax', (')
Noise
Distilation
[1
ga
1-SD
Distilation
(()))
Activation
2a Mp
ib B
argmin L" L'(w", 15")
(iú B;=,) m.)
V B
Generalzationn Error
Figure 6: A schematic diagram of the t-SD model.
C Replica calculation
This appendix outlines the procedure for deriving Result Jusing the replica method. The process can be summarized in the following steps:
Present the complete statements of the results, including Results 1
Subsection C.1
Introduce the average of the p-th moment of u .... ro denoted F which character- izes macroscopic quantities such as generalization error (Subsection
Show that the joint probability distribution of u' and 3 coincides with the correlation function of system obtained by duplicating the original system, referred the replica system (Subsection C.3 for = 0 and Subsection C.4 for > 0).
Outline the procedure for evaluating F3 by incorporating certain assumptions into the replica variables (Subsections C.5 and C.6).
Derive equations determine the parameters necessary for calculating Ft (Subsections C.7 C.8).
Use these equations to derive Result C.I (Subsection C.9
Finally, present remarks the rigorous proofs in Section C.10
C.1 Full Statement of the Result
Here we give the complete statements of the results to be proved.
Theorem C.I. (Statistics of the T-SD model) Under the proportional asymptotic limit (N, M + 0c.
constrained by M/N E (0, oo)), we have
m + 20)
(11)
Q00
1o
T-1
W
T +
Σ
03
(T21) 1)
(12)
QTT + XT
==0
+ 1TT
1TT
+
T
(13)
VN
25

---

where the parameters satisfy the following equations:
Qte m R
2 Q
Q
nn m
n E1= Q"10"
(12821)
(14)
x+
1e00
400
(15)
Rst
(t 1)
Q"+AA'
m
th
20
(16)
ml
m²
=
X44Qu
(t2 1)
X=
(17)
x
4 Xxr
(t s)
Qat
d==
x y.gerae x
ah?
rir
Ax Ey y.ytrin true . <( 1)21
(18)
Xat
Ax ee Ey yyggiro g [2 [22
By.gtrun (a=gttung(=((]
O.
Here, Q = =Q""), x=
= (xx*) and X = (xxt) are symmetric matrices in [(++))x(++1) and we
introduced the following notations:
(2t)²
2 argmin
+ -ly', oh' + +21))
(19)
22
21
yo
y
(20)
y (3t-1(h² + 2)- (1 1)
110 d 50 + (2y 11)m" + 6000
(21)
h² $ gt + Σ y 2. + (2 1)m² + (t2 1),
Xxe
where & = (5*) E RT+1 ~ N(0,, AQ). = E RT+I N'O, 8). and y, gru E {0, 1} that are
generated as ply, rru = ply p( with pry = pd" (1 (1-p)2 -) ppy # yru
y yrrue)= =8.
C.2
What to calculate
Our primary interest lies in understanding how macroscopic quantities, such as the generalization error, behave under fluctuations in the training data. These macroscopic quantities can generally be expressed, excluding the bias, as functions of
E(ol) ... (w!)p', N
1
(22)
where p", ... ,P' E N. In the asymptotic limit N + co, we expect that Eq. 22 converges with probability to their expected values
lim 1 E(wPy (we)p' lim N
Ep
(wf) almost surely.
(23)
This kind of concentration is called the self-averaging property Mezard et al. (1886) in the context of statistical mechanics. Although it is not obvious that the self-averaging property holds, this property has been proved in several convex optimization problems Thrampoulidis et al. (2015).
Thus, the quantity we need to compute is
F = lim Ep (109pp ... (00)''
(24)
N
000
26

---

where p = (p",...,p"
Since the optimization problem for each time step t in our model is convex, wh and D' are deterministi- cally determined as (t0), Bo) + (ub² B²) + + (ιv', B³) given the data D = { 1/ U4
However, facilitate our analysis, intentionally treat this deterministic process as a stochastic one. Specifically, we model the transition from ( B--1)
1) to (2o", B*) using the following distribution
exp(->< "sp B*)) (s 1.,,,..,t) ,t,, (25)
w. B ~ p(w0, B" 100-1 Be-1,
p D) lim +00
z
where Lg defined in Eq. (2) and
z z²= = / dw dB exp(<>> BB)) (s=1,
(26)
the marginal likelihood (partition function). Observe that in the limit y co, the distribution concentrates on argmin BB L., B*). Similary, the case / = is defined as
exp(->0 B²)) (s= 18..,....1
ú 3 ~ p(w", B D) = lim
(27)
700000
Zo
where
-0= = dw dB exp(_>" Loww",, BO).
(28)
Following the probabilistic interpretation of these dynamics, the quantity defined in Eq. reformulated as
(4 can be
K (wtt) (wr (u)-1)=--1
(109)00) Da
F F =
DD Ep
(29)
where (f(w))) is expectation under the distribution p(w", 1 wn p(w", Bd D) if 0.
1= BB,,,,1100 D) if
and
In summary, our computational task is to calculate the data average of statistical quantities F (Eq. (29) for sequence of random variables (w. ,B) (w.,B") B') - (ιb', B¹) following
Markov process defined by Eqs. (25) and
C.3 One-stage replica method
To grasp the outline of the calculation in the replica method, we first consider the case of \ = 0. For simplicity of notation, in the calculations within Subsections C.3 and C.4 treat w' as one- dimensional, omitting the subscript from wf. However, it is straightforward to extend this to the general N --dimensional case.
The data average of the p-th moment of the solution S0 (Eq. 27 is given by
FØ ED [(00)]]] ED [ du a w pww B D)
P
(30)
lim Ep If du dB w exp exp(->0 (oo'" Lo(w", BO)) 7 20000
(31)
zo
Direct computation of this is challenging due to the presence of the marginal likelihood 7o in the denominator of Eq. (3T). To circumvent this difficulty, we introduce the following identity that holds for any p E
Ep [1) lim lim Wp(nn, 70) -0-0 Ep(n", 7)
(32)
27

---

where
Wp(n",>)) =Fo ErD /// du dB w exp((>><((w) B²1) / (20gz
(33)
E(n", 70)= =
(34)
For the expectation with respect data, resort calculation method known replica method. First, we assume that n' E N and n² p B and express (33) and (34) by using n' reepliccteed variables
w1, wi..... w n²
= ED / dwo dB wf... wff exp
No Σ
=ED
-70
(ww
B
(35)
on=1
u = =fdw dwo dB w/ ... up
ED
cxp
L(w no:
B
(36)
Gn=1
E(n", = ED / /dw" dB
exp
-70
L(w
B
(37)
o0=1
= f dw dB ED
no
exp
L(
B
(38)
where w and 8dd are shorthands for wi. w and BT Bq,...
respectively. These expression can be regarded as the expectation the p-body correlation of
Ba n
indicates that (n / 70) replica variables obeying the joint distribution
n
)]
p(w0, Bd) = lim 70 700 E(no, 70)
FD
exp
0
L( Bao
(39)
The primary challenge in our initial calculations was necessity of averaging over the data, which significantly complicated the process. However, by employing the replica method, shown in Eq. (39). we effectively incorporate the data average into the probability distribution of the variables, thereby enabling us to derive their statistical properties. The system that follows the probability distribution given by Eq. (39) is called replica system.
C.4 Extention to multi-stage replica method
While the preceding analysis follows the conventional replica method prescription, the current scenario presents a unique challenge: the dependence of each estimator ú' on its predecessor u'-1 significantly increases the complexity of the problem. To address this issue, we employ an innovative approach that involves the recursive application of the replica trick at each stage the process.
To illustrate this approach, let us consider how Eq. (39 evolves along step t under our recursive methodology. First we define the following recursive function:
fe-1(w²-1) (w0))²
(40)
fo(w²) = (o+1( (wett))r
10<s<t-a 1).
(41)
Then, what we want to calculate (Eq. 22 is expressed as
ED So (w0
(42)
"This assumption is not consistent with taking the limit n² remains necessary to verify whether results obtained under the condition n" > p can correctly extrapolated to the regime where nt" + O. While the mathematical validity of this analytic continuation has not yet been rigorously proven, no counterexamples to its validity have been identified so far, at least in cases where the optimization problem determining the parameters 6 convex.
performed in Eq. Therefore,
28

---

On the other hand, assuming that n., .....
.....'' n'' E N, n² > p pand n. ,n > 1, one can deduce
st-1(10-11= (u
(43)
{ du' dB' w² (exp(=>> B²)) }d (20)ri-
(44)
- =ʃ dw' dB' wf....... ....f exp
n²¹
-7 Σ
Li(waa Bla))
(45)
1--1(w0~1) = (u("")(wo)pr
(46)
= { dw dB (a(w")("))'' exp(->" (xp(->><o(w",, BB)) {" (2o)ne-p'
(47)
ff dw- dB* fa(w))uf uigo exp -7x Σ Lo(wg, BBa) (1 (s8st)) (48)
n"
F ((w)))
(49)
)20 x0
I du dB (w"))) exp(~>" C, BB)) }
(20)
(50)
= =f dw dB fo(w))of ... wgo exp -70 Σ Lo(w no: Buo
n
(51)
ag
where w' and B' are shorthands for wf: w and Bi...., B respectively. These equations demonstrate that when considering the parameters at time s while keeping the parameters time
1 fixed, the distribution of replica variables w" and B at time s depends on the first replica variables wi and b7- at time 1. By recursively substituting Eqs. (45) and (48) into (5)). we have
(n. 7)
Fi lim
lim
(52)
-000 n 00
-00
E'(n, 7)
00
where n (re.... .n,,,n)), (70, ,7*) and
(T = dwo /dw"...dud' dwo dB"...dB' (luf ..feef)) x (w/-lpp 1 x ... x (woteo)
(53)
Lo Bis)) )
x ED
exp Σ 7
(54)
8=1
a.=l
Yn = ED (Z)gn x (21-1gn' .... (z
(55)
=ED E / /duo".. dw' dB ....B'' exp
Lo B
(56)
= -f /dd"....ur dB ...... Ep cxp
(w B
(57)
4,=1
29

---

where D(wa, B²)) represents the loss when the first replica at t 1 is used as pseudo-label for training. i...
E(w'a, Y :x)) + X²
1 11?
(58)
2
(g(10 109
Fp + B1 b²=)
1/ BB
(1 0)
(59)
N
-0 Uu 1p
(60)
These expression indicates that W (n, 7) / 7) can be regarded as the expectation of the (p" + -body correlation of replica variables obeying the joint distribution
p(w0, , B' ,
,B) = lim guin +00 B'(n, 7)
ED
exp
L B²
8=1
a,=1
(61)
Ultimately, our problem reduces to calculating the statistical properties of the replica variables wo .... wt that follow the distribution given by Eq. (61). Following the standard prescription for analysis in the asymptotic limit, it is crucial to investigate the behavior of the replica partition function Eq. (57) for this analysis.
C.5 The calculation of the replica partition function
From this subsection, we recover the subscript of dimension in wa Our next step is to calculate data average in the replica partition function (Eq. (57p. To achieve this, we first calculate the partition function for finite n², ... nn', and then consider the limit n ;... ,n² 0.
First, we define the lincarly transformed variable u'
(ui, " u/g ... u as
u
Σ
wizi
(62)
N
where is standard normal random variavle defined in Eq. and w (wf...".. w " w )T. Then, u ((u))T. ... (u²)T
also follows Gaussian
distribution, with the mean and covariance given by
Ep wall = O, ED [ua veal = AQat
(63)
where Qzy is defined as
Q wa w/
(64)
Then, the partition function of the replica distribution Eq. (7 can be expressed as
30

---

(-
(n, 7) / w0 wo.....''' ...'''"'' dB dBB...'"' B² exp
- Σ
Σ | 11
M
(wi 2x
E
pp V. exp
ΣΣ
+ BBy
(65)
VN
= dQ dm /
/wo w0 wo.....' w JABo.. BT
I 8(NQai
w w'
NN - w/o :0)
41
Σ X
exp
M
w'a IH
x
T Ba Mas thee exp
ΣΣ
x'
Bi.
(66)
VN
where fd is an integration over {0at he and f dm is an integration over hsessssssss Using the following integral representations of the Dirac delta function
Qat 2
5(NQ2, wa, w'y) = dô exp
(NQ2,,, waa w':)
(67)
fdin, 8(NmL, w/y/ driaa exp(-mi, (Nm's w
(68)
One can find that Eq. 666 can be separated into three terms: (1) an interaction term G,, which shows the interaction between order parameters (parameters without hat) and conjugate parameters (parameters with hat); (2) an entropic term Gs. which scales N; and (3) an energy term Gr, which scales as M. Based on these observations, calculating each component the partition function yields the following equation:
/ dQ dm dm dB .B. (GI) (Gs) (GE))
=" =mn (n, 7)
(69)
where
( Σ ΣΣ Sat Qut + ΣΣ ma, Mas
G cxp
(70)
Gs = dw ... ... exp
ΣΣ
2 (w!)² + +ΣΣ
m we + ΣΣ
Qct wa, wi
a
apcs
(71)
GE Eyne, En exp
-00 Σ Eyy ((2 1)m², u + B
BBo)) )
-
Σ
(2 1)ml 1-1 ui + B(-1) 1
m2 ((2
1)m', + 2h, + By,))
1=1
(72)
'The integrations in Eqs.
67 and
68
performed along the imaginary axis.
31

---

In the asimptotic limit (N, .M + x0,a = M/N (1)), Eq. 69
can evaluated using the saddle
point method. Using this technique, the partition function (Eq. 69 is evaluated as
Ex En 7) = exp N max (Q. m. Q m. B
B³] ]]
(73)
= exxp N (Q" m ,,, mn" B }]
(74)
where
(QQ, m, Q m. Bº, ,B)) log G + log Gs + a log Gg.
(75)
and
,",m", Q' m B
B T
VQQ m. Q m. By, argmax Q.m. Q .o...
B²).
(76)
The saddle point equations to minimize VQQ, m, Q m, B B²) are given by
1 aGs
aGs Qa = a Ge aQ
dGg my = = GE dm!.
aGg
ma. = Gs dri
Gs oQat ase
(77)
from the saddle point conditions of Q mar Qat and m respectively, and
1 0Gg
0,
(78)
Gg a 0t
from the saddle point condition of B In the limit where n², ... + 0, we have GE 1 and G3 - 1. Therefore, Eq. become
77 specifically
Qaye Ew [wa wiy]
(79)
maa = Ew (w'al
(80)
T
Qat 2a aQate
E Eu.y. me exp
Σ
(of
(81)
1=0
a
T
AaEE..g.
exp
2-1
(82)
Dus u
t=0
41
①
my
E Eu.yy mr exp
Σ
v
(83)
dmL,
1=U
T
a.u,y.y y.e dut
exp
Σ
Σ
v
(84)
1=0
a2
where we assumed the random variables (wwa raa in Eqs. distribution:
79 and
81
follow the following
p(w) x exp ΣΣ
X' (win)) 2 ΣΣ maa wa, + Σ
Σ
Qate wā, waa
(85)
2
a
and
0
1 = ((2 - 1)ma, + w/, + +Ba)
(1 2 0)
(86)
"1
1
(87)
C.6 RS assamption
From Eqs. 61) and 74 the solutions of saddle point equations related to the statistical properties of the replica variables, i.e..
m
N
E W
Qut
= N
1
Σ E W Wua w(.a)
(88)
32

---

in n n22 + limit, with the expectation taken over the probability distribution defined by
Eq. 6I). From the fact that the p-body correlation functions between replicas correspond the p-th moments Fp in the original Markov process, we obtain
Fe = Ep [0 = m
(89)
F Felo.1) = ED [ wi]
Qaya
(90)
where e(!) and els,t) are T T-dimensional vectors defined as follows:
{
if 6 otherwise,
( (21.e,,..... ,g, er), where e,
and
if s ( (e1, eT). where et 0, otherwise,
{1
and (a., ax) (1, if = = T, and (0s,a)) = (1,1) otherwise. Equation (00) holds for an arbitrary i since the integrand of Eq. (69) can be written independently of
However, the replica parameters (m Qai mi and Qxt) are ill-defined in the limit as n " n²" To further advance our calculations and obtain well-defined quantities, we invoke the replica symmetry (RS) assumption, which posits symmetry under permutation between different replicas, i.e,
m m
(91)
Q" x"r
Qu
? daice
(92)
Qat = Qtt + xe
da,1 (s
(93)
za
my = m'm
(94)
Qu, = (1¹)² Xth q'Q""aaaa
2x))"x²
(95)
Qui xxxx -²² 22220 (t
(96)
By = b'
(97)
where dab is the Kronecker delta function. Using this parameterization, one can deduce
m = [
(98)
Qat LE W
(99)
for an arbitrary i. Higher-order moments can immediately determined to zero from Eq. (69). given that the distribution in Eq. (61) indicates that w wT follow a Gaussian distribution. Although the RS assumption not mathematically rigorous in general, has been empirically validated in many practical scenarios, particularly in convex optimization problems. To date, there are no known examples where the RS assumption leads to incorrect predictions convex settings.
C.7 Saddle point equations: order parameters
Our next step is to derive the saddle-point equations for the order parameters (m' Q*t and x²*) under the RS assamption from the ones for replica parameters (m Qat and X.b). First, substitution of the
33

---

RS assumption in eq.
85 yields
p(w) x cxp
Σ 7t
Q"
2
+
X
Σ ( 2
+ Σ 7
Σ
a
+
Σ
uq
Σ
w
41
de
(ac))
Re
+ 2
Σ
X
9/ah
Σ
Wa
72
Σ
W
(100)
a
at
= /
Q" +
D exp
Σ
(wa)
2
+
Σ
7
m
w +
2' wi Σ
7
w
2
a1
(e<()
+
Σ
7/
Σ
Wh
x st
gt
(101)
st.
a.
=
- D exp
-71 Σ
Q A
(waa)
2
ni + Σ Vx
0s 5³
wao ]
D exp
-7"
Σ
Qth
+ Xa
(wax) )²=
mn' + Σ VX 5"
Σ
]
2
a1
(102)
where vx Gaussian measure.
is the 8, i element of the cholesky decomposition of the matrix X, and D is the normal
Now we define following notations:
(w²))
-(mh0 (m= E. VX vx(=()) )))=u uh (t= 0)
flwni wf....,.f$")
+x 2
(w²g) (mn"' + Σ, vX 5" -E' (n' (121) 1)
(103)
m + vx- (t=
w'(w (., , = argmin w/ ,,,,
1-1 ('."
2 100
E, vX Σi= 6'1mf
w
(t 1)
2' x'
(104)
w¹ 1 (wo ww wf) = w (wf wf,
w = wt (w., w (w))
(w...... ..... ,., 2)
(105)
w²))
(106)
Then, the saddle- -point equations of order parameters are simplified as (in the n'..... n
0, 72, ... If + 0 limit)
m
E [
(107)
= / / DE
II / dw exp(->p go) S dw' w/ exp(->'ff ʃ dw' exp(->² 1*)
1-1
(108)
==
= & [w:]]
(109)
Qt Bllua w
(110)
= / D II
1-1
/ d'w' exp(- p4) I dwo w (3p(--9(-))
S dwe expl -20
1=0
II dwo' exp(->' I dww' w' exp(-ga11
x
(111)
I dwt exp(->28")
1=++1
= =E [w
(s t)
(112)
34

---

Q" E [wu w²y
(113)
/ DE II
dw" exp(->"f") ( (Idw' w² expl-
( dw' exp( ft)
(114)
==U
BB [(
(115)
x" = 7 w we, - wfw''
(116)
=>² 7"' /
5-1
d'o' l4) [ dwf wi exp(->" (- (wwf))
DE
dw' expl->7 (((=)) w (w), wef)
1=0
S dw w exp( - (((")) f duuf wt (wt . w0s)
(117)
dd exp(- exp(->"fe))
/ D II
d S dwo exp( ff (wi)) (w), w S dw exp( --pp- f2)
dw exp( exp(->!(")
(118)
dmn
I=0
dw
(s << t)
(119)
drit"
x" y TIE (wa.) 2² w/a wē,)
(120)
= /
1-1
ʃ dw' (wt)² "exp(=2'f dw exp(->2*
(
{ dw' wt exp(-
27
D
II
dw expl-x" (e)
den expl exp(--*f1)
==
(121)
/ DE II
L-1
θ f dw' wt exp(->"
dw" exp(->8'*)
(122)
Omn f f exp(->"1
Ow:
Eg
(123)
am!
By introducing the helper variable Rst = Bg (VX 18 (onat] these equations yields
for simplicity, the explicit calculation of
200+10 800 (t=
1
ast = E
18 Vx
E
(124)
Tuex (x -i- (((1pe) (t21) 1)
(m + +ROI) (s= =0 0)
400
Q"h = E
(125)
m²m² + Rst -E Ql=QUt) (s2 1)
+x
+00 m
(t =0
m' = E [0]]
gu m"
(mnt -t- (+)m+) (t21)
(126)
dw
(s=t)
x 8t
E
dm
Ox EiznQu x (s <))
(127)
C.8 Saddle point equations: conjugate parameters
Similar to the previous section, we now derive the saddle-point equations for the conjugate parameters (mr ghh x based on the RS assumption.
35

---

The covariance matrix of the Gaussian variables u (Eq.
63
is rewritten as
ED 1u Wea = A
3000 Qt
x e
+
òase,
(128)
yp
X
ED 4 Wey =
@at
Da,1
(s < ().
(129)
Under these conditions, we can introduce the random variable with an equivalent distribution as follows:
AtrSo
(130)
1=0
==0
where Ast are the cholesky decomposition of the covariance matrix of u, i.e., E, Air = AQ" z4 VAx" and S < NO 1) are independent standard normal random variables.
Following the same procedure as the previous section, taking the limit of ot -> in order, the expectation calculation is transformed into the solution of the optimization problem, and finally the following relationship is obtained:
d
Qai
Eg
E
dhh
(131)
m'
Axtt By.girue
|2y - 1)24
(132)
8at axas E yrun [[x]=A]]
(133)
By,,truc & [
(134)
where z is the solution of the optimization problem follows:
(=0)²
argmin
00 + E(z h2 =9)
(135)
h AooEi
(2y 1)m² +
(136)
(zt)²
2 argmin
+ ((0 (1¹ + 1))
(h (1' (M++)1) (1SIST)
(137)
2
t-1
AutC + Σ
Bu + (2y 1)m² +0 (1SLST).
(138)
C.9 Derivation of Result C1
To summarize the results obtained far, have derived the first- and second- statistics of the estimator w/ which are determined by the constants m' and Q (Eqs. (98) and (99)). Furthermore, we have shown that these constants can be computed by solving the saddle-point equations defined in Eqs. (124)- and Eqs. (131) 134). As a representation of the distribution of oi that satisfies all the conditions for the integer moments,
express it as
6 d
1
m + )))
(139)
200 + 1o
1-1
1
m +
Σ
20
(t tt1 1)
(140)
Qu
+
and this representation is unique up equivalent forms. Furthermore, by proceeding with similar calculations while taking into account the correlation with the data, Eqs. (155) and (137) yield
x
+
B
h +
(141)
N
for the pre-activation distribution.
36

---

C.10 Remarks on Rigorous Proofs
Beyond the replica method, a fully rigorous proof in our setting remains open. The main difficulty is the temporal correlation across multiple SD stages. AMP/state-evolution techniques Donoho al. [2009]. . Liu and Ma [2024) have been applied successfully to handle such correlations, but basically only for first-order iterative algorithms, and thus do not directly extend to multi-stage SD. CGM'T-based approaches [Thrampoulidis et al. [015] could in principle be adapted, but they are typically used in on-line settings where new data are introduced at each stage. Developing tools that combine these techniques rigorously capture the multi-stage dynamics would be an interesting direction for future work.
C.11 Remarks on Numerical Calculations
In numerical calculations of the saddle point equations Eqs. 124 127 and Eqs. (33))((34)) in replica method, the order parameters are usually obtained through iterative method. The procedure starts from an initial guess of the order parameters. These values substituted into the left-hand side of the saddle-point equations, and the resulting right-hand side gives an updated estimate of the parameters. This process is repeated until the values converge within a chosen tolerance. In this way, the order parameters are determined as the fixed point of the equations.
Iterative Method
Discrete Expectation When the theoretical formulation includes averages over discrete random variables such the class label and the label-flip noise, each possible configuration is evaluated sepa- rately. the present model, the class label takes values O or 1, and the label may be either flipped or unflipped. Hence, there are four possible combinations: (y, Vonee) (0. 0), (0, 1), (1, 0), (1, 1). For generic quantity f(y, ), the expectation appearing in the saddle-point equations is computed as
Ey..om (v: Mrsee] = p(1 (0, 0) + (1 - p) 1) + p(1 - 0) 0) + (1 - p) (1 1). (142)
In the numerical implementation, the value of (y, Mtme)) is computed and maintained for all four cases.
Sequential Update in Time The calculation also proceeds sequentially with respect to the time index t. The result at step & depends only the quantities determined earlier steps from 0 - t 1. Therefore, the parameters can be updated step by step, starting from = 0. This recursive approach makes it straightforward to implement the computation and to monitor convergence at each stage.
We describe the implementation of evaluating the total derivatives appearing on the left-hand side of Eq. [13 during the numerical calculation of the saddle point equations.
Evaluation of Total Derivatives in Eq. 131
For notational simplicity. we define the partial derivatives of the loss function as
al(a (h' + 1) (h' + +²)) E, al(ar(' + 2 a (h + 2²))
(143)
ah
th--1
Introducing the function
2¹
F' 22-1111 /hhn1111 "1111) = Ax + E.
(144)
we have the condition
Ft 2
0.
(145)
By employing the implicit function theorem, obtain
OF dzf 0r dh dzt dh
OF dh dh-1 dh Oh dh d=t-1 dh 0.
OF dzf-1
(146)
since z represents the solution of the optimization problem given by Eq. 137) Solving for obtain
dzf
we
dæ = a' dh dhe-1
dh + ccc dz!-!
dh + b²
(147)
dh
dh
37

---

where the coefficients a², by, are given by
OF
8"
d²
(148)
0PL
8" +1/
8F
0h
(149)
0F1
8" *11x((x"1)
82
8Fd
0=.-T
e" + +1/(Ax = b¹.
(150)
aF
0zt
Furthermore, using the following relations,
dh
rt dz
(151)
d/
x dhi
r<t
dzf
x dz
(152)
dh
x dh
one can iteratively compute for fixed f, as illustrated in Algorithm which presents a single iteration step of the update procedure. The full derivative is obtained by repeatedly applying this update until convergence.
d=
Algorithm 1: UpdateQhatColumm(() Self-consistent update of column (past columns are fixed)
Input: Results up time
Fx
lat-
h
1it-1
G(srr]
d= dh
H(s,r)
dh dh
(sSr
(153)
Output: Results at time
G|s,t]
dzl dh
H[s
dh th
(s t), Q[s, 5t}
(154)
Step 1: Calculate d²
Cy
a' +
(155)
C + 1/(Ax
fi,--1
b th + +//
(156)
d' 4+b' b'
(157)
Step 2: Calculate [t, s]
for s <- to do
H[s,t]
X x
H[t. +-1
Step Calculate G[t, s]
for s 1- to do
G[s,t] + a' H[s [[]ss]]++ H' [d[sst
10 G[t,t] < a
Step 5: Update Q1,, s]:
12 for s - 1 to
Quew [s, - [s
13
38

---

D Theoretical and Experimental Validation
In this appendix, we present evidence demonstrating the strong agreement between theoretical predic- tions derived from the replica method and numerical experiments for the linear t-SD model. Figure compares the generalization error, weight distribution, and pre-activation distribution obtained from replica method with those from numerical experiments, revealing remarkable consistency between the two approaches.
A
B
C
Generalzztion enror
0.4
0.8-
PDF
2-
02-
-0.2
0.6 0.8
Weight
Pro activation
Figure 7: Comparison of theoretical predictions derived from replica method and numerical ex- periments for the linear t-SD model statistics. (A) Generalization error derived by the replica method (solid lines) and numerical simulations (dots with error bars). (B) Distributions of opti- mal weights derived by the replica method (solid lines) and their empirical distributions obtained from single experiment (histograms). (C) Pre-activee rreaactvvttion distributions predicted by theory (solid lines) and empirically observed from single experiment (histograms). Parameters for (A-C): 0.4, A = 0.6, 0 = 0.2, (A1, A2, Å3. X4) (1.5, 0.5, 2.0, 1.0), (31, 3a) (0.8, 1.2, 1.0); (B, C) a 3.0. Numerical experiments: (A) N 10 (Error bars represent the standard error of the
mean over 20 trials per point.); (B, C) N = 10a.
E Optimization of the hyper parameters
The above results describe the statistical properties of the estimators { B he2o and the gener- alization error for a fixed hyperparameters {Xi (X')20. {3 J1>1- In order to find the optimal hyper parameters, used the Nelder-Mead (NM) method Dixit and Rackauckas 2023). which is versatile black-box optimization algorithm. At each optimization stage in NM, we numerically solve the set of equations in Result C.1 which can be efficiently solved using simple fixed point iteration to evaluate the generalization error
F Exact results for the linear i-SD model
In this appendix, we present several simplified analyses of the linear t-SD model and conclude by proving a generalized version of Result 2
F1 Integrated saddle point equations in the linear t-SD model
In the case of the mean squared envor loss and the linear activation function, the saddle point equations for the conjugate variables and b' integrable. For some algebraic manipulations, we have the following equations:
Ax
2 2 + Ax
(2n¹ 1²' -1)
(158)
1
3 2
(B(h' + + +1) (+21)
(159)
39

---

A
Q"
(160)
2+ Ax
A
1-1 Σ
-x (s<t)
Q
81-1 8t-1,1
1
Qal
(161)
2 4x
1=s
(162)
2ap
m 0
(2(1- 8) - (m + bb) - 1)
(163)
2+ Ax
m²
20
1-1
3 - x x² ))1)"))))
+ +/3 p(mmm +b--)) -p(m² -00'm+ +b))
(1565T)
2 + Ax
2
0==0
(164)
8
2a
QU1Q" -m mmm) m')
(165)
2 A
A Ax it
1=0
8
-38-1
Q 1.0.ii-__.)maxx(-__1
Σ
x h-- xx mm= mm--mm))
(166)
2 + Ax
1=0
1=0
+
QU'QUe
Σ
2 100mmmmmm m²m!)
(167)
1=0
4x
80 2 A 00
10 + 0- -20)) - {pp11 - (m +b0) (1 -1- (1-m) +b
(168)
Ax"
Σ
1
( x =20
(169)
2 + A
La=0 X
+ + {p(1 (m
1-1 + +b'=1) + (1- (1-)))((mmmm fp(1 0) + b²) (1 (1-)))(-mm 60) (1SIST)
(170)
(171)
8 = 2(p + 2p0) (2p - 1)m² -
(172)
b' = 8 - (12 - (2p-1)mi (1
(173)
where we introduced the auxiliary variable ft = Ep ED(y [v>²2]] for simplicity.
F22
Case of = 1/2 and 1o X - 0c
One can solve the saddle point equations explicitly in some specific cases. For example, in the case of
1/2 and X A + oc, the explicit form of the geneliztion error is given by following proposition.
Proposition F.1. In the linear t-SD model with 1/2 and Xo, A + co, the generalization errors 0 and are given by
/ 20)
8 = H
(174)
/ (A a -20)2)
a +a+ Aa) -29)
El H
/A2 + 3a 1)A3a a² (A2 + 5a +3) A(2a + 3)a + a²) (1 2002
(175)
In particular, g0 = 0 and 1+1 <
40

---

F33
Case of p 1/2, _A= + oo and T + 0c
Another specific solvable case is 2² ..... A + 00 and T + cc. For ease of reference, we first restate the generalized version of Result 2 Las R
We then proceed to its proof.
Result 3. (The generalization error at + oo) For an arbitrary choice of the set of the temperature parameters {3t Je2o. the generalization error of the linear I-SD model with = 0.5, ,", ,... X² +
oo and - OC given by
0.5
(a < A2)
lim Ed 1-000
(176)
a-4² H
2 Aa)
A(++A)
Under these conditions, equations 172 and A X" = X, e 1/X, and 7
yield bo ..... , For simplicity, we set 7 without loss of generality. The scaling of
each parameter with respect to e and
7
m' (7
m' 0 (0(")))
(177)
Qat 0())))) (
2tr = 0((-==))0(--) (8 < (), Q" =0= 0(1)(1)
(178)
x= 0 ()) Rst (0()+)-))(
X (0 (+--2)
(179)
(180)
Based on this scaling, we rescale each variable as
m² + c
m' - e² 1 1-1
(181)
Qtt + e ettgttt-22) 2Qttt xt - c xee
Q=t e 1- 200=022 (s (s< t), Q - Q²
(182)
x 2
(183)
1xth + - t 2 patt
(184)
Taking the limit as + 0, obtain the following simplified recurrence relations:
Da
x =
(185)
2
Q"r Aa
Q-11it
Aa
(186)
2
2
A(1 + +a)
1-8-2
A
Qet_
(12822) 2)
(187)
2
4
m
m
Q (1 20)
(188)
2
m
AA + +a)m
m =
(A + a Da)m
(189)
2
2
A
Aa
1-1--1
(-1
Aa
-6
m
1²
m (t
(190)
2
2
2
2
==0
m'
a
A
(1
(1+a)
1-2 1-2
Σ
2 (1+))
m² + m + Da m1
(1 2 2)
(191)
R00 x0s
(192)
ast emin(e.d),max((11)
Σ Qt Rs! (t 1)
(193)
Qº = =m² ml + Re00
(194)
Qat mmm + Re
a A
(1 +a)
2-28-2
Σ
(1
Aa
(1 2 1)
4 2
2
2
1=
(195)
41

---

Aa
800
(196)
4
201
A
1 11-1x x00 -f =1 28) 20)m
(t 1)
(197)
2
X
A
mmin -- max (1)1) Q3
Σ 1:a- 8hr
m m==mm11p 'nn)) (12 1). (198)
We now consider the solution to these recurrence relations for sufficiently large (1 > 1). Let us propose trial solution of the form m' = cM', nn' = cLM's where M > A(1+a)/² and c is a constant depending on the initial condition 0. Note that in the left-hand side denotes the step number, while in the right-hand side is exponent. We have:
Aa
1-1
Aa
Σ
m
LM O(()
(199)
t-2
1-2
(1 +
+a)
m² = Σ
2 (1)
M + 011
(200)
From equations (990) and 191 we obtain the solution satisfying the condition M A(1 + a2/2
m' = + cc(=+AAA)MW A)M
(201)
m² cM
(202)
where
1
M
22 a (2A
(203)
Next, we consider solutions of the form Q = eqtatt RRt = er.att,, 8"= cxx for
=c²==x
f >>>>> 1. Substituting these into (933, we obtain
A1+
(204)
q (1+ (1 +x)
(205)
A
x a
1+
(1+ A 79
(206)
Solving these equations yields
A(1 + A)²
+ A)" a A)
A(1+A) A)
x=
(207)
2
A
2
Consequently, the generalization error as 00 is given by
m
a a- A
+ H
A(a A) V
(208)
VAQU
However, when < 2 this solution becomes inappropriate as Q" < 0. In this case, the scale N of Q²t = cag.eet satisfies N M, and the generalization error becomes (()) = 0.5. This completes the proof of Result
Result B reveals phase transition phenomena a 22 A generalization error 0.5 means that the performance of t-SD is equivalent to random guessing; hence we refer to the phase < A as the performance collapse phase. The independence from the choice the temperature is natural since it only affects on the scale of the weight in the linear t-SD model. In Figure 8 dependence of El on a and A / - oc shown, with phase transition boundary represented by dashed line. The generalization error at guessing, but remains higher than the optimal error; hence, we refer to the phase a 2 A as the intermediate performance phase.
for 2 A is below 0.5, indicating performance better than random
42

---

5
0.5
4
0..4
Generalization Error
Performance Collapse
3
0.3
1-
Intermediate Performance
0.1
0.0
10
10
10a
Figure 8: Theoretical prediction of generalization error for the linear t-SD model with ,",... A - 00 and oo with the phase transition boundary indicated by the dashed line.
G Experimental Details
This appendix provides the detailed experimental settings employed in Section S All experiments were executed on CPU workers equipped with an AMD EPYC 9654 processor and 512 GB of main memory. Each rum required approximately two hours of wall-clock time using all available cores.
G.1 Data and backbone selection
We consider the binary 'cat vs. dog' subset of CIFAR-10 Krizhessky et al. (2099] (licensed under the MIT License) and employ two deep backhones, ResNet- 18 and ResNet-50, pre-trained on ImageNet.
G.2 Feature Extraction
Each CIFAR-10 10 image is first resized and normalized to match the preprocessing used during ImageNet training. We then remove the backbone's final classification layer and take the output of the penultimate layer as a fixed embedding of dimension N 512 for ResNet-18 or N 2048 for ResNet-50 maintainers and contributors] (2016) (licensed under BSD 3-Clause "New" License). All embeddings and their clean labels are saved for downstream use.
G.3
Label noise injection
To simulate noisy supervision, we flip each training label independently with probability 0. Test labels remain untouched.
G.4 Training subset sampling
From the pool noisy embeddings, we uniformly draw M samples (with fixed class balance when desired) to form the actual training set used in the SD experiments.
G.5 Self-distillation and hyperparameter tuning.
Using those M examples, we perform the logistic /-SD procedure defined in Section The key hyperparameters, X and 3 are selected by minimizing the estimated generalization error the test embeddings via Bayesian optimization. Concretely, we model the test error as a function of (A, (9) with Gaussian proccess surrogate and optimize its expected improvement.
43

---

H
Additional experiments
H.1
Self-distillation on noiseless dataset
We repeat our 1-SD experiments with no label noise (0 = 0) to isolate the denoising effect from the dark knowledge effect. Since there is no label noise, any gain in generalization must arise solely from the teacher's soft outputs. As shown in Figure 9 no meaningful improvement is observed under realistic settings.
These findings validate the hypothesis of Section in linear models under the Gaussian mixture model, dark knowledge alone yields only marginal benefit. The dominant mechanism by which SD enhances performance may be denoising, not the transmission of refined probability information even in more realistic scenarios.
ResNet18
ResNet50
2:50
+ 0:50
Generalization Emrr
Gereralization Error
0:152
1:50
0:12
0.102
10 12
12" 10
10"*
Figure 9: Comparison of the optimal generalization error of the logistic O-SD model, 1-SD model and 1-SD model using hard pseudo labels for CIFAR-10 dog versus cat classification using pretrained ResNet- 18 (N- 512) and ResNet-50 (N 2048) feature representations. Parameters: 0.0. Error bars represent the standard error of the mean over 10 trials per point.
H.2 Optimal soft labels for self-distillation
natural question is whether the soft labels produced by the optimal 1-SD teacher simply mirror the probabilities predicted by the optimal O-SD model, or whether they differ in a systematic way. Figure 10 shows, for several randomly chosen training samples, the ground-truth label, the noisy observed label, the predicted probability under optimal O-SD, the pseudo-label assigned by the optimal 1-SD teacher, and the student's prediction under optimal 1-SD. The optimal teacher consistently issues more extreme confidence scores than the base model. This observation suggests that the most effective labels for student improvement can deviate from the model's optimal activations.
I Further remarks on limitations and future works
In this appendix, we outline several promising avenues that address the limitations of our current study and extend its insights.
Our theoretical analysis assumes purely linear models under Caussian-mixture noise and relies on asymptotic N, M - oc formulas, so its accuracy may degrade on finite-sample, non-linear settings. Likewise, although on larger or more complex vision tasks, Addressing these gaps suggests following several natural directions for further work.
CIFAR-10 probes demonstrate feasibility, they do guarantee performance
I.1 Extension to anisotropic data distributions
In anisotropic settings, the impact of SD inherently inreetionn-deeeddent tends to amplify useful signals along high-variance directions, while potentially neglecting even distorting low-veriance ones. As a consequence, the overall benefit of SD may become larger when the task-relevant information aligns with principal components, but smaller when crucial information lies in weak directions. Understanding this interplay between pseudo-label dynamics and the spectral structure of the data covariance is important direction for future work.
44

---

cat (clean)
cat (cieen)
cat iclean)
cat (clean)
cat (clean)
cat (cieen)
cat Iclean)
cat (cleen)
true-1 cbs-1
trac=1 c0s-1
true-1 ats=1
true- oo5-1
true=1 abs-1
true-1 obs-1
true-1 ots-1
true- obs-1 ore=0.501400
one 500498
onead 501990
oneao 502612
one 49 7548
uneso 499533 9a-0-499911
one50.501958 1ea-0.582755
uneaD 498572
Na-00.52121 01521211
tea-0.50
team0 0.6005553
tea-0-409422
tea-0-433370
bea-0.56005]
250022
stu-0.501233
staw 5.518787
stu stu-0.98811
0399994
stu
5t3-0.439389
stu 5tu-00500335
(clean)
(cieen)
cat Iclean)
(clean)
(clean)
cat (cleen)
Iclean)
(cleen)
buee1
550053
are=0.496805
0,500800
mee-0.501190
ane=0.499347
one-0.501.90
mee0..50agg
one=0.450973
m,500002000
team0.5 ..536665
tea 0,550620
0a4-0.472102
tea 580359
team0.666 tea-0.662678
tea-0.498p
50000.50004
stu--14499880
stu=0.50n333
stu #u=-0.500322
stu-0.499770
50u00.501188
4tu-0.501785
stu-0.499915
cat Inolsy)
cat (nolpy)
cac (nolsy)
cat (nolsy)
cat (no sy)
cat (nolsy)
cas Inolsy!
cat Inolsy)
true-1 obs-o
true-1 obs-0
true1 obs-0
true-] obs-0
true=1 obs-0
true- obs-0
truem obs-q
true=1 obe-0
one-0 500264
one-a DCc-0.4657as
ann 500006
one-0.502 392
ana-o 400153
one
mee-D. 00-505001520
anc-0.500000
tea:0 510757
tes 426975 419-04
tee:00 5.507225
tea 600509 stu-0.501137
979-0,479884 490853
5:54400 stu-0.5001
lea:00 564587 ftu00.5001
stu-0.500533
0.4996
stu-0. stu-0.500259
(noly))
(noloy)
can inolsy)
(nalsy)
(no
cat Inoloy)
Ineley!
(noisy)
true=1
true=1
true=1 one-0.500155 500155
true-1
true=1
true=1 one-0.500252
501372
502363
ona-0 4.9336
one-0.50044.
ene-0. 501116
tea 558557744
tes 033-0.599297
0.500458
tea 955-0,994877
tea 073977
518895
tea 54733333 stu=0,500599
155-0.511425
stum0 500564
stum0,50115
stumo, 500206
3um00.50000 0,5010997
9t9-0.499978
stum0, 5.000099
stum0.499911
dag (clean)
dog (clean)
dog Iclean)
dog (clean)
dog (clean)
dog (clean)
dag (clean)
dog (clean)
true=0 cbs=00
true-0 obs=0
true=0 abs-d
true obs=00
true=0 obs=0
true obs-ng
true-0 obs=D D
true obs=0
one-0 500909 0
one
mne-0-49662 0.49607
0ne-0-197900
one-0/996224
one-0. one-0.197666 09786658
098-0-498033
one-0.19619)
to3=0-538708
toa-0 480102
0.1501415001
tsa-0 441625 499349
4401644
416921
toa-0.473340
500339
stum0, 499373
stum0 499695
stum0. 0.4996
stum0, 498805
4u911499114
stum0 4.9933
dog (clean)
dog (clean)
dog Iclean)
dog (clean)
dog (clean)
dog (clean)
dog (clean)
dog (clean)
true-0
true=0 dbs-0
true- obs-0
true=0
true-0
true=0 obs-
true=0 obs-D
0 obs-0
499428
499
497794
unem0.49975
4.9200 4244447
tea-O A75530 stu=0 439944
450161 stu=0.499334
433622 439448
tea-0.477 474422266
A07252 stu=0.498835
5355255 stu 0,502331
tea-0. 356106 #tu=0.499064
0,499744
stu 0949304
dog (noisy)
dog (noisy)
dog (noisy!
dog (ncisy)
dog (noisy)
dog (noisy)
dog (noisy)
dog Incisy)
true-0 ebs-
true-0 oo2-1
true- abs-1
true-0 obi-1
truc-0 cbs-1
true- obs-1
true-3 obs-
true-0 obs-1
0.4900
499963
one=0 one=0-49002 498029
one=0 498524
one=0 one=0.500340 500340
one=0 4882331
mee=0 0.50200
one=0 498733 tea-0.4422
tea-0.100 -82206
tea-0 4564 30
046-0416552
(as----57399 293
tea-0.5 0a-0.514600
6ea-D-467225
Pa-00.511400 20000250433
stu-0 4499685
stu- 499006
sta-0. 499282
stu Ruu00.89333
stu-0 0.50040
stu- mm--..4990 459688
0.45515
dag (noisy)
dog (noisy)
dog (naisy)
dog (ncisy)
dog (noisy)
dog (ncby))
dog (npby)
dog (noioy)
true-0 cbs-1
true-0 cbs-]
true-0 obs-1
true dbs-1
true-0 cbs-1
t obs-1
true0 obs-1
true-0 obs-1
me-0-501007
one-0.4998)
438404
mec-0.457530
one-0
one-O one-0.500044
one maa0-4436760 stu-0.499 499468
one-0.498130
team0
445088 stu-0 499682
51888
463312
500011
0.48882
493353
-0.458800
stu- 0.500240
stu-D. 4594455
Figure 10: Sample images from the CIFAR-10 training set used in our experiments. 'true" and 'obs'
indicate the ground-truth and noisy observed labels (0 for dog, for cat), respectively. 'one', 'tea', and 'stu' denote the predicted labels under the optimal O-SD model, the pseudo-labels provided to the student by the optimal 1-SD teacher, and the predicted labels under the optimal 1-SD student, respectively. Parameters: 0.4, M 1000, N = 512; ResNet-18 18 features and logistic I-SD.
1.2 Extension to other distillation strategies
While this study focuses on linear models, extending the analysis to deep learning presents promising directions for future research. In deep models, dark knowledge may differ significantly and hold greater significance than in linear analysis, due to their feature learning capabilities. For instance, models propagating intermediate layer information Zhang et al.) 2019) might depend more on transfer of feature representations rather than predictions alone. Additionally, another avenue lies in exploring the interplay between SD and security, particulaly optimizing defense against model stealing attacks Ma et al. [202T] Yilmaz and Keles 2025). This line of inquiry extends our problem setting to a min-max framework, aiming to minimize the effectiveness of SD. Advancing these directions could contribute to both KD robustness and secure machine learning.
45

---

1.3
What are the best pseudo-labels to learn?
Because the optimization goal of t-SD is the generalization error of the final (t-th) student, every intermediary teacher must issue labels not to mirror the true decision boundary but to maximize the downstream student's performance. Intriguingly, these student-centric labels differ substantially from the teacher-centric labels that would be chosen for standard classification, and more confident soft labels tend to produce stronger students. A more detailed analysis of these label distinctions likely to reveal the answer to the fundamental question of which labels are most beneficial for learning.
1.4 Application of multi-stage replica theory to other learning problems
We believe that the multi-stage replica method employed for theoretical analysis can be extremely useful for probing learning dynamics that have, to date, remained inaccessible to DMFT. Problem formulations which the outcomes of one learning phase are reused in subsequent phase setting ups to which our framework can be applied directly include domain adaptation Blitzer et al. [2006), curriculum learning Bengio et al. [2009] and meta-learning Hospedales al. [2022] It would also be intriguing to investigate the generalization-emi dynamics of deep neural networks under regime where layers are trained sequentially and iteratively Cui et [2024]
46


Reviews:
[{"review_id": "Jzsp5BYOqd", "reviewer": "Reviewer_F2RL", "summary": "This paper demystifies self-distillation (SD), a puzzling yet effective machine learning technique where models bootstrap their own performance by learning from their previous outputs. While widely adopted, the mechanisms behind SD's success have remained elusive until now. The researchers tackle this knowledge gap through a rigorous theoretical analysis, applying statistical physics' replica method to examine multi-stage SD with linear classifiers operating on noisy Gaussian mixture datasets. Their investigation reveals a crucial insight: SD's effectiveness primarily stems from its noise-reduction capabilities when generating hard pseudo-labels, with particularly pronounced benefits in datasets of moderate size. The analysis yields two practical enhancements for practitioners: implementing stage-limiting early stopping procedures (which shows broad applicability), and stabilizing bias parameters when dealing with imbalanced label distributions. Moving beyond theory, the authors validate these findings through real-world experiments on CIFAR-10 using pretrained ResNet architectures. This work bridges the gap between theoretical understanding and practical application, providing both fundamental explanations and actionable strategies for leveraging SD in environments where noise poses significant challenges.", "questions": "- The authors of the paper identified to two practical heuristics to enhance self distillation. How do they perform when applied to more realistic scenarios (e.g. ResNet for CIFAR10/100 when the entire neural network is trainable)?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper is well structured and well written. \n- The problem studied is of general interest to lots of researchers in the field. \n- The theoretical study was rigorously carried out and technically sound. \n- The conclusions drawn were interesting and insightful. \n\n### Weaknesses \n- My major complaint is the disconnection between the simplified linear classification of Noisy Gaussian Mixture Model and deep neural networks that distillation has found success in. While the authors of the paper tried to supplement the theory with experiments on CIFAR-10 classification using pretrained ResNet backbone, the experiments are still quite different from the real-world setting that self-distillation is applied. Indeed, the entire neural network weights are typically trained during self distillation. It would be much more interesting if the authors of the paper can also show that the conclusions drawn with linear classifier can be observed in the more general and realistic scenario as well.", "was_revised": true, "final_justification": "The authors argued that training full scale neural networks can be challenging during reviewer discussion. I still believe that training the full ResNet models on CIFAR datasets should be doable. Including the additional experiments on these could further enhance the paper. Nevertheless, I think the theoretical insights are valuable. As such, I am retaining my original score.", "rebuttal": "We thank you for your valuable time and insightful feedback. We address specific comments below.\n\n**Reply to the Questions:**\n\n> The authors of the paper identified to two practical heuristics to enhance self distillation. How do they perform when applied to more realistic scenarios (e.g. ResNet for CIFAR10/100 when the entire neural network is trainable)?\n\nWe appreciate the reviewer's interest in the applicability of our heuristics to more realistic scenarios, which are indeed practically important.\n\nUnfortunately, conducting large-scale experiments with fully trainable networks is challenging due to the significant effort required for hyperparameter tuning. However, our analysis of the linear model may provide a strong theoretical foundation to discuss the potential utility and outlook for these heuristics in more complex scenarios. Below, we describe our current outlook on the utility of each heuristics.\n\nEarly Stopping: Our results for the linear model indicate that once performance begins to degrade, it does not recover with further training (as seen in Figure 3C). If we assume this monotonic degradation beyond the optimal point holds universally for deep models, then early stopping is likely to be a beneficial heuristic for improving performance. \n\nBias-fixing: In the linear model, we demonstrated the difficulty of simultaneously optimizing two distinct components (the bias and the decision boundary's orientation) via SD (Figure 4B). It is natural to expect that this difficulty would persist in more complex models. Therefore, fixing the bias term is likely a better strategy than leaving it unconstrained, even in large-scale settings. Furthermore, even if this is not directly applicable, the concepts from our analysis can inspire more sophisticated heuristics for larger models. As mentioned in the manuscript (lines 344-345), exploring strategies like the sequential optimization of different parameter groups via SD is a promising future direction."}, {"review_id": "l6cx5y61nz", "reviewer": "Reviewer_7Ees", "summary": "This paper presents a theoretical and empirical study of multi-stage self-distillation for binary classification under label noise. Using the replica method from statistical physics, the authors derive exact asymptotic expressions for the generalization error of a linear classifier trained via an optimally-tuned SD schedule on a noisy Gaussian-mixture model. The analysis shows that the dominant benefit of SD arises from a denoising effect of (hard) pseudo-labels, while the additional “dark-knowledge’’ contained in soft probabilities plays only a marginal role. Two practical heuristics naturally emerge from the theory: (i) early stopping after a few distillation stages to avoid performance collapse, and (ii) bias fixing in imbalanced-label regimes. Both predictions are confirmed experimentally on CIFAR-10 (cat vs dog) using frozen ResNe backbones, where distillation yields an error reduction at medium data sizes and behaves as predicted by the theory.", "questions": "* Please clarify the logical status of Theorem 4.1 (and others) is it a conjecture supported by extensive evidence, or is there a rigorous proof in this particular Gaussian-mixture/linear-classifier regime?  If it is still conjectural, could the authors outline a plausible proof strategy (e.g. AMP/state-evolution or convex Gaussian min-max techniques) and discuss outstanding technical obstacles?  Conversely, if a proof does exist (even for a restricted loss/activation setting), please cite or sketch it explicitly, so readers can distinguish rigorously established results from physics-style predictions.  To be clear, I think it is ok to use the replica method, but statement like \"theorem\" should be used precisely. There are a number of rigorous predictions, that the authors cite so I was wondering... for instance, \n\n* Extension beyond linear models: Could the authors discuss how the order-parameter decomposition might generalize to shallow networks with learned features, or suggest what additional assumptions would be required?\n\n* Computational cost of exhaustive tuning: The theory assumes hyper-parameters are chosen to minimize the asymptotic error. How sensitive are the results to imperfect tuning in practice, and can the replica formula guide efficient hyper-parameter search?\n\n*Robustness to non-random (adversarial) label noise: Have the authors investigated whether the denoising advantage persists under structured or instance-dependent noise?\n\n* Scaling of the early-stopping heuristic: The stopping rule is motivated by the peak of the signal term . Could the authors propose a data-driven criterion that does not rely on inaccessible order parameters?\n\n* Broader experimental validation: The CIFAR-10 cat-vs-dog split is convenient for binary evaluation; do the authors expect similar gains on multi-class tasks such as CIFAR-100 or ImageNet subsets? Evidence here could strengthen the empirical story.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper’s primary strength lies in its systematic analytical framework for studying multi-stage self-distillation under label noise: by wielding the replica method, the authors derive closed-form asymptotic expressions for the generalization error of a linear classifier, giving the community a rare, almost “microscopic” view of how successive distillation stages denoise hard pseudo-labels and why soft-label “dark knowledge” contributes comparatively little. This theoretical clarity is complemented by carefully designed CIFAR-10 experiments that track the predicted order parameters and reproduce all qualitative trends—most notably the early-stopping sweet spot and the bias-correction benefit in imbalanced regimes—lending empirical credibility to the analysis. Beyond these immediate results, the work is significant because it reframes a widely used heuristic (self-distillation) as an optimal-control problem, yielding actionable guidance: practitioners can tune the number of stages, stop training when the signal term peaks, and apply bias fixing to counter class imbalance, all with principled justification. The presentation is generally lucid: major claims are boxed, proofs (albeit non-rigorous) are deferred to well-labeled appendices, and the code release fosters reproducibility. From an originality standpoint, extending replica calculations—traditionally applied to single-shot learning or convex optimization—to a sequential learning protocol and extracting concrete training heuristics is, to the best of my knowledge, novel.\n\n Nevertheless, weaknesses remain. Most conspicuously, the theoretical guarantees are confined to linear models trained on Gaussian-mixture data; while this stylized setting captures the essence of label noise, it leaves open the quantitative fidelity of the formulas when one moves to deep, finite-data regimes. Relatedly, the empirical validation, though well executed, is narrow in scope: a single binary split of CIFAR-10 shows roughly a five-percentage-point error drop—a respectable but not game-changing improvement—and no large-scale or multi-class experiments are reported. Accessibility is another concern: the dense replica derivations, coupled with the replica-symmetry assumption, may deter readers unfamiliar with statistical-physics techniques, and the paper does not fully clarify whether its “Theorem 4.1” is rigorously proven or best viewed as a conjecture supported by evidence. Finally, while the authors note hyper-parameter tuning is guided by their closed-form error expression, they do not quantify how sensitive real-world gains are to imperfect tuning, nor do they assess robustness to structured or adversarial noise. In sum, the paper offers a well-crafted, theoretically insightful contribution with clear practical hooks, yet its impact could be amplified by broader empirical validation, a more explicit discussion of rigor versus heuristic derivation, and guidance on extending the framework beyond the linear-Gaussian sandbox.", "was_revised": true, "final_justification": "This paper provides a novel and insightful theoretical treatment of multi-stage self-distillation under label noise using the replica method, complemented by experiments that qualitatively confirm its predictions. The rebuttal was detailed and transparent, clarifying the logical status of the replica-based results, the obstacles to rigorous proofs, and the practical scope of the heuristics. While the analysis remains confined to linear Gaussian models with limited empirical validation, the work offers valuable conceptual clarity on why and when self-distillation helps. These strengths outweigh the limitations, and I therefore maintain my borderline-accept recommendation.", "rebuttal": "We thank you for your valuable time and insightful feedback. We address specific comments below.\n\n**Comments on the weakness:**\n\n> it leaves open the quantitative fidelity of the formulas when one moves to deep, finite-data regimes.\n\nThank you for raising the concern regarding the realistic setups. As for the finite-data regime, we confirm in Appendix D that our predictions for the asymptotic limit are empirically consistent with numerical experiments of finite size, where the input dimension is on the order of 10^2. Therefore, we expect that the finite-size effects are not a major concern. As for the fidelity in deep cases, it is true that our results do not directly extend to such cases. Nonetheless, we suspect that some of the insights obtained in the paper may hold broadly. For further discussion on this point, please refer to our response to Reviewer F2RL.\n\n> Relatedly, the empirical validation, though well executed, is narrow in scope: a single binary split of CIFAR-10 shows roughly a five-percentage-point error drop—a respectable but not game-changing improvement—\n\nOur primary goal in this study is to obtain reliable insights into the mechanisms underlying self-distillation (SD), which can remain unclear based on limited empirical evidence. The CIFAR-10 experiment was intended as a sanity check to examine whether the key qualitative findings predicted by our theory also emerge in a more practical setting beyond linear classifiers. While the scope of these experiments is limited, the results are consistent with our theoretical predictions, which we believe supports the validity of our analysis. We agree that more extensive empirical validation on large-scale and diverse benchmarks would be an important direction, and we leave this for future work.\n\n> Accessibility is another concern: the dense replica derivations, coupled with the replica-symmetry assumption, may deter readers unfamiliar with statistical-physics techniques\n\nWe acknowledge that the replica-based analysis inevitably involves mathematically dense derivations, which may present an accessibility challenge to readers unfamiliar with statistical physics. However, given the goal of precisely characterizing the behavior of the trained model,  we believe that a certain level of technical detail is intrinsically unavoidable. To address this, we defer the core derivations and technical steps in the appendix, keeping the main text focused on conceptual explanations and key results. We believe this structure allows readers to adjust the depth of engagement depending on their background and interests.\n\n\n**Other comments regarding the weakness overlap with the content of the question, so I will describe them below as answers to the question.**\n\n\n**Reply to the Questions:**\n\n> Please clarify the logical status of Theorem 4.1 (and others) is it a conjecture supported by extensive evidence, or is there a rigorous proof in this particular Gaussian-mixture/linear-classifier regime? If it is still conjectural, could the authors outline a plausible proof strategy (e.g. AMP/state-evolution or convex Gaussian min-max techniques) and discuss outstanding technical obstacles? Conversely, if a proof does exist (even for a restricted loss/activation setting), please cite or sketch it explicitly, so readers can distinguish rigorously established results from physics-style predictions. To be clear, I think it is ok to use the replica method, but statement like \"theorem\" should be used precisely. There are a number of rigorous predictions, that the authors cite so I was wondering... for instance,\n\n\nOur results are derived from the replica method under the Replica Symmetric (RS) assumption. Although formally proving the validity of the analytic continuation from integer replica number to real value is a long-standing mathematical problem, for the convex optimization we consider, predictions based on the RS assumption are widely believed to be exact. Therefore, while our findings are not yet rigorously proven in the strict mathematical sense, they are considered to be reliable predictions within the statistical physics community.\n\nRegarding a rigorous proof without the replica method, the presence of temporal correlations in our model presents a significant technical obstacle. For example, the AMP-based proof strategy to handle such a time-correlation used in [1, 2] is limited to the so-called first-order algorithms, which does not cover the current multi-stage SD procedure. Also, the application of CGMT for the dynamics, which could be applied to multi-stage SD, is often done in on-line settings [3], where a new fresh batch of data is used at each stage of optimization. Hence, formally proofing the current result would require developing new mathematical tools.\n\nFinally, we were unsure about the point you were beginning to make with \"for instance,\"​​, but we hope our explanation clarifies the status of our results. \n\n> Extension beyond linear models: Could the authors discuss how the order-parameter decomposition might generalize to shallow networks with learned features, or suggest what additional assumptions would be required?\n\nFirst, the primary contribution of our paper is to provide a precise, analytical solution for the linear SD model, which in itself was a significant and previously unaddressed challenge. We believe that establishing this rigorous baseline is a necessary first step before tackling more complex architectures.\nRegarding generalization, an extension might be possible under certain simplifying assumptions. For instance, our analytical framework could likely be adapted to a two-layer random feature model under one step SGD, as the analysis in that setup is similar to a random feature model with a spike signal.\nHowever, extending the analysis to a shallow network where all features are fully learned is a much more formidable challenge. This is an important open problem that is of great interest to the broader field, and it extends beyond the specific scope of our current work. \n\n>  Computational cost of exhaustive tuning: The theory assumes hyper-parameters are chosen to minimize the asymptotic error. How sensitive are the results to imperfect tuning in practice, and can the replica formula guide efficient hyper-parameter search?\n\nBased on our numerical results for the linear model, any sufficiently large value of the regularization parameter $\\lambda$ yields near-optimal results. We also found that the temperature parameter has a minimal impact on the outcome. Consequently, the replica theory indicates that an exhaustive hyperparameter search is not critical for this model. \nHow these sensitivities would change in more general and complex settings remains an open question for future investigation.\n\n> Robustness to non-random (adversarial) label noise: Have the authors investigated whether the denoising advantage persists under structured or instance-dependent noise?\n\nNo; however, we agree that this is a valuable and interesting direction for research. There is a wide variety of ways to model structured noise, and a meaningful investigation would require carefully specifying the type of noise in a concrete context. We will add this to future work in the camera-ready version.\n\n> Scaling of the early-stopping heuristic: The stopping rule is motivated by the peak of the signal term . Could the authors propose a data-driven criterion that does not rely on inaccessible order parameters?\n\nOur results indicate that once performance drops, it does not recover. Therefore, we believe a practical and sufficient data-driven criterion is to simply stop training as soon as performance on a validation set starts to degrade.\n\n\n> Broader experimental validation: The CIFAR-10 cat-vs-dog split is convenient for binary evaluation; do the authors expect similar gains on multi-class tasks such as CIFAR-100 or ImageNet subsets? Evidence here could strengthen the empirical story.\n\nWe hypothesize that in the multi-class setting, the importance of dark knowledge would become even more pronounced. As the output dimension grows, the richer information contained in the multi-dimensional probability distribution of the logits would likely play a larger role.\nFor this reason, we believe that extending our analysis to multi-class classification is a crucial and exciting direction for future research. For this, the present result would provide a theoretical framework necessary for tackling this important extension.\n\n\n\n\n\n\n[1] Z. Fan. Approximate message passing algorithms for rotationally invariant matrices. The Annals of Statistics, 50(1):197–224, 2022.\n\n[2] S. Liu. and J. Ma. Unifying AMP Algorithms for Rotationally-Invariant Models. arXiv preprint arXiv:2412.01574.\n\n[3] K.A. Chandrasekher, A. Pananjady. and C. Thrampoulidis. Sharp global convergence guarantees for iterative nonconvex optimization with random data. The Annals of Statistics, 51(1), pp.179-210. 2023"}, {"review_id": "cMp2RqLCbt", "reviewer": "Reviewer_qzrp", "summary": "This paper studies the self-distillation technique on the binary classification problem on GMMs with corruption in the labels in the asymptotic regime and focuses on the generalization error of linear classifiers obtained through ridge regression and $\\ell_2$-regularized cross-entropy loss. By employing the non-rigorous replica trick, precise expressions characterizing the generalization error are provided through self-consistent nonlinear equations. Using these derivations the utility of the dark knowledge and the effects of soft/hard labels in self-distillation across different dataset sizes and noise levels are investigated. Finally, it is claimed that naively applying self-distillation will deteriorate the performance, and a heuristic for appropriately learning the bias terms in the unbalanced setting is provided.", "questions": "1. Some comments:\n\n    **1.1.** I could not find the definition for the term \"pseudo-label\" in the abstract (line 8). \n\n    **1.2.** Does the definition of the temperature parameter only apply for the sigmoid and logistic loss in line 146? \n\n    **1.3.** I could not find the definition of $\\Delta$ in the main body and it is only defined in Appendix B.1.\n\n    **1.4.** The sentence \"These findings suggest that, at least within our toy model, dark knowledge is not the key factor driving the success of SD\" in line 70 is in contrast with the sentence in line 213.\n\n    **1.5.** What does \"diversity of teacher prediction\" mean in line 281?\n\n\n**2.** In Figure 2A, is there an optimal corruption level in the overparametrized regime ($\\alpha<1$)? \n\n\n**3.** In Section 7, how does regularizing the bias term affect the performance?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n**1.** By considering self-distillation, an important paradigm in training foundational models, this work has provided theoretical insights into when SD can prove useful and what its limitations are.\n\n**2.** The authors have provided extensive discussion in Sections 5-7 around Result 4.1 to make the derived expressions interpretable. In the single-step case, they investigated the efficacy of soft and hard labels for various noise levels, corruption levels, and overparametrization ratios. The paper demonstrates that depending on the dataset size, the improvement in the performance through self-distillation may be marginal in the multi-stage setting.\n\n**3.** The problem setting and the notation are defined clearly, and the paper is well written in general.\n\n**Weaknesses**\n\n1. The authors report their findings through the non-rigorous replica method as \"Theorem\". I believe that the title of statements should be changed from \"Theorem\" to \"Result\" or \"Finding\" to reflect the lack of rigor. It is recommended to use the word \"Computation\" instead of the word \"Proof\". There are also other instances in the paper which require rephrasing to indicate that the approach is non-rigorous.  Adding a discussion on potential approaches to proving the derived expressions will improve the quality of the paper.\n\n2.  The literature review in the paper could be more comprehensive:\n\n    **2.1.**  Starting from the introduction, referring to more recent papers using self-distillation in practice motivates problem formulation further as the currently cited papers in lines 28-29 are from 2019. Is self-distillation being used actively? How close is the formulation of Self-Distillation technique considered in this paper to what is being employed in practice? I would appreciate authors' comments on how applications motivate the current problem setting.\n\n    **2.2.** In lines 58-59, the authors mention CGMT and Gordon's inequality as separate approaches for analyzing high-dimensional inference problems in the asymptotic regime. CGMT relies on Gordon's inequality to provide precise results in the asymptotic regime. Citing Gordon's inequality as a separate approach is confusing. I would appreciate the author's comment on this. Furthermore, I believe that adding additional references on the asymptotics of linear binary classification and providing a brief overview of known results will improve the exposition. Related works on the asymptotics of GMM classification are as follows : [1], [2], [3], [4]\n\n    **2.3.** [3] rigorously analyzed the linear binary classification problem using ridge regression and other regularizers in the presence of label noise (corruption). I recommend referring to this paper in the paragraph \"Fixed point analysis and learning dynamics\" in line 255 as [3] also focuses on the effect of corruption and the large $\\lambda$ regime.\n\n3. While I appreciate the extensive discussion following Result 4.1, I believe the current result is limited in scope, given the theoretical nature of the work. As shown in [3] (Remark 2), taking the regularization strength to be large recovers the oracle optimal classifier in the balanced setting for arbitrary corruption level $< 0.5$. This challenges the need for successive use of self-distillation and training the model from scratch for each $t$; whereas by taking the regularization strength to be large and training only once yields the optimal performance. I would appreciate the authors' comment on this.\n\n    **3.1.** Regarding the early-stopping heuristic in line 72 and Result 6.1, I am not convinced that only considering the setting $\\lambda_t \\rightarrow \\infty$ would suffice to claim that early-stopping is a sensible strategy. \n\n    **3.2.** Potential tangential directions to complement the results include performing the computations for a more general mean and covariance structure as it is not immediate what the optimal performance of the ridge regression would be for arbitrary covariance matrices.\n\n    **3.3.** Furthermore, characterizing the evolution of the optimal regularization strength through self-consistent equations and the generalization error can further strengthen Result 6.1.\n\n    **3.4.** The discussions in Sections 5-7, despite being intuitive, lack the necessary mathematical details. I recommend presenting some of the mathematical expressions of the predicted generalization error from the Appendix in the main body (such as (158) and (192) in the Appendix). Discussing how the authors arrive at the observations in Section 5-7 by referring to the mathematical expressions adequately would help in understanding the computations.\n\n4. Other comments:\n\n    **4.1.** The overparametrizated regime ($\\alpha < 1$), despite being fundamentally important, remains underexplored in this paper. For instance, in Figures 1 and 2, it is difficult to discern how the error behaves when $\\alpha < 1$ varies.  I suggest adding separate figures in the spirit of Figures 1 and 2 for the overparametrized and underparametrized regimes. \n\n   **4.2.** To support their proposed heuristic in Sections 6 and 7, the authors should conduct additional experiments on real-world dataset.\n\n**[1]** Kini, Ganesh Ramachandra, and Christos Thrampoulidis. \"Analytic study of double descent in binary classification: The impact of loss.\" 2020 IEEE International Symposium on Information Theory (ISIT). IEEE, 2020.\n\n**[2]** Deng, Zeyu, Abla Kammoun, and Christos Thrampoulidis. \"A model of double descent for high-dimensional binary linear classification.\" Information and Inference: A Journal of the IMA 11.2 (2022): 435-495.\n\n**[3]** Akhtiamov, Danil, Reza Ghane, and Babak Hassibi. \"Regularized linear regression for binary classification.\" 2024 IEEE International Symposium on Information Theory (ISIT). IEEE, 2024.\n\n**[4]** Ghane, Reza, Danil Akhtiamov, and Babak Hassibi. \"Universality in transfer learning for linear models.\" Advances in Neural Information Processing Systems 37 (2024): 125729-125779", "was_revised": true, "final_justification": "I would like to thank the authors for their responses. The authors have adequately addressed the questions 1-3 raised in my review.  However, I remain unconvinced about some of the points outlined in the Weaknesses section (points 3.1 and 3.2):\n\n1. The assumptions on the class means being antipodal and the covariances being isotropic are too restrictive.\n\n2. The authors did not respond to my latest comment regarding whether their proposed early stopping strategy is a sensible approach.\n\nIn the light of this discussion, I would like to maintain my score.", "rebuttal": "We thank you for your valuable time and insightful feedback. We address specific comments below.\nPlease note that due to word count limitations, we may omit quotations from the original comments. We kindly ask you to refer to the original review for the complete questions.\n\n**Comments on the weakness:**\n\n> 1. The authors report their findings through the non-rigorous replica method as \"Theorem\". I believe…\n\nOur results are derived from the replica method under the Replica Symmetric (RS) assumption. For the convex problem we consider, predictions based on the RS assumption are strongly believed to be exact. Therefore, while our findings are not yet rigorously proven in a mathematical sense, they are considered to be solid predictions within the statistical physics community. Our initial choice to use \"Theorem\" was intended to clarify the paper's structure and highlight our main findings, a convention also seen in other related works [1, 2]. However, we acknowledge your point that this term could be misleading. We will revise the wording accordingly in the camera-ready version.\n\n> The literature review in the paper could be more comprehensive:\n> **2.1.** Starting from the introduction, referring to more recent papers using self-distillation in practice motivates problem formulation further as the currently cited papers in lines 28-29 are from 2019. Is self-distillation being used actively? How close is the formulation of Self-Distillation technique considered in this paper to what is being employed in practice? I would appreciate authors' comments on how applications motivate the current problem setting.\n\nSD is indeed actively applied to practical problems in recent studies [3, 4], with current research often focusing on specialized methods like propagating internal representations. **However, our focus—propagating the output—also remains a practical and current research topic [5, 6], and we believe our work offers significant insights for development of modern algorithms. Furthermore, our mathematical model serves as a valuable foundation for the theoretical analysis of mainstream contemporary SD methods, including those that propagate internal representations [3].**  The potential for such extensions to more modern methods is also discussed in Appendix I.\n\n> **2.2.** In lines 58-59, the authors mention CGMT and Gordon's inequality as separate approaches for analyzing high-dimensional inference problems in the asymptotic regime. CGMT relies on Gordon's inequality…\n\nThank you for your advice and giving additional reference. We agree that presenting CGMT and Gordon's method separately was confusing. We will fix this point. Also, we will add to and revise the manuscript based on these for the camera-ready version.\n\n> **2.3.** [3] rigorously analyzed the linear binary classification problem using ridge regression and other regularizers in the presence of label noise (corruption). I recommend…\n\nThank you for your advice and giving additional reference. We were not aware that it was valid for such a general setting. We plan to add to and revise the manuscript based on these for the camera-ready version.\n\n> 3. While I appreciate the extensive discussion following Result 4.1, I believe the current result is limited in scope, given the theoretical nature of the work. As shown in [3] (Remark 2), taking the regularization strength to be large recovers the oracle optimal classifier in the balanced setting for arbitrary corruption level . This challenges the need for successive use of self-distillation and training the model from scratch for each t ; whereas by taking the regularization strength to be large and training only once yields the optimal performance. I would appreciate the authors' comment on this.\n\nIt is sure that $\\lambda_0 \\to \\infty$ leads to the Bayes optimal classifier. However, the situation changes after 1-SD because the labels used have different nature from the original labels. This is the non-trivial aspect of SD that should be investigated in detail.\n**Our results show that SD can further improve performance beyond the initial Bayes optimal estimation of the standard supervised setup (referred to as \"optimal 0-SD\" in our notation)**, as seen for example in Fig 3A. Therefore, applying SD remains a valuable approach in the current setting.\n\n> **3.1.** Regarding the early-stopping heuristic in line 72 and Result 6.1, I am not convinced that only considering the setting $\\lambda_t \\to \\infty$ would suffice to claim that early-stopping is a sensible strategy.\n\nWhile we did not include it in the current draft, we have confirmed in our linear model setting that the optimal hyperparameter schedule, derived from the black-box optimization, follows the following pattern: it begins with a very large regularization ($\\lambda\\to\\infty$) for an initial period and then switches to zero regularization $\\lambda\\to 0$ thereafter.\nThis is precisely what motivates our claim that early stopping is a sensible strategy through the $\\lambda_t \\to \\infty$ hyper-parameter schedule. To provide full justification for this claim, we will add the detailed results of this optimal hyper-parameter schedule to the appendix in the camera-ready version.\n\n> **3.2.** Potential tangential directions to complement the results include performing the computations for a more general mean and covariance structure as it is not immediate what the optimal performance of the ridge regression would be for arbitrary covariance matrices.\n\nWe agree that analyzing the model with a more general covariance structure is an　interesting extension. We will add this as a key direction for future work.\n\n> **3.3.** Furthermore, characterizing the evolution of the optimal regularization strength through self-consistent equations and the generalization error can further strengthen Result 6.1.\n\nSee the reply to question 3.1.\n\n> **3.4.** The discussions in Sections 5-7, despite being intuitive, lack the necessary mathematical details. I recommend presenting some of the mathematical expressions of the predicted generalization error from the Appendix in the main body (such as (158) and (192) in the Appendix). Discussing how the authors arrive at the observations in Section 5-7 by referring to the mathematical expressions adequately would help in understanding the computations.\n\nThank you for your advice. Generally, it is not feasible to write a simple, closed-form expression for the generalization error. For this reason, we have validated our theoretical results with numerical calculations, which is one of the standard practices in asymptotic analysis. Furthermore, we wish to prioritize an intuitive presentation for a broad readership. In addition, the expression for the generalization error is given in Eq. (9), and we believe the necessary mathematical expressions in the main body are self-contained.\n\n**Reply to the Questions:**\n\n>  **1.1.** I could not find the definition for the term \"pseudo-label\" in the abstract (line 8).\n\nThe detailed definition of pseudo-label is described in line 144-146. We will clarify this by adding the following definition in the camera-ready version: \"A pseudo-label is an artificially generated label added to the training set to retrain and improve the model.\" to avoid any misunderstanding.\n\n> **1.2.** Does the definition of the temperature parameter only apply for the sigmoid and logistic loss in line 146?      \n \nYes, as written in line 153-154. However, we will add an explanation to avoid any misunderstanding in the camera-ready version.\n\n> **1.3.** I could not find the definition of  $\\Delta$ in the main body and it is only defined in Appendix B.1.\n\nThe $\\Delta$ is the size of each cluster, as defined in the right hand side of Eq.(1).\n\n>  **1.4.** The sentence \"These findings suggest that, at least within our toy model, dark knowledge is not the key factor driving the success of SD\" in line 70 is in contrast with the sentence in line 213.\n\nThe statement in line 70 argues that the **absolute effect** of dark knowledge is small. In contrast, the statement in line 213 argues that the **relative effect** of dark knowledge (from soft labels) is large compared to that of hard labels. While these two statements are consistent, we agree the phrasing could be confusing. We will add a clarification to avoid any misunderstanding in the camera-ready version.\n\n> **1.5.** What does \"diversity of teacher prediction\" mean in line 281?\n\nDiversity is a measure of the population spread of a teacher model's predictive probabilities across different samples in a dataset [7].  We insist this concept is corresponding to our \"predictive uncertainty\". We will add this explanation to a camera-ready version.\n\n> **2.** In Figure 2A, is there an optimal corruption level in the overparametrized regime ($\\alpha < 1$)?\n\nWe believe the answer is no. The apparent peak is likely an artifact of numerical instability in our black-box hyperparameter optimization, which becomes particularly challenging in that regime.\n\n> **3.** In Section 7, how does regularizing the bias term affect the performance?\n\nRegularizing the bias term is tempting to balance the magnitude with the weight. However, we do not investigate the bias-regularized setting since we believe it is not a general setting.\n\n[1] Barbier et al., _PNAS_ **116**, 5451 (2019). \\\n[2] Cornacchia et al., _Mach. Learn.: Sci. Technol._ **4**, 015019 (2023). \\\n[3] Yang et al., arXiv:2402.13669 (2024). \\\n[4] Chen et al., arXiv:2505.21501 (2025).\\\n[5] Yilmaz and Keles, arXiv:2403.05181 (2024). \\\n[6] Ma et al., _ICLR_ (2021). \\\n[7] Zhang and Sabuncu, _NeurIPS_ (2020)."}, {"review_id": "se69kZleEn", "reviewer": "Reviewer_q2Aw", "summary": "This paper explores the effectiveness of multi-stage self distillation based on linear classifier hyperparameter optimization in binary classification tasks of noisy Gaussian mixture data using the replica method in statistical physics. The main driving force for the improvement of self distillation (SD) performance was found to be the denoising effect achieved through hard pseudo labeling, and two practical heuristic strategies were proposed to enhance SD performance.", "questions": "1. Only exploring binary classification tasks, the generalization of this work needs to be verified.\n2. Is the CIFAR-10 dataset validated on a larger dataset as the image resolution is relatively small? For example, ImageNet.", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：1. This method provides an environment that can simultaneously analyze the \"dark knowledge\" hypothesis and the denoising hypothesis, and analyze both.\n2. Theoretical analysis and experimental verification are complete.\n\nWeaknesses：\n1. Only exploring binary classification tasks, the generalization of this work needs to be verified.\n2. Is the CIFAR-10 dataset validated on a larger dataset as the image resolution is relatively small? For example, ImageNet.", "was_revised": true, "final_justification": "Thank you for the author's clarification. I have clarified the previous questions and will maintain my score.", "rebuttal": "We thank you for your valuable time and insightful feedback. We address specific comments below.\n\n**Reply to the Questions:**\n\n> 1. Only exploring binary classification tasks, the generalization of this work needs to be verified.\n\nWe hypothesize that in the multi-class setting, the importance of dark knowledge would become even more pronounced. As the output dimension increases, the richer information contained in the multi-dimensional probability distribution of the logits would likely play a greater role.\nFor this reason, we believe that extending our analysis to multi-class classification is a crucial and exciting direction for future research. For this, the present result would provide the theoretical framework necessary for tackling this important extension.\n\n> 2. Is the CIFAR-10 dataset validated on a larger dataset as the image resolution is relatively small? For example, ImageNet.\n\nThe primary goal of our experiments was to provide a qualitative validation of our theoretical findings in a real-world data setting, rather than to achieve state-of-the-art performance. CIFAR-10 is a standard and well-understood benchmark that is sufficient to demonstrate that the findings obtained from our theory hold true in practice. We agree that validation on larger-scale datasets like ImageNet is an important direction for future work."}]

Author rebuttal:

Meta-review: This paper presents a theoretical analysis of multi-stage self-distillation (SD) for a linear classifier on noisy Gaussian mixture data. Using the replica method from statistical physics, the authors derive a precise characterization of the generalization error, allowing for an exhaustive, optimal tuning of the SD process. The main findings are that the primary benefit of SD stems from a denoising effect via hard pseudo-labels, with the most significant gains observed in moderately-sized datasets. The paper also proposes two practical heuristics: early stopping to prevent performance degradation over many stages, and bias-fixing to handle label imbalance. The primary strength of this work lies in its rigorous analytical framework, which provides a rare, microscopic view of a complex learning dynamic. This theoretical clarity leads to principled insights that are otherwise difficult to obtain from purely empirical studies, where exhaustive hyperparameter tuning is often computationally infeasible. The main weakness, acknowledged by the authors and all reviewers, is the gap between the simplified theoretical model and the complex, deep neural networks where SD is often applied in practice.

The paper initiated a helpful discussion among the reviewers and authors. Reviewers raised several important points, including the need for broader experimental validation on more complex models, the non-rigorous nature of the replica-based "theorems", and the restrictive assumptions of the theoretical model. The authors provided a thorough rebuttal, clarifying the scope of their work and agreeing to several key revisions for the final version. They convincingly argued that their experiments serve as a valuable sanity check for their theoretical insights and promised to adjust terminology to more accurately reflect the nature of their theoretical results. One reviewer (qzrp) raised a substantive concern that the benefits of multi-stage SD might be unnecessary if a single-stage Bayes-optimal classifier is attainable. In response, the authors clarified that a key finding of their work is that multi-stage SD can surpass the performance of this single-stage baseline, a point supported by their theoretical results.

My recommendation is to accept this paper. The primary reason is that the work provides a clear, precise, and insightful theoretical treatment of a widely used but poorly understood technique. While the model is simplified, the analysis yields valuable conceptual clarity on why and when self-distillation is effective, particularly regarding its denoising properties. The authors' response to the reviews was constructive and addressed the main concerns adequately. While the reservations about the model's simplicity are valid, the consensus is that the new theoretical perspective and the practical heuristics that emerge from it represent a solid contribution to the field. This work provides a strong foundation that can guide both future theoretical analysis and practical applications of self-distillation. Acceptance is contingent on the authors incorporating their promised changes, particularly regarding the terminology of their theoretical results and the addition of supporting evidence for their claims, into the camera-ready version.
Decision: Accept (poster)