Title: The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: knowledge distillation

self distillation

statistical physics

replica method

dynamics

machine learning
Abstract: Self-distillation (SD), a technique where a model improves itself using its own predictions, has attracted attention as a simple yet powerful approach in machine learning. Despite its widespread use, the mechanisms underlying its effectiveness remain unclear. In this study, we investigate the efficacy of hyperparameter-tuned multi-stage SD with a linear classifier for binary classification on noisy Gaussian mixture data. For the analysis, we employ the replica method from statistical physics. Our findings reveal that the primary driver of SD's performance improvement is denoising through hard pseudo-labels, with the most notable gains observed in moderately sized datasets. We also identify two practical heuristics to enhance SD: early stopping that limits the number of stages, which is broadly effective, and bias parameter fixing, which helps under label imbalance. To empirically validate our theoretical findings derived from our toy model, we conduct additional experiments on CIFAR-10 classification using pretrained ResNet backbone. These results provide both theoretical and practical insights, advancing our understanding and application of SD in noisy settings.


Peer reviews:
comments: SUMMARY:
This paper explores the effectiveness of multi-stage self distillation based on linear classifier hyperparameter optimization in binary classification tasks of noisy Gaussian mixture data using the replica method in statistical physics. The main driving force for the improvement of self distillation (SD) performance was found to be the denoising effect achieved through hard pseudo labeling, and two practical heuristic strategies were proposed to enhance SD performance.

STRENGTHS AND WEAKNESSES:
Strengths：1. This method provides an environment that can simultaneously analyze the "dark knowledge" hypothesis and the denoising hypothesis, and analyze both.
2. Theoretical analysis and experimental verification are complete.

Weaknesses：
1. Only exploring binary classification tasks, the generalization of this work needs to be verified.
2. Is the CIFAR-10 dataset validated on a larger dataset as the image resolution is relatively small? For example, ImageNet.

QUESTIONS:
1. Only exploring binary classification tasks, the generalization of this work needs to be verified.
2. Is the CIFAR-10 dataset validated on a larger dataset as the image resolution is relatively small? For example, ImageNet.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the self-distillation technique on the binary classification problem on GMMs with corruption in the labels in the asymptotic regime and focuses on the generalization error of linear classifiers obtained through ridge regression and $\\ell_2$-regularized cross-entropy loss. By employing the non-rigorous replica trick, precise expressions characterizing the generalization error are provided through self-consistent nonlinear equations. Using these derivations the utility of the dark knowledge and the effects of soft/hard labels in self-distillation across different dataset sizes and noise levels are investigated. Finally, it is claimed that naively applying self-distillation will deteriorate the performance, and a heuristic for appropriately learning the bias terms in the unbalanced setting is provided.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**1.** By considering self-distillation, an important paradigm in training foundational models, this work has provided theoretical insights into when SD can prove useful and what its limitations are.\n\n**2.** The authors have provided extensive discussion in Sections 5-7 around Result 4.1 to make the derived expressions interpretable. In the single-step case, they investigated the efficacy of soft and hard labels for various noise levels, corruption levels, and overparametrization ratios. The paper demonstrates that depending on the dataset size, the improvement in the performance through self-distillation may be marginal in the multi-stage setting.\n\n**3.** The problem setting and the notation are defined clearly, and the paper is well written in general.\n\n**Weaknesses**\n\n1. The authors report their findings through the non-rigorous replica method as \"Theorem\". I believe that the title of statements should be changed from \"Theorem\" to \"Result\" or \"Finding\" to reflect the lack of rigor. It is recommended to use the word \"Computation\" instead of the word \"Proof\". There are also other instances in the paper which require rephrasing to indicate that the approach is non-rigorous.  Adding a discussion on potential approaches to proving the derived expressions will improve the quality of the paper.\n\n2.  The literature review in the paper could be more comprehensive:\n\n    **2.1.**  Starting from the introduction, referring to more recent papers using self-distillation in practice motivates problem formulation further as the currently cited papers in lines 28-29 are from 2019. Is self-distillation being used actively? How close is the formulation of Self-Distillation technique considered in this paper to what is being employed in practice? I would appreciate authors' comments on how applications motivate the current problem setting.\n\n    **2.2.** In lines 58-59, the authors mention CGMT and Gordon's inequality as separate approaches for analyzing high-dimensional inference problems in the asymptotic regime. CGMT relies on Gordon's inequality to provide precise results in the asymptotic regime. Citing Gordon's inequality as a separate approach is confusing. I would appreciate the author's comment on this. Furthermore, I believe that adding additional references on the asymptotics of linear binary classification and providing a brief overview of known results will improve the exposition. Related works on the asymptotics of GMM classification are as follows : [1], [2], [3], [4]\n\n    **2.3.** [3] rigorously analyzed the linear binary classification problem using ridge regression and other regularizers in the presence of label noise (corruption). I recommend referring to this paper in the paragraph \"Fixed point analysis and learning dynamics\" in line 255 as [3] also focuses on the effect of corruption and the large $\\lambda$ regime.\n\n3. While I appreciate the extensive discussion following Result 4.1, I believe the current result is limited in scope, given the theoretical nature of the work. As shown in [3] (Remark 2), taking the regularization strength to be large recovers the oracle optimal classifier in the balanced setting for arbitrary corruption level $< 0.5$. This challenges the need for successive use of self-distillation and training the model from scratch for each $t$; whereas by taking the regularization strength to be large and training only once yields the optimal performance. I would appreciate the authors' comment on this.\n\n    **3.1.** Regarding the early-stopping heuristic in line 72 and Result 6.1, I am not convinced that only considering the setting $\\lambda_t \\rightarrow \\infty$ would suffice to claim that early-stopping is a sensible strategy. \n\n    **3.2.** Potential tangential directions to complement the results include performing the computations for a more general mean and covariance structure as it is not immediate what the optimal performance of the ridge regression would be for arbitrary covariance matrices.\n\n    **3.3.** Furthermore, characterizing the evolution of the optimal regularization strength through self-consistent equations and the generalization error can further strengthen Result 6.1.\n\n    **3.4.** The discussions in Sections 5-7, despite being intuitive, lack the necessary mathematical details. I recommend presenting some of the mathematical expressions of the predicted generalization error from the Appendix in the main body (such as (158) and (192) in the Appendix). Discussing how the authors arrive at the observations in Section 5-7 by referring to the mathematical expressions adequately would help in understanding the computations.\n\n4. Other comments:\n\n    **4.1.** The overparametrizated regime ($\\alpha < 1$), despite being fundamentally important, remains underexplored in this paper. For instance, in Figures 1 and 2, it is difficult to discern how the error behaves when $\\alpha < 1$ varies.  I suggest adding separate figures in the spirit of Figures 1 and 2 for the overparametrized and underparametrized regimes. \n\n   **4.2.** To support their proposed heuristic in Sections 6 and 7, the authors should conduct additional experiments on real-world dataset.\n\n**[1]** Kini, Ganesh Ramachandra, and Christos Thrampoulidis. \"Analytic study of double descent in binary classification: The impact of loss.\" 2020 IEEE International Symposium on Information Theory (ISIT). IEEE, 2020.\n\n**[2]** Deng, Zeyu, Abla Kammoun, and Christos Thrampoulidis. \"A model of double descent for high-dimensional binary linear classification.\" Information and Inference: A Journal of the IMA 11.2 (2022): 435-495.\n\n**[3]** Akhtiamov, Danil, Reza Ghane, and Babak Hassibi. \"Regularized linear regression for binary classification.\" 2024 IEEE International Symposium on Information Theory (ISIT). IEEE, 2024.\n\n**[4]** Ghane, Reza, Danil Akhtiamov, and Babak Hassibi. \"Universality in transfer learning for linear models.\" Advances in Neural Information Processing Systems 37 (2024): 125729-125779\n\nQUESTIONS:\n1. Some comments:\n\n    **1.1.** I could not find the definition for the term \"pseudo-label\" in the abstract (line 8). \n\n    **1.2.** Does the definition of the temperature parameter only apply for the sigmoid and logistic loss in line 146? \n\n    **1.3.** I could not find the definition of $\\Delta$ in the main body and it is only defined in Appendix B.1.\n\n    **1.4.** The sentence \"These findings suggest that, at least within our toy model, dark knowledge is not the key factor driving the success of SD\" in line 70 is in contrast with the sentence in line 213.\n\n    **1.5.** What does \"diversity of teacher prediction\" mean in line 281?\n\n\n**2.** In Figure 2A, is there an optimal corruption level in the overparametrized regime ($\\alpha<1$)? \n\n\n**3.** In Section 7, how does regularizing the bias term affect the performance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical and empirical study of multi-stage self-distillation for binary classification under label noise. Using the replica method from statistical physics, the authors derive exact asymptotic expressions for the generalization error of a linear classifier trained via an optimally-tuned SD schedule on a noisy Gaussian-mixture model. The analysis shows that the dominant benefit of SD arises from a denoising effect of (hard) pseudo-labels, while the additional “dark-knowledge’’ contained in soft probabilities plays only a marginal role. Two practical heuristics naturally emerge from the theory: (i) early stopping after a few distillation stages to avoid performance collapse, and (ii) bias fixing in imbalanced-label regimes. Both predictions are confirmed experimentally on CIFAR-10 (cat vs dog) using frozen ResNe backbones, where distillation yields an error reduction at medium data sizes and behaves as predicted by the theory.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper’s primary strength lies in its systematic analytical framework for studying multi-stage self-distillation under label noise: by wielding the replica method, the authors derive closed-form asymptotic expressions for the generalization error of a linear classifier, giving the community a rare, almost “microscopic” view of how successive distillation stages denoise hard pseudo-labels and why soft-label “dark knowledge” contributes comparatively little. This theoretical clarity is complemented by carefully designed CIFAR-10 experiments that track the predicted order parameters and reproduce all qualitative trends—most notably the early-stopping sweet spot and the bias-correction benefit in imbalanced regimes—lending empirical credibility to the analysis. Beyond these immediate results, the work is significant because it reframes a widely used heuristic (self-distillation) as an optimal-control problem, yielding actionable guidance: practitioners can tune the number of stages, stop training when the signal term peaks, and apply bias fixing to counter class imbalance, all with principled justification. The presentation is generally lucid: major claims are boxed, proofs (albeit non-rigorous) are deferred to well-labeled appendices, and the code release fosters reproducibility. From an originality standpoint, extending replica calculations—traditionally applied to single-shot learning or convex optimization—to a sequential learning protocol and extracting concrete training heuristics is, to the best of my knowledge, novel.\n\n Nevertheless, weaknesses remain. Most conspicuously, the theoretical guarantees are confined to linear models trained on Gaussian-mixture data; while this stylized setting captures the essence of label noise, it leaves open the quantitative fidelity of the formulas when one moves to deep, finite-data regimes. Relatedly, the empirical validation, though well executed, is narrow in scope: a single binary split of CIFAR-10 shows roughly a five-percentage-point error drop—a respectable but not game-changing improvement—and no large-scale or multi-class experiments are reported. Accessibility is another concern: the dense replica derivations, coupled with the replica-symmetry assumption, may deter readers unfamiliar with statistical-physics techniques, and the paper does not fully clarify whether its “Theorem 4.1” is rigorously proven or best viewed as a conjecture supported by evidence. Finally, while the authors note hyper-parameter tuning is guided by their closed-form error expression, they do not quantify how sensitive real-world gains are to imperfect tuning, nor do they assess robustness to structured or adversarial noise. In sum, the paper offers a well-crafted, theoretically insightful contribution with clear practical hooks, yet its impact could be amplified by broader empirical validation, a more explicit discussion of rigor versus heuristic derivation, and guidance on extending the framework beyond the linear-Gaussian sandbox.\n\nQUESTIONS:\n* Please clarify the logical status of Theorem 4.1 (and others) is it a conjecture supported by extensive evidence, or is there a rigorous proof in this particular Gaussian-mixture/linear-classifier regime?  If it is still conjectural, could the authors outline a plausible proof strategy (e.g. AMP/state-evolution or convex Gaussian min-max techniques) and discuss outstanding technical obstacles?  Conversely, if a proof does exist (even for a restricted loss/activation setting), please cite or sketch it explicitly, so readers can distinguish rigorously established results from physics-style predictions.  To be clear, I think it is ok to use the replica method, but statement like \"theorem\" should be used precisely. There are a number of rigorous predictions, that the authors cite so I was wondering... for instance, \n\n* Extension beyond linear models: Could the authors discuss how the order-parameter decomposition might generalize to shallow networks with learned features, or suggest what additional assumptions would be required?\n\n* Computational cost of exhaustive tuning: The theory assumes hyper-parameters are chosen to minimize the asymptotic error. How sensitive are the results to imperfect tuning in practice, and can the replica formula guide efficient hyper-parameter search?\n\n*Robustness to non-random (adversarial) label noise: Have the authors investigated whether the denoising advantage persists under structured or instance-dependent noise?\n\n* Scaling of the early-stopping heuristic: The stopping rule is motivated by the peak of the signal term . Could the authors propose a data-driven criterion that does not rely on inaccessible order parameters?\n\n* Broader experimental validation: The CIFAR-10 cat-vs-dog split is convenient for binary evaluation; do the authors expect similar gains on multi-class tasks such as CIFAR-100 or ImageNet subsets? Evidence here could strengthen the empirical story.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demystifies self-distillation (SD), a puzzling yet effective machine learning technique where models bootstrap their own performance by learning from their previous outputs. While widely adopted, the mechanisms behind SD's success have remained elusive until now. The researchers tackle this knowledge gap through a rigorous theoretical analysis, applying statistical physics' replica method to examine multi-stage SD with linear classifiers operating on noisy Gaussian mixture datasets. Their investigation reveals a crucial insight: SD's effectiveness primarily stems from its noise-reduction capabilities when generating hard pseudo-labels, with particularly pronounced benefits in datasets of moderate size. The analysis yields two practical enhancements for practitioners: implementing stage-limiting early stopping procedures (which shows broad applicability), and stabilizing bias parameters when dealing with imbalanced label distributions. Moving beyond theory, the authors validate these findings through real-world experiments on CIFAR-10 using pretrained ResNet architectures. This work bridges the gap between theoretical understanding and practical application, providing both fundamental explanations and actionable strategies for leveraging SD in environments where noise poses significant challenges.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is well structured and well written. \n- The problem studied is of general interest to lots of researchers in the field. \n- The theoretical study was rigorously carried out and technically sound. \n- The conclusions drawn were interesting and insightful. \n\n### Weaknesses \n- My major complaint is the disconnection between the simplified linear classification of Noisy Gaussian Mixture Model and deep neural networks that distillation has found success in. While the authors of the paper tried to supplement the theory with experiments on CIFAR-10 classification using pretrained ResNet backbone, the experiments are still quite different from the real-world setting that self-distillation is applied. Indeed, the entire neural network weights are typically trained during self distillation. It would be much more interesting if the authors of the paper can also show that the conclusions drawn with linear classifier can be observed in the more general and realistic scenario as well.\n\nQUESTIONS:\n- The authors of the paper identified to two practical heuristics to enhance self distillation. How do they perform when applied to more realistic scenarios (e.g. ResNet for CIFAR10/100 when the entire neural network is trainable)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}