Title: Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Retraining

Predicted Labels

Label Noise

Approximate Message Passing
Abstract: Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model’s performance. While prior works have demonstrated the benefits of specific heuristic retraining schemes, the question of how to optimally combine the model's predictions and the provided labels remains largely open. This paper addresses this fundamental question for binary classification tasks. We develop a principled framework based on approximate message passing (AMP) to analyze iterative retraining procedures for two ground truth settings: Gaussian mixture model (GMM) and generalized linear model (GLM). Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels, which when used to retrain the same model, minimizes its prediction error. We also quantify the performance of this optimal retraining strategy over multiple rounds. We complement our theoretical results by proposing a practically usable version of the theoretically-optimal aggregator function and demonstrate its superiority over baseline methods under different label noise models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of learning under noisy labels via retraining. Specifically, it investigates the optimal way to combine a model’s own predictions with noisy labels for binary classification, using the theoretical machinery of Approximate Message Passing (AMP). The authors derive the Bayes-optimal aggregator function for both Gaussian Mixture Models (GMMs) and Generalized Linear Models (GLMs), and analyze the iterative retraining dynamics through state evolution. They also propose a practical version for linear probing with binary cross-entropy loss, showing improved performance under high label noise.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nTheoretical originality: The work provides the first principled derivation of optimal label aggregation in retraining using AMP. This paper addresses a key question on how to optimally combine labels and predictions.\n\nWeaknesses:\nThe analysis is restricted to binary classification with linear models. No results are presented for multi-class settings or nonlinear deep networks.\nThe evaluation is limited to two binary classification datasets with artificially injected noise. Broader empirical validation (e.g., on naturally noisy datasets) is needed.\nThe paper is heavy on derivations and could be difficult to follow for non-experts. Additional intuition, figures, or explanations would aid understanding.\n\nQUESTIONS:\nThe authors claimed that this paper is the first work that addresses optimally using the predicted and given labels. To my knowledge, the following paper addresses how to combine the prediction and given labels to do label correction. \n\"SELC: self-ensemble label correction improves learning with noisy labels\"\n\nLIMITATIONS:\nThe analysis and evaluation both restricted to binary classification with linear models. Broader empirical validation is necessary.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary**\nThis paper investigates how to optimally combine model predictions with noisy labels during iterative retraining in binary classification tasks. The authors develop a theoretical framework based on Approximate Message Passing (AMP) under both Gaussian Mixture Model (GMM) and Generalized Linear Model (GLM) settings. They derive the Bayes optimal aggregator function for label fusion and characterize the asymptotic performance via state evolution. A practical variant is also proposed for linear probing with cross-entropy loss, demonstrating empirical gains in high label noise regimes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Clean Theoretical Results: The paper provides a rigorous analysis of retraining procedures using AMP. The state evolution analysis provides transparent insight into how retraining affects generalization over iterations.\n2. Optimal Aggregator Derivation: The derivation of a Bayes optimal aggregator for combining predictions and noisy labels is a significant theoretical contribution.\n3. Empirical Gains: The theoretical insights are translated into a practical aggregator that can be used for linear probing to fine-tune deep neural networks. Empirical results show that this strategy performs particularly well in high label noise settings.\n\n**Weakness**\n1. Limited to Linear Models and Binary Classification: The current results are restricted to binary tasks and linear predictors. Extension to multi-class or non-linear models remains open.\n2. Assumption of Uniform Label Noise: The theoretical analysis assumes uniform random label flips, which may not hold in many practical settings.\n3. Lack of Theoretical Guarantees for Practical Variant:\nWhile the proposed aggregator function is shown to work well in practice, the prediction error under this retraining strategy is not theoretically characterized, especially when the optimal aggregator function is estimated from data (e.g., it seems very difficult to obtain a consistent estimator for the optimal aggregator function for GLMs). A formal analysis of its generalization behavior or error bounds would strengthen the practical significance and reliability of the method.\n4. Limited Empirical Evaluation: Experiments focus on synthetic and semi-synthetic settings. More diverse benchmarks would better support claims of practical relevance.\n\nQUESTIONS:\n1. Extension to Non-Uniform Noise Models:\nCould the authors comment on the feasibility of extending the analysis to more realistic, non-uniform or instance-dependent label noise models? A discussion or simulation could help assess robustness beyond uniform noise.\n2. Interpretation of the Aggregator Function:\nThe Bayes optimal aggregator is theoretically elegant, but can the authors provide more insight into its behavior (e.g., monotonicity, dependence on noise level, bias-variance trade-off)? This would help practitioners better understand when and how to use it.\n3. Scalability and Runtime Considerations:\nHow computationally efficient is the AMP-based retraining approach compared to existing heuristics? A brief runtime comparison or complexity discussion would help readers assess its practicality for large-scale use.\n4. Path Toward Non-Linear Models:\nGiven the growing use of deep models, can the authors elaborate on what specific challenges they foresee in extending their approach to non-linear architectures, and whether AMP or related techniques may still apply?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the open question of how to optimally combine a model’s own predictions with the original (potentially noisy) labels during retraining. Focusing on binary classification, the authors develop a theoretical framework based on Approximate Message Passing (AMP) to analyze iterative self-retraining for two settings: a Gaussian Mixture Model (GMM) and a Generalized Linear Model (GLM). \nThe main contributions are: Bayes-Optimal Aggregator: Derivation of the Bayes-optimal aggregator function gt that combines the model’s current predicted label and the given noisy label at each retraining round such that retraining on this combination minimizes the model’s prediction error. To our knowledge, this is the first work providing a theoretically optimal way to mix predicted and given labels in retraining, extending prior retraining methods (which used heuristics like “full” or “consensus” retraining) by rigorously identifying the optimal mixture. AMP-based Analysis of Multiple Retraining Rounds: Introduction of an iterative retraining scheme formulated as an AMP algorithm. The authors derive a state evolution recursion that exactly characterizes the asymptotic behavior (as data dimension and sample size grow proportionally) of the model’s accuracy over successive retraining rounds. Using this analysis, they quantify how the optimal retraining strategy performs over multiple iterations, and they prove theoretical results such as Theorem 3.1 (asymptotic Gaussianity of model estimates via AMP) and Theorem 3.2/4.3 (formulas for the optimal aggregator in GMM/GLM).\nKey Theoretical Insights: They show that using the Bayes-optimal aggregator strictly improves the model’s signal-to-noise ratio each round (when the model is not already near-perfect) and dominates earlier retraining schemes. A notable insight is that retraining is not universally beneficial: if the initial model is very strong, further retraining can hurt performance, whereas if the initial model is weaker, retraining helps (they formalize a condition for this crossover).\nPractical Method and Experiments: Building on the theory, the authors propose a practical version of the optimal aggregator for real data. In particular, for a pretrained model fine-tuned with a linear probe, they use a bimodal GMM fit on the model’s logit distribution to approximate the Bayes-optimal combination of labels. This method, termed “BayesMix RT”, is evaluated on image classification tasks with artificial label noise. In experiments on a medical imaging dataset and a Food-101 subset, BayesMix RT significantly outperforms two baseline retraining strategies from prior work (full retraining and consensus-based retraining) under high label noise (45% flipped labels). For instance, after 10 retraining iterations on one task, BayesMix RT reached ~76.6% accuracy vs ~64–65% with the baselines, demonstrating a substantial performance gain in noisy conditions. \nThe paper’s contributions are thoroughly evaluated through theoretical proofs (with detailed appendices) and simulation/empirical results, providing a holistic understanding of optimal self-retraining.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper answers a clear open question - what is the optimal way to mix predicted and given labels during retraining? They obtain this  by deriving the Bayes-optimal solution for that mixture. This is a notable advance since prior works on label-noise retraining either retrained on predicted labels alone or used heuristics (e.g. ignore or trust labels based on agreement) but did not determine an optimal combination rule. The authors’ use of AMP theory to derive a closed-form aggregator (e.g. Equation (13) for GMM) is elegant and appears novel. \n\nThe paper is technically solid. It provides precise asymptotic analysis of the retraining dynamics via state evolution equations (Theorem 3.1, 4.2) and proves the optimality of their proposed aggregator (Theorem 3.2, 4.3) with all assumptions clearly stated. The proofs are given in a detailed appendix, and the authors reference where each result is proven. The analytical work is thorough. For instance, authors even discuss how to handle non-Lipschitz aggregator functions via approximation to satisfy AMP conditions.  This level of rigor is required for  high quality paper in the theory development.\n\nThe theoretical results yield interesting insights into when retraining helps vs hurts. Proposition 3.3, for instance, shows that if the initial model’s accuracy is below a certain threshold (related to a fixed point of their state evolution), each retraining round will improve accuracy, but if it’s above that threshold, retraining could degrade accuracy. This explains and quantifies a phenomenon that practitioners might have observed anecdotally. The paper also neatly recovers prior strategies as special cases – e.g., “full retraining” corresponds to using only model predictions. \n\nDespite the heavy theory, the authors connect to practice by devising the BayesMix RT method for training a linear probe with noisy labels. This method is directly inspired by their theory (essentially implementing a softened version of the theoretical aggregator using a fitted mixture model). The experiments, though limited in scope (two datasets), demonstrate significant empirical gains in extremely noisy scenarios. This suggests the theory has practical merit. Furthermore, the authors conducted simulations (Appendix D) to verify that their state evolution predictions match finite-sample behavior and that the optimal aggregator improves over simpler retraining approaches.\n\nQUESTIONS:\nThe work is restricted to binary class labels. How challenging do the authors expect it to be to extend the Bayes-optimal aggregator derivation to a multi-class setting?  Could the authors elaborate on what the optimal aggregator might look like for $K>2$ classes?\n\nThe optimal aggregator formulas (e.g., Eq. (13) for GMM) explicitly depend on the label flipping probability $p$ (and class priors). In practice, $p$ might not be known exactly. Did the authors experiment with estimating $p$ from data or treating it as a tuned parameter? How sensitive is BayesMix RT to mis-specification of $p$? It would be useful to know if using a slightly wrong noise rate (or an adaptive estimate of it) significantly hurts performance or not. Similarly, the practical method requires fitting a GMM to logits and if this fit is imperfect (e.g. if the logit distribution is not a clean mixture of two Gaussians), does it degrade the results? Clarifying the robustness of the approach to these assumptions would help practitioners know how to apply it. \n\nThe theory (Proposition 3.3) indicates a scenario where retraining can degrade performance if the initial model’s accuracy is above a certain threshold relative to noise. This is a compelling insight. Could the authors provide more intuition or empirical confirmation for this phenomenon? In the experiments presented, BayesMix RT always improved accuracy over 10 iterations, even when starting from a reasonably good initial model (e.g., ~79% in Table 5 for p=0.30). Did the authors observe any instance where continuing to retrain actually made things worse (for either their method or the baselines)? It would be useful to know, for example, if one should stop retraining after a certain point. \n\nThe empirical demonstration is for retraining only the last layer of a neural network. Do the authors have thoughts on extending BayesMix retraining to fine-tuning an entire deep network? In principle, one could use the model’s predictions (or soft probabilities) in the loss for all layers. Would the aggregator still be applicable or would new complications arise (e.g., the model’s feature representation shifting while we alter labels)? This is related to the assumption in GLM analysis that features are fixed i.i.d. random vectors. How might one integrate the optimal label aggregation into an end-to-end training pipeline?\n\nThe paper assumes uniform random noise (each label flipped with the same probability p). If the noise were class-dependent (each class has its own flip probability) or feature-dependent (hard examples more likely mislabeled), the current aggregator might not directly apply. In such cases, do the authors expect the strategy to change?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work and potential societal impacts. They explicitly discuss key limitations in Section 6 (second paragraph): notably, the restriction to linear models, binary classification, and uniform noise. They frame these as avenues for future work, indicating awareness that the method needs extension to handle multi-class data and more complex noise distributions. They also acknowledge that their current practical method is tailored to linear probing and that extending to nonlinear deep models is a future goal. This openness about scope is commendable and readers are clearly informed about where the results apply or not which is good practice.\n\nOne limitation not deeply discussed (but worth reflecting on) is the assumption of knowing the noise level $p$. In practice how would one estimate it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious work has proved the benefits of retraining, but few work have answered what is the optimal combination of the model's predictions and the provided labels. This paper addresses the fundamental question for binary classification tasks. They develop a framework based on approximate message passing to make an analysis for GMM and GLM. The contribution is the derivation of the bayes optimal aggregator function. Based on this, the authors propose a practically usable version of their theoretically optimal aggregator function. This practical version is tailored for linear probing with the cross-entropy loss. Through analysis and experiments, they demonstrate its superiority over baseline methods (like full retraining and consensus-based retraining) in high label noise regimes. The paper distinguishes its method from existing \"full retraining\" and \"consensus-based retraining\" schemes, which use hard predicted labels or only samples where predictions match noisy labels, respectively.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1. The paper presents a good theoretical framework based on AMP to analyze iterative retraining. And based on the theoretical analysis, the paper develops a strategy to combine given labels and predicted labels for linear probing with the binary cross-entropy loss.\n2. The paper is the first work to analyze the optimal way to combine predicted and given labels for any retraining-like idea.\n3. The paper has a good writing structure and clearly articulates the problem statement and its motivations\n\nWeakness:\n1. The current theoretical results are primarily for linear models and binary classification. Although not necessary for this paper, it should be better to try the framework in a more general setting, like self-training for other deep learning models. At least one case should make the method more scalable and convincing research communities that it can be incorporated into recent AI system.\n2. Although the theoretical results are solid, the experimental results are using uniform noise. It is unknown how it can generalize to more complex scenarios like data dependent noise.\n\nQUESTIONS:\n1. How can you make a connection of the cases you study in the paper to other non-linear deep learning models? I would like to see the discussion on the practical computational overhead of the optimal method, particularly concerning the complexity of the optimal aggregator and its derivatives within the AMP updates, and how this might impact its usage on very large models or datasets.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}