Title: On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
Venue: NeurIPS 2025
Accepted: yes
Keywords: weak-to-strong generalization

deep learning theory

linear regression

feature learning

regularization

ridge penalty

over-parameterization
Abstract: Weak-to-strong generalization—where a student model trained on imperfect labels generated by a weaker teacher nonetheless surpasses that teacher—has been widely observed, but the mechanisms that enable it have remained poorly understood. In this paper, through a theoretical analysis of simple models, we uncover three core mechanisms that can drive this phenomenon. First, by analyzing ridge linear regression, we study the interplay between the teacher and student regularization parameters and prove that a student can compensate for a teacher’s under-regularization and achieve lower test error. We also analyze the role of the parameterization regime of the models and show that qualitatively different phenomena can happen in different regimes. Second, by analyzing weighted ridge linear regression, we show that a student model with a regularization structure better aligned to the target function, can outperform its teacher. Third, in a nonlinear multi‐index learning setting, we demonstrate that a student can learn easy, task-specific features from the teacher while leveraging its own broader pre-training to learn hard‐to‐learn features that the teacher cannot capture.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines weak-to-strong generalization in several theoretical settings focused on ridge regression and multi-index regression. The authors identify a number of settings in which weak-to-strong generalization may occur, namely when the teacher is under-regularized, the student has partial access to ground truth weights via weighted ridge regularization, or the student has partial access to ground truth weights in a multi-index setup.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**. The paper touches on a timely and fascinating topic. The results are precise. The writing is clear.\n\n**Weaknesses**. I am likely not the target audience for this paper, so please take my comments with a hefty grain of salt. Please also defer to comments from reviewers who are more firmly placed in the target audience.\n\nThe results are impressively precise and the paper overall appears to be a nice application of increasingly popular RMT ideas, but I'm left unsure what to take away from this paper. The general finding seems to be that weak-to-strong generalization occurs when 1) the teacher is under-regularized for a ridge regression target (where regularization has a very concrete meaning), or 2) when some aspect of the target leaks to the student, either through a regularization weight or multi-index component.\n\nFor the first case, the result is quite interesting, though I'm unsure how to interpret the magnitude of difference. At best, for a very large regularization mismatch, the student outperforms the teacher by ~0.05. Is this a large or significant margin? Based on the plotted scale, this seems to be a fairly small difference. Should I interpret this result to mean the student can very slightly outperform the teacher when the teacher is virtually un-regularized? More broadly, what would regularization mean in an applied setting outside of ridge regression? Drawing from Burns et al. example you highlight as motivation, would GPT-2 be in some sense \"under-regularized\" for natural language, while GPT-4 compensates?\n\nFor the second case, it is perhaps unsurprising that a student would outperform the teacher if some aspect of the target function leaks to the student while remaining inaccessible to the teacher. Repeating the question from before, would you imagine that GPT-4 has access to some ground truth about natural language that remains out-of-reach for GPT-2, and for this reason succeeds at weak-to-strong generalization?\n\nOverall, while the paper presents a nice theoretical argument, I'm unsure what the broader impact of these results are. Is there an actionable practice I can take away for performing weak-to-strong generalization on a real world task? Do we have a deeper understanding of weak-to-strong generalization beyond a linear setting or when the target leaks?\n\nQUESTIONS:\nPlease see Strengths and Weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends current research on weak-to-strong generalization by identifying three mechanisms that facilitate such phenomena: (1) compensation for teacher under-regularization in ridge regression; (2) better alignment of student regularization structures with the target function under weighted ridge regression; and (3) effective utilization of pretraining by the student in nonlinear settings.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. The paper is well-written with clear logical flow and appropriate formalism.\n2. Derivations are detailed and reflect strong theoretical proficiency.\n3. The extension to nonlinear settings is interesting and adds depth to the study.\n\n\n## Weaknesses:\n1. In the nonlinear analysis, the definition of the *easy* link function appears to reduce the setting effectively to a linear case. This makes the learning of $\\beta_e$ trivial.\n2. If I understand correctly, in the nonlinear case (Theorem 11), the $\\beta_h$ component in the student model originates entirely from pretraining, with early stopping controlling its degradation during subsequent training. However, even though the student learns $\\beta_h$ while the teacher does not, if the constant $c$ in Proposition 10 is significantly larger than $c_e$ in Theorem 11, it is still unclear whether the student outperforms the teacher. If the presence of $\\beta_h$ alone sufficed, the student would already outperform the teacher without any fine-tuning, which is not the intended conclusion.\n3. Overall, while I consider the paper to meet the acceptance threshold, many of its key results and proofs hinge on the assumption that the student benefits from better pretraining while only modestly learning from the teacher’s supervision. As a result, the conclusions—though valid—are somewhat predictable, which limits the novelty of the contribution.\n\nQUESTIONS:\nI have already raised concerns in the Weaknesses section，and none of them is very critical.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the question of when a student can surpass a teacher in isotropic-gaussian linear regression and a two-index model trained with a single step of gradient descent. In the linear case, the ability to surpass the teacher hinges on increasing regularization. In the two-index model, the student is initialized with a non-trivial alignment with the desired direction.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is clearly written and easy to follow. The underlying topic is timely and well motivated.\n\n- It is always nice when new observations are made in classical settings. I find Theorem 5, which clearly characterizes when the student can surpass the teacher in the linear setting (as long as equation 5 holds), to be a nice addition to the literature.\n\n- The contour plots nicely complement the results.\n\nWeaknesses:\n- The isotropic-Gaussian setting is strict. By now, there are many papers analyzing linear regression in much more general settings. \n\n- I'm not sure that the results of section 3 actually showcase weak to strong generalization, since the student is assumed to be initialized with weights that are already aligned with $\\beta_h$. Training follows only a single gradient step, which prevents the student from changing significantly in the $\\beta_h$ direction. This is fundamentally different from the case where the student achieves lower loss than the teacher despite being trained in a similar way. \n\n- I believe the way the authors discuss the setting of section 3 in the abstract and introduction is misleading. Rather than saying things like \"We assume that the easy component is highly specialized and task-specific, but, the hard component is a component shared across many tasks\", it would be much more faithful to the theorems to state directly that the student weights are already aligned with the hard component. The storytelling regarding multiple tasks is not directly reflected in the math. It is fine to use this as motivation for the actual assumptions, but not to replace stating them.\n\nQUESTIONS:\n- The statements of Proposition 10 and Theorem 11 appear strange when it comes to the constants. Are these absolute constants? Shouldn't $c_h$ depend on $\\tau$?\n\n- In figure 2, why did you increase the noise from the left and right figures? Is equation 5 not satisfied when $\\sigma=1$?\n\nLIMITATIONS:\nSee the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the mechanisms of weak-to-strong generalization, on three different models, linear models trained using standard and weighted ridge regression, and generalized linear models with feature learning in the asymptotic regime. Their analysis includes both the over-parametrized and the under-parametrized regime and they provide explicit condition in which the student can outperform the teacher and when it fails to do so.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\nWeak-to-strong generalization is a modern paradigm important to the improvement of performance in foundational models. This paper has provided precise conditions for linear models, when using a stronger regularization strength, the student's model's generalization error can outperform that of the teacher's. Furthermore, by analyzing the weighted ridge regression problem with a spiked covariance model, they show that the student can surpass the teacher if it has been adequately pre-trained. Finally on a generalized linear model which involves learning the two \"easy\" and \"hard\" directions, they showcase that applying one step of the gradient descent to the teacher's model can be still useful if the student has been sufficiently pre-trained. \n\nOverall the paper is well-structured, well-written and easy to follow and their numerical experiments on synthetic supports their claims which align well with intuition.\n\n**Weaknesses**\n\n**1.** I believe that adding a discussion about the effect of the noise level improves the comprehensiveness of the paper. Namely, the question is as follows; given a budget on $\\lambda_s$, what is the best the student can do in the face of various SNR levels. An study of low-SNR vs high-SNR regime is pertinent to the discussions presented in the main body.\n\n**2.** The discussion around the exhibition of double-descent phenomenon in Appendix E should be included in the main body.\n\n**3.** I recommend going beyond the synthetic and conducting experiments on real-world data such as MNIST and CIFAR-10 datasets.\n\nI would like to add that I am open to increasing my scores in case the authors adequately address the points raised here.\n\nQUESTIONS:\n**1.** Line 129, \"that\" should be changed to \"than\".\n\n**2.** In lines 209-210, what does the statement \"if $\\gamma_s > 1$ then some information is lost\" mean?\n\n**3.** In line 267, doesn't taking $\\zeta \\rightarrow 0$ lead to recovering the setting in section 2.1? In that case, there are values of $\\lambda_s$ for which the student can still outperform the teacher. I would appreciate the authors' comment on this.\n\n**4.** In line 351, \"teaching\" should be changed to \"teaches\".\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}