On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
Authors: Behrad Moniri
Hamed Hassani
Venue: NeurIPS 2025 poster ()
Keywords: weak-to-strong generalization
deep learning theory
linear regression
feature learning
regularization
ridge penalty
over-parameterization
Primary area: theory

Abstract: Weak-to-strong generalization—where a student model trained on imperfect labels generated by a weaker teacher nonetheless surpasses that teacher—has been widely observed, but the mechanisms that enable it have remained poorly understood. In this paper, through a theoretical analysis of simple models, we uncover three core mechanisms that can drive this phenomenon. First, by analyzing ridge linear regression, we study the interplay between the teacher and student regularization parameters and prove that a student can compensate for a teacher’s under-regularization and achieve lower test error. We also analyze the role of the parameterization regime of the models and show that qualitatively different phenomena can happen in different regimes. Second, by analyzing weighted ridge linear regression, we show that a student model with a regularization structure better aligned to the target function, can outperform its teacher. Third, in a nonlinear multi‐index learning setting, we demonstrate that a student can learn easy, task-specific features from the teacher while leveraging its own broader pre-training to learn hard‐to‐learn features that the teacher cannot capture.

On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
Behrad Moniri University of Pennsylvania Philadelphia, PA bemoniri@seaz upenn edu
Hamed Hassani University of Pennsylvania Philadelphia. PA hassani @aens.@@ees... upenn. edu
Abstract
generalization -were a student model trained on imperfect la- bels generated by weaker teacher nonetheless surpasess that teacher- has been widely observed, but the mechanisms that enable it have remained poorly under- stood. In this paper, through theoretical analysis of simple models, we uncover three core mechanisms that can drive this phenomenon. First, by analyzing ridge linear regression, we study the interplay between the teacher and student reg- ularization parameters and prove that student can compensate for teacher's ' under-regularization and achieve lower test error. We also analyze the role of the parameterization regime the models and show that qualitatively different phenomena can happen in different regimes. Second, by analyzing weighted ridge lincar regression, we show that a student model with a regularization structure better aligned to the target function, linear multi-indes learning setting, we demonstrate that a student can learn easy, task-specific features from the teacher while leveraging its own broader pre-training to learn hard-to-learn features that the teacher cannot capture.
outperform its teacher. Third, in non-
Introduction
Weak-to-strong generalization refers to the phenomenon where a strong (student) model trained on data produced by weak (teacher) model can sometimes significantly surpass the teacher's performance. This concept was first introduced by Burns et al. [2024], where the authors fine-tuned the GPT-2 model [Radford et al., 2019] (the teacher) for specific task using ground-truth labels. subsequently employing the fine-tuned model to generate synthetic samples for the same task. These synthetic samples were then used fine-tune GPT-4 [Achiam al., 2023] (the student). Remarkably, the fine-tuned student model outperformed its teacher in certain settings despite having access only to the imperfect synthetic data generated by the teacher.
Weak-to-strong generalization an especially important phenomenon from a practical perspective because of its implications for the emerging question of superaliguuentt [OpenAI, 2023]; i.c., can humans steer models with potentially superhuman capabilities to become aligned to human norms and values [Burns al., 2024]? Considering the weak model a proxy for humans, the possibility of the weak-to- to-ttoo- generalization phenomenon suggests that the answer can be affirmative.
Despite its practical importance, the mechanisms that enable weak- weak-to-strong generalization still not fully understood. Regularization has empirically been shown to play critical role in enabling weak-to-strong generalization. However, despite recent theoretical progress demonstrating that regularizing the student is necessary in some settings [Medvedev et al., 2025], the full picture of the effects and the interplay of the regularization of both the student and teacher models in weak-to-strong generalization is still unclear. For example, prior work on weak-to-strong generalization mainly focus on ridgeless regression (see e.g.. Dong et al. [2025], Xue et al. [2025], Ildiz al. [2025], etc.).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Additionally, prior work assumes that the teacher and student models have frozen representations, and only a linear head is trained through convex objective (see e.g.. Ildiz al. [2025], Medvedev et al. [2025], Dong et al. [2025], Xue et al. [2025], Charikar et al. [2024], etc.). However, fine-tuning can in practice go beyond this linearized regime and update model features as well. Burns et al. [2024] empirically demonstrated that updating the features yields substantially stronger weak-to-strong gains than only updating a linear head. For these cases, a linearized theoretical model might not suffice to capture all the relevant phenomena. This motivates a theoretical study beyond the linearized regimes and an analysis of the role of feature learning on weak-to-stropp generalization.
To take steps towards better understanding these aspects of training on weak-to-strune generalization, in this paper, we conduct a thorough theoretical study of this phenomenon in prototypical theoretical models. For the linear setting. we let the student and the teacher be high-dimensional standard and weighted ridge regression models and study how the explicit regularizations of the teacher and student models affect weak-to-stte -t-rnn generalization. We also investigate the role of the parameterization regime of the models. As the choice of regularization, we consider ridge and weighted ridge penalties. For the nonlinear case, we consider the problem of learning from multi-index models where the models learn relevant features through non-convex optimization objective. By studying this setting, we characterize how knowledge propagates between the models.
1.1 Contributions
Here we discuss the main contributions of the paper. We characterize three mechanisms that can enable weak-to-stronp generalization.
In Section 2.1, we consider a setting where the student and the teacher are trained with ridge regression. We fully characterize the test error of the models in the high-dimensional proportional regime by deriving asymptotic expressions for the test errors. Using these expressions, we study the conditions where the student model outperforms the teacher. We show that the student model can outperform the teacher by adequately compensating the vnder--egullrizztioon of the teacher. We further prove that different parameterization regimes of the student model can result in qualitatively different phenomena.
In Section 2.2, we consider a setting where the teacher again trained with ridge regression. However, we train the student with weighted ridge regularization. We again fully charac- terize the limiting test errors of the models in the high- sigh-dimensional proportional limit, and show that weak-to-strong generalization can happen when the regularization structure of the student is better suited the task.
In Section 3, we study learning form a nonlinear multi-index learning function that can be decomposed to a mix of easy- and hard-to-learn components by applying a single step of gradient descent on the first layer weights of two-layer neural networks. We assume that the easy component is highly specialized and task-specific, but, the hard component is a component shared across many tasks. We consider case where the strong student model has already learned the hard component during pre- treitrain but not the easy (yet task specific) component. We show that even if the teacher model is not able to learn the hard components on its own, a pre-trained student can learn the easy component from the teacher while still retaining the knowledge from pre-training for the hard component.
1.2 Related Works
The machine learning community has shown growing interest in weak-to-strong generalization. In this section, we review these results.
Theoretical Results. Prior work examines scenarios in which both the student and teacher rely on fixed, pre-trained feature representations. Wu and Sahai [2024] analyze stylized classification task under an over- paramaaraeteei spiked-cove covariice model with Gaussian covariates, where the teacher model does not have the capability to fit the target function, and the student has structure that is better aligned with the target. Ildiz al. [2025] investigate weak-to-strone -t--strong generalization for high- dimensional ridgeless regression. Building on this line and in similar setting, Dong et al. [2025], Xue et al. [2025] study how mismatches between student and teacher features affect generalization: Dong et al. [2025] focus a ridgeless, variance-dominate have negligible bias and show that weak-to-strong transfer occurs when the student's features have lower intrinsic dimension. Xue et al. [2025] consider the same setting and propose that the overlap
linear regime in which both models

---

between the subspace of features that teacher model has not learned, and the subspace of features that the student model has learned during pre pre-training govem took-to- generalization. In contrast, this paper analyzes linear models in the high-dimensional proportional regime, covering both under- and wver-parameterized cases, where models can have large bias. We also explicitly investigate the role of regularization on weak-to-strong generalization. Complementing these model-centric studies, Shin et al. [2025] take a data-centric approach and study the aspects of data enable weak-to-strong generalization.
Relatedly, Medvedev et al. [2025] consider two-layer neural networks with random first-layer weights (randon-features models) and show that, when the student is much wider than the teacher, early stopping essential for weak te-tono- generalization. However, they assume the teacher is already optimally trained and do not analyze the role of its training. Charikar et al. [2024] propose that the erroneous knowledge that the strong model does not obtain from the weak model characterizes how much the strong model improves over the weak model.
Empirical Studies. Following the pioneering work of Burns et al. [2024], different variants and applications of weak-to-strong generalization have been studied. Bansal et al. [2025], Yang al. [2024] let the weak model generate data with chain-of- of-thought to supervise the student models. Ji al. [2024], Tao and Li [2024] use weak-to-strong generalization for the problem of alignment. Guo et al. [2024] study this phenomenon in vision foundation models. Liu and Alahi [2024] propose a hierarchical mixture of experts method boost weak-to-strong generalization. Mulgund and Pabbaraju [2025] characterize the gain in performance of the student model over the teacher model in terms of the misfit between the models.
1.3 Notation
We denote vector quantities by bold lower-case, and matrix quantities by bold upper-case, We use IH-llog I1- to denote the operator (spectral) and Frobenius norms. Given an indexed set of vectors
{x; [ we use the upper case to denote the (row-wise) stacked matrix, e.g. X [x] xn]". Throughout the paper, we use the standard asymptotic notation +p to denote convergence in probability.
((-). @()) Finally, we use
2 The Linearized Case
During the fine-tuning of pre-trained large-scale models, the training dynamic often falls into kemell regime where the features are not evolved [Wei et al., 2022, Malladi et al., 2023]. Motivated by these observations, in this section we cast the fine-tuning problem as a linear regression problem over Gaussian features. We aim to analyze the role of student and teacher regularization, and also the parameterization regimes of the models in weak-to-strong generalization.
Assume that the teacher model has access to n, independent samples S, = {(x,, W)) drawn according to
x; ~ N(O. Ia.). y: = BT x; Ei
where B. E IRxx is an unknown target vector, and E; ~ N(O. o2) is an independent additive noise. The teacher model fe Rxd + R is fit on the features (x, Vi using these labeled samples. The teacher is then used to generate synthetic labels for n, E N unlabeled covariates S, { 1n,
drawn independently from the same distribution according to X; N(O, Lax) as ÿ. = (x;). These samples are then used to train the student model f. 12 R.
We focus on the following two settings, each showcasing different mechanism that can enable weak-to-strong generalization.
Setting Ridge Regression. We train the teacher f((x) = ST_ x and the student fx()) BT using (standard) ridge regression. We prove that properly regularized student can outperform the teacher, in the case where the regularization parameter of the teacher is set be smaller that the optimal regularization parameter. This is an example of weak-to-strong generalization through adequately compensating umdeer-egullrizztioo. Furthermore, we show that two qualitatively different scenarios can arise depending whether the student model is over- or paraet-duaaeetiized This aligns with Burns et al. [2024] and Medvedev al. [2025], which show that student regularization necessary for weak-to-strone generalization; we extend their work by analyzing the role of teacher
3

---

regularization, and a finer- anee-grained analysis of student regularization and model parameterization, revealing new phenomena.
Setting 2: Weighted Ridge Regression. We train fx(x) = 3. x using (standard) ridge regression
and (x) 3 x using weighted ridge regression [Hoerl and Kennard, 1970, Casella, 1980]. We show that the strong student model can leverage better regularization structure and outperform the weak teacher, even if the regularization parameter for the teacher is tuned optimally. We argue that student model can have more suitable regularization either by using architecture that is better tailored to the task or by benefiting from more effective pre-training.
We consider growing n4 The dy following the dineessinnni limit. Although results are proven for this asymptotic regime, through numerical experiments, we show that they still match simulations very well, even for moderately large values of n2 n4, dx.
Assumption 1. Assume that nt, ny and dy all tend to infinity with a proportional rate; i.e.,
dg/n,, 7s > 0, and de/ng, + 7/t >
In this high-dimensional limit, we characterize the test errors achieved by the teacher and student models given by
C. Ex.y (y 3r o² + 11 3.112
C, = Ex.y
B. o² + 113, - 3.112
where (x, y) is an independent test sample drawn from (1), We then use these characterizations to study the conditions under which the student model outperforms the teacher.
2.1 Setting High-Dimensional Ridge Regression
In this section, we assume that the teacher fits a linear regression model ft(x) = 3T x trained on the samples S,, and is given by
B, argmin 3ER*
1
Σ
(v. + Ar 1
The
(x.I))ES
where d E RR is the teacher ridge regularization parameter. The student is also a linear model fx(x) = 3. trained on fresh samples S, labeled by the teacher model, and is given by
G = argmin
Σ X.ES,
B x; 3 x.)
1111311
(4)
BER
in which A IR is the student regularization parameter. We characterize the test error of these models in the high-dimensional proportional limit of Assumption Our characterization of the test errors C., Cg will be in terms of the following quantities from the random matrix theory literature (see e.g., Bai and Silverstein [2010]).
Definition 2. Let m():7) be the Stieltjes transform of the Marchenko-Pasiiee law with parameter ? evaluated at -X; i.e.
m({;>)= - d)MP(>)())
27d 7+
-
1
/ (1+1++++ 4n
s ++}
Also, for p E {s,t}. we define mp.1 = m(Ap; 7p) and mp.2
8m 2A 1pp77
The test error of 33 in the high- -igee-immnsional proportional limit has been studied extensively in the literature [Tulino and Verdú, 2004, Dobriban and Wager, 2018, Hastie al., 2022]. The following proposition characterizes the test error of 3, in our setting.
Proposition 3. Under the condition that G_ ~ N(O, Tax) independent of other sources of ran- domness the problem, in the high--imensional proportional limit of Assumption I, we have
-p 0² (2e )² X, 02
where m and m4, are defined in Definition 2.

---

Strnnacton Theory
Simiaatoon Thory
0.04
01
Theary A
Thery
0.15
0.04
0.3
4.1
40
0004
40 0.00
0.2
013
0.2
-0.12
12
43
0.1
0.1
48
0.05
0.2
0.4
0.5 0.8
0.2 0.4 0.5
0.8
Figure I: Test-error difference L, Lt as a function of (Ae, A,) in the setting of Section 2.1. Filled contours are numerical simulations, and the dashed red contours follow the expressions of Theorem 4. The solid curve marks C. = L. and the dashed black curve is X, = X; Left: under-parameterizze student. Right: over-parameterized student. See Section for more details.
In the following theorem, we study test error of the student model 32.
Theorem 4. Under the same assumptions as Proposition 3, the test errors of Gr and Br satisfy L. - L. +p A = (o - >x)) [[mL, - >.m222) (X. - 22,m,,1)] + Xmm (1 Aem1,)) where m1,1, M4,2, Ms,11 m2,2 are defined in Definition 2.
This theorem fully characterizes the limit of the test error of the student model in the high dimensional proportional limit. The formulas for the limiting errors derived in this theorem be used make numerical predictions for L, L.- Figure 1 shows an example, supporting that the theoretical predictions of Theorem 4 match very well with simulations even for moderately large d, n,, ne. See Section for more details on the experimental settings. We use this theorem to study the test error of the models function of overparamterriation in Section E.
In the next theorem, we use the formula for the limiting value - L, from Theorem to study the conditions on Ys Tt, Àg de, of under which the student model outperforms the teacher; i.c., the conditions of weak-to-strong generalization.
Theorem 5. Under the conditions of Theorem 4, the (limiting) test errors of the student and teacher models satisfy the following:
If Ax 2 o2 we have L. 2 Lt.
> < o2 two cases happen:
- 0 < 7s < 1, there exists X 2 such that C. < for all X E (0, X).
- If 7s > and if the parameters "t, Yso de, oe satisfy
1
(5)
V(1 71. A1)2 47
1 47 4 7s
then there exists X_ 2 2 0 such that C. < for all >g (2.74) Moreover, if (5) does not hold, we have L. 2 Lt.
Note that under the setting of this section, the optimal ridge regularization parameter for the weak model known be equal to Xニ o²7 [Dobriban and Wager, 2018, Theorem 2. Theorem states that if the teacher is over-regularized (A, 2 X;), the student can never outperform it. In the case that the teacher under-vegularized (A₃ < X;), the parameterization regime of the student model % plays key role. In particular, if % < (i.e., the student is inder-parameterizedl, the student model can outperform the teacher by further regularization as long as < X < X. However, long as (5) holds, X, should be larger than
if 7/44 > (i.e., the student in puereeerraelrriiee))) certain threshold for it to outperform the teacher. Otherwise, the student will always have a worse performance compared to the teacher.
5

---

The phase transitions predicted in Theorem 5 can be seen in Figure I, where for each (A,, A.) pair, we plot the contours of L. C, for given Ys Tts oz. In these plots, the solid red curves show the pairs (Aw; X,) for which L. L.. The left plot corresponds to the case where 7/8 < 1. It can be seen that when X₂ < >; the student model outperforms the teacher as long X₂ < x(x;; 7s: Tt Tz). Moreover, the student is always worse than the teacher when X > X The right plot corresponds to the case with T'+ > 1. In this case, it is seen that as predicted in Theorem 5, for some values of X, the student outperforms the teacher only if X₂ E (A) for some 0 < x_ < X4. See Section 4 for more details on the experimental setting.
Mechanism of error stems from compensating for the teacher's 's insufficient regularization. Thus, intuitively, similar to what is proven in Theorem 5, when the teacher is already over-regularized, the student is unable to achieve better performance by leveraging this mechanism. Also, note that when 7 > 1, the student model is weer-parameterized and as result, some information is lost. Thus, more regularization is required for the student to outperform the teacher. This can be seen as the reason why in this regime, non-zero lower bound exists for X to ensure this. Our results complement the results of Medvedev et al. [2025] who demonstrated that regularizing the student is essential to avoid overfitting to the mistakes in setting where the teacher is optimally trained.
Generalization. In this section, we show that the student's reduced
Non-Monotone Student Test Error Curves. Studying the test error L. as a function of the the teacher overparameterization y using Theorem 4, we observe that student model also exhibits the double descent phenomenon, where we can see a second bias-variance tradeoff in the test error beyond the interpolation limit [Belkin et al., 2019]. See Section E for more details. This is in line with findings in different linear regression setups such as standard ridge/ridgeless regression [Hastie al., 2022, Nakkiran et al., 2021] (which corresponds to double descent in the teacher model in our setting), ridge regression with correlated samples [Atanasov et al., 2024, Moniri and Hassani, 2025], and weighted ridge regression [Wu and Xu, 2020].
Universality. Similar to other high high-dimensional ridge regression problems (see e.g. Hastie et al. [2022]. Hu and Lu [2023], Montanari and Saeed [2022], etc.), we can use a simple Lindeberg exchange argument [Lindeberg, 1922, Korada and Montanari, 2011] 1] to show that A from Theorem 4 exhibits universality; i.e., the actual distribution of the samples does not matter and the limiting value of C. L, is determined only by the first and second moment of the covariates. Thus, our results in theory do hold much more broadly than the Gaussian case. The study of this universality is not central to our discussion and we leave it as future work.
Other Related Work. The high- nigh-dimensional ridge regression setting considered in this section related to the linear regression setting considered by Dohmatob et al. [2024] to study model collapse. However, in their setting, only the downstream model (which corresponds to the student model in our setting) has a non-zero ridge regularization. Similar settings have also been studied in the self-distillation literature (see e.g., Das and Sanghavi [2023], Pareck et al. [2024], etc.). However, the training procedure of the models are different; e.g. in their setting, the teacher generates synthetic labels for its own training set and not for a fresh set of covariates. Additionally, in the self distillation setting, the student model still has access to ground truth labels.
2.2 Setting 2: High- Bigh--imensional Weighted Ridge Regression
In this section, we consider a setting where the strong model is a linear model trained using weighted ridge regression [Hoerl and Kennard, 1970, Casella, 1980, Wu and Xu, 2020, Richards et al., 2021]. Given samples S C Rdd x IR, the estimator WRidge(S, A, I) is defined as
WRidge(S, A) == argmin BER'x
Σ 3Tx)) -|-r- B1B (X.D))SS
(6)
n
where P E RRdxxxx is a weighting matrix and À E R is a scalar. In this section, we assume that the teacher is still a (standard) ridge regression estimator. However, unlike the previous section, we let the student model be a weighted ridge estimator; i.c.,
3 (S,,, ^o, Láx), A. = WRidge (S., >s, I)-
In this model, the matrix r is assumed to be given and fixed. The matrix T determines the structure of the student regularization enforeing different levels of regularization in different directions, In the next remark, we provide linear neural-network interpretation for r.

---

4, -
4 4-4
14
Tarse
Sundatton
0.8
144
0.04
Saadekko Tlamy
Tarney
0.08
1.2
Thamr C,
0.02
2 0.8
0
0.8
0.04
6.6
0.08
0.6
4.04
0.02
008
0.4
-0.0
0.
4.1
0.5
1.5
2.5
3.5
Lt as a function of (A₁. A,) in the setting Section 2.2. Filled contours are numerical simulations; dashed red contours follow the theory of Theorem 8. The solid curve marks L. = Lt, and the dashed black curve is A = A Left: Right: peeepaaameteri student. See Section 4 for more details.
Figure 2: Test-error difference L, -
student.
Remark 6. The weighted ridge estimator WRidge(S, X, F) can also be seen training the second layer of a two-layer linear neural network fNN x To; i.e., f To in which
â = argmin acR*
(yi x, Ta) + Alla
(x::B:)ES
In light of the connection to linear neural networks in Remark 6, as pre-trained neural network. During the pre-training, we assume that the student model has had access data from various sources with shared structure with B.. The goal of the pre-training is to use this data to learn features that align well with the underlying task 3. [Sun et al., 2021J. Motivated by a recent line of results in deep learning theory where the updated first layer weights are shown to have a spiked structure with a few directions having information about the target function [Ba et al., 2022, Moniri et al., 2024, Cui al., 2024, Zhang et al., 2025, Ba al., 2024, Demir and Dogan, 2024, Moniri and Hassani, 2024, Li and Sonthalia, 2024, Mousavi-Hosselie et al., 2023, Radhakrishnan al., 2024], we model the alignment of T with the task structure using oon-informative bulk component plus an informative low-rank component.
think of the student model
Assumption We assume that the matrix ddxxxxx given by
T [.]
T= Iay dx 3,9 with
||0||2 |. dr
where Ç E [0,
the correlation of the learned direction / to the target direction B. which is measure of how much ( aligns with the target direction B..
The prefactor dy for the spike term in (8) is chosen in a way to ensure that ||]| x |d 3 || This closely resembles the scaling of the updated weights with maximal update parameterization in the feature learning theory literature YYang and Hu, 2021, Ba et al., 2022]. the following theorem, characterize the test error difference of the models L, L, for this setting in the high-dimensional proportional regime of Assumption 1.
Theorem 8. Under the conditions of Proposition 3, the test errors of the student and teacher model from (7) with T from Assumption satisfy L. L. to 22 where the expression for A is given in Theorem 4, and
Ar = de (-1+ (-1+Amm,))) 22mm,,1m..1 + A.m,,2( (-1g2(1110 + \emr,1)
where m1,1, m4,2 M8.11 ma,2 defined in Definition 2. Additionally, we have Ar 0.
7

---

In the limiting formula for L C, the term A is equal to the limiting value L. L. in the setting of Section 2.1 where both models are trained using (standard) ridge regression, and the benefit of the learned features for the student is due to the term - Ar: which is always non-positive. Because of this term, even in the settings that A 2 0 (i.e., the mechanism of Section 2.1 not enough on its own for weak-to-strong generalization), the student may still outperform the teacher.
Figures 2 and 3 demonstrate that the asymptotic characterization of Theorem 8 match simulations very well even for moderately large na, n4, dx. In Figure 2, we fix Ys Yts and o and plot the contours of L. - L. for different (Ar, As) pairs. We show that unlike Section 2.1, in both settings with 7 > 1 or Y < 1, pairs Xx) with d > X; ob (i.e., over-regularized teacher) exist where the student model outperforms the teacher. In Figure 3, we set A = X and plot C, as function of X for different values of the feature quality parameter Ç. We see that for small ç, the student never outperforms the teacher, similar to the case in Section 2.1. However, this changes when Ç is increased, and the student can have smaller test error for some values of X
Mechanism of Weak-to-Strong Generalization. Theorem 8 shows that if the student model has been pre-trained and has learned features that are better suited for the task of predicting the target function (or equivalently is fine-tuned using a better regularization structure), it can leverage this advantage to achieve a better performance compared to the teacher, despite being trained on labels generated by the teacher. This shows yet another mechanism of weak-to-strong generalization.
3 The Nonlinear Case
In Section 2, we considered a setting where both the stu- dent and the teacher model trained linear head through convex objective. Although this is a very effective model for the analysis of the roles of regularization and over- parameterization, it coincides to the linearized regime of neural network training where features are frozen at their initialization. As a result, the linearized models are not rich enough study phenomena that happen as a result of (nonlinear) feature learning.
0.00
0.68
45
0.59
0,98
12
Consider the problem of learning few distinct skills and abilities using data from a compositional task, where a blend of different skills are needed order to succeed, We model each skill as a vector in the high-dimensional input space Rdx, and learning a skill as learning the cor- responding direction. Take, as running example the task of holding a coherent conversation in an unfamiliar lan- guage. To succeed, the model needs to blend abilities such as logical reasoning, with language-specifff skills. Abil- ities such as logical reasoning are difficult to acquire, but transferable across domains. However, iagguage-specingg abilities such as vocabulary and grammar, conceptually hrraightfowaad, but they are task-specific and often cannot be learned from other related tasks.
(2
log A.)
Figure 3: Student error L. versus log A in the setting of Section 2.2. plotted for several values of Ç with the teacher op- timally regularized. Circles show simu- lation results, and dashed curves are the predictions of Theorem 8. The dashed black line marks the teacher error Lt- See Section 4 for details.
Motivated by this setting, in our model, we let the samples for the teacher model be generated according to a multi-index function (e.g., Box and Cox [1964], Bickel and Doksum [198]] I with an easy and a hard component. Here, the teacher has access to S; = {x; bi drawn from
X; N(O, Iax), and y: = (x (xTB.) + o Bx),
where Be,B, Rd are two orthonormal directions that we want to learn, and Ce, 0h IR -> R are two link functions. In this problem, the hardness of learning each direction Be, Bn from these samples is known to be characterized by the information-expoeen of their link function [Dudeja and Hsu, 2018, Ben Arous et al., 2021]. For any real-valued function a R + R with Hermite coefficients {Co, o k the information-exponet is defined as Nr := min l E EN Co.k + 0}. We make the following assumption on G. and d₁,
Assumption 9. Assume that Ko. and Kos > 1 (i.e., de is easy and o is a hard link function).
8

---

We let the student fa and teacher fa models be neural networks given by fa(x) = a (We xx and f((x) a a (W,x,, where R + R is activation function with KG = 1, and a, E R a, e RR, W, E Rex and W. E Rexxdx, To learn the relevant directions Per 3r. update the first layer weights to align them to these directions; task also referred to as weak recovery [Ben Arous et al., 2021, Dandi et al., 2023, 2024, Arnaboldi et al., 2024, Lee et al., 2024).
For training, we update the teacher model ft using the samples S and use ft to generate synthetic labels for n2 E N unlabeled covariates S,- {& | drawn from the same distribution according
to X ~ N(O, Tax) as t/s = fx,)). We then use these samples to update the student model fs- We consider the correlation loss defined as
;= - Σ ff((()), and E. := -2, ==1
Σ Us (xx.).
(10)
Fixing a, E IRP with ||r||| - ((), and initializing W, at W.,0 with i.i.d. N(O, d entries, we update W, using one-step of gradient descent on Lt given by
We We,0 Te V Z.lw.....
The following result, which is a corollary of Ba al. [2022. Proposition 21, shows that the after this update, W, aligns to the easy direction Ge. but does not align to the hard direction Bai i.e., the teacher model could not learn the hard direction using these samples.
Proposition 10. Under the high-dimension proportional limit of Assumption 1, and assuming that " O() and pt (dxx) we have
BeBeele +pc> O and W +>p
After this update, we set fx(x) = a, a ( (wx)). We assumed that the teacher has not gone through extensive pre-training
other tasks that depend the directions Br thus, we made the assumption that the W, is initialized at random. Recall that 3r is hard direction, but is relevant for a variety of tasks. Thus, we assume that the student has been pre-trained on a variety of different relevant tasks, and has already learned the hard but cross-domain ability that corresponds Ba; thus W. at initialization is aligned to 3x. In particular, we set W.,0 = W.,0 + 7 3 where W W..o .. E R xdx has N(O, 1) entries, 7 E IR and á E RPs is a unit norm vector. We update W, using one-step of gradient descent on L,; i.e..
W s,0 7/ Ww L.lw.....
Note that training Wa excessively on data generated from the teacher can result in the student to
forget the direction 3r. However, in the next theorem, we show that a single step of SGD on Σ can induce non-trivial aligmment between W, and the easy direction B, while keeping the weights aligned to 3r. This aligns to the empirical and theoretical findings Burns et al. [2024], Medvedey et al. [2025] that show that early stopping the teacher is required for weak-to-strong generalization.
Theorem 11. In the asymptotic regime of Assumptioo I with Pt. O(dx), assuming that n), 7e ((1) and o( dg) we have
W W.Bell2 +p Ce > O and W W.B + Ch
This theorem is a extension [Ba et al., 2022, Proposition 2] the case where the first-layer weight is initialized as a spiked random matrix and the labels are also generated by another one-step updated two-layer neural network, which can be also of independent interest. This theorem shows that, under this setting, the student model is still able to learn the direction B. from the imperfect labels generated by the teacher, and achieve non-vanishing alignment to both the directions (3e, Br. although the student was unable to learn the hard direction Br.
Mechanism of Weak-to-Strong Generalization. In this setting. a teacher model can acquire the easier, yet specialized skills through fine-tuning, even though it cannot learn abilities that are challenging to learn. By contrast, the student is pretrained on vast, heterogeneous corpora and already possesses those hard, yet cross-domain, skills. As a result, weak-to-stron generalization can happen when the teacher teaching the student the specialized language abilities, thereby complementing the student's ' strengths from pre-traiee pre-training.

---

Numerical Validation
In this section, we provide the details of the simulations presented throughout the papers.
Figures 1 and 2. We fix the values of dx, n4, and and plot the contours of L. -. using numer- ical simulations and also the results of Theorem and The simulation results are averaged over ten trials. See Section 2.1 and 2.2 for discussions of the results. In Figure 1, for the under- muder-parameterized regime (left), we set dy = 500, N4, = n2 2000, 0g = 1 and for the over-parameterized regime (right), we set dx 500, T2 2000, nt 416, dg 2. In Figure 2, for the under- pader-parameterized regime (left), we set Ç = 0.8, dx = 500, n, = n2 = 2000, dg = 1, and the over paramettameeeee regime (right), set 0.88, dx 500, n2 2000, n, 416, de 1.
Figure 3. In these experiments, we d 500, n4 n, 2000, 0g and set X = 0.25. We compare the theoretical curves of L. as a function of 7 with numerical simulation, for E {0, 0.68, 0.89, 0.98). The simulations in this experiment have not been averaged over multiple trials. See Section 2.2 for discussion of the results.
Conclusion
In this paper, we show that weak-to-strong generalization not unique to complex language models and can happen even in much simpler theoretical setups. By studying three natural and tractable learning theoretical settings, we identified and theoretically analyzed three distinct routes by which a student model can outperform its teacher: (1) compensating for under-regularizations (2) harness- casy-to-learn ing a more task-aligned regularization structure, and (3) combining saahhrr-tugght components with pretrained, hard-to-lent features. Our results clarify when and why these effects arise, complementing prior empirical and theoretical insights about the roles of regularization and overparameterization, and the role of feature adaptation.
6 Acknowledgments
Behrad Moniri gratefully acknowledges the gift from AWS AI to Penn Engineering's 's ASSET Center for Trustworthy AI. The work of Behrad Moniri and Hamed Hassani is supported by The Institute for Learning-enables Optimization at Scale (TILOS), under award number NSF-CCF-21 22CFF-2116665. and the NSF CAREER award CIF-1943064.
References
Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical tables, volume 55. US Government printing office, 1968.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. OpenAl Technical Reports, 2023.
Luca Arnaboldi, Yatin Dandi, Florent Krzakala, Luca Pesce, and Ludovic Stephan. Repetita iuvant: Data repetition allows SGD to learn high-dimensional multi-index functions. arXiv preprint arXiv:2405. 15459, 2024.
Alexander Atanasov, Jacob Eavatone-Veth, and Cengiz Pehlevan. Risk and cross validation in ridge regression with correlated samples. arXiv preprint arXiv:2408.0448 046007 2024.
Jimmy Ba, Murat Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How gradient step improves the representation. In Advances in Neural Information Processing Systems, 2022.
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, and Denny Wu. Learning in the presence of wew-dmmensionall structure: a spiked random matrix perspective. In Advances in Neural Information Processing Systems, 2024.
Zhidong Bai and Jack W Silverstein. Spectral Analysis of Large Dimensional Random Matrices. Springer, 2010.
10

---

Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training LLM reasoners via compute-optimal sampling. In International Conference on Learning Representations, 2025.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learnin practice and the classical varinnae trade-off. Proceedings of the National Academy of Sciences, 116632):15849-1555 15854, 2019.
Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research, 22 (106):1-51, 2021. 2021.
Peter J Bickel and Kjell A Doksum. An analysis of transformations revisited. Journal of the American Statistical Association, 76(374):296-31. 1981.
George EP Box and David R Cox. An analysis of transformations. Journal of the Royal Statistical Society Series B: Statistical Methodology, 26(2):211 1--223,, 1964.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar. Jan Leike, al. Weak-to-strone generalization: eliciting strong capabilities with weak supervision. In International Conference on Machine Learning, 2024.
Mireille Capitaine and Catherine Donati-Martn Strong asymptotic freeness for wigner and wishart matrices. Indiana University mathematics journal. pages 767-803, 2007.
George Casella. Minimax ridge regression estimation. The Annals of Statistics, 8(5), 1980.
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to-strong generalization. In Advances in neural information processing systems, volume 37, 2024.
Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue Lu, Lenka Zdeborova, and Bruno Loureiro. Asymptotics feature learning in two-layer networks after one gradient-step. In International Conference Machine Learning, 2024.
Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan, Learning two-layer neural networks, one (giant) step at time. arXiv preprint 2..iv:2305..827777 2023.
Yatin Dandi, Emanuele Troiani, Luca Amaboldi, Luca Pesce, Lenka Zdeborova, and Florent Krzakala. The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. In Forty-first International Conference on Machine Learning, 2024.
Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In International Conference on Machine Learning, 2023.
Samet Demir and Zafer Dogan. Random features outperform linear models: Effect of strong
input-label correlation in spiked covariance data. arXiv preprint arXiv:2409.20250 2024.
Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1):247-279, 2018.
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. In Advances Neural Information Processing Systems, 2024.
Yijun Dong, Yicheng Li, Yunai Li, Jason D Lee, and Qi Lei. Discrepancies are virtue: Weak-to- strong generalization through lens of intrinsic dimension. International Conference on Machine Learning, 2025.
Rishabh Dudeja and Daniel Hsu. Learning single-index models in gaussian space. In Conference On Learning Theory, 2018.
Adina Roxana Feier. Methods of proof in random matrix theory, volume 9. 2012.

---

Michael J Feldman. Spectral properties of elementwise-rrasffored spiked matrices. SIAM Journal on Mathematics of Data Science, 7(2):542-571, 2025.
Alice Guionnet, Justin Ko, Florent Krzakala, Pierre Mergny, and Lenka Zdeborová. Spectral phase transitions in non-linear wigner spiked models. arXiv preprini arXiv:2310. 1000:555,0 2023.
Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vision superalignment: Weak-to-strong generalization for vision foundation models. arXiv preprint arXiv:2402.03749, 2024.
David Lee Hanson and Farroll Tim Wright. A bound on tail probabilities for quadratic forms in independent random variables. The Annals of Mathematical Statistics, 42(3):1079-1083, 1971.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-
dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949 986, 2022.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55 1970.
Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features.
IEEE Transactions Information Theory, 69(3), 2023.
M Emrullah Iidiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. In International Conference on Learning Representations, 2025.
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong. Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. In Advances in Neural Information Processing Systems, 2024.
Satish Babu Korada and Andrea Montanari. Applications of the lindeberg principle in communications and statistical learning. IEEE Transactions on Information Theory, 57(4):2440-25 24500, 2011.
Arun Kumar Kuchibhotla and Abhishek Chakrabortty. Moving beyond sub-gaussianity in high- dimensional statistics: Applications in covariance estimation and linear regression. Information and Inference: A Journal of the IMA, 1144) 2022.
Jason D Lee, Kazusato Oko, Taiji Suzuki, and Denny Wu. Neural network learns low-dimensional polynomials with SGD near the ceammation-thooretiee limit, In Advances in Neural Information Processing Systems, 2024.
Jiping Li and Rishi Sonthalia. Generalization for least squares regression with simple spiked covariances. arXiv preprint arXiv:2410.13991, 2024.
Jarl Waldemar Lindeberg. Eine neue herleitung des exponentialgesetzes in der wahrscheimlichkeit- srechnung. Mathematische Zeitschrifi, 15(1):211-25) 1922.
Yuejiang Liu and Alexandre Alahi. Co-supervised learning: Improving weak-to- toak--to-strong generalization with hierarchical mixture of experts. arXiv preprint arXiv:2402.15500, 2024.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, 2023,
V. A. Marchenko and L. A. Pastur. Distribution of eigenvalues in certain sets of random matrices. Mathematics of the
1(4):457-483, 1967.
Marko Medvedev, Kaifeng Lyu, Dingli Yu, Sanjeev Arora, Zhiyuan Li, and Nathan Srebro. Weak-to- strong generalization even in random feature networks, provably. arXiv preprint arXiv:2503.02877, 2025.
Behrad Moniri and Hamed Hassani. Signal-plus-noise decomposition of nonlinear spiked random matrix models. arXiv preprint arXiv:2405. 2024.
Behrad Moniri and Hamed Hassani. Asymptotics of linear regression with linearly dependent data. In Annual Learning for Dynamics and Control Conference, 2025.
12

---

Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. In International Conference on Machine Learning, 2024.
Andrea Montanari and Basil N Saced. Universality of empirical risk minimization. In Conference Learning Theory, 2022.
Alireza Mousavi-Hosseinn, Denny Wu, Taiji Suzuki, and Murat A Erdogdu. Gradient-based feature learning under structured data. Advances in Neural Information Processing Systems, 2023.
Abhijeet Mulgund and Chirag Pabbaraju. Relating misfit to gain in weak-to-strong generalization beyond the squared loss. arXiv preprint 2rXf::2501.19105 2025.
Preetum Nakkiran, Prayaag Venkat, Sham M Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. In International Conference on Learning Representations, 2021.
OpenAl. Introducing superalignment. OpenAl Blog, 2023.
Divyansh Pareek, Simon Shaolei Du, and Sewoong Oh. Understanding the gains from repeated self-distillation. In Advances in Neural Information Processing Systems, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mechanism for feature learning neural networks and backpropagatton-fiee machine learning models. Science, 383(6690) 2024.
Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge (less) regression under general source condition. In International Conference on Artificial Intelligence and Statistics, 2021.
Mark Rudelson and Roman Vershynin. Hanson- inequality and sub-gaussian concentration. Electronic Communications in Probability, 18: 18:1 2013.
Changho Shin, John Cooper, and Frederic Sala. Weak-to-strowe to-kknng- generalization through the data- centric lens. In International Conference on Learning Representations, 2025.
Yue Sun, Adhyyan Narang, Ibrahim Gulluk, Samet Oymak, and Maryam Fazel. Towards sample- efficient overparameterizee meta-learning. In Advances in Neural Information Processing Systems, 2021.
Leitian Tao and Yixuan Li. Your weak LLM is secretly strong teacher for alignment. arXiv preprint arXiv:2409.0883 2024.
Antonio M Tulino and Sergio Verdú. Random matrix theory and wireless communications. Commu- nications and Information Theory, 1(1):1-182, 2004.
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. Yonina C. Eldar and Gitta Kutyniok, editors, Compressed Sensing: Theory and Applications, page 210-268. Cambridge University Press, 2012.
Dan Voiculescu. Limit laws for random matrices and free products. Inventiones mathematicae, 104 (1):201-220, 1991.
Zhichao Wang, Andrew Engel, Anand Sarwate, Ioana Dumitriu, and Tony Chiang- Spectral evolution and invariance in linear-wide width neural networks. arXiv preprint arXiv:2211.06506, 2022.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning, 2022.
David Xing Wu and Anant Sahai. Provable weak-to-strong generalization via benign overfitting. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024.
13

---

Denny Wu and Ji Xu. On the optimal weighted & regularization in overparameterized linear regression. In Advances in Neural Information Processing Systems, 2020.
Yihao Xue, Jiping Li, and Baharan Mirzasoleiman. Representations shape weak-to-strong general- ization: Theoretical insights and empirical predictions. In International Conference Machine Learning, 2025.
Greg Yang and Edward J Hu. Tensor programs IV: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, 2021.
Yuging Yang, Yan Ma, and Pengfei Liu. Weak-to-strong reasoning. Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
Thomas T Zhang, Behrad Moniri, Ansh Nagwekar, Faraz Rahman, Anton Xue, Hamed Hassani, and Nikolai Matni. On the concurrence of layer-wise preconditioning methods and provable feature learning. In International Conference Machine Learning, 2025.
14

---

A Preliminaries
Given two matrices A, B E x"2, we denote their Hadamard product (element-wise product) by A o e R Kn2, Also, for for E N, we define the Hadamard power
A ok
E
R R1x22
Lemma 12. For any v E R" E I" and C E Ru X we have
(vu C diag(v) C Cdiag(u).
Proof. The proof is immediate by writing the entries of the two sides.
We also heavily leverage the following theorem prove the concentration inequality for quadratic forms. See e.g., Rudelson and Vershynin [2013] for modern proof.
Theorem 13 (Hanson- Wright
Wrriht Inequality [Hanson and Wright, 1971]). Let x (X 1..... x,) Rd
be a random vector with independent sub- sub-gaussian components X; with EX; = O. Let D be an n x matrix. Then, for every 2 we have
t2 I
P
> 2xp
-emin
D
where e is a constant that depends only on the sub- uub-gaussian constants of X;.
To analyze spiked random matrices, will use the following matrix identity.
Lemma 14. (Sherman-Morrison Morrison Formula). Let A E gax" be an invertible matrix, and let u, v E RR" 'u be column vectors such that 1 + ++v A -'uu * 0. Then the inverse of the rank-one update A + uv is given by:
A A T A
(A + uv
--- = -
1+ A -'u
A.1 Hermite Polynomials
We let Hg be the k-th (probabilist's) Hermite polynomial on R defined by
Hx(x) = (-1)* exp(x>/22
dx exp( x² 222 Va E R.
These polynomials form an orthogonal basis the Hilbert space L measurable functions f R IR such that
/ ( & dr <0
with inner product
(f.g) = / ((=))(x))== dx.
The first few Hermite polynomials are
Ho(x) = H1(x)= x, and H2(æ) =
Lemma 15. For any lo E N and x, y E R. we have
Ha + v)
Σ 3=0
x5 (
Hu))(1)
Proof. Note that using [Abramowitz and Stegun, 1968, Equation 22.8.8] we have
d H (x)
kH 1- (x).
dx
15

---

Thus, the j-th derivative of Hş is given by
d'
k! (k - j)!
Hx(x)
H (k-)(
dx.
By Taylor expanding H&(x + 3/ at y, we find
x di
# Σ
k
H₂ + v) = Σ
(v)
x3 Hk-)(u))
oز dy
1=0
f=0
proving the lemma.
A.2 Random Matrix Theory
We first define the following empirical covariance and resolvent matrices that will appear throughout the proofs.
Σ X X X/n1, Σ X X/na: Naa
R E + Ae and R R= ( 1xxx))
We will use the following characterization of the eigenvalues of E, Σ in the high-dimensional proportional limit by Marchenko and Pastur [1967].
Theorem 16 (Marchenko- Theorem). In the high- high-dimensional proportional limit where D4, Dgg dx co such that dy - " and dx - Tth the empirical spectral distribution (ESD) of ② (and Σ) converges almost surely to the Marchenko-Pastur distribution (MP())) (and (MP()))
Recall from definition that the function ml(:.) - IR is defined as
m(>:7) duMP()(s) + X
Hence, taking derivatives with respect to A, we get
m' m' (1: Ox m()=)) =- duMrMi)s))
(s 1)
In the following section, write the test enror L, and L, function of and its derivatives. In particular, note that using the Marchenko- -oostt theorem, we have
dz 'r ( +p me,1, d Tr (it2)
+p m2,2,
and
d Tr R +p Ma.1. d Tr Ri.)
+p ms2.
where for p e {s we define mp.1 = m(>p: 7p)) and mp.2 ax Ap,7%
0m
In the following proof', we will also use the asymptotic freeness of independent Wishart random matrices |Voiculescu, 1991, Capitaine and Martin,, 2007]. See [Feier, 2012, Section 3.4 and 4.4] for brief overview of free probability theory for random matrix theory.
B Proof of Proposition 3
To prove this theorem, note that the vector Br can be written as
3, = (x T x AprpTtet) -Xx y
1 = =X X (en -IxT (X,3, +E)
=(S + X.Taa -1
1EB, + + 1 + -IXT e,
16

---

where we have used the fact that y = X,3, +e. Thus, recalling the definition of Σ and R from the
Section A, we can write
B 3. = (R - Iaxx B, + RX
The test error of the teacher model is given by L. - + ||3 B- 112 in which
|| 3. 3. (RÉ
Lax) (RE - + = (XR²XT) -
(1),
where have used the Hanson- Wrightt inequality and the facts that and B, independent mean zero random vectors. Again, by using the Hanson- Wright inequality and recalling that (e. 33)) is independent of other sources of randomness in the problem, expression to arrive at
further simplify the above
1| 3-12 to ' Th ( T [an]
og Tr [
Note that R = Lax R. Thus, denoting the eigenvalues of by {or) R= we can use the Marchenko-Pastur Theorem for covariance matrices write
dx d [ Iax) (R lax) = Ai Tr Ii.
+p A2m...
Similarly, we have
d Tr R =d d Tr (R. 2.1(3) pp m4,1 A
Putting these together yields
1|3 3.112 +r X2ma, + o. (m4.1 Aem4,2)
= = (N+ - -² + +
Plugging this into the expression of L; gives
L. -p of + Amm1a (Ne - -2 + +
which concludes the proof for Proposition 3.
C Proof of Theorem 4
Recalling that y = X,3, and Gr = (X X A - -IXT y. the vector &, can be written as
B, (X X n, Iax) - y
XX x
x
x A In
x X x AmtxTax) - (X,3, +
where we have used y = X,3, + e. Using the definition of the matrices Σ, R. E, R from Section A, can simplify the above expression
B. - 3, (RÊRÉ - 14x)),, + n -RERXX
Hence, we have
||β, - B. (RERÉ- Iax) T (ÉÈRÉ I(xx B, + n, 38 XRER'ERX o1
= Tr (RER - Iix) (RÉRÉ - Iix)
+ oo Tr (EÉR) + op))
where have used the Hanson- Hanson-Wright Wright inequality and the facts that e and B, are independent of other sources of randomness in the problem. Now, it remain to analyze the following traces in the high-dimensional proportional limit:
T. d Th
REÈS LN (RERÉ Iax]
and
:22 = d TT Tr
(ER
17

---

Analysis of 71 Note that RE Idx - X. R and RE = Idx ArR which gives
RERS Lax -, XR AAR
Plugging this into the expression for 71, arrive
J1 x2 Tr
R + x2 TF
R2
X Ir
R²R²
2 22. 'rr
RR
22.
RR.
2222, RR."
The limiting values of these traces be computed follows:
Term 1 and 2: Let & be the k-th eigenvalue of Σ. We can use the arguments in Section A to write
d Tr
R
+p M2,2;
where m, is defined in Definition 2. Similarly, for the second term, we have
Tr R
>> m2,
where the term ma is defined in Definition 2.
Term 3, 4, 5, and 6: To analyze d R we can use the asymptotic freeness of independent Wishart random matrices [Voiculescu, 1991] (see also Capitaine and Donati- Martin [2007]), and the Stone- Weierstrass theorem to approximate the function f(z) = (x + s) using polynomials, write
TT R TT.
R²
ad 'Tr [
[ o1(1)
be 222
Similarly, for the remaining terms, we have
Tr
R.
m 17,,11
R
d Tr
->r m m,,
and
'r R
>p m
Putting these together, we arrive at the following conclusion:
T1 + X2 m2, + X2 m2 + X2,
+ -2, m,,,1 m,,
22, 22. X, m4,2 2,,
Analysis of 72. Again, we can use the identities RI Lax R and R Lix Xr R,, and the
asymptotic freeness of independent Wishart random matrices to write
T2 = d Tr
(ÉÉ))))
= d 'Tr
(R- XxR²²)
-2.R) 27
[R - xx²² T [IAx- Ne
2
= d
Hence, we can use an argument similar to the argument for the term above to write
T2 >p (m4,1 - Xe mm,22 (1 X3ma, 22.ma.1.
18

---

Putting everything together. can be written as
Putting the conclusions above together, the limiting loss difference
C. Le o3 + + +p
+p (ož 2x) [(mm, 2) (A3ma.2 -2).m,,)]] X2ma,2 (1 >,m),1) A
which completes the proof of the theorem.
D Proof of Theorem 5
From Theorem 4, know that in the high-dimensional proportional limit, have
C. -.- + A (o? Ae) [m mm,22 (A.ma,2 mA.), Xmma,2 (1 A
To study the (As; Xt) pairs for which the strong model can outperform the teacher, study the non-zero roots of the nonlinear equation A = 0, which are the solutions to
M4,1 - A, 10,.1--,100.22
A M,
(02 - -+))
(11)
Xem4,1 -1
XgM,, 2m,,1
where the left-hand side is a function teacher parameters, and the right-hand side is a function of the student parameters.
Hight-Hand Side.
First, recall from Definition 2 that
ms,1 = m(Aii
M,, = OY
m mm():)))an (X:
in which
m():7) = - 1 1
29d
Thus, after simple Algebraic manipulations, we write the right-hand side of (11) as
AaM22
H A,M,, 2m,,1
A X2 7s Xa) 4ys + (1 + 7s X,)
S0.
1 2%,
We plot H1 function of A₂ for different values of 7. in Figure 4. be seen that this function H1 undergoes phase transition at has only one solution for A and the function is strictly decreasing. However, for Y > 0, the function Hi is non-monotone and H1 (A, 7) = can have either none, onc, or two solutions for X₃. We will algebraically prove this fact below.
1. When 7s < 1, the equation H1 (A,,)) 7) = for any c < 0
After simple algebraic manipulations, we find that setting H 7a) = c for some c S 0, we can have potential solutions for X given by
1 + 2c + 5c2 + 407s + 402 (1+ 3c) (c- 1)2 8c(1 + c)>+
(c 7x) =
(12)
4c(1 + c)
By inspecting the solutions, find that when < 1, only the solution A5 is valid. However, when % 1, both X and A= become valid solutions as long as
1
(13)
1 - -% + V-e(Ts-1)
which ensures that (c-
1)2 1)2
8c(1 c)y, 20.
Figure shows the values of X, for which (2,,, 7s) = e as function of e, for different values of
7s. The case of 7/ 1 is shown with blue dashed line. When 7. > 1, two solutions can exist for
A,, In this case, the largest e for which two solutions exits are given by (13). However, for 7s < 1,
there is always solution.
19

---

Ys <
-0.1
Y
-0.2
Y
>
-0.
0.1
-0.4
To
0.6
-0.5
0.6
0.7
-0.8
-0.9
0
0.2
0.4
0.6
0.8
1.2
144
1.6
1.8
2
As
Figure 4: The function H1, as a function of A, for different values of 7s. For the case with y > 1, the equation (X,))) with < 0 can have solutions. However, for 7 1, there always one solution.
10
0.2 %
7/4 1.5
7/
-1
-0.8
-0.6
-0.4
-0.2
Figure 5: The parameters A₂ for which H, (A,) = c, for different values c. Two solutions can exist
when 7/4 > 1. However, for 7s 1, only one solution exist.
20

---

Left-Hand Side. We will now turn our attention to the left-hand side of (11). Recall that
a
M1,1 me))
M2,2
π% m();
We plug these into the right-hand side of (11), and after simplification, we arrive at
[ A
of d
H2 = (o As)
> ( -
V-4% (1 74 + X₂)
Hence, oi7 d determines the sign of H2.
Putting Everything Together. After characterizing the functions H1 and H2, we can use these characterizations prove the theorem.
When of7 < Xe, we have H₂ Noting that H1 S 0, we find that in this case, there is no solution for X2 such that H1 H2.
When Te dc, have H2 Based on the analysis above for the right-hand side of (11), two cases can happen:
If % 1, we always have solution À given by X (s)7s) 7a) from (12) with
(14)
-- + (1 7t + )x)²
such that H1 H2.
If Jo 1, long (13) holds; i.e.,
02 d 4-4++ (1 X,)² 1 4% + V%5(%.- 11)
1
S
two solutions exists for X that satisfy for H1 H2. The solutions are given by
xt) from (12). Consequently, we have A < 1 as long as X, E (A5,
These together finish the proof.
E Non-Monotone Error Curves
We use the theoretical prediction from Theorem 4 and plot the test error of the student model C, as function of Yt. Figure 6, we set the teacher regularizer A -> (ridgeless regression) and set = 0.1, = 0.2. We consider the same setting but with a, = 1 in Figure 7. We observe that both settings, the student has a non-monotone behavior for different values of X,, with peak happening at the interpolation threshold % 1.
In Figure 8, we consider the same setting Figure 7 with 7. 0.1,
and set = o² (i.e.. the optimal ridge regularizer). We observe that optimal regularization of the teacher model completely mitigates double descent in the student model; i.c., the test loss of the student model becomes monotone as a function of 72. This is in line with the findings of [Nakkiran et al., 2021 1] for standard ridge regression. Also, we observe that as predicted by Theorem 5, the student can never outperform the teacher.
In Figure 9, we consider the same setting 8 but set X 0.15A;. We observe the in this setting where the teacher is still under-regularized, the student model still exhibits a non-monotone test error function of it.
Proof of Theorem 8
First, recall from Definition that
T= Iay + dy 333
Using the Sherman-Morrise formula, we have
d₂138T
r-1
Iax
Inl - -(1 + o(1)) 2,3
1 d
21

---

0.2
0.8
0.6
0.4
Lt
Ler A, = 0.01
L., 0.044
0.2
L. À = 0.19
L., = 0.83
L., A = 3.6
La A 16
0.5
1
1.5
2
7
Figure 6: The error of the student model L, as function n for og =0.2, no 0.1, X + (ridgeless), and different values of A
0, =1
Lar A = 0.01
3.5
L. 0.044
L., À - 0.19
3
L. = 0.83
L., X 3.6
2.5
LaT A. = 16
1.5
0.5
2
4
7/t
Figure 7: The test error of the student model L. as a function of 7. for 0. 1,,7 0.1, A, 0 (ridgeless), and different values of X..
22

---

1.2
1
0.8
3 0.6
L
L3; A = 0.01
0.4
Lss A - 0.044
L., A 0.19
0.2
C., A 0.83
Lss; A = 3.6
Las A 16
2
4
7
Figure 8: The test error of the student model function of "/t for og 11,70 0.1, dr X; oe (optimal ridge regularizer), and different values of X
1.2
1
0.8
3 0.6
L,
Cgp X 0.01
0.4
Ls, A = 0.044
L., X 0.19
0.2
C,, A 0.83
L,, A 3.6
Lor A 16
2
4
%
Figure 9: The error of the student model C. as function 74 for 0s 1, 0.1, X
0.15X; = 0.155"m and different values of Aa.
23

---

In the setting of this theorem, we have 3, (Σ -,- y/n.. Thus, we now focus on the generalized resolvent matrix (Σ + --1-1)-11 Using the Sherman-Morrison formula, this matrix can be expanded as
+ + - + \n.3 -
X 1 2 R
R 3B R.
For simplicity, we define the scaler
As
:==
X
Hence, plugging the Sherman-Morrison expression back into the expression for B., we have
G. = R R 3, R Xý/ns = R + v R R Σ
R v 3 R Σ
RÉB + RX E/mL)
X n
X
RÉRÉR, RER
+ MAABREREAA + RØ3
71g
We define t1, ts Rdx as
ti RÉRÉB, - B., t2= XE
X t RØ9 RERÉA, + RERR
RER
n
With this definition, L. o3 + |3, B.1 can be written as
C, 2 + ||t + t2 t
|t 112 + ||t2||2 + ||talé + 2t, 12 + 21 + 2t/
will analyze each term separately:
The first and second terms ||a 12 + || 22 have already been calculated in in the proof of Theorem have
|t |taali
C. tp A,
where A is defined Theorem 4.
For the third term, we can write
(1-x,
2 ) AT (1-X) )
ts = R2)
From the definition of m,,1 and m,,2 and using the Marchenko- Pastur theorem, we have write
/ R b ms,2, and also 3 R +p m.,1. Also,
BRÉRÉSS = -dd 'Tr ( (RERE) = < < 'Tr R
'T'
-r S (1 )gm,,.) (1 >emm,1)
where Ç is defined Assumption 7, and we have used the asymptotic freeness of independent Wishart random matrices [Voiculescu, 1991, Capitaine and Donati-Martin 2007]. Hence,
2 tt +4 (1 -1- A.m,,
(1 Apm,,1)²(1 22m 2 mz A
For the fourth term, note that
2t tg 2(REÉ Iay Σ1,))
T e/n4 - 0,
using the Hanson- Wright inequality and the fact that e is mean zero and independent of all other sources of randomness in the problem.
24

---

Similarly to the fourth term, for the fifth term we can write
XTE
RER X
2t t3 2v
RER
R33 RÉRÉ3, + R
The
= =20 (B
(ni'e " XR + 2y n RERX E) n4 (n ((TXRRER3))
Using the Saason-Wrightt inequality and the fact that is other sources of randomness in the problem, we have ni ni'e "'RÈÈR >> 0. Hence,
and independent of all
2t ts >
The sixth term can be expanded as follows:
2t t3 2v ( RERÉE, ni B RERX 8) (ÓTR(RERS [..)
2v ( ( ( R B RB3) op(1),
where again we have used the Hanson- Wright inequality and the fact that e is mean zero and independent of all other sources of randomness the problem. Above we have already shown above that
RÉREA, dr 5. Ammg,2) (1 demm,1).
With similar argument, have
8 RRÉERE.. d (R'ÈRE) + or(1)
= 'Ir (R'E) d 'Tr (RE) + or(1)
-77 ((1 (11_(mmm,))) (1 1) (ma.1 - Amma.2)
Also, R. +>p (m,,1. Hence, putting all together, we get
2t ts >r 2ç A, Amm,,1) [A A,m4,1 A,ma,1).
Thus, adding all the terms together, we have
L. - L. +p A &
where the expression for A is given in Theorem 4, and Ar is given by
Ar A. (-1 1,
[- 122mm,,mmm A,m 2- \,m;,))
This concludes the proof.
G Proof of Proposition 10
The training loss for the teacher model is given by
Th
dt ya (W,)))
2,
fx(x))))
14
14
Taking derivatives with respect to the matrix W., we arrive at
1 ooox"")
Vw,
nh
Let C,,1 the first Hermite coefficient of the activation function a, and define a : R as
α (2) o(2) Ca,12, V2 R,
25

---

where (x. = O. Thus, we can write
Vw, L. -- 1
[(aLy]]
o (can + o (Wo, x
ny
Cr.1 a, y
X
n4 [(ayy') o 0' (W x
74g
By construction, the matrix a'L
d'(W, (W 1,0
has mean zero entries. Thus, using [Vershynin, 2012,
Theorem 5.44], we have I|o'1 (W Lo
Ilpp = (()))). Hence, using Lemma 12, we have
- [(.y 00 (W. x x
1
diag (a.) o'1
diag(y) op
op
polylog(p:)
vnr polylog(m)
n
VPr
= Ô
/Pi
where we have used the fact that || = 0 (mz))) [Vershynin, 2012, Theorem 7.3.1], and the sub-gaussian maximal inequality to get ||a|| = p; 1/2 polyloooogg Similarly, using the sub-Weit Weibull maximal inequality [Kuchibhotla and Chakrabortty, 2022, Proposition A.6 and Remark A. J., we have o O(polylo((n))). As a result, for any , E Rddx with || we have
Vw, Bl1 = n, x
We will now study the case where ,3 is the easy or the hard direction.
Easy direction. First, we let 3 3r. In this case, we have
VV Bell2 3 X (o. on (X,3)) + oe(1).
Note that X,3 E R is a vector of i.i.d. N(O, 1) entries. Thus, using the weak law of large numbers, we have
n B X (XB.) +p Ez=.No = Ca..1
Also, recall the assumption that Be and Bh are orthonormal vectors and is a matrix with i.i.d. N(O, 1) entries. Thus, X3, and X,8, are independent and we have
n B x on (X,3)) +> 0.
Thus, the gradient has non-trivial alignment to the easy direction. Consequently, for W W., Ww,C,, with proposition.
@()), we have W.Bollop +p 0, proving the first part of the
Hard direction. For the hard direction B = Br. have
| E, Ball n, 3 x. x (XB,) + (x op(1).
The first term n, 13, X o. (X,3.) can shown to o(1) with an argument identical to the argument above. For the second term, note that X,3, E IR is a vector with independent N(0, 1) entries. Using the weak law large numbers, we have
ni 33 x on (X,B)) +e EsNo.,)) ~ (=)] Co = O,
where have used the fact that the information exponent of σ is lager than one; i.e.. Coa.1 0. This shows that the gradient has no alignment to the hard direction, completing the proof.
H Proof of Theorem 11
From the proof of Proposition 10, we have
W, = W.o + 1 + +A.
26

---

where Be ni X and | lop = o(1).
Given the fresh indepednent set of samples X, the
updated teacher model labels them as
y Fa with F= ox W, XM
and the training loss for the student model given by
1
: Th
oe aaa a W wx)).
Yhs 1=1
Taking derivatives with respect to the matrix W., we arrive at
1 --1 (a. o o's, X.
Vw, Vw,Co W.. =
(15)
To analyze the gradient, we should first characterize d'( W.,uX and y.
Analysis of y. The feature matrix F is given by
F P = (XW 6, +
which is nonlinear transform applied element-wise spiked random matrix. Following the recent results in nonlinear random matrix theory (e.g., Moniri et al. [2024], Wang et al. [2022], Moniri and Hassani [2024], Guionnet et al. [2023], Feldman [2025]), in the regime where 7) O(1), we Hermite expand the nonlinearity as follows:
F
(w.) = (XW.o 1o.1a
H (xW,, 1.0 + Ca, 1 xB.a.)
Using Lemma 15 wise, we can expand this matrix further
F
(fenviicaat (XW..))
(
(1
O5T
k=1j=0
Σ Ho (XWTo)
Σ de co, (K
Caniicasttes (XWTo) @ ((xB))01
k=1Ø=1
Note that the first sum can be written as
Co, Ha (XWT) (. (XW.. (o).
In the second sum, by a simple sub- sub-multiplicativity argument, the k-th term has an operator norm bounded by
(ocnn co.k ((X
O&T !1 = =0 (p)
which is ol PP) when & > 1. Moreover, using Lemma 12, the (k,j) j-thh term of the third sum has an operator upper bounded by
(*())
kb) de co, co, (XWT.) の (X..))01a3)) 11
= =| k Thc. diag (1) Ha-s (XWT.) ) diag (a.*) 1op
Ō (p. 3/2 n1/22) olpp 12).
27

---

Putting everything together, we have
F = o(W (,0 + ( (X,3))a + +A.
where | 0 (Vn.). Hence, recalling that ||ar||2 (1),, we have
= Fa; = oXW,, a, + 17, (X33.) +
(16)
in which ||ö|| = nz))
Derivative Term o'(WW 2.0) Recall that we have W.. W..0 + T Hence,
d' (XWI)) s.0) d' (Xw, 7 (X33)) a
This is again a nonlinearity applied element-wise to spiked random matrix. Similar to the argument for F, we can use Lemma 15 to write
00
k
k (t) co & (XWTO) @ (1x,)) a
d' (XWTo)
k=1Ø=0
30
20
Σ Cor (XWT.)
+ Σ
T cors (xx,,)
s.0)
kml
k-1
Fecors Ha-s (XWT.) ( o (x
k=jj=1
Similar to the reasoning used for F, we have
(1()) P Cort HK-; (WT.) 5.0)
((x,,))
! = (n///²
I
T
Vpr
which is of (T)) as long ol v..) Thus, we have
o' (XWT)) = (XWT.) ΣΣ+ chcor (KK + +
(17)
in which | =op==(ynea o(v
Gradient of the Loss. Now, we have all the ingredients to study the gradient of the loss function of the student model. Plugging (17) into (15), have
Vw, C.
(a y
d' (W.o + > co (a (XBr) OKT + A X,
W..
The
which we decompose as Vw, Z.
G + G2 where G2 and G2 are defined as
W..
1
0 [W,oX ) X.
G1
[ ÿ
Ng
Σ choook a (a0 (X(x)) + +2) ) x.
1
G2
(a,
O
n2
will analyze each component separately.
Analysis of G This term can be written
1 -
(a. 00' (W..x
G Let C,,1 the first Hermite coefficient of the activation function a, and define o :R R as
ni
a (z) o(2) Ca,12, V2 R,
28

---

where E (x) 0. We can write write
x
(a.y)) odi (w (Woox*)) X
G
ag
(18)
Tls
By using Lemma 12 and by similar argument to the one used in the proof of Proposition 10, the operator norm of the second term of (18) be upper bounded as
(a. (a.y")
oo' (W )
X
= d(1)
f1a
op
Using the characterization y in (16), the first term of (18) can be written as - with
x__xT_
x
[o(X (o)a +ar.laatt...... (x21)((B)) + +
11g
This vector aligns to the easy target direction:
B
1
(X
((XWTu)a + (. +o]
1
(x,3,)
(XX (.)))a a + e (X + o(1)
11
2 12
(XG3) T (X33)) + o(1) +o ce 00
(19)
n,
Analysis of Gz.
To analyze this component, first note that we can write
G2 =
c (a. O a
(x
OKT ) +
X.
The operator norm of the k-th term of the sum is given by
T Cho
(a. (s (XBn) OAT
=0
(( \pg
T (vxx.)
= o(1),
ns
op
where we have used the sub-multiplicativitty of the operator norm, ((V)s), and the fact that | O (2,6))l2 @(vn,) and ||a
a 2| Opp */2).
Putting everything together. Using the the results of the analyses above, we have
W 0.0 T n Å
where | = o1). Hence, recalling (19), the updated weight matrix has non non-vanishing correlation with both the easy and hard directions, completing the proof.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
31

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [NA]
Justification: The paper only has simple synthetic experimetns with clear instruction on how to replicate them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.cc cttpssccccbblll
gguiidss/Coeeuumiii ssiiiissiion caeiionooolyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
The small scale experiments can be easily ran on all typical personal computers.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///euriiscccoo httprrppp../ppbbltttthhhseeeeeee
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
33

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The focus of this paper is on theoretical aspects deep learning. We expect the results to be illuminating for the deep learning theory community. We do not anticipate any negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
34

---

Answer: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
35

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standale components.
Please refer LLM policy pettps://neurtie /00t:////eeeeeeeee////0///////// for what should or should not be described.
36


Reviews:
[{"review_id": "uCWBUc1BLg", "reviewer": "Reviewer_V6rD", "summary": "This paper studies the mechanisms of weak-to-strong generalization, on three different models, linear models trained using standard and weighted ridge regression, and generalized linear models with feature learning in the asymptotic regime. Their analysis includes both the over-parametrized and the under-parametrized regime and they provide explicit condition in which the student can outperform the teacher and when it fails to do so.", "questions": "**1.** Line 129, \"that\" should be changed to \"than\".\n\n**2.** In lines 209-210, what does the statement \"if $\\gamma_s > 1$ then some information is lost\" mean?\n\n**3.** In line 267, doesn't taking $\\zeta \\rightarrow 0$ lead to recovering the setting in section 2.1? In that case, there are values of $\\lambda_s$ for which the student can still outperform the teacher. I would appreciate the authors' comment on this.\n\n**4.** In line 351, \"teaching\" should be changed to \"teaches\".", "limitations": "See weaknesses and questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\nWeak-to-strong generalization is a modern paradigm important to the improvement of performance in foundational models. This paper has provided precise conditions for linear models, when using a stronger regularization strength, the student's model's generalization error can outperform that of the teacher's. Furthermore, by analyzing the weighted ridge regression problem with a spiked covariance model, they show that the student can surpass the teacher if it has been adequately pre-trained. Finally on a generalized linear model which involves learning the two \"easy\" and \"hard\" directions, they showcase that applying one step of the gradient descent to the teacher's model can be still useful if the student has been sufficiently pre-trained. \n\nOverall the paper is well-structured, well-written and easy to follow and their numerical experiments on synthetic supports their claims which align well with intuition.\n\n**Weaknesses**\n\n**1.** I believe that adding a discussion about the effect of the noise level improves the comprehensiveness of the paper. Namely, the question is as follows; given a budget on $\\lambda_s$, what is the best the student can do in the face of various SNR levels. An study of low-SNR vs high-SNR regime is pertinent to the discussions presented in the main body.\n\n**2.** The discussion around the exhibition of double-descent phenomenon in Appendix E should be included in the main body.\n\n**3.** I recommend going beyond the synthetic and conducting experiments on real-world data such as MNIST and CIFAR-10 datasets.\n\nI would like to add that I am open to increasing my scores in case the authors adequately address the points raised here.", "was_revised": true, "final_justification": "I find the paper well-written, organized, and its scope of analysis, comprehensive. The rebuttal has properly resolved the points I raised in the original review. I recommend acceptance.", "rebuttal": "We thank the reviewer for their insightful comments. We are glad that you found our analysis of the weak-to-strong generalization in linear and nonlinear models interesting and well-structured. Below we respond to each of the comments raised in the review:\n\n- **Effect of noise level.**  We thank the reviewer for this constructive suggestion. To see the effect of the SNR, we first fix a teacher regularization parameter $\\lambda_t$. As a corollary of Theorem 5, we know that in the high-SNR case where $\\sigma_\\epsilon^2 \\leq \\lambda_t/ \\gamma_t$, weak to strong generalization cannot happen. However, the low-SNR case is completely different. In the low-SNR case ($\\sigma_\\epsilon^2 > \\lambda_t/ \\gamma_t$), if the student in under-parameterized, there is always a $\\bar\\lambda>0$ such that weak to strong generalization happens for $\\lambda_s \\in (0, \\bar\\lambda)$. Moreover, in the over-parameterized case $\\gamma_s >1$, weak-to-strong generalization happens if we have\n$$\n\\frac{\\lambda_t}{\\gamma_t} < \\sigma_\\epsilon^2 < \\frac{\\lambda_t}{\\gamma_t} + \\frac{1}{\\gamma_t} \\cdot \\frac{\\sqrt{(1 + \\gamma_t + \\lambda_t)^2 - 4 \\gamma_t}}{4 \\gamma_s + 4 \\sqrt{\\gamma_s^2 - \\gamma_s}- 1.}\n$$\nThis means that in the overparameterized case, weak-to-strong does not happen in the ultra-low SNR regime; i.e., in the regime with\n$$\n \\sigma_\\epsilon^2 > \\frac{\\lambda_t}{\\gamma_t} + \\frac{1}{\\gamma_t} \\cdot \\frac{\\sqrt{(1 + \\gamma_t + \\lambda_t)^2 - 4 \\gamma_t}}{4 \\gamma_s + 4 \\sqrt{\\gamma_s^2 - \\gamma_s}- 1}.\n$$\nWe will discuss these in detail in the final version of the paper.\n\n- **Discussion of double descent.** In the final version, given the extra space, we will move the main parts of the discussion on double descent from the supplementary to the main body of the paper.\n\n- **Beyond synthetic datasets.** We thank the reviewer for this suggestion. From a theoretical side, we note that the limiting traces that appear in the expressions of our paper exhibit universality; i.e., the actual distribution of the of entries of the matrices does not matter and they can be replaced with a Gaussian matrix with matching first and second order moments and still the limit will be the same. This fact can be proved by a standard Lindeberg exchange argument. Thus, our results in theory do hold much more broadly than the Gaussian case. Experimentally, similar kinds of behavior is also seen with real datasets like MNIST. The new NeurIPS rebuttal rules for this year prevents us from sending new plots, but we will include a detailed discussion of this universality and numerical experiments in the final version of the paper. \n\n\n- **Information is lost.** In the case where $\\gamma_s > 1$, we have $d > n_s$; i.e., the student model is over-parameterized. Information-theoretically, having access only to $n_s < d$ samples generated by the teacher, no algorithm can fully recover the $d$ dimensional vector $\\hat\\beta_t$. This is what we mean by \"some information is lost\". In the final version of the paper, we will clarify this point to avoid confusion in the final version of the paper.\n\n\n- **Limit of $\\zeta \\to 0$.** When $\\zeta \\to 0$, setting 2 reduces to  setting 1.  This can be seen by comparing the limiting expression of $\\mathcal{L}_s - \\mathcal{L}_t$ in Theorem 4 and 8. In setting 1, we have $\\mathcal{L}_s - \\mathcal{L}_t \\to \\Delta$ and is setting 2, we have \n$\\mathcal{L}_s -   \\mathcal{L}_t \\to \\Delta - \\zeta^2 \\Delta_g$\nwhere $\\Delta_g \\geq 0$. In particular, having $\\zeta > 0$ enables the student to outperform the teacher in setups (i.e., values of $\\gamma_s,\\gamma_t, \\dots$) in which a student model with $\\zeta =0$ cannot."}, {"review_id": "OPaHwxhau6", "reviewer": "Reviewer_cVVG", "summary": "The paper addresses the question of when a student can surpass a teacher in isotropic-gaussian linear regression and a two-index model trained with a single step of gradient descent. In the linear case, the ability to surpass the teacher hinges on increasing regularization. In the two-index model, the student is initialized with a non-trivial alignment with the desired direction.", "questions": "- The statements of Proposition 10 and Theorem 11 appear strange when it comes to the constants. Are these absolute constants? Shouldn't $c_h$ depend on $\\tau$?\n\n- In figure 2, why did you increase the noise from the left and right figures? Is equation 5 not satisfied when $\\sigma=1$?", "limitations": "See the weaknesses section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and easy to follow. The underlying topic is timely and well motivated.\n\n- It is always nice when new observations are made in classical settings. I find Theorem 5, which clearly characterizes when the student can surpass the teacher in the linear setting (as long as equation 5 holds), to be a nice addition to the literature.\n\n- The contour plots nicely complement the results.\n\nWeaknesses:\n- The isotropic-Gaussian setting is strict. By now, there are many papers analyzing linear regression in much more general settings. \n\n- I'm not sure that the results of section 3 actually showcase weak to strong generalization, since the student is assumed to be initialized with weights that are already aligned with $\\beta_h$. Training follows only a single gradient step, which prevents the student from changing significantly in the $\\beta_h$ direction. This is fundamentally different from the case where the student achieves lower loss than the teacher despite being trained in a similar way. \n\n- I believe the way the authors discuss the setting of section 3 in the abstract and introduction is misleading. Rather than saying things like \"We assume that the easy component is highly specialized and task-specific, but, the hard component is a component shared across many tasks\", it would be much more faithful to the theorems to state directly that the student weights are already aligned with the hard component. The storytelling regarding multiple tasks is not directly reflected in the math. It is fine to use this as motivation for the actual assumptions, but not to replace stating them.", "was_revised": true, "final_justification": "Thank you for the response.\n\nThe main points that reflect the theory and experiments were addressed. There were also a few points in the rebuttal that I do not fully agree with, but these come down to semantics/storytelling and are not fundamental.\n\nI have raised my score.", "rebuttal": "We thank the reviewer for their insightful comments. We are glad that you found our paper timely, well motivated, and a nice addition to the literature. Below, we respond to the points raised in the review:\n\n- **More general regression settings.** We agree with the reviewer that it is in principle possible to analyze this problem under more general assumptions on the input covariance. However, we note that the resulting expressions in this case will be very complex, and will not have a closed form; i.e., they will involve a fixed point equation. Thus, the results will become much harder to interpret. Moreover, we believe the study of the case of general covariance matrix is not central in our phenomenology of weak-to-strong generalization. In our view, studying the simplest possible settings that exhibit weak to strong generalization without additional clutter makes it easier to understand the fundamental mechanisms that enable it without unnecessary distractions.\n\n- **Results of section 3 and weak-to-strong generalization.** We thank the reviewer for this comment. We completely agree that setting 1, in which  the student and teacher are trained in a completely similar way, and the setting 3, where the weights of the student model are already aligned to some target components, are fundamentally different. But, we believe that setting 3 (similar to setting 1) actually does correspond to a form of weak-to-strong generalization. For instance, note that in Burns et al. [2024], the teacher model is GPT-2 and the student is GPT-4 and these model are indeed trained in a completely different way. GPT-4 already has learned many more capabilities in pretraining compared to GPT-2.  Also note that in even in the setting of Burns et al. [2024], the strong student should also be only fine-tuned (and not fully trained) on the outputs of the weak teacher, otherwise, the pre-trained capabilities of the strong student beyond that of the capabilities of the weak teacher will be eventually forgotten. \n\n\n- **Introduction.** We thank the reviewer for this comment. As suggested by the reviewer, we will adjust the introduction and explicitly mention our mathematical assumption alongside the motivation. Regarding the faithfulness of the motivation, we argue that the motivation of setting 3 is faithful to the mathematical assumption in the following way: the hard direction has been learned during the pre-training phase, and the network weights are already aligned to the hard direction, because it is a direction that is relevant for many different tasks and can be learned from them during pre-training. However, the easy direction is problem specific and could not be learned by the pre-training on other tasks. However, we will make sure that the actual mathematical assumption is also included  in the introduction.\n\n- **Constants.** The constants in Theorem 11 are not universal constants. The reviewer is right and $c_h$ does in indeed depend on the parameter $\\tau$. We will adjust the notation in the final version to avoid this confusion. \n\n- **Choice of parameters in numerical experiments.** We solely made this decision for the choice of $\\sigma$ to enhance visualization. We will also include the plot for $\\sigma = 1$ in the final version of the paper."}, {"review_id": "j7pXTe6rCi", "reviewer": "Reviewer_bbzK", "summary": "This paper extends current research on weak-to-strong generalization by identifying three mechanisms that facilitate such phenomena: (1) compensation for teacher under-regularization in ridge regression; (2) better alignment of student regularization structures with the target function under weighted ridge regression; and (3) effective utilization of pretraining by the student in nonlinear settings.", "questions": "I have already raised concerns in the Weaknesses section，and none of them is very critical.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n1. The paper is well-written with clear logical flow and appropriate formalism.\n2. Derivations are detailed and reflect strong theoretical proficiency.\n3. The extension to nonlinear settings is interesting and adds depth to the study.\n\n\n## Weaknesses:\n1. In the nonlinear analysis, the definition of the *easy* link function appears to reduce the setting effectively to a linear case. This makes the learning of $\\beta_e$ trivial.\n2. If I understand correctly, in the nonlinear case (Theorem 11), the $\\beta_h$ component in the student model originates entirely from pretraining, with early stopping controlling its degradation during subsequent training. However, even though the student learns $\\beta_h$ while the teacher does not, if the constant $c$ in Proposition 10 is significantly larger than $c_e$ in Theorem 11, it is still unclear whether the student outperforms the teacher. If the presence of $\\beta_h$ alone sufficed, the student would already outperform the teacher without any fine-tuning, which is not the intended conclusion.\n3. Overall, while I consider the paper to meet the acceptance threshold, many of its key results and proofs hinge on the assumption that the student benefits from better pretraining while only modestly learning from the teacher’s supervision. As a result, the conclusions—though valid—are somewhat predictable, which limits the novelty of the contribution.", "was_revised": true, "final_justification": "I acknowledge the contribution of this work and will keep my score to be positive.", "rebuttal": "We thank the reviewer for their review. We are glad that you found the analysis interesting, well-written, and detailed. Below, we will respond to each of the points raised in the review.\n\n- **Is learning the easy component trivial?** The easy component is a single-index function with unit information-exponent. It is known that Learning this function using SGD requires $\\Theta(d)$ samples, compared to $\\Omega(d)$ samples required for functions with an information-exponent greater than one. Thus, learning the easy component is still non-trivial (please see Ben-Arous et al. (2021) for more details).\n\n- **Weak-to-Strong generalization in setting 3.** We thank the reviewer for this comment. We note that the magnitude of the rank-one components in the updated weight matrices is controlled by the step-size of the gradient update. Thus, the step-size should be large enough for weak-to-strong generalization to happen. Also note that in the paper, we show that with $n = \\Theta(d)$ samples, the updated weight matrix of the student will become aligned to the easy and hard components. With weights aligned with the target, even a very few $o(d)$ number of samples is enough for generalization. We will clarify these points in the final version of the paper. \n\n- **Novelty of the contributions.** We argue that we provide novel phenomenology that was not known in prior theoretical and empirical works on the subject. We show that weak-to-strong generalization is not unique to deep models and even very simple models can exhibit this phenomenon. We believe that this on its own is a solid contribution that enables the study of waek-to-strong generalization in a more controlled setting.  In setting 1 and 2, we study the case of ridge regression in detail. The analysis of the pairs $(\\lambda_s, \\lambda_t)$ for which weak-to-strong generalization happens is completely new and complements the prior work that do not study explicit regularization. In particular, we believe that the phase transition happens between under- and over-parameterized students, is very surprising and was not expected a priori. The analysis of setting 3 is also generalization of the results of Ba et al. [2022]. We show that the student model can learn the easy direction without \"forgetting\" the hard direction learned during pre-training. In fact, from a mathematical perspective, Theorem 11 is of independent interest and is actually a necessary component to study feature learning beyond the one step setup studied by Ba et al. [2022], Moniri et al. [2024], Cui et al. [2024], etc."}, {"review_id": "EOapljS1Ga", "reviewer": "Reviewer_Rrob", "summary": "This paper examines weak-to-strong generalization in several theoretical settings focused on ridge regression and multi-index regression. The authors identify a number of settings in which weak-to-strong generalization may occur, namely when the teacher is under-regularized, the student has partial access to ground truth weights via weighted ridge regularization, or the student has partial access to ground truth weights in a multi-index setup.", "questions": "Please see Strengths and Weaknesses above.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**. The paper touches on a timely and fascinating topic. The results are precise. The writing is clear.\n\n**Weaknesses**. I am likely not the target audience for this paper, so please take my comments with a hefty grain of salt. Please also defer to comments from reviewers who are more firmly placed in the target audience.\n\nThe results are impressively precise and the paper overall appears to be a nice application of increasingly popular RMT ideas, but I'm left unsure what to take away from this paper. The general finding seems to be that weak-to-strong generalization occurs when 1) the teacher is under-regularized for a ridge regression target (where regularization has a very concrete meaning), or 2) when some aspect of the target leaks to the student, either through a regularization weight or multi-index component.\n\nFor the first case, the result is quite interesting, though I'm unsure how to interpret the magnitude of difference. At best, for a very large regularization mismatch, the student outperforms the teacher by ~0.05. Is this a large or significant margin? Based on the plotted scale, this seems to be a fairly small difference. Should I interpret this result to mean the student can very slightly outperform the teacher when the teacher is virtually un-regularized? More broadly, what would regularization mean in an applied setting outside of ridge regression? Drawing from Burns et al. example you highlight as motivation, would GPT-2 be in some sense \"under-regularized\" for natural language, while GPT-4 compensates?\n\nFor the second case, it is perhaps unsurprising that a student would outperform the teacher if some aspect of the target function leaks to the student while remaining inaccessible to the teacher. Repeating the question from before, would you imagine that GPT-4 has access to some ground truth about natural language that remains out-of-reach for GPT-2, and for this reason succeeds at weak-to-strong generalization?\n\nOverall, while the paper presents a nice theoretical argument, I'm unsure what the broader impact of these results are. Is there an actionable practice I can take away for performing weak-to-strong generalization on a real world task? Do we have a deeper understanding of weak-to-strong generalization beyond a linear setting or when the target leaks?", "was_revised": true, "final_justification": "The author resolved my questions. Together with the other responses, this work seems to be a valuable contribution for this conference, and I recommend acceptance.", "rebuttal": "We thank the reviewer for their review. We are glad that you found the paper timely, precise, and clear. Below, we will respond to each of the points raised in the review.\n\n- **Magnitude of difference.** In the paper, we focused on the conditions under which weak-to-strong generalization can happen. The exact magnitude of the improvement is dependent on the specific parameters of our model and can be smaller or larger, but the key takeaway is that a student can systematically improve upon a teacher by being regularized differently.\n\n- **Regularization in real scenarios.** We use ridge regression as a simple, analyzable proxy for the much more complex set of implicit and explicit regularizers in large neural networks. In a real-world scenario, the choice of architecture and optimization algorithm can also play a similar role. For example, if the student has an architecture that is better suited for the underlying tasks, e.g., it has some invariances that also exists in the target, it can correct some mistakes made by the weak teacher. This is very similar to setting 1 that was studied in the paper. Also recall that in Burns et al., it is seen that early stopping of the student is crucial for weak-to-strong generalization. Our paper propose a potential explanation for this phenomenon. A better inductive bias of the optimization algorithm can also play a similar role.\n\n- **GPT-4 has access to some ground truth about natural language that remains out-of-reach for GPT-2?** We hypothesize that a possible explanation of weak-to-strong generalization in GPT-4 and GPT-2 is that the tasks studied by Burns et al. [2024], e.g., the chess puzzles, have \"leakage\". The more extensive pre-training of GPT-4 (compared to GPT-2) on other tasks, combined with the fine-tuning on the data generated by GPT-2, can help in chess puzzles as well. \n\n- **Broader impact and take-aways of the paper.** Weak-to-strong generalization is a new generalization paradigm and is often considered a mysterious phenomenon and the mechanisms that enable it are not entirely clear. Our paper aims to demystify it. In this paper, we provide a theoretical grounding for weak-to-strong generalization. We show that weak-to-strong generalization is not unique to neural networks; even very simple linear models can exhibit it. We believe that this is an interesting finding that enables more in depth theoretical study of the phenomenon in a more controlled setting. We move beyond empirical observation to provide a rigorous theoretical analysis, identifying concrete and analyzable mechanisms—such as the student compensating for the teacher’s under-regularization or better leveraging pre-trained knowledge—that explain how and why a more capable student model can learn effectively from a weaker teacher."}]

Author rebuttal:

Meta-review: The paper offers a simplified but compelling framework in which to study weak-to-strong generalization. Some of the insightful results that follow are clear conditions, such as alignment with the target, that compel a strong student to learn from a weak teacher, as well as a quantification of the behavior of a pre-trained strong student using its own strength (e.g., better features) to better learn from what information the teacher provides.

Reviewers generally appreciated the clarity of the presentation and significance of the results.There was some discussion on placing the novelty of the contributions more firmly within the literature. Some of the clarifications (about the effect of SNR and about limitations such as beyond regression or beyond synthetic data) should also be part of the paper. Otherwise, the paper is likely to be well-appreciated by the community.
Decision: Accept (poster)