Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Authors: Yihao Xue
Jiping Li
Baharan Mirzasoleiman
Venue: ICML 2025 poster (2025)
Keywords: Weak-to-strong Generalization
Theory
Large Language Models
Alignment
Primary area: theory->deep_learning

Abstract: Weak-to-Strong Generalization (W2SG), where a weak model supervises a stronger one,  serves as an important analogy for understanding how humans might guide superhuman intelligence in the future. Promising empirical results revealed that a strong model can surpass its weak supervisor. While recent work has offered theoretical insights into this phenomenon, a clear understanding of the interactions between weak and strong models that drive W2SG remains elusive. We investigate W2SG through a theoretical lens and show that it can be characterized using kernels derived from the principal components of weak and strong models' internal representations. These kernels can be used to define a space that, at a high level, captures what the weak model is unable to learn but is learnable by the strong model. The projection of labels onto this space quantifies how much the strong model falls short of its full potential due to weak supervision. This characterization also provides insights into how certain errors in weak supervision can be corrected by the strong model, regardless of overfitting. Our theory has significant practical implications, providing a representation-based metric that predicts W2SG performance trends without requiring labels, as shown in experiments on molecular predictions with transformers and 5 NLP tasks involving 52 LLMs.

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Yihao Xue
Jiping Li
Baharan Mirzasoleiman
Abstract
weak model
strong model
span of principal
Weak-to-Strong Generalization (W2SG), where a weak model supervises stronger one, serves as important analogy for understanding how hu- mans might guide superhuman intelligence in the future. Promising empirical results revealed that a strong model can surpass its weak supervisor. While recent work has offered theoretical insights into this phenomenon, a clear understanding of the interactions between weak and strong models that drive W2SG remains elusive. We investigate W2SG through theoretical lens and show that it can be characterized using kernels derived from the principal components of weak and strong mod- els' internal representations. These kernels can be used to define a space that, a high level, cap- tures what the weak model unable to learn but is learnable by the strong model. The projection of labels onto this space quantifies how much the strong model falls short of its full potential due to weak supervision. This characterization also pro- vides insights into how certain errors in weak su- pervision can be corrected by the strong model, re- gardless of overfitting. Our theory has significant practical implications, providing a representation- based metric that predicts W2SG performance trends without requiring labels, as shown in exper- iments on molecular predictions with transform- ers and 5 NLP tasks involving 52 LLMs.
principal kernel
model's error
PredGap b/w W2S and strong ceiling
remaining space
remaining space
Figure 1: An illustration of our main result (Thm. 3.8). The path connecting the two highlighted regions represents the overlap b/w the complement of scaled span of the weak model's principul ker- nel and the scaled span the strong model's principal kernel, de- termining the contribution the weak model's errors to PredGap.
et al. (2023) propose an analogy that can be empirically explored today: can weak model effectively supervise a stronger one? This framework, known Weak-to-Strong Generalization (W2SG), involves leveraging weak model, finetuned on specific task, to supervise the finetuning of stronger model. In this analogy, the finetuning task represents concepts tied to human values or skills, the finetuned weak model represents humans- in capa- bility but aligned with human values, and the strong model represents superhuman intellgence-powerful but initially unaligned. Promising results from (Burns al., 2023) show that the strong model can significantly outperform its weak supervisor. For instance, GPT4 model supervised by fine-tuned GPT-2-level model achieves nearly 20% better performance than the weak supervisor on NLP tasks.
At first glance, this phenomenon seems counterintuitive. Af- all, the strong model is explicitly trained fit the weak supervision. Yet, it goes beyond mere imitation and gen- eralizes better. It is important understand which intrinsic properties of the weak and strong models enable W2SG.
1. Introduction
As AT systems become increasingly capable of performing complex tasks beyond human comprehension, humans will inevitably serve "weak supervisors" in aligning advanced AI. To investigate this fundamental problem, Burns
Efforts have been made toward a theoretical understanding of W2SG. Charikar et al. (2024) demonstrates that the disagreement between finetuned weak and strong models correlates with performance gains in W2SG, However, their analysis assumes high-quality representations in the strong model and does not address the role of the weak model's representations. The analysis of (Lang et al., 2024; Shin et al., 2024) assumes a generalized version of an adversar- ially robust strong model, where W2SG arises solely from underfitting weak supervision. This framework excludes im-
'Department of Computer Science. University of Califor- nia. Los Angeles 'Department of Mathematics. University of Califomiaa Los Angeles. Correspondence to: Yihao Xue <yi- haoxue@g.udo wuwuue@g.cla..du... edu..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
portant scenarios such benign overfitting. where W2SG occurs despite overfitting. Wu & Sahai (2024) particularly studied benign overfitting and examined the impact of number of weakly labeled data points. However, we still lack an overarching explanation that captures the interaction between weak and strong models in enabling W2SG, as well as how it determines which weak supervision errors are corrected in general scenarios. The challenge lies in char- acterizing the abstract concepts including the knowledge embedded in the weak and strong models, their utilization, and their respective roles in W2SG. Striving for results that are general enough to capture spectrum of behaviors with- out overly strict assumptions further adds to the complexity.
2. Related Work
There have been many recent works that theoretically explore W2SG. Somerstep et al. (2024) adopt a transfer learning perspective, focusing on improving W2SG through in-context learning rather than explaining how W2SG emerges. Lang et al. (2024); Shin et al. (2024) analyze W2SG by considering a generalized version of adversarially robust models, showing that certain errors in weak supervi- sion can be corrected by leveraging the good neighborhood structure in the data. However, their argument attributes error correction solely to underfitting-i... avoiding fitting mislabeled finetuning data. This overlooks an important scenario recently discussed in (Wu & Sahai, 2024), known as benign overfiuting, where the strong model overfits mislabeled finetuning data but still achieves accurate test-time predictions. Benign overfitting is particularly relevant in practice, as large neural networks often have the capacity to overfit while still generalizing effectively (Zhang et al., 2021). Closer to our setting, Charikar et al. (2024) formalized W2SG using representation- based perspective. Their work demonstrates that performance gain in W2SG correlates with the disagreement between the finetuned weak and strong models, assuming high-quality represen- tations for the strong model. While insightful, it does not characterize the role of the weak model's representations, leaving the exact conditions for effective W2SG unclear.
To address this, we adopt a representation-basee perspective, analyzing finetuning as a process of learning a function on fixed representations to uncover how the internal structures of weak and strong models influence W2SG. Under a very general assumption about the representations, we demon- strate (illustrated in Fig. 1) that the key quantifiable property governing W2SG is the overlap between two spaces: one representing what the weak model's principal representa- tions (capturing key knowledge gained during pretraining) do not cover, and the other representing what the strong model's principal representations do cover. Errors in weak supervision that fall within this overlap hinder the strong model from reaching its full potential, leading to a predic- tion gap between the strong model finetuned with weak supervision and that finetuned with ground truth labels. A smaller overlap implies that fewer of the weak model's mis- takes are replicated, resulting in better W2SG performance.
Compared to (Lang et al., 2024), we analyze W2SG in more realistic setting where error correction can result from either underfitting or overfitting, allowing for full spectrum of behaviors. While benign overfitting is not our primary fo- cus, we discuss it as special case in See. 4 due to its impor- tance and offer new insights. Compared (Charikar et al., 2024), we explicitly links W2SG performance to the interac- tion between the weak and strong models' representations, providing a more comprehensive view of how the intrinsic properties of the two models jointly determine W2SG.
We then demonstrate an important use case of our main result: explaining benign overfitring, where the W2S model overfits the weak model's mistakes on finetuning data yet paradoxically generalizes better on the test set. Using our theoretical framework, we establish a general condition for benign overfitting and apply it to a toy example to concretely illustrate the role of representations in error replication: errors that do not align with the kernel defined by the strong model's 's principal representations are not replicated by the W2S model, regardless of the extent of overfitting.
3. W2SG from a Representation Perspective
We first formalize finetuning from a representation- based perspective, then introduce the properties of the represen- tations considered, and finally present our main theory.
Our theory offers a metric that predicts trends in W2SG performance in practice without having the finetuning task labels. This metric, which measures the overlap between the two highlighted regions in Fig. 1, shows strong correlation with W2SG performance across various settings. The extensive experiments across 8 datasets, involving 150 small transformers and 52 LLMs, not only validate our theoretical insights but also suggest their potential applications in man- aging W2SG, providing a deeper understanding of I.LMM behavior through their internal representation structures.
3.1. A representation-based perspective
The knowledge model acquires through pretraining enables it interpret inputs, extract relevant information, and organize it into meaningful intermediate states. This can be formalized as a "representation function", h, which transforms data into structured representations. Finetuning leverages this knowledge to produce the desired output, which we formalize as learning a new function f on the fixed h. The entire model thus represented as the composition foh. For simplicity, we consider the outputs
2

---

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
of h as vectors, and focus on the case where f is a linear functions. This is practically relevant because: (1) Training a linear task head on fixed representations is common with large foundation models, e.g., using embedding LLMs (Muennighoff'e et al., 2022), linear probing on intermediate activations (Zou et al., 2023; Nanda et al., 2023; Marks & Tegmark, 2023). (2) fine-tuning of LLMs largely operates in the NTK regime (Jacot et al., 2018), where training dynamics are captured by linear model representations derived from model gradients (Malladi et al., 2023). (3) Our experiments in Sec. 5 show that insights from analyzing linear functions generalize to the complex non-linear setting of finetuning entire LLMs from pretrained weights.
the shorthand notations: the weak model's test error Eryy Err(fw shw), the W2S model's test error Errwz. = Err(Jw2. ha), and the strong ceiling model's test error Errge Err(fe h2). ErTw2s measures the performance achieved through W2SG, while Err, serves as the upper limit.
We also introduce PredGap, the squared difference between the predictions of the W2S and strong ceiling models:
PredGap
- Sso (xx)))
It captures how much the strong model falls short of its full potential due to weak supervision. It is also indicative of ErTw2s, the direct measure of W2SG performance, through these connections: (1) If the strong ceiling model is nearly perfect, it follows that PredGap & ErTw2 the strong ceiling's s predictions are almost identical to the ground truth. This is not unlikely, since the ultimate goal of W2SG is to operate in cases where the strong model is a superhuman- level Al (Burns et al., 2023), plausibly capable of achieving perfect results if provided with ground truth labels. (2) With small regularization and well-conditioned representations, ErTw2s ~ PredGap + Errge (Thm. B.3), analogous to the Pythagorean theorem. Then, PredGap directly determines ErTw2s for fixed Errge (3) For general cases, the upper bound VErr I v /PredGap VErr follows from the triangle inequality. Furthermore, the result obtained from analyzing PredGap helps predict Err in our experiments (Sec. 5). Thus, our main analysis focuses on PredGap.
3.2. Preliminaries
Notations. We sometimes abbreviate a matrix A E Rixm as
when each element Ai, can be ex- pressed as generic term in terms of its indices. Amin, # ( denotes the smallest nonzero eigenvalue of matrix A.
Data. Let DD denote the distribution of the finetuning task's data, defined over the input-label pairs (z.y) E X x V, where y = R. In W2SG, we have two splits of data sampled from D. The first subset, D = { }= consists of ń i..d. samples and is used for finetuning the weak model. The second subset, D= = { }e){ with á i.i.d. samples used for finetuning the strong model. Note that the weak model's s outputs will be used as labels in place of the actual U:'s 's. In our notation, quantities associated with the two splits are marked by the diacritical symbols," and", respectively.
3.3. Setting: representations with a well-concentrated principal part and a manageable non-principal part
Models. We denote the weak and strong models' represen- tation functions as few and hn respectively. The finetuned weak model is represented as Swolww with
We first define two basic concepts, kernel and covariance, before introducing a general assumption on representations.
Definition 3.1 (Kernel Matrix). Given h X' -> IRd we define the kernel matrix on the finetuning dataset D as kh) = (h())) h(t,)) )isss, á x ñ matrix where each element represents the inner product between a pair of representations. K K(h) is defined on D in the same manner.
fw - arg min JEF, Σ - -1) +
where R(-) represents& & regularization.
The W2S model, which refers to the strong model finetuned with weak supervision, is represented as fw2.0/s, with
Definition 3.2 (Population/Emmprrcc Covariance Matrices). Given h :X' + Rd, we define the population covariance over distribution D as E(h) = Ep, [b<=)((=TT] The empirical version on D is defined as (1) = 1/2 Ei h ((.))(().))
à
fwz- arg min( ( JEF, ñ (f
(h) is defined on D in the same manner.
Additionally, as a reference, we define the strong ceiling model as the strong model finetuned with the ground truth labels. It is represented as Sac hi with
Given a representation function and a reasonable sample size, certain components in the representations should concentrate well, meaning they adequately reflect the population distribution. These components are pivotal to the model's generalization. In our analysis, we focus on cases where the remainder- less -esssell-concettrated components- satisfies certain conditions, ensuring their im- pact remains theoretically tractable. The decomposition of representations into these two parts is formalized as follows.
f arg min JEF
E((((E))) - -2))))))0++++ +
Evaluation. At test time, given any labeling function g:X X', we define its test error as the loss on the popu- lation: Err(g) = B(e = We then introduce
snaponents--atiifiess
3

---

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Definition 3.3 ((d, 3, ">ddccomposabiltty) Given D, D, D, and representation function h X' - R, we say that the representations of h are (6, 9, deccopposae w.r.t a subspace V (of R), for some 8 = 011), 9- 011), and 9 = =(1), if there exists a subset of eigenvectors of (h) corresponding to non-zero eigenvalues such that the following holds. Let denote the span of these eigenvectors, and let V denote its orthogonal complement. Let Hy and yy denote the orthogonal projections onto V and V+, respectively, Define p Amin, x (Z((1))) and y= 1=mmnn9, 5)- With high probability of 1 - o(1):
to be approximately orthogonal in the high-dimensional limit. More concrete instances will be presented in Exam- ples 3.4 and 3.5, as well as in Theorem 3.6, along with discussions of their significance and relevance.
Additional explanation for Diminishing population covariance on V+. We note that this condition does not imply negligible impact of representations on V For example, when ô is small, the model can in fact leverage the components in V to interpolate the training data, even when such interpolation cannot be achieved by the components in V (see Example 4.2).
(a) Boundedness. A basic condition that ensures reasonable magnitudes of representations and labels: [ = 01), [ = O(1) |E(h)lopp = 0(1), 1 n. Em= o² = 011 and E == 8 O(1). (b) Concentration on V. Representations are well- concentrated in the subspace V. both in terms of their covariance and their correlation with labels:
We refer to Hyh( the well-comcentrated part of the repre- sentation, as the principal representation, and the remainder, Hyr)(x)), as the con-principal representation.
Examples of Def. 3.3. Def. 3.3 is highly general, covering various representation distributions and dimen- sionalities. One simple case is when all components are well-concentrated, i.e., the entire representation is principal. This occurs when the representations exhibit a certain low-rank structure, which is common in deep neural networks (Huh et al., 2021). Below is a concrete example. Example 3.4 (Arbitrarily parameterized; bounded represen- tations with low intrinsic dimension). Given h : X' - Rd for any (æz, y). [|(x=)]]2 < B and y² S C, where (1). Additionally, [h))l = (()) The intrinsic dimension of E(h) is defined as intdim (Eh)) TY(E) denoted by Tillop
(E) ElIyh)l = o(7² + 5² + p²),
(Iyh) - hllyo) = o(72 + 8² + p²),
Hyh(d:)ū - = +0+ and
Hyh(E,)ū - E E[yylly) =
Kernel-wise o-isotropy on Vー The kernels con- structed using only the components in yー exhibit certain uniformity in all orientations, with the extent of uniformity controlled by d: 1 (II h) =0?? + 5²), and KI - op = o(72 +52)
(d) Small cross-sample inner-product on VL. V (Ty h(@:)) T Hl ((ss)))sssassssllll
q. Let n = min(f. π)) and assume n --c 3 (B (Bgg for some constant e < 1. Then, the representations are (n -0.1e, 0, O)decooposabble w.r.t. IRd.
o(y + 8), which holds when representations on y are nearly orthogonal across samples or have small magnitudes. (e) Diminishing population covariance on V The representations on V have small magnitude in the population: || (IHyy> = o07 + 5).
Remark. The conditions imply low intrinsic dimension relative to the sample size: g log q = o(n²-s) (App. C.I), but without restricting the actual dimension d, allowing both under- (d (2) and pee-pparamterrized (d2n) settings.
The next example is related to the spiked covariance model originating from PCA and widely used in recent theoretical studies across various domains (e.g., (Muthukumar et al., 2021; Nakada et al., 2023)). It is also related to the sparse coding model, which has its roots in computer vision (OI- shausen & Field, 1997), and has been applied language modeling (Arora et al., 2018) and deep learning theory (e.g.. (Allen-Zhu & L., 2020)). More references are in App. C.2. We consider representations that follow aab-Gaussian, which is a very general class of distributions, including, c.g., any bounded random variables and Gaussian.
Additional explanation for Kernel-wise o-isotropy on V-_ . To provide a clearer understanding of this condition, consider the following: If is very small (e.g., = 0), the kernel on D is nearly identical to II, meaning does not exhibit any specific patterns that differentiate between data points. In contrast, with a larger å (e.g.. 8% > t), this requirement is much more relaxed the kernel no longer needs to closely resemble 7 but instead must simply have its magnitude bounded by (8). Thus, it accommodates scenarios where the kernel is highly isotropic, very small in scale, or anywhere in between. This is key to our anal- ysis, as it ensures the effect of the less well-concentrated part of the representations remains tractable. We note that this condition is not only analytically convenient but also practically relevant in real-world scenarios. For example, high-dimensional sub- Gaussian noise satisfies this condition with a small 8 8-a a situation highly relevant to deep neural networks with large internal dimensions, where vectors tend
Example 3.5 (Heavily overparameterized; sub-Gaussian with spiked covariance). Given h +R and randomly drawn æ, M(x) has independent zero-mean sub-Gaussian entries. The first & entries have a (sub-Gaussian) parameter of ((1) and variance 1, while the remaining d k entries
have parameter of ((2-)) and variance d-k- The scalings satisfy: = (()), 0² O(m), á == (l?), and
o²
4

---

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
d=)( The labels have bounded moment, O())) Then, the representations are (0, o² o²) A deccompsabble w.r.t. the subspace corresponding to the first i coordinates. Remark. Compared to Example 3.4, this example accom- modates cases with high intrinsic dimensions. For instance, if we set x² (î)) then intdim = (n)).
Theorem 3.8 (Main result). Under Assump. 3.7, and as- suming reasonable regularization: 8n S [0(=0] and 6,5 B, 01)) let ÿ bū 122 ... Dal Then, w.h.p., we have
1 vh 11l PredGap = [P. -w)
(I)
More complex examples can be constructed from the fact that adding high-dimensional sub-Gaussial to (ō, 0, 0)- decomposable representations preserves decomposability: Theorem 3.6. Given a representation function h whose representations h(x²) E Rd are (6, 0, 0) decomposable W.r.I. Rd, we construct new representations with axx) Mh(x) + M where M E p(d+m)xx and M E (R(4m) xm both have orthonormal columns, and their column spaces are orthogonal to each others. If elements in E(ar)& E Th are independent zero-mean Sab-Gasssian with parameter ((g)) and variance e. assuming ñ = (ii), m m-=we = and o² = ( then o's 's representations are à in ddecomposable w.r.t. the span of M's 's columns. Remark. For instance, one could take h from Example 3.4.
P.(I-Pw) captures "what the weak model is unable to learn but is learnable by the strong model using their respective principal representations". Therefore, it deter- mines the mistakes that will be learned by the strong model, as discussed in the intuition. A more powerful weak model has Pw that covers more space, shrinking P3 P3(I-P, Pw) and potentially leading to a smaller PredGap.
Propagation of Errors. The earlier intuition is reflected in the proof (App. A.7). Given the labeling ÿ. its projection IwPP)) jú is orthogonal to the scaled weak model's s principal kernel and thus cannot be effectively learned, contributing to the weak model's error (Lem. A.1 The projection of this error onto the scaled strong model's principal kernel, (I-Pw) Ú. is learned by the strong model and contributes to PredGap (Lem. A.13).
(8, 0² o²
We assume both models' representations satisfy Def. 3.3: Assumption 3.7. hw's 's representations are (6w. Iw: "- decomposable w.r.t. V.. and h₂ 's representations are (8., 1ss deccmmoooee w.r.t. V/.
4. A Case Study on Benign Overfitting
Our theory can be applied study and provide new insights into benign overfitting, an intriguing special case of W2SG, where the W2S model appears to mimic the weak supervi- sion during finetuning, yet generalizes better at test time.
3.4. Principal representations shape PredGap
Intuition. One implication of Def. 3.3 is that only what is learned through the principal representations will be reflected at test time. Thus, the weak model's mistakes primarily stem from its inability generate certain outputs using its principal representations. For the same reason, among these mistakes, only those expressible through the strong model's principal representations will affect its test performance. Therefore, a key concept affecting W2SG per- formance is "what the weak model unable learn but is learnable by the strong model using their respective principal representations" which we seek to quantify.
4.1. A general condition
Benign overfitting has been studied in the general machine learning context to understand deep neural networks' gener- alization (Bartlett et al., 2020; Wang et al., 2021; Frei et al., 2022; Mallinar et al., 2022). Recently, (Wu & Sahai, 2024) theoretically characterized benign overfitting in W2SG for specific data distribution. Here, we aim to derive broader insights from representation perspective. We consider the scenario where the strong model's ss representations are highly expressive, enabling near-perfect overfitting of arbitrary labelings on the finetuning data, mirroring the behavior of very large neural networks in practice (Zhang et al., 2021). This occurs when 6, = o(f) (Lem. B.4), yielding a highly isotropic non- non-principal kernel. Meanwhile, since generaliza- tion depends solely the principal representations by Thm. 3.8, small P| - Pw) V أزl² suffices for good W2SG performance, regardless of the extent of overfitting. In this way, connect benign overfitting to the general relation- ship between the weak and strong models' representations:
Formalization. To formalize the above idea, we leverage K (II,, (lw) and K (Il, kee-ees computed using only the weak and strong models' principal representations, referred to as principal kernels. We define the following
(1 (vw,) + (Bw (7
Pw
K k hww)
P. = n KII h2))
( K h.) (8. 9))1
Pw and P. represent scaled projections onto the spans of the principal kernels. Each captures the space of output patterns that its respective model can express through its princi- pal representations (with regularization taken into account). Then, the earlier intuition can be characterized as follows.
Theorem 4.1 (A general condition for benign overfitting ) In addition to Assumption 3.7, suppose that (1) 5, = o(1))
'Thm 1 be extended to cases where the strong ceiling not perfect, but we omit this for brevity.
5

---

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
and 6, S 3, = off:). (2) .., the strong ceiling model achieves nearly perfect performance, i.e,, Err, o(1), (3)
which are corrected (overfitted but benignly) by the W2S model, and how representations determine this.
w.h.p., || - Pw) / U11 Errw- - Å with A = (1).
The principal representations for both models are simply at their first coordinates. Thus, the spans of their principal kernels are one-dimensional. Let C E IR denote the
Then, w.h.p., the W2S model achieves an almost zero (o(1))
training error on D. but generalizes better than the weak model: Errwss Errw A + o(1). See proof in App. B.
vector collecting the Ç values on D. i.e. Ç = Sal....,[]]]
Remark. Compared to (Wu & Sahai, 2024), which focuses on demonstrating that benign overfitting can occur under specific assumptions such as bi-level ensemble structure and labels depending 1-sparsely on representations we ex- tract more general insights into when and how benign over- fitting arises. Specifically, we identify single key quantity driving benign overfitting in W2SG: || - Pw) va ûll- When this quantity is small, the strong model can avoid repeating the weak model's s mistakes-rrearddee tent of wverfiting--hheeee achieving error mitigation. This precise mechanism was not revealed in prior work.
Similarly, define ğ [ii...., -9a I We can approx-
imate the projection matrices as: Pw x " 4 and Ps ~ グØ where í VTū + VT- Note that vectors / and VA J are almost orthogonal as the corresponding random variables are uncorrelated: I Oă J. = á E, I& Elyç] = Let cw be the vector whose i-th element is the weak model's error on data point (é,, ūt). By Lemma A.1 we can approximate Ew as:
of the ex-
1 Vil
Ew ~= (I- Pw) ~ 7) 9
Vn(1 - 7) Vn
The strong ceiling model's error Eri,e R 0 as its repre- sentations directly encode y in the first coordinate. Thus, Errw2 R PredGap. By Thm 3.8, PredGap & P.e.. Then,
4.2. Instantiation of Theorem 4.1 on a toy example
We present a concrete example of the scenario in Theorem 4.1 to demonstrate the realizability of the conditions. While more complex examples could be constructed, we focus on a simple one to succinetly illustrate the core ideas.
1 Vú Vn(1 7) Vn ?
WrTw nt úÚ (1 17) Vn
n
replicated
avoided: = since 1 allmost
Example 4.2. The label is a Gaussian: y ~ (0. Given (æt, y). the weak model's representation is hw (x²) =
The first term of the weak model's 's error, (1- 7) A 3. aligns with P. which spans the strong model's principal kernel, and is therefore replicated by the W2S model. The second term, Vnn(1 - 17) vR č. is orthogonal to P. and thus mitigated. Notably, - mT -) n) - aligns with the strong model's non- non-principal kernel, which highly isotropic (7s = w(&))) causing the corresponding enrors to appear mimicked by the W2S model during finetuning. However, they do not manifest at test time. In other words, only errors within the span of the strong model's principal kernel are overfitted harmfully, while overfitting elsewhere remains benign.
[(VTy y + v1 21--1 () E_ where 7 E (0, 1) is some con- stant, (~ (0, 1) and &w (0, I) are both indepen- dently drawn. The strong model's 's representation is (xx) [y ST, where& & (0, - r) independently. The seal- ings satisfy ñ = (î)) w11), d = (nn2), and 0² = o(f) but # 0. Additionally, B. o( ) and S. of )²)
Here, the weak model's 's first coordinate carries a signal about the label y, but corrupted by noise Ç. with 1) controlling the signal strength (i.e., with SNR --m) The strong model's 's first coordinate carries perfect signal about y. The remain- ing coordinates in both models are high-dimensional ran- dom noise. Both models' representations are special cases of Example 3.5 and are therefore (0, o. o²) decomposable.
5. Predicting W2SG Without Labels
Leveraging Thm. 3.8, we derive representation- metric that can predict W2SG performance without labels in experiments across various settings. Notably, this metric strongly correlates with W2SG performance even when we finetune entire LLMs-a scenario significantly more complex than what we analyze in theory.
Corollary 4.3. Benign overfitting occurs in Example 4.2. Specifically, w.h.p.. (1) The weak model's 's errors on both D and the population are (1-n))o(1). (2) The W2S model over- fits the weak model's 's outputs on D, achieving a training loss of o(1). (3) However, compared to the weak model, the W2S model achieves a smaller test error: Errw2s = 2
5.1. A label-agnostic metric for W2SG
For instance, if 17 0.6, then Err, 0.4, while Errw2, 0.16, despite nearly perfect overfitting on D.
We start with upper- bounding the RHS of Thm. 3.8.
Corollary 5.1 (Upper Bound 1). Define C= - R2 Following Theorem 3.8, directly applying the submultiplicu- tive property of the norm yields the following upper bound:
4.3. A closer look at error propagation
We provide a rough derivation of the W2S error (with details in App. B.3.2), illustrating which errors are replicated and
PredGap S C|| - Pu)l13p + +1)),
6

---

Representations Shape Weak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Table I: An overview of the three setups considered in our experiments.
EXP ID
Task
Strong model
Weak models
Finetuning
molecular tasks
MolBERT
150 transformers pretrained on GuacaMol 22 other embedding models
task head task head full model
II
NLP tasks
Bwte//ee-EEbeee Owen/Owen-
NLP tasks
28 smaller LLMs
Corollary 5.2 (Upper Bound 2). Following Theorem 3.8, we can also obtain an upper bound that involves Errsc as long as [E [0)
training a task head the embeddings produced by the base model (2) Activation maps. In Exp. III, we finetune the entire L.M from pretrained weights, so we don't 't have fixed representations as in the theoretical setting. To address this, we adopt a simple heuristic: we treat the layer-wise nor- malized vectorized activation maps of the pre-trained LLM, which encode information about how inputs are represented within the model, as the representations for computing h(2r). This heuristic serves primarily as a proof of concept, demon- strating that even straightforward approach like this can yield meaningful results. More principled definitions of rep- resentations, e.g., those based on NTK (Malladi et al., 2023) or representation engineering (Zou et al., 2023), could be explored in future work. See further discussion in Appx. E.
1 à
Ei= 1i ffl = o(1) (see proof in Appendix B.4):
2 PredGap S ( + v Errse) + (I))
In both upper bounds, C' represents the variance of the labels on DD which can be treated as a constant given a fixed dataset. Therefore, PredGap is governed by the norm [ - Pw) or [|P.(T - Pa) llop. Compar- ing the two bounds, the one in Corollary 5.2 is tighter particularly when Err, is small
This follows from However, in our exper- iments, both are similarly indicative of W2SG performance.
S (.)
I approximating principal representations. We consider two versions of II,, the operation that extracts the principal part from the representations, based on the intuition that principal representations tend to have larger magnitudes (e.g., Example 3.5). (1) In Exps. I and II, we apply PCA by projecting the representations onto the eigenvectors of the covariance (1) with eigenvalues 2 a x (the largest eigenvalue). (2) In Exp. III, we select the top coordinates with variance exceeding ax (the largest coordinate variance), a cheaper alternative to PCA for high- eigh-dimensional activation maps, as it avoids the expensive eigendecomposition. In both cases a is a hyperparameter.
Now that PredGap can be bounded in terms of the above label-agnostic metrics, and PredGap is indicative of the error Errwz. as discussed at the end of Sec. 3.2, we turn our focus to examining the following relationship in real models
L P.(I - Pw) (or ||P - ( Ps)
Errw2s
to evaluate whether the metrics offer practical insights. Specifically, we consider the three setups summarized in Ta- ble with their details discussed in the corresponding sub- sections. In each setup, we fix the strong model and vary the weak model to obtain different Errwas and || P.( - Pw) (or [[ - Pw) Psllo)) pairs and study their relationship.
:er: effective regularization. In Thm. 3.8, (B + 7) is the effective regularization, capturing both the explicit (3) and implicit (6) (Jacot et al., 2020) regularization. In practice, regularization can also stem from factors like early stopping, training algorithms, etc. We summarize these effects using Pen in Eq, 2 and treat Bei as a hyperparameter.
5.2. Empirical measure of P., and P.
Before proceeding, let's address an important question: how can we compute Pw and P3 for real models? In some cases, representations are not fixed during fine-tuning, making h difficult to define. Additionally, determining the principal representation, Hyh, is challenging because the exact V depends on the population, which is unknown in practice. To tackle this, we design heuristics to approximate P as follows
For each model, computing P introduces two hyperpa- rameters, a and B. If every model is assigned unique hyperparameters, the total number of hyperparameters would be twice the number of models. To simplify this, we let all weak models share the same two hyperparameters, aw and Su. For the strong model (only one in each setting), it is treated separately with its own hyperparameters, Os and 8.. Thus, we only have four parameters in total. More details are in App. D.2.
((II,))) 1 ñ (II Ban
(2)
We explain the key components below.
h: extracting representations. We consider two ways of defining the representations, depending on the setup. (1) Last layer embeddings. In Exps. and II, the definition of representation is self-evident, as finetuning is simply
'In the analysis, the linear model does not include a bias term, but it does in our experiments. This is addressed by appending constant 1 to the representation when computing the metrics. *We observed worse results with last-layer embeddings in Exp. III. likely due to complex cross-layer dynamics during finetuning.
"One can also observe this in Example 4.2, where the equality Corollary 5.2 holds, whereas that in Corollary 5.1 does not.
7

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
SC=0.9 0.35
SC=0.841
SC=0.889
5.3. Experimental setups
1.5
niwe 254
120
now 128 suc 64
Exp. I: Molecular prediction. Our first setting follows (Charikar et al., 2024). We use the GuacaMol (Brown et al., 2019) dataset for pretraining both the strong and weak models. For finetuning, we consider three regression datasets-ESOL, FreeSolv, and Lipop-from the MoleculeNet (Wu et al., 2018) benchmark, curated by ChemBench (Charleshen, 2020), which involve predicting molecular physical properties. The strong model is MolBERT (Fabian et al., 2020), a BERT (Devlin, 2018) pretrained for 100 epochs GuacaMol. We use smaller transformers pretrained on GuacaMol as weak models. These weak models have 2 layers and 2 attention heads. We vary the hidden size across 64, 128, 256, and vary the number pretraining epochs from to 50, resulting in 150 weak models. During finetuning, we extract last-layer embeddings and perform linear regression. MSE loss is used for both training and measuring Errw2, as the task is regression. Additional details are in App.D.1.
Efr
EfT
1.3
1.2
1.1
2900
3.285
0,20
0.002
0.38
0.39
0.08
[|P_01
P PwJllop
[P:(1
Pop
|P Pw] lop
(a) Lipop
(b) FreeSolv
(c) ESOL
Figure2 2: Results of Exp. I: our metric strongly correlates
with 2rr and serves as more fine-grained indicator than model size.
5C=0.923
SC=0.711
3.40
Emaan
9:28
1.20
0.40 [P.11 - Polllop
[P,(1 - Pullpp
(a) Justice
(b) Commonsense
Exp. I: NLP tasks with embedding models. We use the "Justice" and "Commonsense" datasets from ETHICS (Hendryeks et al., 2020), which involve binary classification based on basic moral concepts. We consider embedding models pretrai-ed LLMs that convert text inputs into vector-based embeddings, with nvidiaaNV-EEmert (L.ee et al., 2024) (currently ranked first on the MTEB leaderboard (Muennighoff et al., 2022)) as the strong model, and 22 other models as weak models (details in Appx. D. For finetuning, we train a linear classifier on the embeddings with CE loss. Err, is measured as classification error.
Figure 3: A strong correlation between |PP.T - pop and Errw2s is observed in Exp. II where we finetune embedding models.
SC=0.842
SC=0.804
SC=0,853
0.45
.
Erfwos
84-
Errwas
sie [Pa(1 1.(1-P. po
[P P. Palloo
[P
Po
Exp. III: NLP tasks with end-to-end finetuned LLMs. We replicate setup from (Burns et al., 2023) on three datasets: (1) SciQ (Welbl et al., 2017), containing crowd- sourced science exam questions; (2) Amazon Polarity (Zhang et al., 2015), consisting of Amazon reviews; and (3) Cosmos QA (Huang et al., 2019), involving commonsense based reading comprehension. Both data preprocessing and finetuning strictly follow (Burns et al., 2023). The entire model is finetuned with the unembedding layer replaced with a linear head, using CE loss. We use Owen/ /wtn/wenn- (Bai et al., 2023) as the strong model and 28 smaller LLMS as weak models (details in Appx. D.1). Errw2s is measured terms of classification error.
(a) SciQ
(b) Amazon Polarity
(c) Cosmos -QA
Figure 4: A strong correlation between |P( - Pw)
and Errw is observed in Exp. III involving general-purpose LLMs.
SC=0.905
SC=-0.065
SC=-0.048
845
EITw2s
Erfwzs
6.40
Errags
0.2
825
dee ziee dee whep aammact_mapp
200 200 -dim((1h)
203
[P_((- - Poiloo
Figure 5: In Exp. III. for models with activation map dimensions 8000, both the activation map dimension (middle) and the dimension of approximated principal rep- resentations (right) correlate poorly with Errw2s. However, [] - -x) llop remains strongly correlated with Errwas (left). We only show the results for Cosmos QA and defer those for other datasets to App. D.4.
5.4. Results
Strong correlation between Errwss and | - Pw) llop across various settings. For each of the weak models, we perform the W2SG procedure to obtain the resulting W2s model. We then measure Errw2s and P.P Pwll and plot the results in Figures 2, 3 and 4. Across all the setups, observe strong correlation between the two quantities, with high Spearman's correlation values displayed at
8

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
the top of the figures. The results highly similar for [P.(I- Pw) llopp shown in Appx. D.3. Therefore, we only focus on discussing || P.(Z - Pw) lop in the main paper. Notably, the correlation between Errw and | (P)(I- extends beyond the theoretical setting, covering the fol- lowing variations: (1) Loss function and evaluation metric. While Thm. 3.8 is based on linear regression with MSE loss, Exps. II and III demonstrate that the correlation also holds for classification tasks using CE finetuning loss, with 2rr measured as classification error. (2) The form of finetuning Thm. 3.8 assumes that finetuning involves training function on fixed representations. However, in Exp. III, the entire LLM is finetuned. Despite the complex training dynamics in this scenario, a strong correlation between Errw2, and maps are heuristically used representations. These results underscore the broad applicability of our conclusion.
Impact Statement
We see positive societal impacts in our work as it advances the understanding of Weak-to-Stre Strong Generalization, cru- cial problem for aligning superhuman AI in the future. Our results could enhance transparency in AI systems' behavior through analysis of their internal structures and contribute to the broader goal of improving Al safety and reliability.
A Acknowledgement
This research was partially supported by the National Sci- ence Foundation CAREER Award 2146492 and an OpenAl SuperAlignment Grant.
is still observed when activation
References
Allen-Zhu, Z. and Li, Y. Towards understanding ensem- ble, knowledge distillation and self eiff-iittillatinn in deep learning, arXiv preprint orXiv:2012.09816 2020.
Capturing W2SG beyond model size. Smaller weak models can sometimes achieve better Errws than larger ones. For example, in Exp. I, the leftmost yellow point (size 64) outperforms the rightmost teal point (size 128) in Fig. 2, likely because these smaller models were pretrained for more epochs (recall that we have 150 models span different combinations of sizes and pretraining epochs), resulting in better representations. Similarly, in Exp. III, the middle column of Fig. 5 shows poor correlation between Errw,, and size for models with dimension & 8000. Testing another dimension-basee metric-the dimension of approximated principal representations- also reveals weak correlation with Errwas (last column of Fig. 5). This un- derscore the complexity of predicting W2SG performance, as larger models or higher representation dimensions do not guarantee better results. Factors such as the pretraining recipe, the quality and relevance of the pretraining data, etc., all contribute to the final outcome. However, even in these cases, I..(I Pw) ol consistently captures the trend in Errw2s (Fig. 2 and the first column of Fig. 5), demonstrating its robustness metric that surpasses simple dimensional measures and provides meaningful insights for W2SG.
Arora, S., Li, Y., Liang, Y., Ma, T., and Risteski, A. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computa- tional Linguistics, 6:483 49533-44952201 2018.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint ar arXiv:2309. 16609, 2023.
Bartlett, P. I.,, Long, P. M., Lugosi, G., and Tsigler, A. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):3006333000 2020.
Brown, N., Fiscato, M., Segler, M. H., and Vaucher, C. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096-1188. 2019.
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint 2rXh::23/2.093900 2023.
6. Conclusion
In this work, we show that W2SG can be characterized using kernels derived from the principal components of weak and strong models' representations. The theory is applicable to wide range of representation distributions, provides insights into how models' internal structures influence error correction and the conditions for benign overfitting. Additionally, it offers a label-free metric for predicting W2SG performance, validated through experiments on diverse datasets and LLMs.
Charikar, M., Pabbaraju, C., and Shiragur, K. Quantifying the gain in weak-to-strong generalization. arXiv preprint 2rXhv:2405.15116. 2024.
Charleshen. Chembench:
The molecule benchmarks and molmapnet datasets, September 2020. URL https: 5281 /wtp//eeooo..o 4054866.
Demmel, J. The componentwise distance to the nearest singular matrix. SIAM Journal on Matrix Analysis and Applications, 13(1):10-19, 1992.

---

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 1rX1v:1810004855.0 2018.
Kingma, D. Adam: A method for stochastic optimization.
arXiv preprint /4 2014.
Lang, H.,, Sontag, D., and Vijayaraghavan, A. Theoretical
El Ghaoui, L. Inversion error, condition number, and ap- proximate inverses of uncertain matrices. Linear algebra and its applications, 343:171-192 2002.
analysis of weak-to-strome generalization. arXiv preprint arXiv:2405. 1.0.:255555633,, 2024.
Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. Nv-embed: Improved techniques for training ilms generalist embedding models. arXiv preprint arXiv:2405. 74288, 2024.
Fabian, B., Edlich, T., Gaspar, H., Segler, M., Meyers, J., Fiscato, M., and Ahmed, M. Molecular representation learning with language models and dolain-relevant aux- iliary tasks. arXiv preprint arXiv:2011.132300 2020.
Mairal, J., Bach, F., Ponce, J., et al. Sparse modeling for image and vision processing. Foundations and Trends@ in Computer Graphics and Vision, 8(2-3):85-22 283, 2014.
Foldiak, P. Sparse coding in the primate cortex. The hand- book of brain theory and neural networks. 2003.
Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A kernel-based view of language model fine-tuning, In In- ternational Conference on Machine Learning, pp. 23610- 23641. PMLR, 2023.
Frei, S., Chatterji, N. S., and Bartlett, P. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data, In Conference on Learning Theory, pp. 2668-2703, PMLR, 2022.
Mallinar, N., Simon, J., Abedsoltan, A., Pandit, P., Belkin, M., and Nakkiran, P. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Prucessing Systems, 35:1182-1195, 2022.
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J.. Song, D., and Steinhardt, J. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275. 2020.
Huang, L., Bras, R. L., Bhagavatula, C., and Choi, Y. Cosmos qu: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277, 2019.
Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824 2023.
Huh, M., Mobahi, H., Zhang, R., Cheung, B., Agrawal, P., and Isola, P. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2 10427, 2021.
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 2022.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Muthukumar, V., Narang, A., Subramanian, V., Belkin, M., Hsu, D., and Sahai, A. Classification vs regression in overparameterizet regimes: Does the loss function mat- ter? Journul of Machine Learning Research, 22(222): 1-69, 2021.
Jacot, A., Simsek, B., Spadaro, F., Hongler, C., and Gabriel, F. Implicit regularization of random feature models. In International Conference Machine Learning, pp. 4631- 4640. PMLR, 2020.
Nakada, R., Gulluk, H. I., Deng, Z., Ji. W., Zou, J., and Zhang, I.. Understanding multimodal contrastive learn- ing and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics, pp. 1348-4380 PMLR, 2023.
Ji, W. Deng, Z., Nakada, R., Zou, J., and Zhang, L. The power of contrast for feature learning: A theoretical anal- ysis. Journal of Machine Learning Research, 24(330): 1-78, 2023.
Nanda, N., Lee, A., and Wattenberg, M. Emergent linear rep- resentations in world models of self-supervised sequence models. arXiv preprint wxXiv22009000441 2023.
Johnstone, M. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295-33) 2001.
Olshausen, B. A. and Field, D. J. Sparse coding with an overcomplete basis set: strategy employed by v1? Vi- sion research, 37(23)33311-333
1997.
Kalimeris, D., Kaplun, G., Nakkiran, P., Edelman, B., Yang, T., Barak, B., and Zhang, H. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019.
Olshausen, B. A. and Field, D. Sparse coding of sensory inputs. Current opinion neurobiology. 14(4):481-4)) 2004.
10

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Papyan, V., Romano, Y., and Elad, M. Convolutional neural networks analyzed via convolutional sparse coding. Jour- nal of Machine Learning Research, 18(83)11-52, 2017.
Xue, Y., Joshi, S., Gan, E., Chen, P.-.,, and Mirzasoleiman, B. Which features learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression. In International Conference on Machine Learning, pp. 38938-38970 PMLR, 2023.
Pezeshki, M., Mitra, A., Bengio, Y., and Lajoie, G. Multi- scale feature learning dynamics: Insights for double de- scent. In International Conference on Machine Learning, pp. 17669-17690. PMLR, 2022.
Yang, J., Yu, K., Gong, Y., and Huang, T. Linear spatial pyramid matching using sparse coding for image classifi- cation. In 2009 JEEE Conference computer vision and pallern recognition, pp. 1794-1801. IEEE, 2009.
Shen, R., Bubeck, S., and Gunasekar, S. Data augmentation as feature manipulation. In International conference machine learming, pp. 19773-19808. PMLR, 2022.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Shin, C., Cooper, J., and Sala, F. Weak-to-strong gener- alization through the data-centric lens. arXiv preprint arXiv:2412.03881, 2024.
Zhang, X., Zhao, and LeCun, Character-level convolu- tional networks for text classification. Advances in neural information processing systems, 28, 2015.
Somerstep, S., Polo, F. M., Banerjee, M., Ritov, Y., Yurochkin, M., and Sun, Y. statistical frame- work for weak-to-strong generalization. arXiv preprint arXiv:2405. 16236, 2024.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.0142 2023.
Tropp, J. A. et al. An introduction to matrix concentra- tion inequalities. Foundations and Trends@ Machine Learning, 811-2):1-230 2015.
Zou, D., Cao, Y., Li, Y., and Gu, Q. Understanding the generalization of adam in learning neural networks with proper regularization, arXiv preprint arXiv:2108. 13771, 2021.
Vershynin, R. High digg-ddimessionl probability: An introduc- tion with applications in data science, volume 47. Cam- bridge university press, 2018.
Wainwright, M. J. High-dimensional statistics: A non- asymptotic viewpoint, volume 48. Cambridge university press, 2019.
Wang, K., Muthukumar, V., and Thrampoulidis, C. Benign overfitting in multiclass classification: All roads lead to interpolation. Advances in Neural Information Processing Systems, 241799 2021.
Welbi, J., Liu, N. F,, and Gardner, M. Crowdsoure- ing multiple choice science questions. arXiv preprint arXiv:1707.06209 2017.
Wen, Z. and Li, Y. Toward understanding the feature learn- ing process of self-supervised contrastive learning. In In- ternational Conference on Machine Learning, pp. 11112- 11122. PMLR, 2021.
Wu, D. X. and Sahai, A. Provable weak-to-strong generalization via benign overfitting. arXiv preprint arXiv:2410.04638, 2024.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530, 2018.

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
A. Main Analysis
In this section, we provide thorough analysis of the errors associated with the weak model, the W2S model, and the strong ceiling model. Some of these results are used prove our main conclusion, Theorem 3.8, while others are applied in subsequent analyses.
A.1. Notations and additional notes
Symbol definitions. We introduce the following notations. The symbol r represents representation, i.e., h(x)). For the samples in the splits D and D. we denote their representations as fi..... i' and P1...., : Th respectively. We define the sample representation matrices, where each column corresponds representation:
R
Fa
and
R = [ 0² .n].
We also define y which collects the labels of the samples:
V1
Ú1
8/2
ÿ
and
ÿ
:
[
Un
For the covariance matrices, we use the following shorthand notations to avoid clutter:
Σ = (h), Σ = E(,, Σ (h)). E' = M)),,, Σ' = E(IT,), Σ" = E(T,h), E" = E(m N), E" E(Ty, h). Σ" E(IT h).
Use of subscripts. Additionally, we use subscripts 'w' and 's' to indicate the model associated with given quantity. For example, Ry and Ra denote the sample representation matrices generated by the weak model, while RG and R, denote those generated by the strong model. Similarly, this convention applies to covariance matrices; for instance, Σ = EII,, h.).
Mathematical notations. For convenience, whenever we say B + o(1), where and B are matrices or vectors, we mean that A ol = o()). We let (A), Amin (A), Amin. 0 (A), and Amax(A) represent the i-th, smallest, smallest nonzero, and largest eigenvalues of the matrix A, respectively. The expression A B means that the matrix B A is positive semidefinite, and A > B means that A B is positive semidefinite.
Implied proof' techniques. Sometimes, in the proof, we use the triangle incquality and the sub- cubbtuuliplicatiiitty of norms without explicitly stating them when they are straightforward, as mentioning them would make the text unnecessarily verbose.
A.2. Restatement of Definition 3.3
Here, we restate Definition 3.3 with simplified notations for convenience and clarity in the proof.
Definition A.1 ((d, 9, d)ddecooposabilitt (restated) Given D, D, D, and representation function h, say that the representations of h are (ő, 9, decomposablee with respect
subspace V (of the representation space), for some 5 = 0(1), 9 = O(1). and & = (1)), if the following holds. Let U UAUT be the singular value decomposition (SVD) E. There exists a matrix U consisting of subset of columns of U, corresponding to the nonzero eigenvalues, such that the following conditions are satisfied. Let U" denote the matrix that collects the remaining columns of U. Define diagonal matrices A' and A" to collect the eigenvalues corresponding to U' and U", respectively. Additionally, define: E' " and Σ" U" A""",, Let 7 min(f, ?), and let V be the span of the columns of U'. Now, leveraging the fact that the projection Hl be written as U and noting that Amin, y of oEE') Amin (A'), we can reformulate the original Definition 3.3 in terms of U': with high probability 1 o(1).
a. Boundedness. | 0(1) [Bopp = O(1), |Σ op = 0(1) and Σ op = O(1). Additionally, Ely?] 0(1), || O(1) and
12

---

Representations Shape Weak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
b. Concentration V. The original statement is | "'llop = o(1) and '' - E'llop o(). However, since:
| ΣU'
A'loop
U R A
UU RR U'U UU AU'U 1lop
E E'l E'llop=
and similarly for Σ', we can restate as: ||U JU' A A = o(>² + 62 + Amin (A') and | ÊU' A -Allop o(>2 + 82 + Amiu (A ). Similarly, by noting that the operator is invariant under left multiplication by U can restate the statement regarding y as: | vi
Rý U Erylll o(7 + 8 + Amin (A') and | vn Ry
U E Ervill = o 8 Amin (A')..
Wernel-wise 6-isotropy V 1l R R
Tlpp +(7 and |
R UU R R- o(>2+53).
d. Small cross-sample inner-product on V |1 R U"T
1 Rllop o(9 +).
Diminishing population covariance V+. E" op ol7 + 5).
Use of subscripts. Since in Assumption 3.7 assume that the representations of both the weak and strong models satisfy Definition A. all the notations in Definition A. have corresponding versions for the weak model's representations and the strong model's ' representations. We follow the previously mentioned convention and use the subscripts w' and s' to distinguish between them. For example, notations such as U", and U", A and AG, will be used. The meaning of such notations should be clear from the context which they appear.
A.3. Lemmas
Below, we introduce some basic lemmas and prove properties that will be used in the later analysis.
Lemma A.2 (Push-through identity). For any matrices A, B. and any scalar the identity (al AB) - A(al BA)-1 holds as long as (aI + AB) and (aI + BA)-1 are invertible.
A=
Lemma A.3, A classical result on the effect of perturbations on the inverse of a square matrix states that [[A + A)-1 A llopp S ||AA11113 ||
|| lop- where is an invertible square matrix. This result can be found. for example, in (Demmel, 1992) or Equation 1.1 of (El Ghaoui, 2002).
Lemma A.4. If condition Kernel-wise 6-isotropy on V holds, we have that 11 N 1R-(AÑUU' R 51 1lpp 2 52), and a similar conclusion holds for R as well.
Proof. By Kernel-wise 5-isotropy on V-L
PRUURR+ R+51)
R R
7
= R (U''T +U" - R'U R+31) lop
"
1 R -
=11 n
R R
=0(?2 + 82)
Lemma A.5. If condition Kernel-wise 8-isotropy on V holds, then for any 3 0 s.l. we have that II R R -1 R U R (5 - o(1), and similar conclusion holds for R. as well.
13

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Proof. By Kernel-wise d-isotropy on V-,
R R 31-
R (3 (4+ R
L 71
llop
R 3I ( R ++
R (U' +U"U"T
n
R U" R U
=0(?² + 8*).
Then, by Lemma A.3, we have
1 R R + 31)
1 R
UUU
R 8) So(7 + +80 |1( I(-R-U'' R B)T) - -
n
72+ + 82 (5 + 332
=0
Lemma A.6. If condition Concentration on V holds, then for any B 0(1) s.t. 8 8, and 7u E 15.5) we have
IIU ÉU' + (70 - -)) (A' + (70 + --))))-1 llop = o(1),
and a similar conclusion holds for Σ as well.
Proof. By condition Concentration on V, we have
| ÊU' A'llop = o072 + 82 Amin Amin (A')2).
Then, by Lemma A.3, we have
I1 ΣV' + (70 + B)T) - (A' + (70 ()1)) =5o(>
+ + Amin (A') I1( + (70 + - '
32 + 82 + Amin (A')2 (1o + B + Amin ( A (A))2
Lemma A.7. conditions Boundedness and Concentration V hold, then [Mmnn (A')2 Amin (UTTEU = o(72 +
5 + Xmin(A (A)) still holds if we replace with".
Proof. Define Amin (A') Amiul (U" E'U'. By condition Concentration on V and Weyl's theorem, we have
[4] o(72 + 82 *min (A')2). Then, we compute:
Amin (U'n E'U' E'U
= =Amin (A') + 12 - 2t2Amin (A')
Amiin (A')² + =(7² + 82 + Amin (A1)²),
where the last step follows because Amin (A') 0(1) (via condition Boundedness) and ||| o672 +.2 (Ai)n(A)))
26 + Amiia (N')
Corollary A.8. Lemma A.7 further implies that 4 (U' SU"F
O(1) when conditions Boundedness and Concen-
tration on hold. It still holds if we replace" with".
14

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Proof.
72 + 62 + Amin (U"T Σ
7 42-00 + Amin (A
Amin Amin(A)2 Amii EUT
42 82 Amiul (A
12 82 + Amin (A1)2
32 + 82 + Amin (A')2
o(7² + 8² + Amin
>minn (A))
<1 +
72 + 82 Xmin (A
< + o(1).
-2.462+ +.min (
Therefore, 574624 Amim (U'T EUY O())
Corollary A.9. If conditions Boundedness and Concentration on hold, then for any q with ||all = O()), we have
[|U" VA E'U + J/ U' ()) o(1). It still holds if we replace "with.
Proof.
I IV''N/0''''+ 'll2
(U00" (6++))+)+++
"" ()]++))))=+)
+0(13--0-NMMMM)))000X0XXXXX+++10 9 = (10"""""00((000000""">000000"V00
by Concentration on V and ||g| O(1)
72 + + Aminn (A')
(Amin(U" E'U') + B)²
72 + 00² + Amiin (A
""00""0""000"""00000000000000000 EU
E'' "0" 8)1) 9 t
7.2 + 322 +Amin (U" E'UNY²
72 + 5² + Amin (A')
=q =q E0'''
+ -)) - 9
22 + 82 Amia Ami E'UYY
=q -q"(U' EUT + 10 (6 B) ' =
by Corollary A.8
=|1 R RU RR U +
Corollary A.10. If conditions Boundedness and Concentration on V hold, then for any with | 0(1), have [JU' T Vn R R UU R + - 1b1 R. U R R U'U R (9 + () 2112 o(1)
and |IU'T VA
+ (5 = O()). still holds if we replace" with..
Proof. First, we have
R 1
R U+ 3) 2112
Vn
=|UVAA
U
RR +
TT
Robl" 12
by Lemma A.2
Vn.
=|| Vn
R
U
R - (f 1) "T
Rubll + o(1)
Vn
by the fact that
- Rubl 011) (via Boundedness) and invoking Corollary A.9
T RUU R (f + 3) -1612 2161 + o(1)
by Lemma A.2.
llop = I U'U' RIl Additionally, since || I R -+
P D'UPT t S 1, we also have the
bound "AU' 1 R R
0(1).
VA
15

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Lemma A.11. condition Kernel-wise ô-isotropy V holds, then | 1 R oo S -
Rooop vo( + 82) +5. Similarly,
Vn
Proof. By condition Kernel-wise d-isotropy on V and triangle inequality, we have
R U Rllop I o(7² + +0) +
nt
Then,
- Rlpoo
V || R U"U Rellop S vo(7² 82) + +
A.4. Basic expressions for the model weights and errors
Let ww E IRd,, Ww2s E Rdd and w, R&, represent the weights of the linear models fw. fw2s. and fs. respectively, Using the well-known closed-form solution for the minimizer of the MSE loss with lg regularization, we derive their formulas:
Www
Rw
Ray R - Ba1)
11
Vn
Ww2s Vn
Ra(= R. Rs -111
RR ww)
(3)
Vn
W.
Ra ( Rg R. + 311)
ÿ.
Then, we derive the expression PredGap
PredGap =E,, [(r, w. r "wwss) )
=E,, [(r, (wo - ww2s))
(4)
= Ey, [ww, ww22)) "rsrs (wt, ww2s)]
=(w; Ww2s) Er, 1rars]] [ ww2s)
= -(w, Wwzs) Ew Ww2s)
E, (. 2w2x)l1
-
1 Vn
Vn R + 8-1)
R
Ww
Vn
transformation determined by the strong model's representations weak model's normalized error vector on
VE, R
R.
R R. + B
I)-1
y
||.
f
Vil
transformation determined by the strong model's representations
weak model's normalized emvor vector on
(5)
From the above, we see that PredGap can be broken into two parts: the weak model's 's normalized enror vector on D, and transformation applied to this error vector which captures how the weak model's errors propagate to the strong model. In Sections A.5 and A.6, will analyze each part individually.
A.5. The weak model's error
Lemma A.12 (The weak model's error on D ). The weak model's error vector on D can be approximated as follows
1
1
R
R
R
R R
R. + B,
Vn 31
(I P.)
94 o(1),
Vn
Vn
where P., = R.TUUUU Ri -
(IRTUUU Re + (5r + Bw)1
16

---

Representations Shape Weak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Proof. By condition Boundedness and Lemma A.5, we have
1
1
R
R
R.
Bx1)
vn y
ni
11
71
1
Rw
Rw
1
R. U Rw + (iw + Bu)
Vn Vūū +
fii
UU
R
+
R U Vn
Rw
(
R
Ry + (fw + Bu)I)
-1
Vn ÿ + o(1)
Vn
Vn
By conditions Small cross-sample inner-product on V and Boundedness, and noting that [| U R. + (5 + 3) "lopp S 3utB. the preceding can be further bounded as
Rw
RI
R. Ru
+ 1)
Vn
VT
Vi
1
1
(.RTUR Rw (Fr + Bu))
1
Ri
UN
Rww
ÿ + o(1)
Vñ
Vi.
Vn.
U
I R. + (5w + Bw)1
U w
U w n
-
R + o(1)
by Lemma A.2.
By Lemma A.6 and condition Boundedness, the above further leads to
1
R Ru ( RN (.311 y
A1
Vn
Vn
R U (Nr + (5w + Bw)
U ñt
R. + o(1).
Condition Concentration on V implies that | 1 Rwo - U Rwilloo llp = o(Amin (A + Yw + Bw) via the triangle inequality. Then, by condition Boundedness and that (A + (7w + I) !lop Amin(A)) tt we further have
1
1
R
R. + Be 1) R (Nrr
Rew
Vn Ú
vn
1
U.T 1
Ryy Uny
(5w + B)
Ru. + o(1)
Vn.
ñt
1
R U'
1 U Rr R. Un + (Fw + Bw)I)
U
Ruy + o(1)
by Lemma A.6 and condition Boundedness
Vn.
n
-1 ( Pew + (Fw + B.)1)
U.U Roy
ÿ + o(1)
by Lemma A.2.
Vi
Let us define the shorthand P..
UUU R.d U. R, + (5w + +w))))
Then, we conclude that
RT
R
R
Ro
31
ÿ (I -P)
ÿ + o(1).
Vi
Vn.
n
Vn
Vi
A.6. Propagation of the error to the strong model
Lemma A.13. For any up with ||>5|| = 0(1, we have || V,,
L RT R. +
o(1), where
1
P. = R U.
R
R
U'U R + (1))
n
17

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Proof. We first decompose V
L Rg
L R R2 B,1)
11--ap as follows
1
R
R + B11
Vn
1 R
Σ, R.(
U,U R. + + +()+))))+ + o1) by Lemma A.5
Vn
-(U' + U. VA R R. + + (x))
R.
-U,VA R
R U R (3++))))
Vn
n
1
+U A
R
UU R + (8s +
(6)
Vn
n
The second term above can be bounded:
1
II"" VATU
R. R
U RG + (3s + 3,) llop
T
7 + 82))
%
max
(A
by Boundedness and Lemma A.11
% + B.
Vo(n2
+ 82) +
EX ol
i/s + 6
(7's + 6,) (,)2 2 + 62) + 9/6 6,)
by Diminishing population covariance V
(is + 8,)²
o(? 62)
+ 7/8
So
= =()))
i/s 8
R/s + &
Combining Equations 6 and 7 yields
/ Σ 1 B
1
R 1
R.UU R 13 (++++B) - + o(1.
f
Vn
Finally, we consider the squared norm:
Il Σ R. R + -
R
n
-
N.U
R.
R
U.
R (f+++++B)) -
Vn
#o
| 1
R,
RRIRR.
R U. R + (3% + BB)) - -bbll) + o(1)
Vā
= 1 i
R UUU'
R. + (5r + - + o(1) by Corollary A.10.
A.7. Proof of Theorem 3.8
Given that - أl = 0(1) by Boundedness, and that ||| Pullop Amin + R U.U. R.)+B.
S 1, we have |[]
Pw) 7R أ1 = 0()) Then, by Lemma A.1 the weak model's enror on D can be bounded as
(I Pw) VR +o(1) 0(1).
Recalling the expression of PredGap derived in Equation and applying Lemmas A.12 and A.13, we obtain:
PredGap P.(II Pw) 2 o(1)
18

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
B. Additional Analysis
B.1. Additional Lemmas
Lemma B.1. By Diminishing population covariance on V and Boundedness, we have
EUU"U" ry) ol v(
Proof.
1 n U Tiys
lim 1
1
1 U"
EU ry]
lim
U"
R
y
lim
R
llppl
n-+00
71
n Vn
Vn
vn
lim
U RR U"U"T
Ilop
y
op V Eyy = o( +
Lemma B.2. By Boundedness, we have
EUU'T ry] = 0(1)
Proof. The proof follows the same approach as that of Lemma B.1. This conclusion can also derived by bounding EUU'UT ry] in terms of its empirical counterpart using Concentration V. and then applying Boundedness
B.2. When ErT PredGap + Errse
Theorem B.3. Suppose that, addition Assumption 3.7, the conditions B, + 9/5 = o(Amin o(E(Ily,h.) = @(1) and
Amin 0 (E((Iy,))) OOmmux (E(Il,, hold. Then, w.h.p., we have:
Errwzs = PredGap + Errse = o(1.
Proof. First, decompose Errw2s as follows
ErTw2s [r,s Ww2s - y)²1
E(r Ww2s r. Wse r, Wwx y)
[r, Ww2s r.T wxr)² (r. wce y)² 2(w w.. (T. Use y)]
PredGap + Errs: + (W t's w r.) wee - y)]
= -PredGap Errse 2(Ww2s ww) (E, Ws E(r,yl),
(8)
Thus, to prove the theorem, it suffices to show [2W - wse) T TEE, - E(r,y)) = o(1). We decompose (ww2s ww)) WE - B[ray]):
(Ww2. ww)) (. - E[(ryl))
= =Wwws )s (E'wse" Wse Ex U. Eirsl U. B(rsy))
=(Wo2 wse) (Eewx U E/ray)) (ww2s w.)) E u.e (Wwas ) T.TT
E[ryy]
(9)
Ww2s - Wse can be approximated as:
1
Ww2s
Wse
R.
R2
R. + -111
R Ww)
P& Rs R. + +1,11
Vn
11
fn
1
R.
R R2 + +3,1) -1(
12 Ww
RS (1 K + (74 + 8,)1
-1
1
1
R w
4) + o(1)
by Lemma A.5 and that other terms are O(1)
Vn
in
Vn.
vi
(10)
19

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
where K² R. U Rg is shorthand for K (TI,, hg). Then, by Lemma A.11 and Boundedness, we obtain:
/0 52) + 9s
(1 Wse) U =0(
(11)
i/S + 3
We also have the following bound:
UUTH
R
R
R + 3,1) 1
Vi. Vllop
Vn
1 oop + olluUr"
UUT
Rs(
RR
U Rs + (fss + B))
Rsllopp) by Boundedness and Lemma A.5 vn
Vi
11
vn
o(?g + 63) + 9s
by Lemma A.11 and Boundedness
(12)
1/8 + Bg
Combining Diminishing population covariance VL and Equations 11 and 12, the second term in Equation can be bounded as:
(Ww2. wx.) Σx wa =1( Wsc) T A"" Wsel
(0(7) 63) 94)(7s &&)
o
(13)
(9/ + 3.)2
The third term in Equation 9 can be bounded as:
(ww2s w.)) UU" B[ray] S1l(w -w.) T U | T E(r.yll
=0( vo(T3 + 6.) + %
)o( Vo 0.)
by Equation 11 and Lemma B.1
is +
=o(1).
(14)
Now, it remains to bound the first term in Equation 9, We start with approximating E( U Erra]:
Eer U
UUUTT 1 Pr,
1 - -Ū-U,U UUUrT
R.
Vi
R UNUUT (
( R (fx +fx+ (B++B1))
인 UUU
UUU
E(ray] + o(1)
Va
Vn
by Lemma A.5 and Boundedness
-1 +8)1
-U ÉUTT R R.UUT Vn. At
(
" UU Elr.y] + o1)
Vn
by Concentration on and Boundedness
-U,U.'EUU SUTUT 1 ('
R
R U.U. R + (3% + +881)
2 U,U 1 R. + o(1) by Concentration on
Vn n
Vh
nt
(15)
Due to the two additional assumptions the statement of the theorem, along with Concentration on V and Boundedness,
20

---

Representations Shape Weak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
the RHSs of both equation 10 and equation 15 are O(1). Combining equation 10 and equation 15, obtain: (wws Wse) W - -U, B(rsy))
RB ww v
T) ( -
1
K (9s + A11
x
R
U UUT (U.U,T EUU + +6s (68++80) ((++)))² - -U,
R ú + o(1)
Vi
Vn vn
1 1
(=
Ruw ww
105+(69 (is
Vi
Vn
K2
1 K3 +is+
-1
- K' -
y o()
by Lemma A.2
Vn.
1 8)
1
Ruy Ww
- -P.P.-Ps)
Vn 4 + o(1)
(16)
P.P. Ps's eigenvalues are given by:
A. K
7 8
K()++)++++)))
AR +)++++) A. +++++++))
Xe
(+++8)
since 1 K and E, share non-zero eigenvalues, we analyze the relation between 8 + % and E non-zero eigenvalues. By à
Concentration on and Weyl's Theorem
Amin, * o o(E)) Amin, # o(E)) = o(?? + + Amin. # o(E))
Combining this with B, + + o(Amin, # o( o(E))), ; we conclude:
8g + 1/- = ol Amin, o (É))-
(17)
Using Equation 17, we then obtain |P.P, - Pollop = o(1). By Lemma A.12, the term ( R ww vi v) can be bounded by
Pw) A + o(1) 0(1). and |1 1 O(1) by Boundedness. Combining these results, the RHS of Equation 16 is (1). Therefore, ((ww2 - w (Eg - B(rsy])] = o(1), which completes the proof.
B.3. Proof of results in Section
B.3.1. PROOF OF THEOREM 4.1
First, we present the following lemma, which provides a sufficient condition under which any labeling can be fitted by the W2S model.
Lemma B.4 (Condition for overfitting arbitrary labels). As long as 6, olf)) and 6, 9, o(Ts), given any fw bw s.1. ts Ei Sn (hr (2:))) O(1), the weak-to-strong model can almost exactly overfit it, as indicated by an almost zero training error: E (h, - (d, = o(1), with high probability 1 o(1).
Proof. Let T E R denote the weak model's predictions on D. The following holds for all T such that 1 ñ training loss can be expressed as
1//2 = O()). The
1
R Ww2s 1112- =1
R
R.
R
R +
r
2
by Equation
ft
ñ
Vn
Vn
1
1
RT
RA
R R +1,11)
T112
Vn
Vñ.
Vn.
1
1
R
R.
-
R.
+2++111 - 1 3
ft
B
1
1 211
Amin 1/4 Re Ra)) + 3,
Vn
2
3.
because we assume
2 0(1).
(18)
Amin R.T Rs)
+ 8
21

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
By Lemma A.4 and Weyl's Theorem, have
1 Rg Ra) Amin
Amin ( RS U R + +x)) < n Rg R. - (. R. 9xl)
= = + 65)
"
1 UUU R o(7? + +o²) 2 9/ - -07 + +,).
Amin
Rg R Amiu
(19)
f
ht
Substituding Equation 19 into Equation 18 yields
((
B o(> + 62) 3s ) =o(1)
RR Wwz. - T112 -7
o(1)
because we assume 3, = =((s) and ö, = o(1),
which completes the proof.
The first statement in Theorem 4.1 can now be readily proved by invoking the above lemma.
For the second statement in Theorem 4.1, we first apply the triangle inequality, which gives VErrws S /PredGap + v Vrr Given the assumption Errge = o(1) and the fact that Theorem 3.8 implies PredGap = =O(), we obtain Errwa S PredGap + (1). Furthermore, by our assumption combined with Theorem 3.8, we know PredGap = Ertw - A + o(1). Substituting this into the previous inequality yields Errw2s I ErT, A o(1.
B.3.2. PROOF OF COROLLARY 4.3
We begin by presenting the following general result regarding the test emror of the weak model and the strong cciling model.
Lemma B.5 (The weak model's error the population). If |By"] Σ ofl = o(1) w.h.p., then the weak model's error the population, Errw, can be upproximated as follows,
Errw = 11 - Pw)
f O()).
A similar conclusion holds for the strong ceiling's 's error Errse well: Errs: |1(I P,) Js 2
+ o(1).
Proof. We decompose the error as follows
Errw =B((r Ww y)
w En ww 220 I E/rwy] E[y²].
(20)
The first term can further be decomposed as:
1
W. En w. =
A
UT
R
R.
R, + B.I)
Vi ÿ/1
Vn
1
1
= A TT
R. + (Bw + =w)))
5 + o(1)
by Lemma A.4 and Boundedness
Vni
N
1
Ray 1
1
=|1
[R. Rw + (Bw + +
5 - o(1)
A
Vn
-IIVAU
R.
R R. + (Bw + +/w) 1)
أ||2
1
+ IIVAUU
R.
R UUU R... (B,, + kx))
أ = o(1)
(21)
Vi
Vn
22

---

Representations Shape Weak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
We bound the second term in Equation 21:
1
1
Row Ru U. R + (Bw +ww)
//1
Vn
n1
Vi.
< Al "
Rw
1 ||| (
Ru U. R + (Bw 7w ( Tpppl
Vn
Vni
o77 + (w)( + + + 7w)
by Diminishing population covariance on VL, Lemma A.1 and Boundedness
(Bw )
=0(1).
(22)
Then, we approximate the first term Equation 21:
A N Vn R
Ral
RR RN + (Bw + +)w)()=
1
vn
-IIVAL - Re (Bw + 9ro) U
1
1
Ru
أ
by Lemma A.2
Vn
-IIVA,, + (5w) U
ÿ1 + o(1)
by Lemma A.6 and Boundedness
=lly A U RT U/ (8w + 1) U 1
1
R,
- o(1)
by Lemma A.6 and Boundedness
Vn
Vn
=ll A
Roy R U (Bw + 7w ))- -1U -UT
1
Ray
1
111
1 o(1)
by Concentration on V and Boundedness
Vn
Vn
-II 1
Ra h
R P. (8w + 5w)1
1
01 + (()
by Lemma A.2
Vii"
=|1 A
RUU Rm(
R. R., + (3w + I)
1
1 + (
by Corollary A.10 and Boundedness
PM
0 + o1).
(23)
Vn
23

---

Representations Shape Weak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Now, we approximate the second term in Equation 20:
u
Erwwl
1
-( R
RS R
Rw + Bx.1)=
y)
Eiryyl
Vn
[) """""" Eirwel
-(
R
R R + +.111-1
U U U Eirwel
R.,
R. Rw + 1
1)
Vn.
n
Vn
Vn
n
vn"
R Rw 1,.1) 1
/0(0 + 6%)
Tw
Rw,
U U E(rwy] o(
V 7/m + Ow
iw Bw
by Boundedness, Lemmas A.5, A.11 and B.1
-(
Rw
R R + B111
1) U Birwyl o(1)
Vni
Vū
=(
R
R U R. + (Sw + nw
Vn ú) U U Elrwyl "TTwyll + (11)
Vi
n
by Lemma A.5, Boundedness, and Lemma B.2
1
T
1
RR
U
R + (3w + +()))) 1 U.U E(rwe] (1)
Vii
1
=
y
U R,R, + (3w + 9w 1) U E = (11)
by Lemma A.2
Vii
Vn
_ 1
=
R
U R. U + (3w + -)) E(rwo] = (11
Vi
Vi
by Lemma A.6, Lemma B.2, and Boundedness
1 y
U R. R, (Sw + (w) - 1 A
R
R + o(1)
by Concentration on V, Boundedness and Lemma B.2
1 1 ú + o() by Lemma A.2.
(24)
Combining Equations 20, 21, 22, 23, 24, and the assumption about Bly²] yields
Err = (1 - Pw) ا (1).
(1).
The proof of the result concerning Errye is similar.
We show that the condition regarding Ely?) is satisfied in Example 4.2. Specifically, Σi= O follows a (f) distribution, with a mean of AE(y²) and variance of 2n. For simplicity, we demonstrate the following result using Chebyshev's inequality, while noting that tighter bounds could be achieved with tail bounds for x" variables or Lemma C.3. For any
k > O, we have: Pr (1> i oe A 2 kv2n)
I + Letting k = f/4/4, we find that with probability 1 1 0 7
E O ( Thus, Lemma B.5 applics to Example 4.2.
Now, based on Lemmas A. B.5, and Theorem 3.8, the key to computing the errors of all these models boils down to simply computing Pw and P.
We first compute the kernels. For convenience, we use the shorthand notations Ky and K, to represent K (II, hww and K (IIy, hs), respectively. Since the representations Example 4.2 are decomposable with respect to the subspace corresponding the first coordinate, for both the weak and strong models, the principal kernels are rank one and can be expressed as K,, qq" and K, カカ" where q VT -T- ./.. Then, for ,/- Kw, it has single nonzero V% q. Similarly, K, has a single eigenvalue || 11 01" with
eigenvalue || + 2 with the corresponding eigenvector
the corresponding eigenvector
VR 9.
Next, we present the following Lemma.
24

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Lemma B.6. We have the following:
1
5 vn 1 Ul - o
1
91 = 1
$12 = - Vn
all = = Vn
Proof. The first two statements can be proved by leveraging classical results on the concentration of Gaussian matrices (see Lemma C.3 for details). The third statement follows as a special case of Lemma C.4. The last statement is implied by the previous three.
Recall that both the weak and strong models' representations in Example 4.2 are special cases of Example 3.5. Given that x² = o(f) and = (ft). we have fwo Two is. and %/S all being o11), 8w = 5, = 0, and B = o(1), B, = o(1). Combining
these with Lemma B.6, we derive:
Pw -
49 1l lop o11),
-: lop = o(1)
Now, leveraging Lemma B.6, we can derive all the emror using the expressions provided in Lemmas A.12, B.5, and Theorem 3.8.
B.4. Proof of Corollary 5.2
Following Theorem 3.8, we bound the RHS as follows
PredGap - -P.(T P.) 1 ÿ + P.(I Pw)(I - P.) 1
U1 + o(1)
Vn
P.(I Ps)
bll + PI. - (I - P.)
vā 01 v1
+ o(1)
(25)
Vi
1
< - PR) VC ([I-PP)
3 + +
Vn
(IR.(I -C(I-P.)) (IP.(I-Po))RA) VC Err + o(1)
2
+ o(( by Lemma B.5
(I -IP.(T-w))PPllae VC + VErree)
+ 01)
C. Proof of Examples in Section 3.3
C.1. Example 3.4
For convenience, let q = intdim(2) and oopp
Firstly, we note that the conditions in the example imply low intrinsic dimension. Here's why: since Tr(E) EE/ B, it follows that
Tr(E) op
B <
intdim Σ)
O(B).
(26)
T
where the last step holds because |olppp @(1). Given that n l-c = w B (og(g))) we then have n 1 = w(g as mentioned the remark.
Additionally, since intdim! (E)= 2 1, Equation 26 also implies
B2T and
B S(1),
(27)
which we will use later.
Next, we introduce the following two lemmas, both of which rely on the matrix Bernstein inequality with intrinsic dimension, as stated Theorem 7.3.1 of (Tropp al., 2015).
25

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
-0.5A Lemma C.1. With probability of least eqexp( Br+(B+r1/3
o(1), the following holds
|| - S 50-50
The same conclusion applies to E as well.
We prove the result for E; the result for can be proved in the same way. Define S; = (f. Σ). The random matrices S; are independent, identically distributed, and centered. Their norms are bounded as follows
Proof.
1 (1lAat) llp [1 B L.
SS A
Then,
1 B(F.r. n22
1
1 E(Bri.
B πta
ES
- Σ)² =
B|
P -2 Σ²) + fi2
- -E²)
Σ
n2
Define Z = Ei S.. We have
B Σ = V
0 0 122
ES! $
1=1
V's norm can be expressed as follows:
E op
BT
V lpp
Define d intdim \ ), which can be simplified as:
d = 2 Tr( 2intdim
B Σ) 2intdim Σ) = 2g. 2
2 Elop
Now we are ready to apply Theorem 7.3.1 of (Tropp et al., 2015). It leads to the conclusion that, for any 1 12 Ve+ + L/3.
-12/2
[ Z lp 2 1) <xdexp
+ Lt/3
-t2/2
T37 =8gexp
+ -n 1//11/3
7
-iil2 /2
expe
(28)
BT (B+ + T)t/3
By assumption:
n wB log
= n I-c w(((T + 1/3) + T/3) log 4)
because 7 0(1)
w(log q)
(T 1/3) B 7/3
ii-e
= 0.5
w(logg)
(T 1/3)B T/3
1-5
exp
w(4))
(r
+ 1/3)B+T/3
-0.5nt-c
gexp
o(1)
(29)
(T 1/3)B+
26

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Therefore, set the value of to á 0.5c o(1) in Equation 28. is easy verify that n-0.5c Vo L/3. Substituting.
get:
-12/2
-it2/2 Br (B (B+))t/3"
-0.5% I-
P(I lop 2 0-563
cxp
Lt/3 ) - exexp(
)=
Sqexp(
Br + (B ()+ 0.5c /
-0.5h 1-c
<8q exp(
because át -0.5e <1
Br (B 73/
=0(1)
by Equation 29.
-0.564 Since Z = E, restating the above, we have that with a probability of at least 8g expp (B+( ) the following holds
a Σ lopop S ft -0.5c
Lemma C.2. With probability of at least
(g + 4) exp(
-0.5A
= 1 o(1), the following holds
4BC+
1
fiye E[ryll < 50.5c
The same conclusion applies to 1/2 Em= Tiyl as well.
Proof. We prove the result for à Ei Piy the result for 1 Σ. P.ys can be proved in the same way. Define S 1 [f:yy E[ry]) The random matrices (vectors) S, are independent, identically distributed, and centered. Their norms bounded as follows
à
2 nt S VVBC = L.
| S /े (1 + |B|rill)> < (1 + B[j|l|bbl)
(30)
Define Z D S,.. We analyze the semidefinite upper bounds for the variances T and EZ Z:
T
ESSST
(Eyfr.A
E[ryy][rry] T)
22
Eyj A.A
62
E = Ví.
11
EZ Z
E.S. S
1=1
-1 112
BC = V2
by Equation 30.
Define = V/ (2llop). It can be simplified as follows
C Eloop BC)
max(1l
n
4 BC
because B 2 Σ op as in Equation 27.
27

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
V1
Define d = intdim
). which can be simplified as
222
I
d inndiim
0
BC
Tr Σ
+ 4
BC
81
max
C Σ ops 16 BC)
Tr [ 1 BC
BC
( Σ)
+1
4 BC
Tr(E)
5q/4 +1
because B 2 as in Equation 27 and
Applying Theorem 7.3.1 of (Tropp et al., 2015), we have that for any 2 vo+ + L/3.
-12/2
Z| 2 + <4dexp
+ Lt/3
-t2/2
<(g + 4) expl
(31)
BC + 2v 2V t/3
By assumption:
n 1-c = ( log q)
n --c = w(B Blog(g 4))
2v BC
= n = w((BCC+ +
log(a + 4))
because C' = =0(1, and B = 22(1) as in Equation 27
3
1-5
wlog(4 + 4))
(4BC 2VBC)
-0.5n l-c
(q + 4) exp
o())
4BC
BC
Therefore, we set the value of to -0.3c = o(1) in Equation 31. is easy to verify that -.5e 2 Vo + L/3. Substituting, we get:
0.5c
0.5m c
2| 2 á -05cc
<(g + exp(
BC + 2VBC ń 0.5c /3
-0.5n
<(q + cxp(
because á 0.5e <
BC + 2/BC /3
-0.5m 1-c
=(q + exp(
4BC + 22 BC/3
=o(
Now, we are ready to show that Example 3.4 satisfies Definition 3.3. We let V be the entire representation space. Then, V the zero space 0. In this case, the conditions Kernel-wise 5-isotropy on V Smaal cross-sample inner-product on V and Diminishing population covariance on y trivially hold. Thus, we only need to prove that Boundedness and Concentration on hold.
28

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
We let 6 n n0.e and 7 0. First, note that 82 n n00.2c
0.2c 2 -0 Then, by Lemma C.I, obtain that || E ol 0-5. o(ñ 0.2c) ((52) o(>2 + 82 p) with probability 1 o()). Similarly, can show that | oloo = ol>; + 8 + p) with probability 1- 11)(1).
0.1c Next, since ò u 001c applying Lemma C.2 gives us
Σ Pay. E(ry] Sn- -ff1-0.1)) d()
F:3/ o(y + + p) with probability (1). Similarly, the same conclusion can be shown for 1 n
Note that there are only four events above, so the probability that all of them occur remains 1 o(1) To now, have proved Concentration on V.
Finally, regarding Boundedness, |ا| oo = (1) is directly given in the assumption. Keeping in mind that V is the entire space, the conditions regarding covariance matrices are readily satisfied through the triangle inequality. For example: |oo S || Sloop + | = o(1) + (1) = O(1). The other two conditions are directly implied by the boundedness of each y.
C.2. Example 3.5
Originating from PCA (Johnstone, 2001), the spiked covariance model has been widely adopted in recent works to theoretically characterize key aspects across various topies (Ji et al., 2023; Nakada al., 2023: Muthukumar et al., 2021; Pezeshki et al., 2022; Wu & Sahai, 2024). Furthermore, Example 3.5 also subsumes the sparse coding model as a special case, which has its roots in computer vision (Olshausen & Field, 1997; Foldiak, 2003; Olshausen & Field, 2004; Yang et al., 2009; Mairal et al., 2014; Papyan et al., 2017), has been used to model language data (Arora et al., 2018 8), and has been extensively employed in recent theoretical studies (Kalimeris et al., 2019; Allen-Zhr & Li, 2020; Wen & Li, 2021; Zou et al., 2021; Shen et al., 2022; Xue et al., 2023).
In the following proof, we start with simple case where the data are Gaussian. We then extend the result to sub-Gaussian data by replacing the technical lemmas for Gaussian data with appropriate alternatives.
C.2.1. OVER-
GAUSSIAN DATA
Suppose that we have R E gdxa R ıdxx with â (n)) and d w(f2) drawn from a high-dimensional E-Gaussian ensemble with zero mean, where
1 Σ
A'
0 0
with o² O(h). wkb)).
(32)
d Id-ki
0
0
A" Ia k
Σ
Σ"
Here the data splits have comparable sizes, and the model heavily over-parameterized. By splitting the matrix R where F k corresponds the k principal features (which form the space V) and A E g(d k)xū
F
A
corresponds to the rest (which form the space VL1,, we can write the sample covariance matrix as
1 RR nt
1 FFT ñ
FÄ AÄT
S
AFT
We note that d - k= = and the corresponding labels have bounded mean and variance. The same decomposition applies to R. Note that here U'= Is and U"
(k)(d=k)) allow to define the projection matrices U Ia-k
x
and U" V and 1 respectively.
In this section, we show that our assumptions hold in the above setting with 5 = 0 and f = 02/0, = 02/h. We only prove for R whenever the same proof can be easily applied to R.
First, let us introduce the following Lemmas:
Lemma C.3 (Restatement of Example 6.2 in (Wainwright, 2019)). Let X Bddx be random matrix with i.i.d. entries n8? /2 for some
drawn from N 1) (that a E-Gaussian ensemble with Σ Ia). Then with probability at least no
29

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
δ > the following inequality holds:
d 2 + ((
(v=+))
d V +
XX 12
Tallop 2
Lemma C.4. Consider two independently sampled Gaussian matrices where A RRdxx xn has columns a; N( TT and B p has columns b; (O, dā Then for some djd2 ô 0 and constant C, with probability at least 1 dj d26, we have
1 AAB lop
0102 (dddznlogg log( 2
n
Proof. Let Q = AB". Then each entry of Q is an inner product Q: a, bj, where a; E RR is the i-th row of A and by E RR" is the j-th row B. Since each entry of a; is (s, of) and each entry of by is (0, a3). by Lemma from (Shen et al., 2022), with probability at least
å (taking 2,d2
0), for some constant Cej.
2 = (a: b,)² & (20/
We define max [Ctj 1 < < i d1, Sj $ d2}. Now we bound the operator norm with
1 AB
llpp < AB | 11 QQ
Σ
O
Isish
Σ
Cooff33noog(//0)
1sidh
K
Cdid2ofo3nloog22000
0102
Cdiddnn (2/0)
with probability at least d2d since the inequality has to hold for each entry,
We now prove that the example satisfies the five aspects of the definition:
1. Boundedness:
First, have Σ Eloop = O(1) from its definition, and
FF Ik
0
FÄ
|S - Eloop <
O
LÄÄT
+
AFT
0
(33)
T-
up
By Lemma C.3, we take 8, 1/4 and have that with probability at least - 2e -667/2 1 - 2e Vi/22 = o(1).
(V++
2
2
- FFT Ikllop 2
o(1) since ft > k.
21/4
12/4
As Å E (d) sampled from implies that
d-k Ia-ks va Å is sampled from Ia-k- With this scaling, similarly, Lemma C.3
T
dd - A A)
(vd-k) k A Ia-k
d k
ÄÄT
Ia-k
1p
og
2
d
2
k
2
1144
ft
1/4
30

---

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
AAT-
0² d- k
I
o² d
2
2
+
k
= 0(1) as o = O(i).
k
ft
1//
ft
h1/4
We have bounded the first term on the right side of Eq. 33 and have that that
F
Tlop
and || 1 vB
Allop are O(1). It follows
v/
1
0
FÁ
lop = 0 FA
- | Σ Fllop = O(1).
AFT
0
=
up
Hence,
| op = (1) directly follows from 11 = O(1)
Er U3. where y; represents the i-th entry of the vector. Since the label has bounded population variance (11), the i.i.d assumption implies
Now we consider A ||
1.
Var
Var
1 22
Σ
0(1)
-
22
1=
1=0
2=0
Then by Chebyshev's inequality, for any e > 0 and some constant C1, we let
1/ 1 || for simplicity and then have
Var(z)
C1
P( - B > <) v
6²
NEE
We take = 1-1/1 Then with probability at least
C₁
= 1 - o(1),
Vn
Var(y,)] = = f2
|101l O())
since the variance the label bounded.
2. Concentration on
Is ((ddk)xk)]
With U"
preserving only the first components, have from above that with probability
least - o(1),
- A'llop = 11 1 FFT Inllop = o(1).
Now we consider
U Rý - E rgill 1 Fý FF R[fylll
where f = U'. We define a new random variable 2 = fy and its sample mean 2 1 Fu E R2 We first show that the variance of each entry of Ż is of magnitude 1
Var 2.) = Var Σ
1 Var Σ PijOs
Vi = 1, ... k.
A2
For each term in the summation,
Var = -E 0(1)
since F.j and új are both bounded. By the i.i.d assumption,
Var 2,) n² Σ 20
By Chebyshev's inequality, for any >> 0 and some constant Cz,
P P(12: E|2il| > 6) <
Var (2x)
C2
22
Rue
31

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
(12. - (12: B|2,]] 11
kC2
ñie
Similarly, by choosing e ft" 1/4 the probability of large deviation decays rapidly as:
(12: - -B|z]]
1
Vi 11-1... 1k) S
kC₂
=(1)
since á w(k)).
1/4
Ví.
This statement implies that with probability at least - o(1),
k
| B[2]|| = 1| ft
UT E(-TU _ vn
= o(1) = o(y + 6+ Amin Amin(())) (A'))
as we sum up the k terms. This shows that our setting satisfies the second part of the definition.
3. Kernel-wise 6-isotropy on D4:
We define Z
vd-k A R(d-k)xá which has standard normal entries.
With the scaling, we plug in U",
9 o² f and have
R U R ol = A
o² 0²
Top d
1
2 2
I Tloop
(34)
à
Now we apply Lemma C.3 and have that with probability at least
-152/22 2--11
for some 82 > 0,
2
nt (
27
ñ d
Z Z llop S
10
2
V d
k
+ 82
+
d
+ 02
The rest follows similarly by taking & = 1/4
4. Small cross-sample inner-product on n4:
0 (d=x) and Lemma C.4 with AT Rdx(d-R) and T Rax(d-)), each having N(0, a
By U"
La-k
entries, the target expression becomes
R
U"T""
Rllop=
1
A A
Allop
(35)
Vn
Vn
Viñ
1
04
C4ñn
(d 1:) log (2/6g)
v Vin
(d - 1:)2
g²
C4
log (2/8g)
- k
2 V C₂ log (2/8a)
V d
for some constant C4 and with probability at least 1 - nndy for some 0 d3 गं We choose some 63 = of nा in this range and then have that with probability at least o(1). ). the previous bound can be expressed as:
/ C2 C log (2/0z) V d e 02
C1 log(An)
o²
o( k max fu nj
o(7 + 8)
since d k (n)) = w()). 5. Diminishing population covariance on VL
By definition, is trivial see that
a²
0²
Aman Aman(A") (A")
o
o(T + 5)
d
max (ú, n)
since d - k k= wi)) wn).
32

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
C.2.2. FURTHER RELAXATION TO SUB-GAUSSIAN DATA
Now, we consider the more general sub-Gaussian setting outlined in Example 3.5. The population covariance is:
0 Σ = 0 Td-k
Ix
where the top left block has corresponding sub-Gaussian parameter (1) and the has parameter of O We adopt the following definitions from Chapter of (Vershynin, 2018) for reference.
Definition C.5. A zero-mean random variable X is sub- Gaussian if there is a positive parameter K₂ such that
Ele xlex2/K3 k 2
Definition C.6. A zero-mean random variable X is sub-exponential there is a positive parameter Ke such that
E x /Kg] < 2.
We can also define the following norms that give the sub-Gaussian sub-exponential parameter:
|| = inf t 0 E /18 S 2 Kg
|X llv = inf > 0 We X//
S 2} = Ke
Remark. There are many different characterizations for these two definitions, each with a different uub-Gaussian/sub- exponential parameter. A detailed summary can be found in Chapter of (Vershynin, 2018). Notably, these parameters differ from each other only by at most a constant factor.
Lemma C.7. (Extension of Lemma 4 (Shen et al., 2022) to sub- cab-Gaassinn) Consider high-dimensional independent sub- Gaussian vectors 21, 22 E Rd, whose i...d entries have variances of, of and sub-Gaussian parameters ( ((22) respectively. Then for 8 > 0 such that vlog(2/8) Ved for some constant c, there exists a constant C such that with probability at least 6,
121 Ø2 S Co,o2v (2/8)
Proof. We consider the product z1 22 result that the product of two sub-Gaussian random variables is sub-exponential. More precisely,
> 21:22
Dí= where we define for simplicity. It is well-known
||ally, < || ( llv
Co102.
By Bernstein's 's inequality for sub sub-ex exponential functions (see Theorem 3.8.1 (Vershynin, 2018)), this summation can be bounded as: for some constant 0.
12
P
0j
2
< 2 exp
-emin
Σd
|a:113
max,
12
$ exp
-emin
dC
Conoe
Lett C 2 (200(2/8) for some 8 that satisfies the condition Vlog(2/0) Ved (e.g. 8 1/d3). The probability /e statement becomes:
C 00 dlo((2/0)
log(2/8)
d log
P
< 2 exp
-emin
Vc
exp
min (log(2/)), Ved log(22)))
33

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Since choice of ó ensures that the first quantity smaller,
C Vc
P
Σ a
o
(log((2))
In other words, letting C" C/Ve, we have that with probability at least - 6,
22 22
dlog02//
Now we ready to show that our assumptions capture the setting in Section C.2.1 but with sub-Gaussian data. That is, we now allow the data to have possibly even lighter tail than that of Gaussian. The proof can be easily replicated, as Chebyshev's inequality still applies here and Lemmas C.3, C.4 find the following "sub-Gaussian" alternatives, namely Lemmas C.8, C.9:
Lemma C.8. (Restatement of Theorem 6.5 in (Wainwright, 2019)) Let X E Rdxx be random lub-Gaussian matrix with purameter Kg and population covariance Ia. Then for all 8 2 0, there are universal constants C1, C2 C3 such that
d + +
1 xx" Iallop S K2
Ci
n
with probability at least 1 Cze Can min(6,6"),
Lemma C.9. Consider two independently sampled row-wise sub-Gaussian matrices E IRdixn, E Rd that have i.i.d. entries with variances oî, o2 respectively. Then for some djda 6 > 0 and constant C, with probability at least 1 d, d2ō, we have
1 AB llp
0102
2 Cdidan log 2
n
n
Proof. The proof is the same as Lemma C.4 except that we now use Lemma C.7 to bound the squared value of each entry in the Frobenius norm.
With these alternative extended results, the proof in Section C.2.1 immediately generalizes to GabuGassiian data. This extension potentially allows us to accommodate more realistic scenario and enhances the theoretical robustness of our assumptions. Sub-Gaussian distributions capture a wider class of data behaviors; for instance, the fact that bounded random variables sub-Gaussian makes the theory more applicable many real-world datasets, which naturally exhibit sub-Gaussie Gaussian characteristics. In the following section, show general result that even more examples can be constructed.
C.3. Proof of Theorem 3.6
The intuition behind this theorem that adding nigh-dimensional aab-Gaussian entries the given representation preserves decomposbility while slightly modifying the parameters. Due to the orthogonality of M and M we let U [M M
and then axx) U [(x))]
naturally, the column space of M can be regarded as the subspace V, and the column [(x)]
space of M is V Given that h(=)'s representations are (8. O, decoooossslll w.r.t. Rd. we now prove that the new representations are (6, a² x²
; deecomposable. Again we only present the proof for one data split whenever it can be replicated for the other.
For notation, we let 7 0²/ max(ū,
1. Boundedness:
1 $ " 0 =O(1) follows from the previous proof using Chebyshev's inequality. For the population
34

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
covariance,
(x))((a)) T M())(())) {(x){(<)
lop = ([Bba(a(z)a(z)) Illop = ED=
[(x))(())
op
h()))((x))) T
h(x){(()) T
ED=
(())
-
+
ED [(()))(a))
op
op
(36)
We have that ( [|(()) 0(1) by the (ò, 0, decompiiccom assumption on h's represen- tations. From the proof for sub-Gaussian data in Section C.2.1, || (((m))(()))] || = 0(1). These bound the first term on the RHS of Equation 36.
By the definition of operator norm,
Ep, (bx))())))))))))()))))))))))))) olop sup sup u² (b,,)((()(())))) sup sup ED (u Eo.l(u ((=)) || - ||||=
(37)
|
By Cauchy-Schward inequality, we can bound this expectation as:
ED. (o.((""(a)))"" 2 ED, (b
where
ED, (ul((()))))) ED, h(z)h(2) h(>))((()())) u E'Ep, (h(x)((2)) S | [h) 0(1)
ED_1( [(v [p=](((( = E0__0' (() u| (<=)) | [g) O().
Combing these results, we have that Equation 37 Equation 36. Hence, [pp))lop (1)) Simiarly, can prove for the empirical covariance:
| ([F_) ( bounding the second term in
1
Σ == ((.))
((E))(())) E(E:)E(0.)
(aa)llp = A
a()))a(;:)1llo
=
Σ ((2.))(2.)
1 HH
H
ËhT
T
op
where the i-th column of E is ((t;) and the i-th column of H is h(w.).
The rest is straightforward: the assumption on h and the existing proof for sub-Gaussian data imply I 1lop (1) and || 1 S llop = O(1). Hence, || + H llop and
Vi Ellop are O()), and we have || 1 - llop is also O(1).
These together bound the empirical covariance.
2. Concentration on V: Since V corresponds to the representation space of h(::), this condition is automatically satisfied by the (6, 0, o)ddccomposiiility assumption on h.
3. Kernel-wise d-isotropy on y4: In this setting, since VL corresponds to the column space of M (the high-dimensional sub-Gaussian part), we have
1 K (II v
0² =
1
σ² I Iloop
-lopp (id))))saaa T with E defined above. Then the equation is essentially in the same form of Equation 34, so the previous proof applies here.
- A
11 ñ (()
ñ
By definition of the kernel matrix, k({) (((t,)
4. Small cross-sample inner product on V Similar to 3, we have
[II a( Hy (z (1 = _
[E(2,)
E
lop
Vin
Vin
where E is defined in the same manner. Then the proof after Equation 35 for sub-Gaussian data applies.
5. Diminishing population covariance on pd: This refers covariance matrix of the sub-Gaussian part, and we simply
have:
[l op op |E (()))(()) "Ilopp = a²
o66 + 7) as m w()) (n)
35

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
D. Additional Experimental Details
D.I. Training details
D.I.1. MOLECULAR PREDICTION.
Our experiment is built on the GitHub codebase provided by (Fabian et al., 2020). The strong model, MolBERT, can be downloaded using the link provided on their GitHub repository. For the weak models, we train small transformers using their SGD train linear model on representations with the following 0.001, weight decay 0.1, and epochs 2000 when using representations
pipeline with batch size of 256. For finetuning, settings: batch size 1024, learning from the strong model; and batch size 1024, learning rate 0.01, weight decay O, and epochs 2000 when using representations from the weak models.
D.1.2. NLP TASKS WITH EMBEDDING MODELS.
We use nvidia/WW-Emeddd ranked first on the leaderboard of the Massive Text Embedding Benchmark (MTEB) (Muennighoff al., 2022), as the strong model. We consider the following 22 embedding models as the weak model:
jxm/cde-small-vl
cherlperr/tee-baee
interaat/steelll-bbseeeezz
BAAI/oge-base-en-11..5
chenlper/gte-small
int.floarre5-bbss-v22
300
TayloraA//gte-tinny> sentenncetrranssfforrrrs/gttttte bumyy/sft-bbe-snell
During fine-tuning, we train a linear classifier on representations using the Adam optimizer (Kingma, 2014) with the following settings: batch size = 200, learning rate = 0.01, weight decay 0.00001, and epochs 200.
1.1.3. NLP TASKS WITH END-TO-END FINETUEND LLMs.
We largely reuse the GitHub codebase provided by (Burns al., 2023). We Qwen/Qwen- as the strong model. We consider the following 28 LLMs the weak model:
biggcience/bboom-5560 biggcieence/lloon=-500
bigsciience//ntO-basee
baidh//ernie-code-5500
biosciencce/mtO-mmaal1
google/umt5-small
google/umt.5-asee
36

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
Table 2: Average Spearman correlation with hyperparameters selected on half of the models and evaluated on the rest.
Justice
Commonsense
0.885 10016
0.67
google/nt5-base
MB2UA//LaMinn-155666
WB2UAA1/aaKini--1an--5--77
MB2UAL/LaHinnj-GPP-1122
MB2UAT/1aViiniiNNe--1555
MBLUAI/LaMinn1-T-2222
applee/0penE1MM-2700
agplee/OpenE1M145500
cleuthhezzA//ptthia--60mm
MB20A1/aaMinni--1aa-1---222
NB2UAA/1aMMin1-277-7778
@erabras//errebra--6771111
google-:5/t5-smal2
facebook/opt-125m
Qwen/Omen2.5-0.52
distilbeert/diitillptt?
GleutheezA//gpt-neo-1122
gpt2
google/nt5-small
EleutherAt//yyhha--000
We finetune all the models using the pipeline provided the codebase, which employs the Adam optimizer with a batch size of 32 and trains for single epoch. The learning rate is set to 5e-5 for weak models and 1e-5 for the strong model, following the default configuration in the codebase, which applies smaller learning rates for larger models.
D.2. Details and discussions hyperparameters
In Exp. I, Ow 0, 0.1 and Bw B. 0.1 for all datasets. In Exp. II, Ow 0.001, a, 0.05, Aw 0.0001, and Å 0.01 for both datasets. In Exp. III, we tune the hyperparameters for each dataset, reporting the best result. Specifically, we set the range (0.2, 0.5, 0.8, 1.0, 2, 4, 8}.
and vary them within the range (0.02, 0.05}, and vary Bw and B, independently within
Effect of hyprparammetrss We vary the hyperparameters to evaluate their impact on performance. In the setting of Exp. II. we vary aw and a, within the range 0.001, 0.01, 0.05 and 8w and 8& within the range 0.0001, 0.001, 0.01. The results visualized in Figure 6. In the setting of Exp. III, vary the hyperparameters while keeping Orw Og described in the previous paragraph, with results visualized in Figure 7. Although certain hyperparameter configurations may lead to lower correlation, non-trivial positive correlation is observed in most cases. Interestingly, in Exp. III, which is seemingly the most 'challenging setting', the results are highly robust to changes in hyperparameters, with the worst-case correlation remaining around 0.6 across all three datasets.
Cross-model hyperparameter transfer. We note that, although each model could technically require different hyper- parameters, in experiments we let all weak models share hyperparameters for simplicity and still achieve strong results, suggesting that our approach is not very sensitive hyperparameters. Further, we present new experiment demonstrating that hyperparameters selected using one group of models (i.e., validation set) generalize to other models. We randomly split the weak models into two groups, select hyperparameters based one group, and evaluate them on the other. We repeat this 20 times and report the results in Table 2. Correlation remains high with low standard deviation, indicating that hyperparameters selected using a few models can reliably generalize to new ones. Additionally, we note that a small number of labeled data should suffice for hyperparameters tuning, as they are only used to measure test performance and not to compute our metric.
37

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
0,-0.001
R,=0.001
0.9
0.7
5091
0.5
dg 01
a,=0 01
041
5.81
0001
680
0881
0.3
6481
200
0001
0.1
a,-0.05
e,=0.05
0.3
8381
A091
680
200t
64m
0.0
A/81
8801
1080
0001
19
680
A
8
Pu
B
0,=0,001
=000.01
=00.05
0,=0.001
0,00000
0.00.55
(a) Justice
Commonsense
Figure 6: Effect of hyperparameters in Exp. II. Colors indicate Spearman correlation.
3.00
0.55
3.76
6035
10
15-
10
30-
63
0.3
30
55
0.5
Px
G, a,-
Ow a
Gr 6 0.02
G, 02
da 02
9, 0
(a) SciQ
(b) Amazon Polarity
(c) Cosmos- QA
Figure 7: Effect of hyperparameters Exp. III. Colors indicate Spearman correlation.
OC=00.94
SC=0,936
SC=0.846
1.5
size 256
14.
size 128
size 128
256
Errw
size 64
Errw2s w25
Err
64
1.3
10
2
1.1
0.96
0.96
0.98
|Ps(1 Psllop
|Ps P.
|Ps(1 s1
Pw) Pw)
(a) Lipop
(b) FreeSolv
(c) ESOL
Figure 8: Results for P.(I Pw,) P.llop in Exp. I.
D.3. Results for [ Psl
Results for Ps are presented in Figures 8, 9, and 10. We observe a strong correlation between Errw and [P.(I Pxllpp across the settings. These correlations are similar to those achieved using P.(I Pu) indicating
38

---

Representations Shape Wcak-to-Strong Generalization: Theorctical Insights and Empirical Predictions
5C=0.897
SC 0.649
0.45
0.35
0.40
Errw2s
Errwas
0.30
0.35
0.25
0.08
0.10
0.12
0.14
0.15
0.20
[1 Pw)
|| Ps) Plopp
(a) Justice
(b) Commonsense
Figure 9: Results for |P. PP) Pollop Exp. II.
SC=0.842
SC=0.810
SC=0.846
0.5
0.4
0.45
0.3
0.40
Errw2s
Err
Er5
0.35
0.2
0.3
0.30
0.1
0.25
0.2
0.20
0.7
0.8
0.9
0.96
0.98
0.90
0.95
[1 Pw] Ps Psllop
[P Pw)P lslop
|Pa(1 Pw) P.ls)sll
(a) SciQ
(b) Amazon Polarity
(c) Cosmos -Aa
Figure 10: Results for P.(I Pw) Pw) Pollop in Exp. III.
SC= 741
SC=0.272
SC=0.428
ErTw2s
Errw2s
Errwzs
0.3
0.3
0.2
0.2
0.2
0.90
10000
8000
6000
4000
-10000
-5000
[P Pw)llop
-dim(act_mapp
dim(
SC=0.786
SC=0.026
SC=-0.060
0.20
Frrwss
0.15
0.15
0.15
0.10
0.10
0.10
0.05
0.98
0.99
5000 4500 4000 3500
-150
-50
|P
P
ol
-iim(act (aannac__mapp)
-dim(hh)
Figure 11: The top panel shows results SciQ for models with sizes 10000. while the bottom panel shows results Amazon
Polarity for models with sizes < 8000, The patterns observed here are consistent with those discussed Figure in the main paper.
that the two metries are similarly informative for W2SG in practice, despite being theoretically derived in different ways.
39

---

Representations Shape Wcak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
D.4. Comparison with model size and effective dimension
Figure Il compares our metric with the activation map dimension and the dimension of approximated principal representa- tions for smaller models SciQ and Amazon Polarity. The results consistent with those presented in Figure in the main paper.
E. Discussion
Using activation maps as representations Exp. III a simple heuristic that yields promising results. However, more principled methods for defining and extracting representations from LLMs, such as those through NTK (Malladi et al., 2023) representation engineering (Zou al., 2023), could be explored. Future research could leverage these approaches to improve results and uncover applications. For instance, (Zou et al., 2023) introduces method for extracting specific concept directions in representations, such as honesty and power-seeking. This could enable computing metric based on topic-specific representations, allowing predictions of W2SG for general tasks within specific topical domains.
40


Reviews:
[{"review_id": "qaDBk9Hk5x", "reviewer": "Reviewer_wiCj", "summary": "This paper provides a theoretical analysis for weak-to-strong generalization (W2SG) from a representation-based perspective. In particular, the authors consider finetuning over fixed representations with mild structural assumptions. \n- It is shown that the overlap between the principal subspace of the strong (student) model's representation and the orthogonal complement of the weak (teacher) model's representation is a key quantity that governs W2SG. \n- The theoretical framework is then leveraged to explain benign overfitting in W2SG -- errors that do not align with the strong model’s principal subspace do not affect W2SG. \n- The overlap between the two subspaces -- the principal subspace of the strong model's representation and the orthogonal complement of the weak model's representation -- provides a metric that theoretically predicts the W2SG performance. In practice, this metric demonstrates a strong correlation with the W2SG performance across various datasets and architectures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our claims well-supported, our explanations insightful and intuitive, the theory and experiments extensive and convincing, and the proposed metric novel. We respond to the comments below.\n\n> the distinction between Sec 4 and Wu & Sahai (2024)\n\nThe main differences are twofold:\n\n(1) Wu & Sahai (2024) show that benign overfitting **can happen**, but they do not extract **general insights about when and how it occurs**. In contrast, we identify a single key quantity driving benign overfitting in W2SG—namely, $||P_s(I - P_w)\\frac{1}{\\sqrt{n}} y||$ in Thm 4.1—which characterizes how much of the label aligns with the intersection between what is missed by the weak model's principal kernel and captured by the strong model’s principal kernel. When this quantity is small, the strong model can avoid repeating the weak model’s mistake, regardless of the extent of overfitting, thereby achieving error mitigation. This very mechanism is not revealed in Wu & Sahai.\n\n(2) Our Thm 4.1 is stated in a very general setting, whereas Wu & Sahai focus on a highly specific distribution with detailed assumptions, making it more of a toy example than a realistic scenario. E.g., there is no evidence that neural network representations follow exactly their assumed bi-level ensemble structure  and that labels depend 1-sparsely on representations. In contrast, our assumptions (discussed on page 4) cover a wide range of realistic cases, supported by literature suggesting that neural network representations often exhibit such properties.\n\n> further explanation of assumptions in Def 3.3; and intuition or practical implications of condition (c)\n\nDue to space limitations, we only provided explanations for kernel-wise isotropy and small cross-sample inner products on $\\mathcal{V}^\\perp$, as we believe these two are the most involved, while the others are relatively natural and self-explanatory. Here, we provide further explanation for all the items, which we’ll include in the revised version. \n\n- (a) is a basic condition that ensures reasonable magnitudes of representations and labels. \n- (b) states that representations are well-concentrated in the subspace $\\mathcal{V}$, both in terms of their covariance and their correlation with labels. This is why the representations on $\\mathcal{V}$ are referred to as the principal representations—they are the part where the empirical distribution closely aligns with the underlying population distribution.\n\n- (c) implies that kernels constructed using only the components in $\\mathcal{V}^\\perp$ exhibit a certain level of uniformity across all directions, with the degree of this uniformity controlled by $\\delta$. In the paper, we discuss two extreme cases—one with very small $\\delta$ and one with very large $\\delta$—to aid understanding. Importantly, this assumption is not made solely for analytical convenience; it is also general and applicable to realistic settings. For example, high-dimensional sub-Gaussian noise satisfies this condition with a small $\\delta$—a scenario highly relevant to deep neural networks with large internal dimensions—since these vectors tend to be orthogonal to each other in the high-dimensional limit. More concrete instances can be found in Examples 3.4 and 3.5 and Thm 3.6, with their significance and relevance discussed in the right column of page 4. Thus, (c) is a key condition that allows us to capture all these diverse scenarios. It is not just analytically useful, but also practically relevant to real-world scenarios.\n\n- (d) holds either when representations on $\\mathcal{V}^\\perp$ are nearly orthogonal across samples or when their magnitudes are small. \n\n- (e) means that the representations on $\\mathcal{V}^\\perp$ have small magnitudes in the population. \n\n> Typos in Def 3.3 (b)\n\nAs the reviewer correctly pointed out, the expression $|| \\frac{1}{\\tilde{n}} \\Pi_{\\mathcal{V}} h(\\tilde{x_i}) \\tilde{y_i} - \\mathbb{E}[ \\Pi_{\\mathcal{V}} h(x) y ]||$ contains a typo—we forgot to include the summation after $\\frac{1}{\\tilde{n}} $. We will fix it in the revised version.\n\nIf there are no other concerns, and given that the reviewer’s feedback is largely positive, we would greatly appreciate it if the reviewer could consider raising the score.", "claims_and_evidence": "The main claims made in the paper are well-supported by analysis and experiments. The theoretical analysis is mostly reasonable. However, some statements seem to have minor issues and are in lack of sufficient explanations (see \"Theoretical Claims\"). The empirical evidences are convincing.", "methods_and_evaluation_criteria": "Yes, the proposed metric is well motivated by the analysis.", "theoretical_claims": "I couldn't verify all the proofs in the appendix, but by quickly going through Appendix A and B, I think the main theoretical results in the paper seem reasonable. However, I feel that some statements are not accurately made or well organized.\n- While the assumptions in Definition 3.3 are relatively mild from the analysis perspective, they are nevertheless dense and not quite well-motivated. For example, only the notion of \"kernel-wise isotropy\" in Def 3.2 is explained, with the explanation focusing on its necessity for the analysis, but not its intuition or practical implications. \n- Some assumptions in Def 3.3 looks counterintuitive. For example, in \"(b) Concentration on $\\mathcal{V}$\", the concentration of correlation with labels, $\\|\\|\\frac{1}{n} \\Pi_V h(x_i) y_i - \\mathbb{E}[\\Pi_V h(x) y]\\|\\|$, should accurately be $\\|\\|\\frac{1}{n} \\sum_{i=1}^n \\Pi_V h(x_i) y_i - \\mathbb{E}[\\Pi_V h(x) y]\\|\\|$?", "experimental_designs_or_analyses": "I review the experiments in the main text (Sec 5) and some of the details in Appendix E. The experimental setup is reasonable, sufficiently detailed, and well-organized. The empirical results align with the theoretical claims and provide convincing evidence for the proposed metric.", "relation_to_broader_scientific_literature": "This paper provides a theoretical analysis for W2SG from a representation-based perspective. Toward the three contributions of this work:\n- The analysis-inspired metric is novel, intuitive, and well-motivated. \n- The explanation on benign overfitting in W2SG intuitive and insightful. But I feel the difference from the analysis in (Wu & Sahai, 2024) is not well explained, especially after stating the results on benign overfitting in W2SG. It seems that both benign overfitting analysis share the same intuition, just using different ensemble models. If more sophisticated mechanisms are involved, it would be helpful to remark on the difference between the two analyses.\n- The empirical verification of the correlation between the proposed metric and W2SG performance is extensive and convincing.", "essential_references_not_discussed": "To my knowledge, the paper discussed the essential references in the field.", "other_strengths_and_weaknesses": "Strengths and weaknesses are discussed in previous sections.", "other_comments_or_suggestions": "Comments are raised in previous sections.", "questions_for_authors": "Major questions are raised in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ei2TSrYdYb", "reviewer": "Reviewer_Yk6a", "summary": "This work provides a theoretical analysis of how a strong model can surpass its weak supervisor by studying the structure of their representations. They key insight (beyond prior analyses) is that even when a strong model perfectly fits the weak model’s predictions at train time, its surpasses its weak supervisor due to its better “principal representations,” which govern generalization. They quantify this, and use it to estimate the error of weak-to-strong models in a few empirical settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our theoretical claims clear and convincing, and for acknowledging the novel insights of our paper.\n\n> “...relative representation structures of the weak teacher and strong student matter…Without controlling for weak supervisor quality, it’s hard to know whether this is a confounder …”\n\n(a) **A new experiment**. We note there are cases where the weak teacher’s performance cannot indicate W2S performance. We conducted a new experiment, where we control the weak supervisor’s error to lie within a narrow range, and observe that **the weak supervisor’s error correlates poorly with the W2S error, while our metric shows a strong correlation**. \n\nSince in our main experiments in the paper, the weak supervisors’ errors span a relatively large range, it is difficult to find a sufficient number of weak supervisors with similar errors. Therefore, we explicitly construct weak supervisors with similar error levels in the following way: We take a single checkpoint of a weak supervisor from Experiment I (the one with hidden size 256 pretrained for 5 epochs), and generate 20 modified versions of it by randomly masking out 100 coordinates of its features each time. We then run the weak-to-strong pipeline for each of these 20 weak supervisors. We compare the correlation between $Err_w$ and $Err_{w2s}$, and between $| P_s(I - P_w) |{\\mathrm{op}}$ and $Err_{w2s}$ across all three datasets. As shown in the table below, the weak supervisor’s error correlates poorly with the weak-to-strong error, while our metric maintains a strong correlation. This further demonstrates that the detailed relationship between the weak teacher and the strong student plays an important role in weak-to-strong generalization—beyond what can be explained by the weak supervisor’s error alone.\n\n|  | Lipop | FreeSolv | ESOL | \n|----|----|----|----|\n| Err_w  | 0.24  | 0.29 | 0.13   | |\n| $\\|\\| P_s(I-P_w) \\|\\|_{op}$  |  **0.62**    | **0.65** | **0.61** | |\n\n(b) We illustrate our intuition with a simple example. Suppose a downstream task consists of 40% advanced linear algebra and 60% advanced calculus. We have two weak pretrained models: model A specializes in basic linear algebra, and model B in basic calculus. Assume fine-tuning mainly builds on existing knowledge rather than learning from scratch. Then, after fine-tuning, model A would likely achieve ~40% performance and model B ~60%, reflecting their alignment with the task. Now consider a strong student pretrained only on linear algebra. According to our main theory, model A—being aligned with linear algebra—should be a better supervisor for this student leading to better W2SG performance, even if its own performance is lower. Our proposed metric captures this alignment and should correlate with W2SG performance, whereas weak supervisor performance alone does not. This highlights a case where our metric offers meaningful insight beyond what weak model performance alone can explain.\n\n(c) **Practical perspective**: Measuring the weak supervisor’s error requires access to labels. In contrast, our metric is label-agnostic. The fact that it achieves such a high correlation while using less information is already impressive.\n\n> “The writing throughout the first part of Section 3 could be more clear. Lots of notations and intermediate results are presented without a lot of guiding intuitions towards the final claims.”\n\nThanks for the suggestion. Due to space constraints and the large number of results, we focused on conveying the most important messages and were unable to include more detailed explanations. We will add more intuitive explanations in the revised version.\n\nIf there are no further concerns, and given the reviewer’s largely positive feedback, we would greatly appreciate it if the reviewer would consider raising the score.", "claims_and_evidence": "In their experiments, the authors argue that their representation-based metric captures weak-to-strong error beyond model size, i.e., it predicts weak-to-strong error in a more fine-grained manner than model size. Besides model size, it would be valuable to also consider grouping according to the error of the weak supervisor. For a given error-level of the weak supervisor, does the representation-based metric predict weak-to-strong error?\n\nThis would be a more convincing experiment that illustrates that the relative representation structures of the weak teacher and strong student matter rather than just the quality of the student model. Without controlling for weak supervisor quality, it’s hard to know whether this is a confounder that causes a high correlation between weak-to-strong error and their representation-based metric.", "methods_and_evaluation_criteria": "See \"Claims and Evidence\"", "theoretical_claims": "I read through the theoretical claims presented in the paper and they are clear and convincing. I did not check the proofs provided in the supplementary material.", "experimental_designs_or_analyses": "See \"Claims and Evidence\"", "relation_to_broader_scientific_literature": "The paper studies a topic of recent interest: the ability of a strong model to exceed the performance of its weak supervisor when trained on labels produced by this weak supervisor. There are several existing theoretical analyses studying the same phenomenon. This paper specifically studies representation structures, and yields novel insights on benign overfitting.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The writing throughout the first part of Section 3 could be more clear. Lots of notations and intermediate results are presented without a lot of guiding intuitions towards the final claims.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gjE629tSAA", "reviewer": "Reviewer_BNKN", "summary": "This paper studies weak to strong generalization, where a strong model is fine-tuned on a task using data labeled by a weaker supervisor model (it is known that perhaps surprisingly the strong model can outperform its weak supervisor). Specifically, the paper introduces estimators, depending only on internal representations of the strong and weak models on the input data distribution for the fine-tuning task, that correlate with the test set error Err w2s of the weakly supervised strong model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback on our theoretical analysis, extensive experiments, novelty and broader impact.\n\n> Cor 5.1-2: why label agnostic\n\nAs noted in L306 right, once we factor the label-dependent term out of the operator norm (becoming the label variance C), it can be treated as constant for a fixed dataset. When varying the models, the only factor that changes on RHS of 5.1 is $||P_s(I-P_w)||$, independent of labels. Thus the trend of RHS can be predicted w/o labels. Same for 5.2\n\n> L110,163\n \nThanks for the suggestions. We’ll incorporate them in the revision.\n\n> L175 Left: L180 typos\n\nThis is a typo. We didn’t intend to use Einstein notation. We’ll add the missing summation after $\\frac{1}{n}$. \n\n> “Conditions d and e closely related”\n\nOne possible way to make (d) and (e) more “unified” is to rephrase e as a variant of the cross-sample statement but with the size of one sample taken to infinity. We’ll note this in the revision.\n\n> Thm 3.6 \n\nWe’ll include the following discussion. Thm 3.6 highlights the generality of Def 3.3, showing more examples can be constructed. Any representation composed of a part satisfying Def 3.3 with δ=0 (eg, from a very low-rank distribution) and a high-dimensional sub-Gaussian will, as a whole, satisfy Def 3.3. Eg, Example 3.4 can be extended by concatenating sub-Gaussian.\n\n> discussion on hyperparameters\n\n(1) How we tuned the hps: See Sec D.2 for detailed hp values. We select the hps that maximizes correlation with test Err_w2s. Our metric is computed on the w2s split, consistent with theory.\n\n(2) **Cross-model hp transfer:** We note that, although each model could technically require different hps, in experiments we let all weak models share hps for simplicity and still achieve strong results, suggesting that our approach is not very sensitive to hps. Further, we present **a new experiment** demonstrating that hps selected using one group of models (ie, as a validation set) generalize to other models. We randomly split the weak models into two groups, select hps based on one group, and evaluate them on the other. We repeat this 20 times and report the results on 5 datasets https://anonymous.4open.science/r/icml2025figures/table.png. Correlation remains high with low std, indicating that hps selected using a few models can reliably generalize to new ones. Additionally, we note that a small number of labeled data should suffice for hp tuning, as they are only used to measure test performance and not to compute our metric.\n\n(3) **intuitions for hp selection**\n\n- **For β**: β captures the effect of regularization and should be set higher when stronger regularization is used. This could explain why the optimal β in Exp III is much higher than in II: in III finetuning for one epoch only (following prior work) introduces very early stopping and thus strong regularization.\n\n- **For α**:  The choice of α depends on the underlying dimensional structure. While the relationship can be complex, one intuition is that larger models tend to require a higher α. For small models whose dimensionality is relatively low compared to the sample size, most components can concentrate well. There may be few or no non-principal ones, so a small α suffices to filter them out. In contrast, larger models have high-dimensional yet low-rank representations (Huh et al 2021), where only a few top components concentrate with finite data. There are more non-principal components, and moreover, their magnitudes can appear inflated in the finite sample due to MP law—necessitating a larger α. These intuitions align with the reviewer’s observation from Fig 6 Exp II that the strong model requires a larger α than the weak ones\n\n> “explicitly targeting the label agnostic property; common crawl”\n\nWe note that all experiments demonstrate the label-agnostic property. In Figs 2-4, our label-agnostic estimator (x-axis) strongly correlates with Err_w2s (y-axis). For the second half of the comment, if the reviewer was asking whether our metric computed on Common Crawl (CC) could predict performance on arbitrary tasks, this is not feasible: label-agnostic≠task-agnostic. Predicting performance on task A still requires unlabeled data from A. It is not reasonable to expect that CC could be used to indicate performance on any tasks. If the question was instead about predicting performance on CC itself, the bottleneck is evaluation: since CC lacks labels, we can’t compute W2SG performance as ground truth to compare our metric against.\n\n> beyond human comprehension regime\n\nWe will revise the sentence to reflect: (1) in the future AI may surpass humans in certain tasks; (2) even today, AI can outperform *average* humans on certain tasks—yet W2SG shows that *average* humans can still help improve AI in those tasks. In particular, our results indicate which humans (or weaker LLMs) can best teach a strong AI —and interestingly the answer is not necessarily the strongest human or the strongest weak LLM.", "claims_and_evidence": "Theoretical claims are precisely formulated with definitions and assumptions made clear, and proofs are included in the appendix (although I didn't read them). Experimental evidence supports the main claims.\n\nOne comment:\n- Cor. 5.1-2: the *variance* of the labels does appear in these corollaries, so some discussion of the precise sense in which they are labeled agnostic would be helpful.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I did not read proofs, but I did read definitions, assumptions, statements, and claims carefully. I have a handful of mostly minor comments on the math.\n\n- L110, top left: might also be worth mentioning using L2 regression as the objective function, as opposed to e.g. cross-entropy.\n- L163, right: \"w.r.t. a subspace V\" in the italicized definition is strange because the subspace gets defined explicitly as part of the criterion. To me the following would make more sense: \"representations of $h$ are $(\\delta, \\hat{gamma}, \\tilde{gamma})$-decomposable for some $\\delta$ ...\" and then V gets defined as part of the existence criteria.\n- L175 Left: I don't agree with the well concentrated characterization. This condition seems to be about the empirical quantities $\\tilde{\\Sigma}, \\hat{\\Sigma}$ closely approximating the expectations. There's also some Einstein notation on L180-181, unless that's introduced a summation symbol is warranted.\n- Conditions d and e seem closely related: I.e. assuming (e), and that $hat{\\mathcal{D}}, \\tilde{\\mathcal{D}}$ are both IID samples from the population $\\mathcal{D}$, and some relationship between operator, norms of covariate and kernel, that the estimate of the full population statistic one gets from estimating with $hat{\\mathcal{D}}, \\tilde{\\mathcal{D}}$ is also small? Okay, some work to be done here, but my point is if possible rearranging to: (e) first, and something should be said about (d) being \"(e) but with hats and tildes\" would be easier to grasp.\n- Thm 3.6: this is interesting, but as a reader, it's not clear to me how important it is for the flow of the paper. Some commentary on what the theorem says in natural language would be helpful.", "experimental_designs_or_analyses": "- hyperparameters $\\alpha, \\beta$: I wish there were additional discussion on this topic. How are these tuned? On what split of the data set? To maximize what (i.e. correlation with Err w2s)? I also wonder if there are some characteristics that would allow for choosing good $\\alpha$ without hyperparameter tuning (it's effectively a cut off on eigenvalues, there are lots of heuristics for that). In the appendix, I see that for the embedding model experiments, low $\\alpha_w, \\beta_w$ and higher $\\alpha_s, \\beta_s$ do better, but for the LLM experiments e.g. higher $\\beta_w$ does better. There's also notable dataset dependence in the LLM experiments. \n- it would be interesting to see an experiment explicitly targeting the label agnostic property of this estimator of Err w2s. For example, does computing $\\lvert P_s (I-P_w)\\rvert$ on text from the common crawl correlate with Err w2s in the LLM experiments?", "relation_to_broader_scientific_literature": "The key contributions of this paper are motivated by human-AI alignment research. They may have broader implications for distillation fine-tuning in machine learning.\n\nAs the introduction and related work outline, results of a similar flavor (decomposing Err w2s using one of various triangle inequalities and showing an easier-to-estimate term that emerges correlates with Err w2s). As I understand that the novelty of this paper lies in:\n- label agnostic-ness\n- estimates based on model representations\n\nFrom looking at the references mentioned in related work, I feel that this paper has a relatively comprehensive experiment suite.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "- \"As AI systems become increasingly capable of performing complex tasks beyond human comprehension\" do you have an example of an an AI system performing a task that's \"beyond human comprehension\"? I get that it's a first sentence, and motivational introductions like this are very common today. I just want to know if we are verifiably in the \"beyond human comprehension\" regime (and impressive task completion in the sense of very few humans could complete the task or a human couldn't complete the task nearly as quickly, etc. don't count, because in those cases, at least some humans could at least comprehend what the system is doing).", "other_comments_or_suggestions": "Overall, I felt that the balance of the main body of the paper leaned very heavily towards theory, leaving very little space for discussion of the experiments. Especially since some of the theoretical content is (useful, but not essential for the flow) examples, differing some of it to the appendix to allow for a more detailed experiment section (including, for example, more discussion about hyper parameter tuning as I asked about above) would improve the paper (in my view).", "questions_for_authors": "None beyond the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an extensive theoretical analysis of weak-to-strong generalization from a representational perspective, supported by some empirical validation. While the reviewers and I generally found the work useful, the presentation is often unclear, including both issues in the communication of the theoretical results and their assumptions, and the details of the empirical experiments. Improving the clarity of the presentation would help to communicate the value of the perspective offered.
Decision: Accept (poster)