Quantifying the Gain in Weak-to-Strong Generalization
Authors: Moses Charikar
Chirag Pabbaraju
Kirankumar Shiragur
Venue: NeurIPS 2024 poster (2024)
Keywords: Weak-to-Strong Generalization
Primary area: learning_theory
TL;DR: We show that the improvement in performance achieved by a strong model supervised by a weak model is quantified by the misfit error incurred by the strong model on labels generated by the weak model.

Abstract: Recent advances in large language models have shown capabilities that are extraordinary and near-superhuman. These models operate with such complexity that reliably evaluating and aligning them proves challenging for humans. This leads to the natural question: can guidance from weak models (like humans) adequately direct the capabilities of strong models? In a recent and somewhat surprising work, Burns et al. (2023) empirically demonstrated that when strong models (like GPT-4) are finetuned using labels generated by weak supervisors (like GPT-2), the strong models outperform their weaker counterparts---a phenomenon they term *weak-to-strong generalization*.

In this work, we present a theoretical framework for understanding weak-to-strong generalization. Specifically, we show that the improvement in performance achieved by strong models over their weaker counterparts is quantified by the *misfit error* incurred by the strong model on labels generated by the weaker model. Our theory reveals several curious algorithmic insights. For instance, we can predict the amount by which the strong model will improve over the weak model, and also choose among different weak models to train the strong model, based on its misfit error. We validate our theoretical findings through various empirical assessments.

enable these highly capable models to realize Quantifying the Gain in Weak-to-Strong Generalization
Moses Charikar Stanford University
Chirag Pabbaraju
Kirankumar Shiragur
Stanford University
Microsoft Research
mosesucs. stanford.
cpabbara@cs. brabbaraccs.stanford..oo edu
stanford.
kshiraagurdm.crosoft.com
Abstract
Recent advances in large language models have shown capabilities that are extraor- dinary and near-superhuman. These models operate with such complexity that reliably evaluating and aligning them proves challenging for humans. This leads to the natural question: can guidance from weak models (like humans) adequately direct the capabilities of strong models? In recent and somewhat surprising work, Burns et al. [BIK+2] 223] empirically demonstrated that when strong models (like GPT-4) are finetuned using labels generated by weak supervisors (like GPT-2), the strong models outperform their weaker counterparts a phenomenon they term weak-to-strong generalization.
In this work, we present a thcoretical framework for understanding weak-to- strong generalization. Specifically, we show that the improvement in performance achieved by strong models over their weaker counterparts is quantified by the misfir error incurred by the strong model on labels generated by the weaker model. Our theory reveals several curious algorithmic insights. For instance, we can predict the amount by which the strong model will improve over the weak model, and also choose among different weak models to train the strong model, based on its misfit error. We validate our theoretical findings through various empirical assessments.
Introduction
Present-day Al models demonstrate incredible capabilities at variety of extremely difficult tasks. For this reason, they are frequently described being superhuman, in that seems hard to imagine a human displaying the same abilities as the Al model, For example, move 37 in AlphaGo's 's famous victory against Go expert Lee Sedol [Met16] has been described as being beyond the realm of human imagination. In this sense, today's 's AI models are well the path of exhibiting abilities [WTB+ 22].. Ultimately, we want these new abilities to be aligned with what would be beneficial to humanity. This rationale is what primarily guides the training of large-scale Al models through human feedback [CLB* (ILB++1]] 17]. However, given that we expect AI models to pick up skills that ourselves don't fully grasp as humans, how their potential?
and emergent
A recent work by 23] shows that not all hope is lost in this endeavor. To model humans as being weak supervisors for increasingly strong Al models, they conduct the following "weak-to-strong generalization" experiment. Suppose we finetune small language model like GPT-2 [RWC+ 19] on data with ground-truth labels for task. What happens if we then finetune a large language model like GPT-4 [Ope23a] on data labeled by GPT-2, instead of data having ground-truth labels? Would GPT-4 simply overfit to GPT-2's s labels and do no better, or would outperform GPT-2, given that inherently much stronger model? The surprising experimental result is that GPT-4 trained in this manner outperforms GPT-2 when evaluated on the true data, for a variety of finetuning tasks. Note
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

that GPT-4 is able to outperform GPT-2 without ever seeing true labels when it was being finetuned. One plausible explanation for this is that GPT-4 was able to glean the essence of the finetuning task from GPT-2's labels, and since it is fundamentally a stronger model than GPT-2, this knowledge was sufficient for to outperform GPT-2.
In this work, we seek theoretical justification for why we might expect to see such a gain in accuracy in weak-to-strong generalization. Concretely, we ask:
Does a weakly supervised strong model provably attain smaller error than its weak supervisor, and if so, can this gain be formally quantified?
Towards answering this question, we show (Theorem 1) that in the concrete setting of regression, the true error of a strong model trained on weak labels smaller than the error of the weak model, by at least the error of the strong model on the weak labels itself. We call this latter quantity the misfir between the weak and strong model. Our result can be stated as the following simple principle:
Gain in accuracy in weak-to-strong generalization = Misfit between the weak and strong model
Intuitively, the misfit quantifies the erroneous knowledge that the strong model does not obtain from the weak model, and hence also the amount that the strong model improves over the weak model. We note that the work of [BIK 23] does empirically show that the performance gain of the strong model scales directly with its misfit (or disagreement) with the weak model; our result thereby provides a precise quantification of this observation.
Key obtaining results is representation- perspective [TJJ20] towards weak-to-strong generalization. We posit that the main difference between weak and strong models is in the disparity between the quality of their data representations. This disparity in representation quality can manifest, among other reasons, due to difference in the expressivity and complexity of the weak and strong models, and the amount of pretraining data that they have seen. For example, in the experiments by [BIK 23], the weak and strong models used are GPT-2 and GPT-4 respectively; the latter is a significantly larger transformer architecture, pretrained on a much larger dataset than the former. As a broader analogy. consider the task of learning new language. This is an easier task for a multilingual person than a monolingual person. A multilingual person has a richer representation for language, drawing from their knowledge of different syntax, lexical structures, and sounds in multiple languages. With this perspective, we can imagine finetuning tasks to be relatively simple functions (e.g., linear functions) composed with the appropriate representation. For example, if the task is about learning Italian, a suitable Iitalian-specifee representation of the problem (including features learned from Spanish and French, say) might allow them to better understand the new language, while the same might not work so well for a monolingual whose representation only has features learned from English.
linear combination of the multilingual's
Armed with this perspective, we model the task of learning a real-valued finetuning task under the least squares loss in the veak-to-strong generalization framework. We assume that there exists a ground-truth representation h* of the data, which makes it amenable to learn a finetuning task f* of interest. We imagine that the weak and strong models come equipped with representation maps hy and h2 respectively, which are possibly obtained via pretraining on corpus of data. Next, we imagine that the weak model sees data labeled by the target function f o h²" and after finetuning, learns some arbitrary function fur o kw.- At this point, the weak supervision pipeline begins. The strong model is fed with data labeled by fw hy (instead of the true labels f* o h²",, and as part of finetuning, outputs a function faw. from a function class F., that minimizes the discrepancy between faw h2 and the data labeled by fw h that it sees. Ultimately, we care about the error of few h, with respect to the true finetuning task, namely f h" Our main result (Theorem 1) precisely quantifies the gain in the accuracy of few he over fw hw in terms of the misfit between them, under the assumption that the set of functions F, is a convex set.2 In many practical applications, the representation map is generally the forward pass of the data through a suitable neural network architecture, and the finetuning task is performed by the last linear layer (KR++22) of the network. In such cases, our assumption that the set F, is convex readily holds true.
We note however that this methodology did not allow GPT-4 to fully reach its performance achieved by training on true data.
Note that the functions in Fs need not be convex themselves.

---

We validate our characterization of the gain in weak-to-strong generalization through various exper- iments (Section 5) on synthetic and real-world data. The experiments corroborate our theoretical findings. Namely, we observe that upon performing the above weak-to-strong supervision pipeline, the gain in accuracy of the weakly-superviser strong model over its weak supervisor more or less exactly aligns with the misfit between the weak and strong models (Figure 2). We also demonstrate (Section 5.3) that the labels "weak" and "strong" models are nuanced and not solely dependent on expressive power, in fact, in a low-sample regime, a less expressive model produces a higher quality representation and should be considered a strong model. Our theory and experiments lead to several algorithmic insights and open up interesting questions. For example, one algorithmic heuristic that arises from our theory is the following: given access to different weak models, choose to deploy the strong model that achieves the smallest difference between the weak model error and misfit (Table 1). Our results also motivate the perhaps counterintuitive algorithmic question of obtaining weak models that lead to large misfity with the strong model. Another possible line of inquiry could look into ensembling across different weak models, and obtaining a gain close to the sum of their individual misfits. At more philosophical level, this is akin a superhuman AI model assimilating knowledge from various humans, while correctly identifying and discarding each of their flaws.
2 Related Work and Preliminaries
2.1 Related Work
The idea of converting a "weak" learner to a "strong" learner can be traced all the way back to the famous paradigm of boosting [Fre95, FS97], if not earlier. The recent work by [BIK+23] frames this problem within the context of superalignment [Ope23b, J0pe22,, which seeks to reliably align AT models smarter than humans human intent. Thereafter, several works that study the training of a "strong" model guided in some capacity by a "weak" model have emerged. Some of these include instruction filtering by weak models [L.ZH+24], easy-to-hard generalization [SYS+ 24], weak-to-strong correction [JCL+24] 24] and weak-to-strong hallucination inducement [ZCBS23].
The weak-to-strong generalization paradigm is perhaps most closely related to the teacher-student model of training [LA16, TV17] (sometimes also referred to as knowledge distillation [HVD15, GYMT2IJ. where student model (typically smaller) is trained using data labeled by teacher model (typically larger), and possibly some additional ground- truth data. The remarkable phenomenon of the student model outperforming the teacher has been observed in many works [BCNM06, HVD15, 18. Most relevant to us are formulations where the student model is equally [FLT+18] 18) or more powerful [XLHL20] than the teacher model. There has been theoretical work explaining superior generalization of the student in some specialized settings, e.g., the work of [MFB20] where the student also has access ground-truth labels, or the work of [WSCM20], which operates under a certain expansion criterion and consistency loss. In contrast, our work does not assume that the student model has any access to ground-truth labels, and also does not incorporate a regularization term in the objective. Finally, the conceptual insight in our work about the performance gain of the student model scaling with its disagreement with the teacher model is closely related to the theory of generalization bounds based on disagreement between different classifiers [DLM01, WZ17, YHY+19)
2.2 Preliminaries
We assume that the data domain is Rd, and assume that there exists a ground truth representation function h+ : IRd + Rd that maps the data x to an enriched representation h (z). We assume the existence of pretraining tasks, through which strong models obtain representations of the data from a function class H. : Rd - Rdd, and weak models obtain representations from a function class Hi Rd d. For example, H. can be the class of deep neural networks, and H., can be the class of shallow neural networks. The target finetuning task (composed with the ground truth representation) is denoted as I h², and the function learnt by the weak model is denoted by fw o hw. We assume that the strong model learns finetuning tasks from a function class F, : Rdd + R. and assume that the set F, a conver set. The convexity assumption requires that, for any f. g E Fe, and for any A E [0, 1], there exists h E F, such that for all x E Rd, h(z) = Xf(2) + (1 A)g(z). For example,
'I other words, weak models that largely misfit strong models are the ones that most effectively elicit [CCX22] what the strong models already know.
3

---

F. can be the class of all linear functions from Rd to R. However, we do not assume anything about either f or fwi in particular, they need not belong to Fs. We denote the marginal data distribution by P. For any two functions g.g RRd -> IR, we define the distance dp (f. g) = (f=r(((r) g(x))²
i.e. it is the average (with respect to P) squared distance between the images of the functions.
3 Results
We first state a quantitative version of our main result that characterizes the gain in weak-to-strong generalization in terms of troong-to-weak misfit in the so-called realizable setting. Namely, we assume that the target finetuning task f* o hi can be equivalently written as Js o h2 for some Is E F,,
Theorem WWeakttoSStron Generalization under Realizability). Let h d + RRd be ground truth representation map, and let f 12d R be a finetuning task of interest. Let h2 E + R. and hw Rd -> IRdw be the strong and weak model representation maps respectively. Given some data labeled by f* ht,, let fw bww be the function learnt by the weak model, for some arbitrary function fw R - IR. Now, for convex set of functions F, mapping Rd to IR let
sew = jrgmin dp(ff fw o hw)
(1)
be the function learnt by the strong model under weak supervision. Lastly, let assume that there exists fa E Fg such that fa o h2 = f o lt",. Then, we have that
dp(Ssm hay S hh) dp(S hwm I hhhh")doh dp(ssw hay smn hww)-
(2)
On the left-hand side in (2) is the error of the weakly-supervised strong model on the true data. The first term the right-hand side is the true error of the weak model, and the second term is the error of the weakly-supervised strong model on data labeled by the weak model (i.e., misfit). Thus, the inequality directly says that the least) an amount equal to the misfit. Note again that in practice, a popular way to finetune a pretrained model task task-specific data is by tuning the weights of only the last linear layer of the model. In these cases, Fs is simply the set of linear functions, which is convex. We emphasize that neither of f* or fw need to belong Fg; as long as the strong model finds the minimizer over a convex set of the loss on the weakly labeled data (as in (I)), the inequality in (2) holds.
strong model improves over the weak model by (at
Next, we relax the realizability assumption that the target task f ht belongs to the space of functions that the strong model optimizes over. Instead, suppose that by composing he with functions in F,, it is possible for the strong model to get small distance to the target task. The strong model could obtain such a powerful representation map after having seen an abundance of pretraining data; the realizable case corresponds to & O. We also relax the assumption that the strong model is able to obtain the true minimizer with respect to the data distribution P as in (1). In reality, we can imagine that the strong model only sees a finite sample labeled by the weak model, and obtains fow by minimizing the loss over this finite sample. Even with these relaxations, we can show that the same qualitative result as in Theorem 1 continues to hold, upto small error terms.
and Finite Samples). Let h" Rd + Rd be ground truth representation map, and let f .Rd + R be finetuning task of interest. Let hg R > Rd and hww : Rd > Rdw be the strong and weak model representations respectively. Given some data labeled by f ht, let fw o oww be the function learnt by the weak model, for some arbitrary function Fw dR + R. For a convex set of functions F, mapping Rdd - TR, let
Theorem 2 (Weak-to Poeeak-to-Strone to-Stoong Generalization under Non-
fa argmin dp(s ha, I o h²),
(3)
and suppose that dx (Sa hg 1 h()) e. Now; suppose we obtain weakly-labeled i.i.d. samples (æ1, y1)...., (Ins )n), where each z, ~ P and y = fww o hw Let
1 n faw argmin JeF, n
E(So
2
Finally, assume that the range of f fw and all the functions in Fg is absolutely bounded. Then, have that with probability at least 1- 8 over the draw of (I1, y1). (In, Vn))
dxp (faw o fn o )²²== < dp(ff o hews f o h²) - dp(fow o h2 fwo o kww
CF,
log(1
+ +0(6)) +0
+0
(5)
n2

---

where Cy, is a constant capturing the complexity of the function class Fs, and the asymptotic notation with respect to - 0, OC.
As compared to (2), the bound in (5) has two sources of error terms: the first error term of (E)) arises (via standard triangle inequality arguments) due to the non-realiaabiltyy assumption, and goes to zero as the strong model becomes stronger and more expressive. The latter two error terms arise (via standard uniform convergence arguments in (TJJ20J) because the strong model only sees finite weakly-labeled cmnple-thes terms vanish too as the sample size becomes large.
4 Main Proof Technique
In this section, we outline the proof of realizable weak-to teak-to-stroon. to-sto generalization (Theorem 1). The proof of Theorem uses the same main idea and is given in Appendix A. Recall that the strong model learns from a convex set Fs Rd, + IR of finetuning tasks. Recall also that we denote the strong model representation map by hi IR Rd., Let V, (ʃ o h2 f E FF) be the set of all tasks in F, composed with the strong model representation. We first observe that V, is also a convex set. Claim 3. V, is convex set.
Proof. Fix S:9 E Fs, and consider f o lis g e h2 E V. Fix any À E [0. 1]. Since Fs a convex set, there exists p E F, such that for all y RR,, p(y) \f()) + (1 A)g(y). Now, fix any x RRd. Then, we have that
X(ʃ (x) + (1 >)(g o hx) A/(h, (x)) (1 ())(H.(1)) p(h) (po (1))
and hence X(ʃ o h2) + (1 (1-A)(g - po h2 E V..
We are then ready to prove Theorem 1.
Proof of Theorem I. The setting under consideration is depicted Figure 1. Since we assume realizability, f o h E Vg. Let A df(few a hg, f a h"), B = dp(f o h,s fw hw) and C'= = dp (fw hw. f o h²). We want show that C A + B. Recall that fau = argmin dp(f o h,s fw hw).
In other words, few o h2 is the projection of fw by onto the convex set Vz. We can therefore apply the "Pythagorean theorem" for projections onto convex set [Haz16, Theorem 2.1].
B
Tow oh,
f'kk -,uh,
V, VF=F.or hr
Figure 1: ssw h2 is the projection of s hop onto the convex set V.
Concretely, for any g Vs, observe that
df Nw, 9) Ez-p(g(x) (Fw o (x:)2
Exup (g(x) (Ssw o hx) (fow o ha) (fw a ha (x))
= Exnp(g(x) (faw o ho) hz))2))² Eamp((Sew h.) - (fw hw (x))
+ 2. Eynp [g((x) (Ssw e ha))))) o ho)((r) - (Sw ha (x))
ddo(Son o h1, 9) + dp(ssm hs, Sw o kww)
+ 2. Eyap [g((z) - (faw a h (x) o h.) - (Fw o h (x)))
(6)
5

---

But note also that by definition of projection, dr(fu hw 2 dp o ho, fw hw). and hence
dp(Ssw an hssgg) 9) + 2 EE- [[g(a) - (Ssw he (2))( o hx)( - (Sw hw) 2 0. (7)
Now, fix E (0, 1), and consider the function
fsw he). w(t) = sow o hi + (f h -
Namely, for any x, (())(x)) = (faw hz) +t ((f* o h2) (faw o h.) hx)() Because V,, is convex set (Claim 3), w() E Vs Also,
dp(ssw o hs, w(l)) = Bypp ((S (1)(()) - w(1)x))2
= t² Exup((f (x))(a)) (few o h.)(2))²
Hence, substituting wil)) for y in (7), we get
12. E.E-p((S 1i (xx) (Ssu i (x))2 +t EEWP [(f o (=) (fow he (()) h.) (far o hww )x))) 2 0.
Taking the limit as 1 O, we get that
Ezap [([ h² )xx (Ssw (z)(()) hx) (Sw hwD ) 20 Substituting f o htd for in (6), and using (8), we obtain the desired result
(8)
dp(fw o hw: f h²) 2 df(fsw hn, f h*) + dp(sow hgs fw hw).
5
Experiments
We perform experiments* on synthetically generated data as well real-world molecular prediction and natural language datasets to verify the guarantees on weak-to-strong generalization given by our theorems. The results for the natural language tasks are given in Appendix C.
5.1 Synthetic Experiments
We set the target data representation ht R - R16 be a randomly initialized 5-layer multi-layer perceptron (MLP) with ReLU activations, with input dimension 8 and hidden layer dimension 16. The class F, of finetuning tasks from which the strong model (as well as the weak model) learns is simply the class of linear functions from IR 16 + R; F, is thus convex set (see Appendix D for instances where F, is a non-convex set). The marginal data distribution P in our experiments is always (0, o² I2).. To ensure that the data is well-spread, we
500.
Representation Learning. We experiment with two different ways of obtaining the weak and strong representations hw and hg:
(1) Pretraining: We randomly sample T finetuning tasks f()).. " f(T) F.. For each
N. E [T). we generate data {z]5) Vj Jj=1, where o. (1) ~ P and Vi = f(t o (x.").
Loosely following [TJJ20], we obtain hw and h2 as
T No
ΣΣ o h(aaa)) 90 for l E {w,s).
h4 = argmin T N,
(9)
[=1==1
We Hi and H. (both R + R.16 be the classes of 2-layer and 8-layer neural networks respectively with ReLU activations and hidden dimension 16. We obtain Atw and ht via gradient descent on the representation parameters find the minimizers in (9). We set T' = 10, N, 2000. Additionally, we also consider the realizable setting (Theorem 1), where we explicitly set h h" and only obtain fy as above.
(2) Perturbations:
We also consider another way obtain the weak and strong representations as direct perturbations ht" Namely, we perturb every parameter in every weight matrix in h² by independent Gaussian noise N (00 o.) to obtain h, Similarly, we obtain ht by perturbing each parameter in h² by (0, to be closer approximation of h" than hag Hence, we set 0.01 and aw 0.05.
Ideally, we want the strong representation hi
'https://githun wtt.chww...oocccgga//taetaaeaaee regresssoo

---

0.0-
Gein accumeyy
A.9
0.5
6TA Misfit on weak labels
062
Mistt on weak lobels
Misti on weak labels
(a) Realizable (pretraining).
(b) Non-realizable (pretraining).
(c) Non-realizable (perturbation).
h" is 5-layer MLP, h. = h..
h" is 5-layer MLP, h.. is
h = h² (0, ), h2 = h'
Nw is a 2-layer MLP.
2 2-layer MLP, h. is an 8 8-layer MLP.
(1 0.01%. h² is 5-layer MLP.
27
12.00
31
Cain acurscy
accurccy
Gain accuracy
Lo-
23 Nisfit on weak labels
Misfit on weak labels
Misfit on weak labels
(d) MolBERT on ESOL
(c) MolBERT on FreeSolv
(f) MolBERT on Lipop
Figure (b).b))(c) Experiments on synthetic data. (d).(e)(() QSAR tasks over MolBERT rep- resentations on the ESOL, FreeSolv and Lipop datasets. For each dataset, ChemBench [Wan20] provides three different train. test and validation splits; multiple points of the same color correspond to
supervision for the same weak model (as specified in legend) across these splits.
Weak Model Finetuning. Once the representations hw and h2 have been obtained and fixed, (8)
randomly generate M new finetuning tasks ( (MM) Fss and obtain data (z) yj 3 for each these tasks. Here again, 5 and / = (() lt* (xx We set M 100, N, 2000. For each task, we train the weak model on the data generated for the task, to obtain
N₂ of (ar /0
argmin JEF, N
(10)
Here, the representation parameters he are frozen, and / is obtained via gradient descent. Note again that we are training the weak models on true data labeled by the finetuning task f() o h"
Weak-to-Strong Supervision. Once our weak models are trained for each finetuning task, we (() Ny generate weakly labeled data. That is, for each [M], we generate { Mj 1i=1 where x P.
But crucially, 0/57 = ((6 ha (2(().. We now train our strong models on this weakly labeled data. Namely, keeping the strong representation ht fixed, obtain, via gradient descent again
N₁ (sohn 1/0
(() argmin JEF, N₁
(11)
At this point, our weak- weak-to-strong training procedure is complete.
Evaluation. For each finetuning task, wish to evaluate the accuracy of our weak-to-strone model ((i) he with respect to the true task f(i) ht*
Towards this, we estimate 3 quantities:
(a) Error of the weak-to-strong model f o hy the true finetuning task: Enn (f o h1 (α)
f(i) ht (x))

---

(b) Error of the weak model [o Nww on the true finetuning task: Eew (S oww(x) - - ht (x))².
(c) Misfit error of the weak-to-strong model on the weakly labeled data: Ep 11) fi (z)))
Each of these quantities are estimated from a fresh sample of size N, drawn from P. For each task i E [M]. we plot the difference (b)-(a), namely the Gain in Accuracy, on the y-axis, versus the Misfit (c) on the x-axis. Figure 2a has the results for the realizable case where h2 hh* and h is obtained by pretraining. Figure 2b has the results for the non-realizable case where both hw and h, are obtained by pretraining. Figure 2c has the results for the non-realizable case where ha and he are obtained by directly perturbing the weights in ht. For reference, recall that Theorem 2 indicates that the gain in accuracy is (upto error terms) at least the misfit. The plots in Figure 2 suggest that the gain is more less exactly the misfit, which is in agreement with our theory!
5.2 Molecular Prediction
We also validate our conceptual insights on real-world molecular prediction datasets. Specifically, we follow the Quantitative Stricture-Activity Relationship (QSAR) task setup in the MolBERT [FEG+20] paper. These tasks involve predicting physical properties of molecules like solubility, lipophilicity, etc. We consider three regression datasets: ESOL, FreeSolv and Lipop. These datasets are part of the MoleculeNet [WRF+ 18] benchmark suite, and have been curated into train, test and validation splits by ChemBench [Wan20]. The MolBERT paper provides weights for a standard-size BERT [DCLT18] architecture (hidden dimension 768, 12 layers, 12 attention heads) pretrained for 100 epochs on the GuacaMol [BFSV V 19] dataset. Wc use these weights as the strong representation h,, For the weak representations hus we run the pretraining pipeline for substantially smaller transformer architectures and lesser compute time. Specifically, we consider transformers with just layers and attention heads, and vary the hidden size in (8, 12, 16, 32, 48, 64, 96]. For each of these settings, we run the pretraining tasks for a mere 2 epochs to obtain different weak representations hi...
Once we have the representations hts and leg, we can finetune a linear layer on top of these for each of the three regression datasets. We run the entire teeak-to-stoong supervision pipeline from above, where we weakly supervise the strong model ha on labels given by each of the weak models hw- The results given in Figures 2d, 2c and 2f. Again, see that the gain in accuracy of the weakly supervised strong models is accurately characterized by their misfit on the weak labels.
We were also able to see an otherwise useful algorithmic insight in these experiments. Consider a setting where we have at our disposal various weak models, and have performed the weak-to-strong supervision pipeline separately on each of them. We now want to deploy one of the weakly trained strong models; our goal is to choose the one that gets the least error on the true data distribution. Recall that Theorem 2 guarantees that the error of a weakly supervised strong model is upper bounded (upto error terms) by the difference between the weak model's error and misfit. This suggests a natural heuristic: sort the strong models by the difference between the corresponding weak model's error and the misfit, and choose the one for which this quantity is smallest. We observed that this heuristic ends up working quite well-Tabe I shows the numbers for the Lipop dataset, while the results for ESOL and FreeSolv are in Appendix B.
5.3 Strong-to-Wert Weak Generalization and Low Sample Regime
In our simulations, we also consider an additional thought experiment, where we reverse the weak and strong models. That is, in the non-realiann non-rcalizable case with pretraining (Figure 2b), we can have H. be the class of 8-layer MLPs, and H. be the class of 2-layer MLPs. Similarly, in the case with perturbations (Figure 2c), we can set ow = 0.01, and a₂ = 0.05. In this case, because the weak models have now become powerful, and can represent the true data well, the weak labels are essentially the true labels. Hence, if we were to obtain the same plots, we would now expect the misfit on weak labels to essentially correspond to the loss in accuracy of the strong model on true data, compared to the weak model. This is confirmed in Figures 3a and 3b: the plots are mirror reflections of Figures 2b and 2c!
Now, suppose that we are in setting where the number of samples available for the representation learning task is scarce. Concretely, consider the original setting of Figure 2b with H.p and H. back to being 2-layer and 8-layer MLPs respectively. Recall that for learning the representations hww ha
8

---

Hidden dimension
Weak error Misfit
True error of weakly-supervised strong model
96
0.8969 + 0.0327
1.0713 + 0.0489
48
0.9731 + 0.0707
1.1293 + 0.0418
24
1.0331 + 0.0449
1.1204 0.0261
64
1.0619 + 0.0441
1.1436 0.0124
32
1.0624 + 0.0527
1.1302 + 0.0220
16
1.1456 = 0.0276
1.1950 + 0.0484
12
1.1499 + 0.0177
1.1869 0.0297
1.1958 0.0194
1.2396 0.0310
Table 1: Heuristic rule choose among different yeakly-supervise models finetumed on Lipop: choose the strong model that has the smallest difference (averaged across the 3 splits) between weak model error and misfit (= is the std across splits). As we see, this model has the smallest true error.
-063
Gain in arcuracy
Gain accurayy
-008-
-025
-032
Misfit labels
Misfi weak labels
Mixft weak labels
(a) Non-realizable (pretraining).
(b) Non-realizahle (perturbation). (c) Non-realizable (pretraining). few = ht + N(O, 0.01"), hi' h": 5-layer MLP. lew: 2-layer MLP.
ht: 5-layer MLP. l::: 8-layer
MLP, le: 2-layer MLP.
+ +, 0.055). ). h" 5-layer MLP. h,; 8-layer MLP. 5, N. 250.
Figure 3: Strong-to-weak generalization. The roles of the weak and strong models have reversed.
we sampled T = 10 finetuning tasks f(),..., F., and obtained N, 2000 samples labeled according to each f(t) o ht. Now instead, consider setting T N, = 250. The number of samples N2 in the weak model finetuning and teek--o-- supervision stages is still maintained at to-sttong supervision pipeline for this parameter setting. The
N, 2000. We run the entire rationale is that, when the representation learning task is data-deprived, the weak model, by virtue of being simpler, learns better representation than the strong model, which is this is what we observed, as shown in Figure 3c. Observe that the trend in the plot is very similar to Figures 3a and 3b, where we had explicitly swapped the weak and strong models. This suggests that in the low-sample regime too, the weak and strong models have reversed roles. Thus, the definition of weak and strong models the framework of weak-to-strong generalization should not solely be based on expressive power; instead, these roles should be assigned based on the quality of representations.
complex. Indeed,
Conclusion
Employing a representation--teorrtiee perspective, we characterized the gain in performance in weak- to-strong generalization. Our results apply in the setting of learning real-valued functions with the least squares loss, where the strong model learns the finctuning task by optimizing over a convex set of functions. We quantify the gain in accuracy of the weakly-supervised strong model over its weak supervisor in terms of the misfit between the strong and weak models.
Our work has natural limitations. Our theorems notably do not apply when the set from which the strong model learns the finetuning task is not convex. Nevertheless, our experiments Appendix D do suggest that our results should (at least qualitatively) hold even beyond the convex case. Our work also does not address classification tasks (see also Appendix E), and would interesting to see similar results could be obtained for more general loss functions. Finally, while we do demonstrate results on real-world datasets, we anticipate that significantly larger-scale experiments on regression datasets used to train modern AI models will yield further interesting insights.

---

Acknowledgments and Disclosure of Funding
This work is supported by Moses Charikar and Gregory Valiant's Simons Investigator Awards. The authors would like to thank Percy Liang and Tengyu Ma for helpful discussions, and also the anonymous reviewers for engaging with the paper and pointing out references.
References
[BCNM06]
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-547 2006. 2.1
[BFSV19] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, (093))1096611111 108, 2019. 5.2
[BIK Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to- strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:23/2.093390, 2023. (document), 1, 2,
[CCX22]
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge. Technical report, Alignment Research Center (ARC), 2022. 3
[CLB + 17] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. 1
[DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0e 04805, 2018. 5.2
[DLM01]
Sanjoy Dasgupta, Michael Littman, and David McAllester. Pac generalization bounds for co-training. Advances neural information processing systems, 14, 2001. 2.1
[FE + 20] Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and delaintr auxiliary tasks. arXiv preprint arXiv:20/ 1. 13230, 2020. 5.2
FFT 18]
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pages 1607-1616. PMLR, 2018. 2.1
[FMB+22] Alex Franklin, Maggie, Meg Benner, Natalie Rambis, Perpetual Baffour, Ryan Holbrook. Scott Crossley, and ulrichboser. Feedback prize english language learning. https:// kaggle, com/ ctwpptiiiioss//eeeaaaaaaaaaa porirdbbcc--iiieeeee. english-language learning. 2022. Kaggle. C.1
[Fre95]
Yoav Freund. Boosting a weak learning algorithm by majority. Information and computation, 121(2):256-285, 1995. 2.1
[FS97] Yoav Freund and Robert E Schapire. A decision-theorett generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139) 1997. 2.1
[GYMT21] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 17896):1889-88811 2021. 2.1
[Haz16] Elad Hazan. Introduction to online convex optimization. Foundations and Trends@ in Optimization, 2(3-4):157-325, 2016. 4
[HVD15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint (503:05500.2255 2015. 2.1
[JL++24] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak- to-strong correction. arXiv preprint arXiv:2402.024111 2024. 2.1
10

---

[JQC 23]
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He. Jiayi Zhou, Zhaowei Zhang, al. Ai alignment: comprehensive survey. arXiv preprint 23100. /9852, 2023. 2.1
[KB14]
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint (rXiv:1412.6980) 2014. F
[KR+ 22] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In Interna- tional Conference on Learning Representations, 2022.
[LA16]
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2016. 2.1
[LT13]
Michel Ledoux and Michel Talagrand. Probubility in Banach Spaces: isoperimetry and processes. Springer Science & Business Media, 2013. A
[LVF*20] 20] Hang Le, Loic Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. Flaubert: Unsupervised language model pre-training for french. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2479-2490, Marseille, France, May 2020. European Language Resources Association. C.2
[L, Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: ste--ng data filtering for fast instruction-tuning. arXiv preprint (xXix:2442.005530 2024. 2.1
[Met16]
Cade Metz. In two moves, AlphaGo and Lee Sedol redefined the future. https: //wp:: wired. com/2016/03/taa m0000//////tooooee----e-ee 2016. 1
moves moves-alphago -ee- reedol-redefieed-e
[MFB20] Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020. 2.1
[MMS+20] Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoit Sagot. Camembert: tasty french language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. C.2
[Ope23a] OpenAl. Gpt-4 technical report, 2023.
[Ope23b]
OpenAL. Introducing Superalignment. introducing-superall.
https://openai. cow..log.ooo/////
2023. 2.1
[R&D22]
La Javaness R&D. BERT and Transformers. cerssinngbbee--ee bert- wtw.t-rraarrrmrraaa1-a------ serression-with-tex.e in 2022. C.2
Regression with Text Input
Using
https://lajavanes .ewerrannsssmedddummmm.mm medium com/
[RWC+19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[SYS+24] Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint XXXi:22403.09472 2024. 2.1
[TCLT19] Iulia Turc, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read stu- dents learn better: On the importance of pre-training compact models, arXiv preprint arXiv:1908, 08962v2, 2019. C.I
[TJ220] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852-7862. 2020. 1, 5.1, A
[TV17] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017. 2.1

---

[Wan20] Shen Wanxiang. ChemBench: The molecule benchmarks and MolMapNet datasets. https: https://github /gthhtgtthu..... www.coo//cssaae wwap//henwaxiiiammmm ang/ ChemBench. 2020. 5.2
[WRF+ 18] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9()::13--330 2018. 5.2
[WSCM20] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2020. 2.1
[WTB*22] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[WZ17] Wei Wang and Zhi-Hua Zhou. Theoretical foundation of co-traiti co-training and disagreement- based algorithms. arXiv preprint arXiv:1708. 04403, 2017. 2.1
[XLHL20] Qizhe Xie, Minh-Thang Luong, Eduard Hovy. and Quoe V Le. Self-training with noisy student improves imagenet classification. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10687-10698, 2020. 2.1
19] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International conference machine learning, pages 7164-7173. PMLR, 2019. 2.1
[ZCBS23] Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:231.115710, 2023. 2.1
12

---

A Non-realizable Weak-to-Strong Generalization
Proof of Theorem 2. The setting under consideration is depicted in Figure 4.
ooh.
fa
V, - o
Figure 4: Non-realizable weak-to-strong generalization where f h* & Vs, and we use a finite sample to perform weak-to-strong supervision. The Pythagorean theorem, along with uniform convergence and triangle inequalities, yield the desired result.
Let
A dp(fow o har o ha)
B dp(fow o has fw o kw)
C df(fw o lwss fo o h2))
D df har f o h²)
E = dp(Sew her oht)
F df o h f o lt))
G dplfow heg f ht)
H dfllow o hos fow o ha)
dpl hes fw hw).
Note that by virtue the range of f fw and all functions Fg being absolutely bounded, dp also bounded above by constant.
We want to show that G F ( Cr,
From the proof of Theorem I, we know that
C A + B.
(12)
But note also that by "triangle inequality" (Claim 5),
VF. < VA+
A A
D D+
(13)
Combining (12) and (13), we get
E - BB2ADD.
(14)
By the same triangle inequality argument,
(15)
CSD+F+2VDF
(16)
Substituting (16) in (14), get
ESF- 2D + 2v DF + AD
(17)
13

---

By uniform convergence argument (Lemma 4), we have that with probability at least 1 å over the draw of (1),. (on, Un)) that were used to construct fsw>
+0 (vV 1og(1/0)
Cr. n
I I+B+0
(18)
72
Substituting (18) in (17), we get
(Voz
Cr.
log (1/)
E 2VDF + 2V AD AD
(19)
Because few h2 is the projection of fu o bn onto Va, we know (e.g., by the argument in the proof of Theorem 1) that
12 H B.
(20)
Combining (18) and (20), we get
CJ.
1og(1/6)
HSO
+0
(21)
Finally, by another triangle inequality, note that
VC V V VH
G E + + +VE
(22)
Substituting (19) in (22), we get
CF.
log(1/8)
+0
+ + + +
log(116)) #1
CF.
SF-1+20 +2 AD + 2VE
0
+0
(from (21))
Cr.
1 +0
log(1/8)
< F - - +0
where in the last inequality, substituted D used that dp is bounded above by constant, and instantiated asymptotics with respect to E - and n - DC.
Lemma 4 (Uniform Convergence). Let 11)........ (In, Un) be an i.i.d. training sample, where each F4 P and yi g(2;) for some unknown target function g- For fixed strong model representation hg, let
feF, n [f ho(x:))))) - y1)2
few = argmin JEF, df o has g),
Assume that the range of and functions in F, is absolutely bounded. Then, with probability at least 1 over the draw of (x1, v1)..... (In: Gn), we have that
|do(fow o ho.9) - dp(fow o ho, hh. 9) (
+ +0
log (10)
Proof. Given sample (I1, .1),..,, ... (Ens Vr) where each 24 ~ P and Bi = g(xz), define
dp(s.g) Σ (s(az) v1)²
14

---

for any function s. Then, we have that dp(ssw h).9) dp(suw 1).9) dp (Sso hssg) (fsr o has
+ dp(Son o (l..)) - dp (psww o haigg)
+ dp(fow o ha: 9) dp(few ho.9)
(23)
By the definition of fmm.. the second term b & in (23). The terms and measure the difference between the empirical distance and true population distance, and can be controlled by standard uniform convergence argument, in [TJJ20, Theorem 2J. For completeness, we sketch the main parts of the argument here.
Let S = (11....)....... (In, Bn)}, where T; ~ P and =(()). Using McDiarmid's inequality, and by double-sampling and symmetrization argument, first holds that with probability least 1 - 6,
sup Idp(t (f o h2.g) - df h, g)l s (R. h,))) + - (V SEF,
log(1/6)
n
where (I a h2)) is the Rademacher complexity of the loss class of F, heg
1 Σ E: (f:))( -;)) JEF, n i=1
R2 (IFFs h.)) Es sup
We then use the assumption that the range of g and Fs is absolutely bounded, which implies that the squared loss function is both bounded and Lipschitz both arguments. This allows us to use the contraction principle [LT13, Theorem 4. 12]
move from the Rademacher complexity of the loss class I(F o h.) to that of F, o h, itself, and claim that with probability at least 1 5,
sup [ ha, 9) - dpls o ha, g)1 s OR o h.)) +0 log(1/6) SEF.
(24)
n2
Finally, the Rademacher complexity RR,(Fs h2)) can be upper bounded by a quantity known as the worst-case Gaussian complexity of Fg; in any case, for a majority of parametrie function classes
Fs, this quantity scales as v Cre, where Cx, a constant capturing the inherent complexity of F.- Plugging this into (24) yields the desired bound.
Claim ("Triangle Inequality"). For any functions f,g, h,
Vdp(f.9) S vdo(f,h) + dh
Proof.
(Vdo(f,h)+ + ddp(h.)) = -f ( g(x))²
g(x)) (25)
but also,
(r(l.g) = En-p ((nn))))n-((-----eeee)))))))>))) g(x))²
= =EENp (f()) h(x) h(x)+ +(x)) h(x))
EENp h(x))²² Ex-p(h(z) - +2 En-p((((r) (x)))((()) - ((x)))
- -(Vdr(f,)) + V de (h, +2. Eewp((f())) ((x))( g(x))]
- -2V Enappppp (f(z) ( g(x))²
(using (25))
$ (Vdo(f,h) V dp (h. (h.
where the last step uses the Cauchy-Schiee Schwarz inequality.
15

---

B Heuristic to Choose Among Weak Models
Similar to Table report additional results the ESOL and FreeSolv datasets about heuristically choosing amongt different weakly-supervised models based on the one that minimizes the upper bound given by Theorem (upto error terms). The numbers are averaged across 3 train-fest-validation splits. For ESOL (Table 2), weakly supervising with the weak model that results in the smallest difference between weak error and misfit also results in the strong model with least error. This was not exactly the case with FreeSolv (Table 3); however, we can still see the general trend that weak models that exhibit smaller upper bound also tend result strong models that have better errors. We note that the upper bound in some cases is negative, which suggests that the error terms (as in Theorem 2) are non-negle
Hidden dimension
Weak error Misfit
True error of strong model trained on weak
96
0..880 0.4314
1.3914 0.0906
48
-0.634 0.1884
1.4565 0.1860
32
00.3016 + 0.5698
1.6583 0.2160
64
0..2450 + 0.7978
1.5232 + 0.2291
12
0.0083 = 0.2838
3.4396 = 0.1433
0.2582 0.1028
1.8794 0.2185
0.2618 0.2065
3.4200 0.0753
24
0.4904 + 0.1188
1.6342 + 0.3086
Table 2: ESOL
Hidden dimension
Weak error Misfit
True error of strong model trained on weak
32
0.1588 1.0254
4.9967 0.5656
96
1.0896 + 1.0111
4.7828 + 2.2787
64
2.9062 2.8130
3.8404 0.5906
2.9758 2.8664
9.5102 2.4511
48
3.0038 1.3063
4.5212 0.3936
16
4.8560 + 2.5910
7.3637 + 1.1048
24
6.4952 + 4.9463
6.5863 + 1.2709
6.7586 0.9997
10.4474 = 1.6261
Table 3: FreeSolv
C Experiments on NLP Datasets
We include additional experimental results on NLP regression tasks corresponding to two publicly available natural language datasets. The gain-misfit plots in these experiments largely follow the same trend as in Section 5.1 and Section 5.2.
C.1 Essay Scoring
We use the dataset from the Kaggle competition "Feedback Prize English Language Learning" [FMB+22] conducted 2022. Here, the task is to assign a score to essays written by 8th- 12th grade English Language Learners (ELLs). An essay is assigned a score from 1-5 on each of 6 rubrics: cohesion, syntax, vocabulary, phraseology, grammar, conventions. Thus, a separate regression task be obtained by treating each of these scores the target label. The data source provides labels only for the train split; hence, we randomly do a 70-30 split of this data into train and test data.
We fix the strong model representation h2 to be standard pretrained prising 10M parameters. We vary the weak model representation hyp amongst different pretrained miniature BERT architectures [TCLT19]. Namely, vary the hidden size H {128, 256, 512, 768)
model com-
16

---

and the number of transformer layers L E {2, 4, 6, 8) (the number of attention heads is always fixed H /64). Pretrained weights for each of these is available online.
We perform the entire weak-to-strong supervision experiment identically as in Section 5.2-the gain-misfit plots are shown in Figure 5.
100
012
PATL 147
04412
8.10
Galin in accuracyy
0.28
8.6
Gain in
1--121-2
1.00
14224
825
033
0.9
CM
036
03
018
0.6
045
027 0.0
355
031 015
Misiit on weak bebe
Misfit on weak labels
Misfi on week labels
(a) cohesion
(h) syntax
(c) vocabulary
0.50
Calin in arrurwy
0.55
thodle
Gain acCuraCY
0.2
14452 0.10
0.3
6.25
L14
100
1-5
10412
1200
ain R2e
Misfit on weak abbl
Misfit on wesklabels
Misfit weak woaklabels
(d) phraseology
(c) grammar
(f) conventions
Figure 5: Results on the Essay Scoring dataset. Each plot corresponds to the task of predicting the score based on a different rubric.
C.2 French Public Service Reviews
We also perform experiments French language collected from Google Maps and Trustpilot which accompa- nies the blog post of [R&D22] available https://lajavaat
dataset of public service reviews in weteessiii-vitttteeeeeee te--r---mmp---ssi---------------
medium. com/ 711.err3417115555111331 task assign score in 0-4 the reviews, indicating whether is negative or positive. The data source provides train and test splits.
The
We use pretrained models from the CamemBERT [MMS+20] 20] and FlauBERT [LVF+20] family for this experiment. We fix hgg to be camembert-large (335M parameters), and range hy amongst camembert- base, bass,, camembert-base-cone
camembert- base- wiiise-wiiddda-4gbb camembert- 4gb, camembert- naneebber-bas--cceee-eo base-ccnet-4gb, saubert-small-casee flaubert-bart basbeett-bse-aseddd and flaubert- bastrttbbee-unee unceeed The results can be seen in Figure 6.
conamaerraatocor aetira padie-ig cememaen mencrmes loe Bessen asrcaved
Gain in accuracy
Reshers uncerd
Misfit labels
Figure 6: Results on the French Reviews dataset.
17

---

D Non-convex Fs
Recall that in our Theorems I and 2, crucially require the set F, of finetuning tasks from which the strong model learns to be a convex set. Even in our experiments (Section 5), the set F, is simply the set of linear functions (which is convex set).
In this section, we instead consider the scenario where the set Fs is not convex. Concretely, consider the setting of the synthetic experiment in Figure 2b from Section 5.1, where we obtain the strong and weak representations via pretraining. the weak model is 2-layer network and the strong model an 8-layer network. Recall that the set F. from which the strong and weak models learnt was the set of linear functions mapping IRLG -> R (this was also the ground-truth set from which the tasks were being generated from). Here instead, activation & following the linear map. Specifically, we consider the set F, to be
consider the scenario where there is an additional non-linear
F. = {g The non-linear activation renders the set F. to no longer be a convex set, and our theorems do not directly apply this case. Thus, if we were to repeat the whole weak-to to-t-on generalization experiment for this Fs, we have priori reason expect the characterization of the gain in terms of the misfit. However, we observe that this characterization continues to hold for the three (popular) choices of d that we tried, namely tanh, relu and sigmoid. The plots are shown in Figure 7. As we can see, even when the set Fg clearly not convex, the gain is well-characterized by the misfit, suggesting that the result from Theorem may hold even under more general settings.
&o f f.fix is linear).
6.5
E37
Gainn in accuracy
Gain in accuracy
6.2M
0900
0.04
0000 labols
9.00
041
062
0.00
6.5
300
600
si
Misfit
wwwk labelss
weak labels
Φ sigmoid
tanh
(c) 6 relu
Figure 7: Weak-to-strong generalization when F, is non-convex set.
E
Discussion Regarding Extensions to Classification
While our weak-to-strong generalization theory is tailored to regression, we do believe that the qualitative conclusion of our work should also extend other settings. including classification. In particular, as evidenced by numerous plots on student-supervisot disagreement in [BIK 23], the performance gain in classification ought also be (mathematicallyy characterized, at least some extent, by the misfit between the weak and strong model; however, additional reaoonaalreegeeeee regularization terms in the objective (along with other error terms) seem be necessary. This is supported by Figure 8 in [BIK BIK] which shows that adding auxiliary losses in the objective helps reduce student-supervisor agreement as compared to naively finetuning (without any auxiliary loss) to the weak labels, and thereby improves the perfomance of the strong student model.
Of particular relevance here is also the study in Appendices E.1, E.2 in [BIK+233, which studies the qualitative structure of weak-to-strong disagreement in the setting of classification. Namely, the question studied there is: do different forms of disagreement between the strong and weak model (e.g., random eттотs vs correlated errors), for the same weak model accuracy, lead to differing beak-to-strong generalization? While this nuance arises in the setting of classification, it may be worth noting that in the setting of regression that we consider, projection onto the convex set only results in movement towards the true function, assuming realizability in this sense, any misfit is only ever "in the correct direction". and its full quantitative benefit is realized. Thus, such qualitative difference among different types of misfits does not manifest for us. Nevertheless, while true that in other settings, the nature of disagreement might matter, the general principle of decreasing student-supervisor imitation (alternatively, increasing misfit) to foster weak-to-stron tt-sonnn generalization, either via auxiliary losses otherwise, does constitute a significant theme in the work of [BIK+ 23].
18

---

Implementation Details
All our synthetic experiments were run on a personal MacBook Pro 2021 with an Apple MI Pro Chip (10 CPU cores) and no GPUs. All the gradient descent optimization procedures (pretraining tasks to obtain hw; hgg weak model finetuning, strong model finetuning on weak labels) used the Adam optimizer [KB [KB14], with a batch size of 32, learning rate of 10 and 1000 epochs. We used the same random seed in our synthetic experiments. All experiments completed execution within hours.
The experiments on MolBERT used 2 GPUs with 8 GB memory on an internal GPU cluster. All ex- periments completed execution within 6 hours. We reuse much of the pretraining as well finetuning codebase from the MolBERT repository (https: / //gittttbbb.// com/ main). The pretraining GuacaMol dataset as well weights for full-sized BERT model pretrained on this dataset can be downloaded from the repository. The MolBERT repository is under the MIT license, and the ChemBench repository is under the Python Packaging Authority license.
/tre//
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim quantification of the gain in performance in weak- to-stoo generalization in the abstract and introduction, and our results (Theorem 1, Section match our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We are upfront about our assumptions throughout the paper, and also elaborate the limitations of our work in the second paragraph of the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[Yes]
Justification:
We state all the assumptions in our theorems, and also provide complete
proofs/references.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have included all the details of our experiments in Section 5 and Ap- pendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: A Python notebook representative of our main experiment (Figure 2a) is available https:// http//ggthtbb.t com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Seaiii//gidess//oeeee wwwpabiisssiooooooooooooo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. htc//ublic//ae wwwlcc/gggss//eeee htt////ooeubiseeeeee wwwninisssispooooooioooooii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included all the details of our experiments in Section 5 and Ap- pendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We repeat 100 trials of each synthetic data experiment, and show results for each. We also report standard deviations across different splits for our MolBERT experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide thorough implementation details about all our experiments as well as compute resources in Appendix F. The full research project did not require tangibly more compute than the experiments reported in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the cothiieiisiininnncc
NeurlPS Code of Ethics https: http///eerripp.eee
Answer: [Yes]
Justification: Our work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper studies weak-to-strong generalization, and provides a characteri- zation of the gain when strong models are trained on weak models. We do discuss ways in which our analysis might lead to algorithmic insights that can guide the choice of weak models used to train strong models for better accuracy on finetuning tasks. We do not
23

---

anticipate malicious ways in which this characterization could be used to guide strong models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not believe that our work poses any such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets used in the paper are properly cited and licenses have been men- tioned where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
24

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcodeccooee.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not entail crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable for the same reason above.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "UMFVONRxlw", "reviewer": "Reviewer_oUtk", "summary": "This paper provides bounds for weak-to-strong generalization, where a strong student model is trained on the labels produced by a weaker teacher model. The authors prove that in a regression setup, under certain assumptions, the strong model gains over the weak model's accuracy by an amount equal to the *disagreement* between the strong and weak model. This gives a natural generalization bound for weakly-supervised regression models and also gives a selection criterion for which weak model to choose in practice, which the experiments show leads to good empirical performance.", "questions": "- L71 \"Next, we imagine that the weak model sees data labeled by the target function $f^* \\circ h^*$, and after finetuning, learns some arbitrary function $f_w \\circ h_w$.\"\n\n    Is this assumption really necessary? Can't we just assume we are given a weak predictor with some error rate? This limits the setup significantly to weak teacher models that are fine-tuned on ground-truth data.\n\n- What happens in the current theory if technically the strong model hypothesis class $\\mathcal{F}_s$ technically contains a function that can exactly fit the weak classifier $f_w \\circ h_w$, but due to regularization this doesn't occur? This is often the case in practice, where regularization terms are required to avoid overfitting to the weak labels. Can the theory be modified to include a regularized version of $\\mathcal{F}_s$?", "limitations": "The paper already discusses what are in my view its main limitations: (1) it only applies to regression and (2) it only applies to settings where the strong model hypothesis class $\\mathcal{F}_s$ is convex.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proves a clean and intuitive theory for weak-to-strong regression.\n\n- Empirical results showing that the proposed bounds are tight (in fact, almost exact).", "weaknesses": "- The results of WSCM20 are not properly contextualized. Their analysis is *not* limited to a self-training scenario and applies for any student model learning from an arbitrary teacher, including a student that is more powerful than the teacher.\n\n- The paper is missing a discussion of and citations to relevant work in other semi- or un-supervised settings that bound generalization error in terms of the disagreement between two classifiers, such as [1], [2], and especially [3].\n\n[1] https://arxiv.org/abs/1708.04403\n[2] https://arxiv.org/abs/1901.04215\n[3] https://papers.nips.cc/paper_files/paper/2001/file/4c144c47ecba6f8318128703ca9e2601-Paper.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reading our paper, and for your review and comments. We are really glad that you like our work!\n\nWith regards to the points you bring up:\n\n> The results of WSCM20 are not properly contextualized. Their analysis is not limited to a self-training scenario and applies for any student model learning from an arbitrary teacher, including a student that is more powerful than the teacher.\n\nThank you for pointing this out. We will clarify that the analysis of [WSCM20] applies to arbitrary student-teacher settings (albeit under expansion assumptions and a consistency loss) in the updated version!\n\n> missing a discussion of and citations to relevant work in other semi- or un-supervised settings that bound generalization error in terms of the disagreement between two classifiers\n\nThank you for pointing out these references on disagreement-based analyses. We will be sure to include them in the revision.\n\n> L71 \"Next, we imagine that the weak model sees data labeled by the target function $f^* \\circ h^*$, and after finetuning, learns some arbitrary function $f_w \\circ h_w$.\" Is this assumption really necessary? Can't we just assume we are given a weak predictor with some error rate? This limits the setup significantly to weak teacher models that are fine-tuned on ground-truth data.\n\nYou are right, our results broadly hold under access to any weak predictor. Note that we do mention in our statement of Theorem 1 that $h_w$ is any weak representation, and $f_w$ is any (arbitrary) predictor (so that $f_w \\circ h_w$ is any arbitrary predictor). Nevertheless, we will rephrase the sentence in the introduction to make it clear that the weak predictor can really be any arbitrary predictor.\n\n> What happens in the current theory if technically the strong model hypothesis class $\\mathcal{F}_s$  technically contains a function that can exactly fit the weak classifier $f_w \\circ h_w$, but due to regularization this doesn't occur? This is often the case in practice, where regularization terms are required to avoid overfitting to the weak labels. Can the theory be modified to include a regularized version of $\\mathcal{F}_S$?\n\nThis is a great question. In the setting considered in our paper, if $f_w \\circ h_w$ can be represented exactly as $f_s \\circ h_s$ for some $f_s \\in \\mathcal{F}_s$, and we were to perform naive finetuning without any regularization, the strong model will exactly converge to $f_w \\circ h_w$, and we will see no weak-to-strong generalization. As also suggested in the work of Burns et al., 23, auxiliary “confidence” losses serve as regularizers to mitigate this phenomenon. Note however, that our theory heavily uses the fact that the minimizer of the (unregularized) mean squared error is in fact the *projection* of the weak model onto the convex set—this is no longer true with a regularization term. However, there is a natural duality between regularized objectives and constrained optimization problems. An interesting future direction then would be to characterize the regularization terms that constrain the space of optimization to still contain the target function, but exclude the weak model itself, so as to provably mitigate student-supervisor imitation.\n\n---\n\nPlease let us know if we can help answer any other questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8kjdcKPXGN", "reviewer": "Reviewer_8aZZ", "summary": "The paper provides a geometric perspective on weak-to-strong generalization [[Burns et al., 23](https://arxiv.org/abs/2312.09390)]. Specifically, the authors show that if the set of strong model-representable functions the following holds:\n$$MSE(\\phi^*, \\phi^{ws}) \\le MSE(\\phi^*, \\phi^w) - MSE(\\phi^w, \\phi^{ws}),$$\n\nwhere $\\phi^*$ is the ground truth labeling function, $phi^{w}$ is the the weak model labeling function and $\\phi^{ws}$ is the weak-to-strong model labeling function. By labeling function I mean a mapping from inputs $x$ to real-valued labels $y$.\n\nThe authors use this result to argue that the gain in MSE loss in weak-to-strong generalization is at least equal to the _misfit_ term $MSE(\\phi^w, \\phi^{ws})$. There are some experiments verifying this result in practice.", "questions": "Please comment on W1-W4.", "limitations": "Limitations are adequately discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The core result is in fact very simple and intuitive: it's a geometric argument about a projection on a convex set.\n\n2. To the best of my knowledge, this main result was not reported previously in the context of weak-to-strong generalization. It provides a toy model where it's trivial to see that weak-to-strong generalization will hold.\n\n3. The authors show some simple experiments where their theory is applicable: synthetic regression tasks and molecular property prediction.", "weaknesses": "W1. My main concern with the paper is that in my opinion the presentation is very confusing. Specifically, the core result is very simple, but it is presented in a way that took me a while to understand. \n\nMy first issue is that the authors use notation $d_{\\mathcal{P}}$ to denote the mean-squared distance, i.e. $\\mathbb{E}(f(x) - g(x))^2$ and refer to it as distance (line 133). If $d_{\\mathcal{P}}$ was in fact a distance, then Eq. (2) is the inverse of the triangular inequality. This confused me for a while. Moreover, Fig 1 shows a triangle with sides labeled as $A$, $B$, $C$, and states $C \\ge A + B$. The resolution is that $d_{\\mathcal{P}}$ is a square of a distance, not a distance.\n\nThen, Theorem 1 is just a combination of (1) law of cosines and (2) and the fact that the angle between the vectors connecting a point to its projection on a convex set, and a vector connecting its projection to any point in the convex set is $\\ge 90$ degrees.\n\nThe proof and the statement of the theorem are in my opinion significantly complicated by the explicit use of representations $h$ throughout. I don't understand what value they add: throughout the paper, the authors always use the same compositions e.g. $f^s \\circ h^s$. Denoting the entire mapping with one letter would simplify the presentation. Currently, the authors also state that they are \"employing a representation-theoretic perspective\" (line 308), but I don't think there is anything added by the parameterization of the model as a function on top of some representations. Possibly the only thing is Claim 3, but it doesn't affect other parts of the paper much.\n\nW2. The result of the paper holds specifically for the MSE loss. Indeed, if $d_{\\mathcal{P}}$ is a distance metric, then Eq.2 is the opposite of the triangle inequality and doesn't hold. Moreover, for the qualitative conclusion\n> Thus, the 148 inequality directly says that the weakly-supervised strong model improves over the weak model by (at least) an amount equal to the misfit.\n\nclearly doesn't hold for the classification settings considered by [Burns et al., 23]: it is possible for the strong model to differ from the weak model on datapoints where the weak model is correct, which would not make it more accurate with respect to the ground truth labels. \n\nW3. In fact, [Burns et al., 23] provide a relevant discussion of _simulation difficulty_ in Appendix E (especially E.1). They also note that in order for the strong model to improve upon the weak supervisor, the strong model should be unable to simulate the weak labels well. A related result is also reported in Fig 8 of that paper. I think it would be good for the authors to comment on how their results connect to these results. \n\nW4. The practical proposition of using the weak model with the highest misfit will not work without further assumptions. This idea is again quite related to the result in Appendix E.1 of [Burns et al., 23]: if the errors of the weak model are not simulatable by the strong model, but the signal is easy to simulate, then we will get very good weak-to-strong generalization. But in general, high weak-to-strong disagreement will not always imply good weak-to-strong generalization.\n\nEven under the assumptions of convexity etc of theorem 1, I believe it is not true that for a given $d_{\\mathcal{P}}(\\phi^*, \\phi^w)$ higher $d_{\\mathcal{P}}(\\phi^w, \\phi^{ws})$ always lead to lower $d_{\\mathcal{P}}(\\phi^*, \\phi^{ws})$. Is that correct?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reading our paper, and for your comments. We address your concerns ahead:\n> W1..presentation very confusing..notation $d_P$ to denote the mean-squared distance..\n\nThe only reason we introduced the notation $d_P(f,g)$ was for ease of reading: it is convenient to have some notation for the long-form term $E_P(f(x)-g(x))^2$. We explicitly state in line 133 that $d_P$ is the “avg squared distance wrt $P$\" (this is already not a valid \"distance\" in the context of metric spaces, which indeed must satisfy the triangle inequality).\n\nWe want to assert that, if anything, we have tried to be as explicit and upfront about the simplicity and intuitive nature of our result as possible. There was no intent to make anything confusing. Nevertheless, we will add a line clarifying that $d_P$ does not satisfy the triangle inequality. We will also consider changing $d_P$ to $MSE$,just as you use in your review, and denoting the sides of the triangle as $A^2, B^2, C^2$ in Fig 1.\n> Thm 1 just a combination of law of cosines and convexity\n\nThis is indeed the right intuition; formalizing it to the space of functions requires a proof, which is what we include. To us, the fact that weak-to-strong generalization (WTSG) (a seemingly complex, and increasingly important phenomenon going ahead) in the concrete setting of regression can be directly traced to the Pythagorean thm for convex sets (a std fact in convex analysis) is both surprising and satisfying. The main contribution of our work is in modeling this phenomenon and formalizing the question. In this capacity, we view simple explanations as a strength and not a weakness, as they help us understand the system better.\n> ...proof complicated by explicit use of representations throughout\n\nThe only reason we frame our results as “finetuning on top of representations” is because this is predominantly the language used for understanding AI models today. Your suggestion about clubbing $f_s \\circ h_s$ to $\\phi_s$ is well-taken; however, if we simply denote the composition by a single function, the result appears more abstract, and the link to WTSG is obfuscated. If anything, our intention was for the reader to appreciate that the working of these models can be elicited at a level where one can instantiate standard mathematical tools.\n> W2. result of the paper holds only for the MSE loss.\n\nIndeed, and this is something we repeatedly make clear to be the scope of the paper (line 67, 133, 309..). This is a first step towards understanding the phenomenon of WTSG, and extending to other settings is an interesting future direction.\n> ...qualitative conclusion clearly doesn't hold for classification...possible for strong model to differ from weak model on points where weak model is correct\n\nIt is a priori not clear that the qualitative conclusion of our work (that the gain in WTSG is effectively characterized by the WTS misfit) does not extend to settings beyond regression. It may very well be possible that an inequality like Eq 2 (with suitable error terms) holds in the classification setting, with $d_P$ measuring (say) the avg binary cross-entropy loss. Moreover, in such a case, it may still be possible that the strong model makes a mistake on a point where the weak model does not; the overall gain in performance will likely stem from the fact that the probability under the data distribution on such points is low, whereas the probability on points where the strong model improves is high. In any case, this is an interesting direction beyond the scope of the present work.\n> W3...Appendix E in Burns et al., 23, where they note that the strong model improves upon the weak supervisor when it is unable to simulate the weak labels\n\nThis is precisely what our main theorem is also saying! Any strong model that suffers a large misfit error when trained (without auxiliary losses) on weak labels (i.e., it is unable to simulate the weak labels well) exhibits non-trivial WTSG! Relatedly, Fig 8 in their paper shows that adding auxiliary losses in the objective helps reduce student-supervisor agreement (alternatively, increase misfit), and thereby improve WTSG.\n> W4...in general, high WTS disagreement will not always imply WTSG\n\nThe study in Appendix E.1,E.2 in Burns et al. 23 is more to do with the *qualitative* structure of WTS disagreement in the setting of classification. Namely, they ask: do different forms of disagreement (e.g., random errors vs correlated errors), for the same weak model accuracy, lead to differing WTSG? \n\nWhile this nuance arises in the setting of classification, in the setting of regression that we consider, projection onto the convex set only results in movement *towards the true function* assuming realizability---in this sense, any misfit is \"in the correct direction”, and its full quantitative benefit is realized. Thus, the qualitative difference among different forms of misfits does not manifest for us. While it is true that in other settings, the nature of disagreement might matter, the general principle of decreasing student-supervisor imitation (alternatively, increasing misfit) to foster WTSG, either via auxiliary losses or otherwise, does constitute a significant theme in the work of Burns et al., 23.\n> Even under the assumptions of convexity etc of Thm 1...\n\nAs our theorem asserts, for the regression setting under the squared loss and with a convex space of finetuning functions, it is (mathematically) true that for a given $d_P(\\phi^*, \\phi^w)$, a higher $d_P(\\phi^w, \\phi^{ws})$ (where $\\phi^{ws}$ is the projection/minimizer of loss) will indeed lead to lower $d_P(\\phi^*, \\phi^{ws})$ (this is verbatim the inequality). In fact, this is also (nearly exactly) corroborated by all our experiments.\n\n---\nWe will definitely add a summary of the discussion above, in relation to Appendix E in Burns et al., 23 in the final revision. We hope that our response helps address your concerns. Please let us know if we can answer any more questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7cZsl5Z4ZN", "reviewer": "Reviewer_322n", "summary": "This paper studies the phenomena of weak-to-strong generalisation (WTSG) from a theoretical angle, aiming to explain why the phenomena occurs. The paper's main theoretical results show that, in a regression setting with a convex function class, the decrease in MSE from the weak to weak2strong model is bounded below by the error of the weak2strong model on the weak labels (the \"misfit\"). The proof of the theorem relies on the pythagoreon theorem for projectsions onto a convex set. The paper then performs synthetic (simulated guassian data) and realistic (molecular property prediction) experiments to test out the theoretical prediction, and find that the decrease in MSE is roughly linearly proportional to the misfit on the weak labels in both settings. They show that this relationship can be used to select which weak2strong model to choose (the one with the highest misfit) correctly. They finally show that in a low-sample regime, \"weak\" models aren't necessarily smaller ones, as larger models may overfit to the small number of samples and hence perform worse. In this setting, their predicted relationship continues to hold, but with the small model being the strong model.", "questions": "-", "limitations": "The authors have adequately discussed the limitations of the work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The weak-to-strong generalisation (WTSG) phenomena is an important one to study and understand. The theoreticaly explanation put forth in this paper is easy to understand, original, and produces novel insight into why WTSG happens. The confirmation of those findings empirically, including in a more realistic setting, strengthens the work's quality and significance. The paper is generally well-written and clear and easy to read.", "weaknesses": "It's unclear why the authors chose the regression setting, when the original work and likely scenario of use is classification setting. I think this difference from the original work should be made clearer.\n\nThe empirical confirmation in realistic data is limited to one setting, which is an unfamiliar dataset to me and likely most of the LLM and NLP community. It would be beneficial to demonstrate the phenomena across more settings, especially more standard ones.\n\nGiving some intuition as to *why* the theoretical result holds from the proof would be beneficial. In particular, pointing out that the inequality achieved relies on the distance measure not being a metric, otherwise the triangle inequality for metrics would force the inequality to be the other way round, if my understanding is correct?\n\nIt's likely that in the setting from the original paper, the weak labelling is within the function class of fine-tuned strong models (where fine-tuning is over all parameters), at least on a naïve interpretation. It would be beneficial to discuss whether the authors think their theoretical results is the reason why WTSG happens in this setting as well (given the assumptions of their theory break), or whether they think WTSG happens in that scenario for a different reason.\n\n### Summary\n\nIn general, I'm in favour of the paper getting accepted, and am giving it an accept (7). I'd be willing to raise my score if additional empirical justification of theoretical claims was made in more varied settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reading our paper, and for your review and comments. We are really glad that you like our work! \n\nWith regards to the points you bring up:\n\n> It's unclear why the authors chose the regression setting, when the original work and likely scenario of use is classification setting. I think this difference from the original work should be made clearer.\n\nRegression with the least squares loss, while arguably being one of the most basic tasks in statistics and machine learning, also turns out to be a setting where we can extract the essence of the weak-to-strong generalization phenomenon in an intuitive, geometric manner — as we show, the phenomenon in this setting effectively boils down to the Pythagorean theorem for projection onto convex sets (a well-studied and standard fact from convex analysis). We do believe that a similar intuition and explanation (in terms of “projections” onto “convex” sets) may also apply to the classification setting (with say the binary cross-entropy loss). It is also worth mentioning that the theory in several previous studies [1,2] on out-of-distribution finetuning is also focused on regression. Nevertheless, we will emphasize more on the difference in our setting (regression vs classification) in the next revision.\n\n[1] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine tuning can distort pretrained features and underperform out-of-distribution.\n\n[2] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts.\n\n> The empirical confirmation in realistic data is limited to one setting, which is an unfamiliar dataset to me and likely most of the LLM and NLP community. It would be beneficial to demonstrate the phenomena across more settings, especially more standard ones.\n\nThe molecular prediction task seemed natural for our setting and also stood out to be a standardized task [1] in computational chemistry for regression over sequential data, with a tractable pre-training dataset, and a good variety of fine tuning tasks to demonstrate the applicability of our results. Additionally, the community had also previously developed specialized transformer architectures for these tasks [2]. That is why we decided to go with this dataset. \n\nAs can also be seen, for example, in Burns et al., 23, a major bulk of NLP/LLM tasks seem to be tailored to classification/generation; nevertheless, we would be happy to include results on a suitable NLP regression task (suggestions welcome!) in the next version.\n\n[1] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning\n\n[2] Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevant auxiliary tasks.\n\n> Giving some intuition as to why the theoretical result holds from the proof would be beneficial. In particular, pointing out that the inequality achieved relies on the distance measure not being a metric, otherwise the triangle inequality for metrics would force the inequality to be the other way round, if my understanding is correct?\n\nYou are indeed right that if the distance measure (average squared distance) was a metric, we would get the reversed inequality. An intuitive reason for the inequality being the in other direction is the textbook cosine law on the sides of a triangle: $a^2 = b^2 + c^2 - 2bc \\cos(\\theta)$, where $\\theta$ is the angle between sides $b$ and $c$. Convexity guarantees that this angle is obtuse, which makes the cosine negative. Hence, $c^2 \\le a^2 - b^2$. Here, $a^2$ is the weak model loss, $b^2$ is the weak-to-strong misfit, and $c^2$ is the strong model loss.\n\n> It's likely that in the setting from the original paper, the weak labelling is within the function class of fine-tuned strong models (where fine-tuning is over all parameters), at least on a naïve interpretation. It would be beneficial to discuss whether the authors think their theoretical results is the reason why WTSG happens in this setting as well (given the assumptions of their theory break), or whether they think WTSG happens in that scenario for a different reason.\n\nThis is a great question, as in such a case, our theory (which works in the setting of regression) would indicate that no weak-to-strong generalization should be exhibited. This is also what the authors of Burns et al, 23 refer to as *perfect student-supervisor agreement/imitation*. To mitigate this (Section 4.3.2 in Burns et al., 23), notice that the authors of that paper suggest incorporating an auxiliary \"confidence\" loss in their finetuning optimization objective to enable weak-to-strong generalization—this serves as a regularizer, and avoids the student (strong) model to naively overfit the weak labels. Note however, that our theory heavily uses the fact that the minimizer of the (unregularized) mean squared error is in fact the *projection* of the weak model onto the convex set—this is no longer true with a regularization term. However, there is a natural duality between regularized objectives and constrained optimization problems. An interesting future direction then would be to characterize the regularization terms that constrain the space of optimization to still contain the target function, but exclude the weak model itself, so as to provably mitigate student-supervisor imitation.\n\n---\n\nPlease let us know if we can help answer any other questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Paper summary**

The paper contributes bounds to characterize weak-to-strong generalization (WTSG) in a regression setting. In this setup, a strong student model is trained on labels produced by a weak teacher. The teacher is trained on the true observed labels. The main result is (slightly rewritten for simplicity)

$$\mathrm{MSE}(\phi_w, \phi_{sw}) \le \mathrm{MSE}(\phi^*, \phi_w) - \mathrm{MSE}(\phi^*, \phi_{sw}),$$

where $\phi^*$ is the ground-truth data generating function, $\phi_w$ is the weak teacher, and $\phi_{sw}$ is the strong model trained on labels produced from the weak teacher. The inequality says that the reduction in error from the strong model (right-hand side) is at least the difference between the weak and the strong model’s predictions (left-hand side), which the authors refer to as “misfit”. Experiments on both synthetic and real data verify the correctness of the bounds.

**Review**

The paper presents clean theoretical bounds to characterize WTSG in a regression setting (oUtk). The results produce novel insights into why WTSG happens (322n), and are intuitive, relying on a geometric argument about a projection onto a convex set (8aZZ). Experimental  results confirm that the bounds are tight (oUtk, 8aZZ).

The rebuttal from the authors was strong and to the point, which eventually convinced the reviewers. Several concerns were raised, including confusing presentation (mainly about the notation), insufficient discussion on relevant results from Burns et al., 2023, limited experiment on real data (only one dataset), and finally limited study of only the MSE loss (8aZZ, 322n). All have been addressed sufficiently. The authors promised to include more experiments on NLP regression tasks.

Of all the concerns, the last point (limited study on only the MSE loss) appears to be the most important. The authors have made it clear in the paper that the results only hold for this setting. In AC’s view, explaining WTSG in, say, classification tasks can be more nuanced than in regression. The cleanliness of the theoretical results and the nice geometric argument are made possible because of the squared loss. Despite the limited setting, explaining WTSG intuitively is of value.


Recommendation: accept. 

Please include more experiments on NLP regression tasks in the next revision.
Decision: Accept (poster)