Quantifying the Gain in Weak-to-Strong Generalization

Authors: moses  charikar
chirag  pabbaraju
kirankumar  shiragur
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:54:02

Abstract: Recent advances in large language models have shown capabilities that are extraordinary and near-superhuman. These models operate with such complexity that reliably evaluating and aligning them proves challenging for humans. This leads to the natural question: can guidance from weak models (like humans) adequately direct the capabilities of strong models? In a recent and somewhat surprising work, Burns et al. (2023) empirically demonstrated that when strong models (like GPT-4) are finetuned using labels generated by weak supervisors (like GPT-2), the strong models outperform their weaker counterparts---a phenomenon they term *weak-to-strong generalization*.

In this work, we present a theoretical framework for understanding weak-to-strong generalization. Specifically, we show that the improvement in performance achieved by strong models over their weaker counterparts is quantified by the *misfit error* incurred by the strong model on labels generated by the weaker model. Our theory reveals several curious algorithmic insights. For instance, we can predict the amount by which the strong model will improve over the weak model, and also choose among different weak models to train the strong model, based on its misfit error. We validate our theoretical findings through various empirical assessments.

Main paper: 
 introduction: 
 Present-day AI models demonstrate incredible capabilities at a variety of extremely difficult tasks. For this reason, they are frequently described as being superhuman, in that it seems hard to imagine a human displaying the same abilities as the AI model. For example, move 37 in AlphaGo's famous victory against Go expert Lee Sedol [Met16] has been described as being beyond the realm of human imagination. In this sense, today's AI models are well on the path of exhibiting new and emergent abilities [WTB + 22]. Ultimately, we want these new abilities to be aligned with what would be beneficial to humanity. This rationale is what primarily guides the training of large-scale AI models through human feedback [CLB + 17]. However, given that we expect AI models to pick up skills that we ourselves don't fully grasp as humans, how can we enable these highly capable models to realize their potential?A recent work by [BIK + 23] shows that not all hope is lost in this endeavor. To model humans as being weak supervisors for increasingly strong AI models, they conduct the following "weak-to-strong generalization" experiment. Suppose we finetune a small language model like GPT-2 [RWC + 19] on data with ground-truth labels for a task. What happens if we then finetune a large language model like GPT-4 [Ope23a] on data labeled by GPT-2, instead of data having ground-truth labels? Would GPT-4 simply overfit to GPT-2's labels and do no better, or would it outperform GPT-2, given that it is inherently a much stronger model? The surprising experimental result is that GPT-4 trained in this manner outperforms GPT-2 when evaluated on the true data, for a variety of finetuning tasks. Note that GPT-4 is able to outperform GPT-2 without ever seeing true labels when it was being finetuned. One plausible explanation for this is that GPT-4 was able to glean the essence of the finetuning task from GPT-2's labels, and since it is fundamentally a stronger model than GPT-2, this knowledge was sufficient for it to outperform GPT-2. 1In this work, we seek theoretical justification for why we might expect to see such a gain in accuracy in weak-to-strong generalization. Concretely, we ask: Does a weakly supervised strong model provably attain smaller error than its weak supervisor, and if so, can this gain be formally quantified?Towards answering this question, we show (Theorem 1) that in the concrete setting of regression, the true error of a strong model trained on weak labels is smaller than the error of the weak model, by at least the error of the strong model on the weak labels itself. We call this latter quantity the misfit between the weak and strong model. Our result can be stated as the following simple principle: Gain in accuracy in weak-to-strong generalization ≈ Misfit between the weak and strong model Intuitively, the misfit quantifies the erroneous knowledge that the strong model does not obtain from the weak model, and hence also the amount that the strong model improves over the weak model. We note that the work of [BIK + 23] does empirically show that the performance gain of the strong model scales directly with its misfit (or disagreement) with the weak model; our result thereby provides a precise quantification of this observation. Key to obtaining our results is a representation-theoretic perspective [TJJ20] towards weak-to-strong generalization. We posit that the main difference between weak and strong models is in the disparity between the quality of their data representations. This disparity in representation quality can manifest, among other reasons, due to a difference in the expressivity and complexity of the weak and strong models, and the amount of pretraining data that they have seen. For example, in the experiments by [BIK + 23], the weak and strong models used are GPT-2 and GPT-4 respectively; the latter is a significantly larger transformer architecture, pretrained on a much larger dataset than the former. As a broader analogy, consider the task of learning a new language. This is an easier task for a multilingual person than a monolingual person. A multilingual person has a richer representation for language, drawing from their knowledge of different syntax, lexical structures, and sounds in multiple languages. With this perspective, we can imagine finetuning tasks to be relatively simple functions (e.g., linear functions) composed with the appropriate representation. For example, if the task is about learning Italian, a suitable "Italian-specific" linear combination of the multilingual's representation of the problem (including features learned from Spanish and French, say) might allow them to better understand the new language, while the same might not work so well for a monolingual whose representation only has features learned from English.Armed with this perspective, we model the task of learning a real-valued finetuning task under the least squares loss in the weak-to-strong generalization framework. We assume that there exists a ground-truth representation h ⋆ of the data, which makes it amenable to learn a finetuning task f ⋆ of interest. We imagine that the weak and strong models come equipped with representation maps h w and h s respectively, which are possibly obtained via pretraining on a corpus of data. Next, we imagine that the weak model sees data labeled by the target function f ⋆ • h ⋆ , and after finetuning, learns some arbitrary function f w • h w . At this point, the weak supervision pipeline begins. The strong model is fed with data labeled by f w • h w (instead of the true labels f ⋆ • h * ), and as part of finetuning, outputs a function f sw from a function class F s , that minimizes the discrepancy between f sw • h s and the data labeled by f w • h w that it sees. Ultimately, we care about the error of f sw • h s with respect to the true finetuning task, namely f ⋆ • h ⋆ . Our main result (Theorem 1) precisely quantifies the gain in the accuracy of f sw • h s over f w • h w in terms of the misfit between them, under the assumption that the set of functions F s is a convex set.2 In many practical applications, the representation map is generally the forward pass of the data through a suitable neural network architecture, and the finetuning task is performed by the last linear layer [KRJ + 22] of the network. In such cases, our assumption that the set F s is convex readily holds true.We validate our characterization of the gain in weak-to-strong generalization through various experiments (Section 5) on synthetic and real-world data. The experiments corroborate our theoretical findings. Namely, we observe that upon performing the above weak-to-strong supervision pipeline, the gain in accuracy of the weakly-supervised strong model over its weak supervisor more or less exactly aligns with the misfit between the weak and strong models (Figure 2 ). We also demonstrate (Section 5.3) that the labels "weak" and "strong" models are nuanced and not solely dependent on expressive power; in fact, in a low-sample regime, a less expressive model produces a higher quality representation and should be considered a strong model. Our theory and experiments lead to several algorithmic insights and open up interesting questions. For example, one algorithmic heuristic that arises from our theory is the following: given access to different weak models, choose to deploy the strong model that achieves the smallest difference between the weak model error and misfit (Table 1 ). Our results also motivate the perhaps counterintuitive algorithmic question of obtaining weak models that lead to large misfits with the strong model. 3 Another possible line of inquiry could look into ensembling across different weak models, and obtaining a gain close to the sum of their individual misfits. At a more philosophical level, this is akin to a superhuman AI model assimilating knowledge from various humans, while correctly identifying and discarding each of their flaws. 
related work: 
 The idea of converting a "weak" learner to a "strong" learner can be traced all the way back to the famous paradigm of boosting [Fre95, FS97] , if not earlier. The recent work by [BIK + 23] frames this problem within the context of superalignment [Ope23b, JQC + 23] which seeks to reliably align AI models smarter than humans to human intent. Thereafter, several works that study the training of a "strong" model guided in some capacity by a "weak" model have emerged. Some of these include instruction filtering by weak models [LZH + 24], easy-to-hard generalization [SYS + 24], weak-to-strong correction [JCL + 24] and weak-to-strong hallucination inducement [ZCBS23] .The weak-to-strong generalization paradigm is perhaps most closely related to the teacher-student model of training [LA16, TV17] (sometimes also referred to as knowledge distillation [HVD15, GYMT21] ), where a student model (typically smaller) is trained using data labeled by a teacher model (typically larger), and possibly some additional ground-truth data. The remarkable phenomenon of the student model outperforming the teacher has been observed in many works [BCNM06, HVD15, FLT + 18]. Most relevant to us are formulations where the student model is equally [FLT + 18] or more powerful [XLHL20] than the teacher model. There has been theoretical work explaining superior generalization of the student in some specialized settings, e.g., the work of [MFB20] where the student also has access to ground-truth labels, or the work of [WSCM20] , which operates under a certain expansion criterion and consistency loss. In contrast, our work does not assume that the student model has any access to ground-truth labels, and also does not incorporate a regularization term in the objective. Finally, the conceptual insight in our work about the performance gain of the student model scaling with its disagreement with the teacher model is closely related to the theory of generalization bounds based on disagreement between different classifiers [DLM01, WZ17, YHY + 19].We assume that the data domain is R d , and assume that there exists a ground truth representation function h ⋆ : R d → R d ⋆ that maps the data x to an enriched representation h ⋆ (x). We assume the existence of pretraining tasks, through which strong models obtain representations of the data from a function class H s : R d → R ds , and weak models obtain representations from a function class H w : R d → R dw . For example, H s can be the class of deep neural networks, and H w can be the class of shallow neural networks. The target finetuning task (composed with the ground truth representation) is denoted as f ⋆ • h ⋆ , and the function learnt by the weak model is denoted by f w • h w . We assume that the strong model learns finetuning tasks from a function class F s : R ds → R, and assume that the set F s is a convex set. The convexity assumption requires that, for any f, g ∈ F s , and for any λ ∈ [0, 1], there exists h ∈ F s such that for all z ∈ R ds , h(z) = λf (z) + (1 -λ)g(z). For example, F s can be the class of all linear functions from R ds to R. However, we do not assume anything about either f ⋆ or f w ; in particular, they need not belong to F s . We denote the marginal data distribution by P. For any two functions f, g : R d → R, we define the distance d P (f, g) = E x∼P (f (x) -g(x)) 2 , i.e., it is the average (with respect to P) squared distance between the images of the functions. 
results: 
 We first state a quantitative version of our main result that characterizes the gain in weak-to-strong generalization in terms of strong-to-weak misfit in the so-called realizable setting. Namely, we assume that the target finetuning task f ⋆ • h ⋆ can be equivalently written as f s • h s for some f s ∈ F s . Theorem 1 (Weak-to-Strong Generalization under Realizability). Let h ⋆ : R d → R d ⋆ be a ground truth representation map, and let f ⋆ : R d ⋆ → R be a finetuning task of interest. Let h s : R d → R ds and h w : R d → R dw be the strong and weak model representation maps respectively. Given some data labeled by f ⋆ • h ⋆ , let f w • h w be the function learnt by the weak model, for some arbitrary function f w : R dw → R. Now, for a convex set of functions F s mapping R ds to R letEQUATION) be the function learnt by the strong model under weak supervision. Lastly, let us assume that there existsEQUATIONOn the left-hand side in (2) is the error of the weakly-supervised strong model on the true data. The first term on the right-hand side is the true error of the weak model, and the second term is the error of the weakly-supervised strong model on data labeled by the weak model (i.e., misfit). Thus, the inequality directly says that the weakly-supervised strong model improves over the weak model by (at least) an amount equal to the misfit. Note again that in practice, a popular way to finetune a pretrained model on task-specific data is by tuning the weights of only the last linear layer of the model. In these cases, F s is simply the set of linear functions, which is convex. We emphasize that neither of f ⋆ or f w need to belong to F s ; as long as the strong model finds the minimizer over a convex set of the loss on the weakly labeled data (as in (1)), the inequality in (2) holds.Next, we relax the realizability assumption that the target task f ⋆ • h ⋆ belongs to the space of functions that the strong model optimizes over. Instead, suppose that by composing h s with functions in F s , it is possible for the strong model to get a small distance ε to the target task. The strong model could obtain such a powerful representation map after having seen an abundance of pretraining data; the realizable case corresponds to ε = 0. We also relax the assumption that the strong model is able to obtain the true minimizer with respect to the data distribution P as in (1) . In reality, we can imagine that the strong model only sees a finite sample labeled by the weak model, and obtains fsw by minimizing the loss over this finite sample. Even with these relaxations, we can show that the same qualitative result as in Theorem 1 continues to hold, upto small error terms.Theorem 2 (Weak-to-Strong Generalization under Non-Realizability and Finite Samples). Let h ⋆ : R d → R d ⋆ be a ground truth representation map, and let f ⋆ : R d ⋆ → R be a finetuning task of interest. Let h s : R d → R ds and h w : R d → R dw be the strong and weak model representations respectively. Given some data labeled by f ⋆ • h ⋆ , let f w • h w be the function learnt by the weak model, for some arbitrary function f w : R dw → R. For a convex set of functionsEQUATIONand suppose that d P (f s • h s , f ⋆ • h ⋆ ) = ε. Now,EQUATION)Finally, assume that the range of f ⋆ , f w and all the functions in F s is absolutely bounded. Then, we have that with probability at least 1 -δ over the draw of (x 1 , y 1 ), . . . , (x n , y n ),EQUATIONwhere C Fs is a constant capturing the complexity of the function class F s , and the asymptotic notation is with respect to ε → 0, n → ∞.As compared to (2), the bound in (5) has two sources of error terms: the first error term of O( √ ε) arises (via standard triangle inequality arguments) due to the non-realizability assumption, and goes to zero as the strong model becomes stronger and more expressive. The latter two error terms arise (via standard uniform convergence arguments as in [TJJ20] ) because the strong model only sees a finite weakly-labeled sample-these terms vanish too as the sample size becomes large.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: A Python notebook representative of our main experiment (Figure 2a ) is available at https://github.com/chogba/wtsg-regression.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
main proof technique: 
 In this section, we outline the proof of realizable weak-to-strong generalization (Theorem 1). The proof of Theorem 2 uses the same main idea and is given in Appendix A. Recall that the strong model learns from a convex set F s : R ds → R of finetuning tasks. Recall also that we denote the strong model representation map by h s : R d → R ds . Let V s = {f • h s : f ∈ F s } be the set of all tasks in F s composed with the strong model representation. We first observe that V s is also a convex set. Claim 3. V s is a convex set.Proof. Fix f, g ∈ F s , and consider f • h s , g • h s ∈ V s . Fix any λ ∈ [0, 1]. Since F s is a convex set, there exists p ∈ F s such that for all y ∈ R ds , p(y) = λf (y) + (1 -λ)g(y). Now, fix any x ∈ R d . Then, we have thatλ(f • h s )(x) + (1 -λ)(g • h s )(x) = λf (h s (x)) + (1 -λ)g(h s (x)) = p(h s (x)) = (p • h s )(x), and hence λ(f • h s ) + (1 -λ)(g • h s ) = p • h s ∈ V s .We are then ready to prove Theorem 1.Proof of Theorem 1. The setting under consideration is depicted in Figure 1 . Since we assume realizability,f ⋆ • h ⋆ ∈ V s . Let A = d P (f sw • h s , f ⋆ • h ⋆ ), B = d P (f sw • h s , f w • h w ) and C = d P (f w • h w , f ⋆ • h ⋆ ). We want to show that C ≥ A + B. Recall that f sw = argmin f ∈Fs d P (f • h s , f w • h w ).In other words, f sw • h s is the projection of f w • h w onto the convex set V s . We can therefore apply the "Pythagorean theorem" for projections onto a convex set [Haz16, Theorem 2.1].Figure 1 :f sw • h s is the projection of f w • h w onto the convex set V s .Concretely, for any g ∈ V s , observe thatEQUATIONBut note also that by definition of projection,d P (f w • h w , g) ≥ d P (f sw • h s , f w • h w ), and hence d P (f sw • h s , g) + 2 • E x∼P [(g(x) -(f sw • h s )(x))((f sw • h s )(x) -(f w • h w )(x))] ≥ 0. (7)Now, fix t ∈ (0, 1), and consider the functionw(t) = f sw • h s + t • (f ⋆ • h ⋆ -f sw • h s ). Namely, for any x, w(t)(x) = (f sw • h s )(x) + t • ((f ⋆ • h ⋆ )(x) -(f sw • h s )(x)). Because V s is a convex set (Claim 3), w(t) ∈ V s . Also, d P (f sw • h s , w(t)) = E x∼P ((f sw • h s )(x) -w(t)(x)) 2 = t 2 • E x∼P ((f ⋆ • h ⋆ )(x) -(f sw • h s )(x)) 2 .Hence, substituting w(t) for g in (7), we gett 2 • E x∼P ((f ⋆ • h ⋆ )(x) -(f sw • h s )(x)) 2 + 2t • E x∼P [((f ⋆ • h ⋆ )(x) -(f sw • h s )(x))((f sw • h s )(x) -(f w • h w )(x))] ≥ 0.Taking the limit as t ↓ 0, we get that 6), and using (8), we obtain the desired resultE x∼P [((f ⋆ • h ⋆ )(x) -(f sw • h s )(x))((f sw • h s )(x) -(f w • h w )(x))] ≥ 0 (8) Substituting f ⋆ • h ⋆ for g in (d P (f w • h w , f ⋆ • h ⋆ ) ≥ d P (f sw • h s , f ⋆ • h ⋆ ) + d P (f sw • h s , f w • h w ). 
experiments: 
 We perform experiments4 on synthetically generated data as well as real-world molecular prediction and natural language datasets to verify the guarantees on weak-to-strong generalization given by our theorems. The results for the natural language tasks are given in Appendix C.We set the target data representation h ⋆ : R 8 → R 16 to be a randomly initialized 5-layer multi-layer perceptron (MLP) with ReLU activations, with input dimension 8 and hidden layer dimension 16. The class F s of finetuning tasks from which the strong model (as well as the weak model) learns is simply the class of linear functions from R 16 → R; F s is thus a convex set (see Appendix D for instances where F s is a non-convex set). The marginal data distribution P in our experiments is always N (0, σ 2 I). To ensure that the data is well-spread, we set σ = 500.Representation Learning. We experiment with two different ways of obtaining the weak and strong representations h w and h s :(1) Pretraining: We randomly sample T finetuning tasks f (1) , . . . , f (T ) ∈ F s . For each t ∈ [T ], we generate data {xEQUATIONj } Nr j=1 , where x (t) j ∼ P and y(t) j = f (t) • h ⋆ (x (t) j ).EQUATIONWe set H w and H s (both R 8 → R 16 ) to be the classes of 2-layer and 8-layer neural networks respectively with ReLU activations and hidden dimension 16. We obtain h w and h s via gradient descent on the representation parameters to find the minimizers in (9). We set T = 10, N r = 2000. Additionally, we also consider the realizable setting (Theorem 1), where we explicitly set h s = h ⋆ , and only obtain h w as above.(2) Perturbations: We also consider another way to obtain the weak and strong representations as direct perturbations of h ⋆ . Namely, we perturb every parameter in every weight matrix in h ⋆ by independent Gaussian noise N (0, σ 2 s ) to obtain h s . Similarly, we obtain h w by perturbing each parameter in h ⋆ by N (0, σ 2 w ). Ideally, we want the strong representation h s to be a closer approximation of h ⋆ than h w . Hence, we set σ s = 0.01 and σ w = 0.05. provides three different train, test and validation splits; multiple points of the same color correspond to weak-to-strong supervision for the same weak model (as specified in legend) across these splits.Weak Model Finetuning. Once the representations h w and h s have been obtained and fixed, we randomly generate M new finetuning tasks f (1) , . . . , f (M ) ∈ F s , and obtain data {x(i) j , y (i) j } N fj=1 for each of these tasks. Here again, x (i) j ∼ P and y(i) j = f (i) • h ⋆ (x (i) j ). We set M = 100, N f = 2000.For each task, we train the weak model on the data generated for the task, to obtainEQUATION)Here, the representation parameters h w are frozen, and f(i)w is obtained via gradient descent. Note again that we are training the weak models on true data labeled by the finetuning taskf (i) • h ⋆ .Weak-to-Strong Supervision. Once our weak models are trained for each finetuning task, we generate weakly labeled data. That is, for each i ∈ [M ], we generate {x(i) j , ỹ(i) j } N f j=1 where x(i) j ∼ P. But crucially, ỹ(i) j = f (i) w • h w (x (i) j ). We now train our strong models on this weakly labeled data. Namely, keeping the strong representation h s fixed, we obtain, via gradient descent againEQUATION)At this point, our weak-to-strong training procedure is complete.Evaluation. For each finetuning task, we wish to evaluate the accuracy of our weak-to-strong model f (i) sw • h s (x) - f (i) • h ⋆ (x)) 2 . (b) Error of the weak model f (i) w • h w on the true finetuning task: E x∼P (f (i) w • h w (x) -f (i) • h ⋆ (x)) 2 .(c) Misfit error of the weak-to-strong model on the weakly labeled data: E x∼P (f(i) sw • h s (x) - f (i) w • h w (x)) 2 .Each of these quantities are estimated from a fresh sample of size N f drawn from P. For each task i ∈ [M ], we plot the difference (b)-(a), namely the Gain in Accuracy, on the y-axis, versus the Misfit (c) on the x-axis. Figure 2a has the results for the realizable case where h s = h ⋆ and h w is obtained by pretraining. Figure 2b has the results for the non-realizable case where both h w and h s are obtained by pretraining. Figure 2c has the results for the non-realizable case where h w and h s are obtained by directly perturbing the weights in h ⋆ . For reference, recall that Theorem 2 indicates that the gain in accuracy is (upto error terms) at least the misfit. The plots in Figure 2 suggest that the gain is more or less exactly the misfit, which is in agreement with our theory!We also validate our conceptual insights on real-world molecular prediction datasets. Specifically, we follow the Quantitative Structure-Activity Relationship (QSAR) task setup in the MolBERT [FEG + 20] paper. These tasks involve predicting physical properties of molecules like solubility, lipophilicity, etc. We consider three regression datasets: ESOL, FreeSolv and Lipop. These datasets are part of the MoleculeNet [WRF + 18] benchmark suite, and have been curated into train, test and validation splits by ChemBench [Wan20] . The MolBERT paper provides weights for a standard-size BERT [DCLT18] architecture (hidden dimension 768, 12 layers, 12 attention heads) pretrained for 100 epochs on the GuacaMol [BFSV19] dataset. We use these weights as the strong representation h s . For the weak representations h w , we run the pretraining pipeline for substantially smaller transformer architectures and lesser compute time. Specifically, we consider transformers with just 2 layers and 2 attention heads, and vary the hidden size in {8, 12, 16, 32, 48, 64, 96}. For each of these settings, we run the pretraining tasks for a mere 2 epochs to obtain different weak representations h w .Once we have the representations h s and h w , we can finetune a linear layer on top of these for each of the three regression datasets. We run the entire weak-to-strong supervision pipeline from above, where we weakly supervise the strong model h s on labels given by each of the weak models h w . The results are given in Figures 2d, 2e and 2f . Again, we see that the gain in accuracy of the weakly supervised strong models is accurately characterized by their misfit on the weak labels.We were also able to see an otherwise useful algorithmic insight in these experiments. Consider a setting where we have at our disposal various weak models, and have performed the weak-to-strong supervision pipeline separately on each of them. We now want to deploy one of the weakly trained strong models; our goal is to choose the one that gets the least error on the true data distribution. Recall that Theorem 2 guarantees that the error of a weakly supervised strong model is upper bounded (upto error terms) by the difference between the weak model's error and misfit. This suggests a natural heuristic: sort the strong models by the difference between the corresponding weak model's error and the misfit, and choose the one for which this quantity is smallest. We observed that this heuristic ends up working quite well-Table 1 shows the numbers for the Lipop dataset, while the results for ESOL and FreeSolv are in Appendix B.In our simulations, we also consider an additional thought experiment, where we reverse the weak and strong models. That is, in the non-realizable case with pretraining (Figure 2b ), we can have H w be the class of 8-layer MLPs, and H s be the class of 2-layer MLPs. Similarly, in the case with perturbations (Figure 2c ), we can set σ w = 0.01, and σ s = 0.05. In this case, because the weak models have now become powerful, and can represent the true data well, the weak labels are essentially the true labels. Hence, if we were to obtain the same plots, we would now expect the misfit on weak labels to essentially correspond to the loss in accuracy of the strong model on true data, compared to the weak model. This is confirmed in Figures 3a and 3b : the plots are mirror reflections of Figures 2b and 2c ! Now, suppose that we are in a setting where the number of samples available for the representation learning task is scarce. Concretely, consider the original setting of Figure 2b Gain in accuracy we sampled T = 10 finetuning tasks f (1) , . . . , f (T ) ∈ F s , and obtained N r = 2000 samples labeled according to each f (t) • h ⋆ . Now instead, consider setting T = 5, N r = 250. The number of samples N f in the weak model finetuning and weak-to-strong supervision stages is still maintained at N f = 2000. We run the entire weak-to-strong supervision pipeline for this parameter setting. The rationale is that, when the representation learning task is data-deprived, the weak model, by virtue of being simpler, learns a better representation than the strong model, which is more complex. Indeed, this is what we observed, as shown in Figure 3c . Observe that the trend in the plot is very similar to Figures 3a and 3b , where we had explicitly swapped the weak and strong models. This suggests that in the low-sample regime too, the weak and strong models have reversed roles. Thus, the definition of weak and strong models in the framework of weak-to-strong generalization should not solely be based on expressive power; instead, these roles should be assigned based on the quality of representations.(b) Non-realizable (perturbation). hw = h ⋆ + N (0, 0.01 2 ), hs = h ⋆ + N (0, 0.05 2 ). h ⋆ is 5-layer MLP. 
conclusion: 
 Employing a representation-theoretic perspective, we characterized the gain in performance in weakto-strong generalization. Our results apply in the setting of learning real-valued functions with the least squares loss, where the strong model learns the finetuning task by optimizing over a convex set of functions. We quantify the gain in accuracy of the weakly-supervised strong model over its weak supervisor in terms of the misfit between the strong and weak models.Our work has natural limitations. Our theorems notably do not apply when the set from which the strong model learns the finetuning task is not convex. Nevertheless, our experiments in Appendix D do suggest that our results should (at least qualitatively) hold even beyond the convex case. Our work also does not address classification tasks (see also Appendix E), and it would be interesting to see if similar results could be obtained for more general loss functions. Finally, while we do demonstrate results on real-world datasets, we anticipate that significantly larger-scale experiments on regression datasets used to train modern AI models will yield further interesting insights.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have included all the details of our experiments in Section 5 and Appendix F.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We repeat 100 trials of each synthetic data experiment, and show results for each. We also report standard deviations across different splits for our MolBERT experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our paper studies weak-to-strong generalization, and provides a characterization of the gain when strong models are trained on weak models. We do discuss ways in which our analysis might lead to algorithmic insights that can guide the choice of weak models used to train strong models for better accuracy on finetuning tasks. We do not anticipate malicious ways in which this characterization could be used to guide strong models.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not believe that our work poses any such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets used in the paper are properly cited and licenses have been mentioned where applicable.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof of Theorem 2. The setting under consideration is depicted in Figure 4 .Figure 4 : Non-realizable weak-to-strong generalization where f ⋆ • h ⋆ / ∈ V s , and we use a finite sample to perform weak-to-strong supervision. The Pythagorean theorem, along with uniform convergence and triangle inequalities, yield the desired result.LetA = d P (f sw • h s , f s • h s ) B = d P (f sw • h s , f w • h w ) C = d P (f w • h w , f s • h s ) D = d P (f s • h s , f ⋆ • h ⋆ ) = ε E = d P (f sw • h s , f ⋆ • h ⋆ ) F = d P (f w • h w , f ⋆ • h ⋆ ) G = d P ( fsw • h s , f ⋆ • h ⋆ ) H = d P ( fsw • h s , f sw • h s ) I = d P ( fsw • h s , f w • h w ).Note that by virtue of the range of f ⋆ , f w and all functions in F s being absolutely bounded, d P is also bounded above by a constant.G ≤ F -I + O( √ ε) + O C Fs n 1 4.From the proof of Theorem 1, we know thatEQUATIONBut note also that by a "triangle inequality" (Claim 5),EQUATIONCombining ( 12) and (13), we getEQUATIONBy the same triangle inequality argument,EQUATIONSubstituting ( 16) in ( 14), we getEQUATIONBy a uniform convergence argument (Lemma 4), we have that with probability at least 1 -δ over the draw of {(x 1 , y 1 ), . . . , (x n , y n )} that were used to construct fsw ,EQUATIONSubstituting ( 18) in (17), we getEQUATIONBecause f sw • h s is the projection of f w • h w onto V s , we know (e.g., by the argument in the proof of Theorem 1) thatEQUATIONCombining ( 18) and (20), we getEQUATIONFinally, by another triangle inequality, note thatEQUATIONSubstituting ( 19) in ( 22), we getG ≤ F -I + 2D + 2 √ DF + 2 √ AD + O C Fs n + O log(1/δ) n + H + 2 √ EH ≤ F -I + 2D + 2 √ DF + 2 √ AD + 2 √ E • O C Fs n 1 4 + O log(1/δ) n 1 4(from (21))≤ F -I + O( √ ε) + O C Fs n 1 4 + O log(1/δ) n 1 4, where in the last inequality, we substituted D = ε, used that d P is bounded above by a constant, and instantiated asymptotics with respect to ε → 0 and n → ∞.Lemma 4 (Uniform Convergence). Let (x 1 , y 1 ), . . . , (x n , y n ) be an i.i.d. training sample, where each x i ∼ P and y i = g(x i ) for some unknown target function g. For a fixed strong model representation h s , letf sw = argmin f ∈Fs d P (f • h s , g), fsw = argmin f ∈Fs 1 n (f • h s (x i ) -y i ) 2 .Assume that the range of g and functions in F s is absolutely bounded. Then, with probability at least 1 -δ over the draw of (x 1 , y 1 ), . . . , (x n , y n ), we have thatd P ( fsw • h s , g) -d P (f sw • h s , g) ≤ O C Fs n + O log(1/δ) n .Proof. Given sample (x 1 , y 1 ), . . . , (x n , y n ) where each x i ∼ P and y i = g(x i ), definedP (s, g) = 1 n n i=1 (s(x i ) -y i ) 2for any function s. Then, we have thatEQUATION)By the definition of fsw , the second term b ≤ 0 in (23). The terms a and c measure the difference between the empirical distance and true population distance, and can be controlled by a standard uniform convergence argument, as in [TJJ20, Theorem 2]. For completeness, we sketch the main parts of the argument here.Let S = {(x 1 , y 1 ), . . . , (x n , y n )}, where x i ∼ P and y i = g(x i ). Using McDiarmid's inequality, and by a double-sampling and symmetrization argument, it first holds that with probability at least 1 -δ,sup f ∈Fs | dP (f • h s , g) -d P (f • h s , g)| ≤ O (R n (l(F s • h s ))) + O log(1/δ) n , where R n (l(F s • h s )) is the Rademacher complexity of the loss class of F s • h s : R n (l(F s • h s )) = E S E εi∼{-1,1} sup f ∈Fs 1 n n i=1 ε i • (f • h s (x i ) -y i ) 2 .We can then use the assumption that the range of g and F s is absolutely bounded, which implies that the squared loss function is both bounded and Lipschitz in both arguments. This allows us to use the contraction principle [LT13, Theorem 4.12] so as to move from the Rademacher complexity of the loss class l(F s • h s ) to that of F s • h s itself, and claim that with probability at least 1 -δ,EQUATIONFinally, the Rademacher complexity R n (F s • h s ) can be upper bounded by a quantity known as the worst-case Gaussian complexity of F s ; in any case, for a majority of parametric function classes F s , this quantity scales asC Fs n, where C Fs is a constant capturing the inherent complexity of F s . Plugging this into (24) yields the desired bound.Claim 5 ("Triangle Inequality"). For any functions f, g, h,d P (f, g) ≤ d P (f, h) + d P (h, g).Proof.EQUATION) but also, d P (f, g) = E x∼P (f (x) -g(x)) 2 = E x∼P (f (x) -h(x) + h(x) -g(x)) 2 = E x∼P (f (x) -h(x)) 2 + E x∼P (h(x) -g(x)) 2 + 2 • E x∼P [(f (x) -h(x))(h(x) -g(x))] = d P (f, h) + d P (h, g) 2 + 2 • E x∼P [(f (x) -h(x))(h(x) -g(x))] -2 E x∼P (f (x) -h(x)) 2 E x∼P (h(x) -g(x)) 2 (using (25))≤ d P (f, h) + d P (h, g) 2 ,where the last step uses the Cauchy-Schwarz inequality.Similar to Table 1 , we report additional results on the ESOL and FreeSolv datasets about heuristically choosing amongt different weakly-supervised models based on the one that minimizes the upper bound given by Theorem 1 (upto error terms). The numbers are averaged across 3 train-test-validation splits. For ESOL (Table 2 ), weakly supervising with the weak model that results in the smallest difference between weak error and misfit also results in the strong model with least error. This was not exactly the case with FreeSolv (Table 3 ); however, we can still see the general trend that weak models that exhibit a smaller upper bound also tend to result in strong models that have better errors. We note that the upper bound in some cases is negative, which suggests that the error terms (as in Theorem 2) are non-negligible. We include additional experimental results on NLP regression tasks corresponding to two publicly available natural language datasets. The gain-misfit plots in these experiments largely follow the same trend as in Section 5.1 and Section 5.2.We use the dataset from the Kaggle competition "Feedback Prize -English Language Learning" [FMB + 22] conducted in 2022. Here, the task is to assign a score to essays written by 8th-12th grade English Language Learners (ELLs). An essay is assigned a score from 1-5 on each of 6 rubrics: cohesion, syntax, vocabulary, phraseology, grammar, conventions. Thus, a separate regression task can be obtained by treating each of these scores as the target label. The data source provides labels only for the train split; hence, we randomly do a 70-30 split of this data into train and test data.We fix the strong model representation h s to be a standard pretrained bert-base-uncased model comprising of 110M parameters. We vary the weak model representation h w amongst different pretrained miniature BERT architectures [TCLT19] . Namely, we vary the hidden size H ∈ {128, 256, 512, 768} and the number of transformer layers L ∈ {2, 4, 6, 8} (the number of attention heads is always fixed at H/64). Pretrained weights for each of these is available online.We perform the entire weak-to-strong supervision experiment identically as in Section 5.2-the gain-misfit plots are shown in Figure 5 . We also perform experiments on a dataset of public service reviews in French language collected from Google Maps and Trustpilot which accompanies the blog post of [R&D22] available at https://lajavaness.medium.com/ regression-with-text-input-using-bert-and-transformers-71c155034b13. The task is to assign a score in 0-4 to the reviews, indicating whether it is negative or positive. The data source provides train and test splits.We use pretrained models from the CamemBERT [MMS + 20] and FlauBERT [LVF + 20] family for this experiment. We fix h s to be camembert-large (335M parameters), and range h w amongst camembert-base, camembert-base-ccnet, camembert-base-wikipedia-4gb, camembert-base-oscar-4gb, camembert-base-ccnet-4gb, flaubert-small-cased, flaubert-base-cased and flaubert-base-uncased. The results can be seen in Figure 6 . Recall that in our Theorems 1 and 2, we crucially require the set F s of finetuning tasks from which the strong model learns to be a convex set. Even in our experiments (Section 5), the set F s is simply the set of linear functions (which is a convex set).In this section, we instead consider the scenario where the set F s is not convex. Concretely, consider the setting of the synthetic experiment in Figure 2b from Section 5.1, where we obtain the strong and weak representations via pretraining, the weak model is a 2-layer network and the strong model is an 8-layer network. Recall that the set F s from which the strong and weak models learnt was the set of linear functions mapping R 16 → R (this was also the ground-truth set from which the tasks were being generated from). Here instead, we consider the scenario where there is an additional non-linear activation ϕ following the linear map. Specifically, we consider the set F s to beF s = {g : g = ϕ • f, f is linear}.The non-linear activation renders the set F s to no longer be a convex set, and our theorems do not directly apply in this case. Thus, if we were to repeat the whole weak-to-strong generalization experiment for this F s , we have a priori no reason to expect the characterization of the gain in terms of the misfit. However, we observe that this characterization continues to hold for the three (popular) choices of ϕ that we tried, namely tanh, relu and sigmoid. The plots are shown in Figure 7 . As we can see, even when the set F s is clearly not convex, the gain is well-characterized by the misfit, suggesting that the result from Theorem 1 may hold even under more general settings. While our weak-to-strong generalization theory is tailored to regression, we do believe that the qualitative conclusion of our work should also extend to other settings, including classification. In particular, as evidenced by numerous plots on student-supervisor disagreement in [BIK + 23], the performance gain in classification ought also be (mathematically) characterized, at least to some extent, by the misfit between the weak and strong model; however, additional regularization terms in the objective (along with other error terms) seem to be necessary. This is supported by Figure 8 in [BIK + 23], which shows that adding auxiliary losses in the objective helps reduce student-supervisor agreement as compared to naively finetuning (without any auxiliary loss) to the weak labels, and thereby improves the perfomance of the strong student model.Of particular relevance here is also the study in Appendices E.1, E.2 in [BIK + 23], which studies the qualitative structure of weak-to-strong disagreement in the setting of classification. Namely, the question studied there is: do different forms of disagreement between the strong and weak model (e.g., random errors vs correlated errors), for the same weak model accuracy, lead to differing weak-to-strong generalization? While this nuance arises in the setting of classification, it may be worth noting that in the setting of regression that we consider, projection onto the convex set only results in movement towards the true function, assuming realizability-in this sense, any misfit is only ever "in the correct direction", and its full quantitative benefit is realized. Thus, such a qualitative difference among different types of misfits does not manifest for us. Nevertheless, while it is true that in other settings, the nature of disagreement might matter, the general principle of decreasing student-supervisor imitation (alternatively, increasing misfit) to foster weak-to-strong generalization, either via auxiliary losses or otherwise, does constitute a significant theme in the work of [BIK + 23].All our synthetic experiments were run on a personal MacBook Pro 2021 with an Apple M1 Pro Chip (10 CPU cores) and no GPUs. All the gradient descent optimization procedures (pretraining tasks to obtain h w , h s , weak model finetuning, strong model finetuning on weak labels) used the Adam optimizer [KB14] , with a batch size of 32, learning rate of 10 -3 and 1000 epochs. We used the same random seed in all our synthetic experiments. All experiments completed execution within 2 hours.The experiments on MolBERT used 2 GPUs with 8 GB memory on an internal GPU cluster. All experiments completed execution within 6 hours. We reuse much of the pretraining as well as finetuning codebase from the MolBERT repository (https://github.com/BenevolentAI/MolBERT/tree/ main). The pretraining GuacaMol dataset as well as weights for a full-sized BERT model pretrained on this dataset can be downloaded from the repository. The MolBERT repository is under the MIT license, and the ChemBench repository is under the Python Packaging Authority license.NeurIPS Paper Checklist We are upfront about our assumptions throughout the paper, and also elaborate on the limitations of our work in the second paragraph of the conclusion. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.We note however that this methodology did not allow GPT-4 to fully reach its performance achieved by training on true data.Note that the functions in Fs need not be convex themselves.In other words, weak models that largely misfit strong models are the ones that most effectively elicit[CCX22] what the strong models already know.https://github.com/chogba/wtsg-regression 


Decision: **Paper summary**

The paper contributes bounds to characterize weak-to-strong generalization (WTSG) in a regression setting. In this setup, a strong student model is trained on labels produced by a weak teacher. The teacher is trained on the true observed labels. The main result is (slightly rewritten for simplicity)

$$\mathrm{MSE}(\phi_w, \phi_{sw}) \le \mathrm{MSE}(\phi^*, \phi_w) - \mathrm{MSE}(\phi^*, \phi_{sw}),$$

where $\phi^*$ is the ground-truth data generating function, $\phi_w$ is the weak teacher, and $\phi_{sw}$ is the strong model trained on labels produced from the weak teacher. The inequality says that the reduction in error from the strong model (right-hand side) is at least the difference between the weak and the strong model’s predictions (left-hand side), which the authors refer to as “misfit”. Experiments on both synthetic and real data verify the correctness of the bounds.

**Review**

The paper presents clean theoretical bounds to characterize WTSG in a regression setting (oUtk). The results produce novel insights into why WTSG happens (322n), and are intuitive, relying on a geometric argument about a projection onto a convex set (8aZZ). Experimental  results confirm that the bounds are tight (oUtk, 8aZZ).

The rebuttal from the authors was strong and to the point, which eventually convinced the reviewers. Several concerns were raised, including confusing presentation (mainly about the notation), insufficient discussion on relevant results from Burns et al., 2023, limited experiment on real data (only one dataset), and finally limited study of only the MSE loss (8aZZ, 322n). All have been addressed sufficiently. The authors promised to include more experiments on NLP regression tasks.

Of all the concerns, the last point (limited study on only the MSE loss) appears to be the most important. The authors have made it clear in the paper that the results only hold for this setting. In AC’s view, explaining WTSG in, say, classification tasks can be more nuanced than in regression. The cleanliness of the theoretical results and the nice geometric argument are made possible because of the squared loss. Despite the limited setting, explaining WTSG intuitively is of value.


Recommendation: accept. 

Please include more experiments on NLP regression tasks in the next revision.

Reviews: review: paper_summary: paper_summary: This paper provides bounds for weak-to-strong generalization, where a strong student model is trained on the labels produced by a weaker teacher model. The authors prove that in a regression setup, under certain assumptions, the strong model gains over the weak model's accuracy by an amount equal to the *disagreement* between the strong and weak model. This gives a natural generalization bound for weakly-supervised regression models and also gives a selection criterion for which weak model to choose in practice, which the experiments show leads to good empirical performance.
strength_weakness: strength_weakness: - The results of WSCM20 are not properly contextualized. Their analysis is *not* limited to a self-training scenario and applies for any student model learning from an arbitrary teacher, including a student that is more powerful than the teacher.

- The paper is missing a discussion of and citations to relevant work in other semi- or un-supervised settings that bound generalization error in terms of the disagreement between two classifiers, such as [1], [2], and especially [3].

[1] https://arxiv.org/abs/1708.04403
[2] https://arxiv.org/abs/1901.04215
[3] https://papers.nips.cc/paper_files/paper/2001/file/4c144c47ecba6f8318128703ca9e2601-Paper.pdf
questions: questions: - L71 "Next, we imagine that the weak model sees data labeled by the target function $f^* \circ h^*$, and after finetuning, learns some arbitrary function $f_w \circ h_w$."

    Is this assumption really necessary? Can't we just assume we are given a weak predictor with some error rate? This limits the setup significantly to weak teacher models that are fine-tuned on ground-truth data.

- What happens in the current theory if technically the strong model hypothesis class $\mathcal{F}_s$ technically contains a function that can exactly fit the weak classifier $f_w \circ h_w$, but due to regularization this doesn't occur? This is often the case in practice, where regularization terms are required to avoid overfitting to the weak labels. Can the theory be modified to include a regularized version of $\mathcal{F}_s$?
limitations: limitations: The paper already discusses what are in my view its main limitations: (1) it only applies to regression and (2) it only applies to settings where the strong model hypothesis class $\mathcal{F}_s$ is convex.
paper_summary: paper_summary: The paper provides a geometric perspective on weak-to-strong generalization [[Burns et al., 23](https://arxiv.org/abs/2312.09390)]. Specifically, the authors show that if the set of strong model-representable functions the following holds:
$$MSE(\phi^*, \phi^{ws}) \le MSE(\phi^*, \phi^w) - MSE(\phi^w, \phi^{ws}),$$

where $\phi^*$ is the ground truth labeling function, $phi^{w}$ is the the weak model labeling function and $\phi^{ws}$ is the weak-to-strong model labeling function. By labeling function I mean a mapping from inputs $x$ to real-valued labels $y$.

The authors use this result to argue that the gain in MSE loss in weak-to-strong generalization is at least equal to the _misfit_ term $MSE(\phi^w, \phi^{ws})$. There are some experiments verifying this result in practice.
strength_weakness: strength_weakness: W1. My main concern with the paper is that in my opinion the presentation is very confusing. Specifically, the core result is very simple, but it is presented in a way that took me a while to understand. 

My first issue is that the authors use notation $d_{\mathcal{P}}$ to denote the mean-squared distance, i.e. $\mathbb{E}(f(x) - g(x))^2$ and refer to it as distance (line 133). If $d_{\mathcal{P}}$ was in fact a distance, then Eq. (2) is the inverse of the triangular inequality. This confused me for a while. Moreover, Fig 1 shows a triangle with sides labeled as $A$, $B$, $C$, and states $C \ge A + B$. The resolution is that $d_{\mathcal{P}}$ is a square of a distance, not a distance.

Then, Theorem 1 is just a combination of (1) law of cosines and (2) and the fact that the angle between the vectors connecting a point to its projection on a convex set, and a vector connecting its projection to any point in the convex set is $\ge 90$ degrees.

The proof and the statement of the theorem are in my opinion significantly complicated by the explicit use of representations $h$ throughout. I don't understand what value they add: throughout the paper, the authors always use the same compositions e.g. $f^s \circ h^s$. Denoting the entire mapping with one letter would simplify the presentation. Currently, the authors also state that they are "employing a representation-theoretic perspective" (line 308), but I don't think there is anything added by the parameterization of the model as a function on top of some representations. Possibly the only thing is Claim 3, but it doesn't affect other parts of the paper much.

W2. The result of the paper holds specifically for the MSE loss. Indeed, if $d_{\mathcal{P}}$ is a distance metric, then Eq.2 is the opposite of the triangle inequality and doesn't hold. Moreover, for the qualitative conclusion
> Thus, the 148 inequality directly says that the weakly-supervised strong model improves over the weak model by (at least) an amount equal to the misfit.

clearly doesn't hold for the classification settings considered by [Burns et al., 23]: it is possible for the strong model to differ from the weak model on datapoints where the weak model is correct, which would not make it more accurate with respect to the ground truth labels. 

W3. In fact, [Burns et al., 23] provide a relevant discussion of _simulation difficulty_ in Appendix E (especially E.1). They also note that in order for the strong model to improve upon the weak supervisor, the strong model should be unable to simulate the weak labels well. A related result is also reported in Fig 8 of that paper. I think it would be good for the authors to comment on how their results connect to these results. 

W4. The practical proposition of using the weak model with the highest misfit will not work without further assumptions. This idea is again quite related to the result in Appendix E.1 of [Burns et al., 23]: if the errors of the weak model are not simulatable by the strong model, but the signal is easy to simulate, then we will get very good weak-to-strong generalization. But in general, high weak-to-strong disagreement will not always imply good weak-to-strong generalization.

Even under the assumptions of convexity etc of theorem 1, I believe it is not true that for a given $d_{\mathcal{P}}(\phi^*, \phi^w)$ higher $d_{\mathcal{P}}(\phi^w, \phi^{ws})$ always lead to lower $d_{\mathcal{P}}(\phi^*, \phi^{ws})$. Is that correct?
questions: questions: Please comment on W1-W4.
limitations: limitations: Limitations are adequately discussed.
paper_summary: paper_summary: This paper studies the phenomena of weak-to-strong generalisation (WTSG) from a theoretical angle, aiming to explain why the phenomena occurs. The paper's main theoretical results show that, in a regression setting with a convex function class, the decrease in MSE from the weak to weak2strong model is bounded below by the error of the weak2strong model on the weak labels (the "misfit"). The proof of the theorem relies on the pythagoreon theorem for projectsions onto a convex set. The paper then performs synthetic (simulated guassian data) and realistic (molecular property prediction) experiments to test out the theoretical prediction, and find that the decrease in MSE is roughly linearly proportional to the misfit on the weak labels in both settings. They show that this relationship can be used to select which weak2strong model to choose (the one with the highest misfit) correctly. They finally show that in a low-sample regime, "weak" models aren't necessarily smaller ones, as larger models may overfit to the small number of samples and hence perform worse. In this setting, their predicted relationship continues to hold, but with the small model being the strong model.
strength_weakness: strength_weakness: It's unclear why the authors chose the regression setting, when the original work and likely scenario of use is classification setting. I think this difference from the original work should be made clearer.

The empirical confirmation in realistic data is limited to one setting, which is an unfamiliar dataset to me and likely most of the LLM and NLP community. It would be beneficial to demonstrate the phenomena across more settings, especially more standard ones.

Giving some intuition as to *why* the theoretical result holds from the proof would be beneficial. In particular, pointing out that the inequality achieved relies on the distance measure not being a metric, otherwise the triangle inequality for metrics would force the inequality to be the other way round, if my understanding is correct?

It's likely that in the setting from the original paper, the weak labelling is within the function class of fine-tuned strong models (where fine-tuning is over all parameters), at least on a naïve interpretation. It would be beneficial to discuss whether the authors think their theoretical results is the reason why WTSG happens in this setting as well (given the assumptions of their theory break), or whether they think WTSG happens in that scenario for a different reason.

### Summary

In general, I'm in favour of the paper getting accepted, and am giving it an accept (7). I'd be willing to raise my score if additional empirical justification of theoretical claims was made in more varied settings.
questions: questions: -
limitations: limitations: The authors have adequately discussed the limitations of the work.

Comments: title: Checking in....
Checking in....
Response to comment
Response to comment
comment: Thank you so much again for taking the time to read our manuscript! As the deadline for the discussion period is approaching, we would really appreciate hearing from you, and ask if any further clarifications are required!
Thank you so much again for taking the time to read our manuscript! As the deadline for the discussion period is approaching, we would really appreciate hearing from you, and ask if any further clarifications are required!
Thank you so much for the clarification. We really appreciate your time engaging in this discussion, and acknowledge that our analysis assumes linear probing of pretrained models, whereas the gains in the classification settings of Burns et al. hold even when all the parameters of the strong model are finetuned. Here are a few thoughts:

1) You are right that if we allow finetuning of all the parameters of the strong model in the weak supervision stage, the weak model function is inside the strong model function class. However, we would like to note that in this case, the class of strong model functions ($f \circ h_s$ where both $f$, $h_s$ are free) is no longer a convex set---thus, our theory doesn't extend to this setting, and doesn't exactly predict whether we should/should not see WTSG. 

2) Nevertheless, we ran a quick experiment on synthetic data (as in Figure 2(a), 2(b) of the paper), where we finetuned all the parameters in the strong model in the weak supervision stage. We would like to remark that we didn't see a clear WTSG trend in these experiments. It still might be true that on large-scale, real data, WTSG is observed in the regression setting, even if we were to finetune all the strong model parameters (like what we see in Burns et al.). If this is so, it would be interesting to characterize why this is happening, given that the underlying convexity assumption from our theory is broken (e.g., does the number of finetuning examples matter in the weak supervision stage?) On the other hand, if no clear WTSG is seen even in large scale experiments (like in the synthetic data), this would also be very interesting and suggest that 1) the conclusion of our analysis may be extendable to non-convex spaces of strong model functions, and 2) even further differences in WTSG between classification and regression.
Thanks for the response and discussion.

My point was that main difference between your analysis and Burns et al. is not classification vs regression, but that your analysis assumes probing pretrained models, as opposed to fine-tuning all the weights. If you were to perform full fine-tuning in the regression setting then I would expect you would still see WTSG, but your theory would not predict that, as I think full fine-tuning of the strong model should be able to completely fit the labels of the weaker model (i.e. weak model function is inside the strong model function class), as it is a bigger model. Your theory would not predict WTSG here I think? If you ran WTSG for regression with full fine-tuning, what would you predict would happen, and how might you extend your theory to cover that case?
Thank you for your comment. We do believe that the performance gains via WTSG in classification settings ought to also be characterized to some extent by the misfit between weak and strong models; however, other terms also seem necessary (capturing, among other factors, the case where the strong model can represent the weak model). Indeed, the gains in accuracy with naive finetuning without auxiliary losses in the classification settings considered by Burns et al. 23 already suggest that such terms should be necessary, and that the phenomenon of accuracy gain via WTSG in classification is more nuanced than in regression (e.g., not only the quantity, but even the *quality* of misfit can matter here). Thus, our theory does not directly explain the gains in performance in these cases, and a different analysis seems to be required.

As for initial thoughts about a theoretical explanation for gains in classification: the phenomenon in classification that we want to capture is the following: the weak model is presenting the strong model with labeled data; however, the labels are really only "pseudolabels", and not true labels. The strong model, while learning from the weak model, is nevertheless still *rectifying* the wrong "pseudolabels". The self-training framework of [1], where the student and teacher are the same model, does indeed capture the case where the student model can in principle fully represent the weak model. Under certain assumptions (like expansion and separation), and along with consistency losses, this theory does explain performance gains of the self-trained student model, and thus seems to be a promising avenue to explain gains in WTSG as well. It would be interesting to see if the high-level analysis in our work (projection onto the space of strong models that are already implicitly biased towards the ground truth) could be combined with such an analysis to explain performance gains, even when no consistency losses are involved. Again, all these are really fascinating directions for future study!

[1] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data.
Thanks for your clarification and response.

In response to your last point, while Burns et al do propose the auxiliary confidence loss, even without this loss (i.e. naïve WTSG), the w2s model outperforms the weak model, which is anti-predicted by your theory (which as you note would predict no performance boost on top of the weak model). Ignoring the auxiliary confidence loss entirely Burns et al's results still don't match your theory, so I'll ask again whether you have any ideas as to why that is the case: Do you think your theoretical results are the reason why WTSG happens in this setting as well (given the assumptions of your theory break), or do you think WTSG happens in that scenario for a different reason?

I agree that it would be interesting to extend your theory to explaining the auxiliary confidence loss' boost in performance, but you would first need to ensure it explains the naïve WTSG effect in the full fine-tuning regime, as it currently does not.

Regardless, I am still happy to recommend acceptance of the paper, and will maintain my score.