Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Authors: Abhijeet Mulgund
Chirag Pabbaraju
Venue: ICML 2025 poster (2025)
Keywords: Weak-to-Strong Generalization
Bregman Divergence
Information Geometry
Alignment
Large-Language Models
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning
TL;DR: We extend prior theoretical results about quantifying weak-to-strong generalization beyond L2 loss, and verify our results with experiments.

Abstract: The paradigm of weak-to-strong generalization constitutes the training of a strong AI model on data labeled by a weak AI model, with the goal that the strong model nevertheless outperforms its weak supervisor on the target task of interest. For the setting of real-valued regression with the squared loss, recent work quantitatively characterizes the gain in performance of the strong model over the weak model in terms of the misfit between the strong and weak model. We generalize such a characterization to learning tasks whose loss functions correspond to arbitrary Bregman divergences when the strong class is convex. This extends the misfit-based characterization of performance gain in weak-to-strong generalization to classification tasks, as the cross-entropy loss can be expressed in terms of a Bregman divergence. In most practical scenarios, however, the strong model class may not be convex. We therefore weaken this assumption and study weak-to-strong generalization for convex combinations of $k$ strong models in the strong class, in the concrete setting of classification. This allows us to obtain a similar misfit-based characterization of performance gain, up to an additional error term that vanishes as $k$ gets large. Our theoretical findings are supported by thorough experiments on synthetic as well as real-world datasets.

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Abhijeet Mulgund I1
Chirag Pabbaraju
2
Abstract
1. Introduction
Weak-to-strong generalization (Burns et al., 2024) posits that strong Al models outperform weaker supervisors when trained on data labeled by them. By virtue of being larger and more complex models that have presumably also seen more data in their pretraining lifecycle, the hope is that strong models will invariably know how to push past the performance ceiling of their weak supervisors. This phenomenon is central towards the eventual goal of super- alignment (OpenAI, 2023), where we expect AI models to exhibit superhuman skills aligned with human principles from human supervision. Perhaps encouragingly, recent studies (Burns et al., 2024; Guo et al., 2024; Ji et al., 2024; Sun et al., 2024; Yang et al., 2024; Zhang et al., 2024; Liu & Alahi, 2024) have affirmed that such a phenomenon can in fact be elicited. Given the extent to which emergent Al would appear influence our future going forward, it be- comes imperative to theoretically understand when and how weak-to-strong generalization may provably be exhibited.
The paradigm of weak-to-strong generalization constitutes the training of strong Al model on data labeled by weak AI model, with the goal that the strong model nevertheless outperforms its weak supervisor on the target task of inter- est. For the setting of vealevalued regression with the squared loss, recent work quantitatively char- acterizes the gain in performance of the strong model over the weak model in terms of the misfit between the strong and weak model. We gen- eralize such characterization to learning tasks whose loss functions correspond to arbitrary Breg- man divergences when the strong class is convex. This extends the misfit-based characterization of performance gain in weak-to-strong generaliza- tion to classification tasks, as the cross-entropy loss can be expressed in terms of a Bregman di- vergence. In most practical scenarios, however, the strong model class may not be convex. We therefore weaken this assumption and study weak- to-strong generalization for convex combinations of k strong models in the strong class, in the con- crete setting of classification. This allows us to obtain a similar mishttbbase characterization of performance gain, up to an additional error term that vanishes l gets large. Our theoretical find- ings supported by thorough experiments on synthetic as well as real-world datasets.
Towards this, Charikar et al. (2024) recently provided a mathematical characterization of wcak-to-strong generaliza- tion for real-valued regression. They consider finetuning the final linear layer of a strong model (keeping other pa- rameters fixed) to minimize the squared loss on labels given by weak model. Using geometric argument involving projections onto convex set, Charikar et al. (2024) show that a strong model trained in this manner provably attains smaller loss than the weak model on the target task. More- over, the quantitative reduction in loss (performance "gain") is at least as much as the loss of the strong model on the weakly labeled data ("misfitt between the strong and weak model). However, their results limited to regression, leaving open whether such misfit-based characterization can also be formally shown for other tasks like classification, one of the primary subjects of study in Burns et al. (2024).
'qqual contribution 'Department of Mathematics, Statisties, and Computer Science, University of Illinois at Chicago, Chicago, II, USA DDparrment of Computer Science, Stanford University, Stanford, CA, USA. Correspondence to: Abhiject Mulgund <mul- gund2 @ @uic..du>> Chirag Pabbaraju <epabbara @ stanford.edu>.
Drawing on the theory of information geometry (Amari, 2016; Ay et al., 2017), we significantly extend the misfit- based characterization of weak-to to-stton-tt generalization be- yond regression tasks. Our main observation is that the
Proceedings of the nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
This notion of misfit is also referred to as "student-supervia" disagreement" in Burns et al. (2024).
Code is available at https: http//ggtttttttt con/ abhmul/ general - miieaa-m-sffie tefiiiinnnn......

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Pythagorean inequality for the squared loss used in Charikar al. (2024) holds more generally for Bregman divergences (Bregman, 1967). Thus, whenever the loss function in the learning task is Bregman divergence of some kind, a cor- responding misfit-based weak-to-strong generalization in- equality can be established. For example, since the squared l2 distance a Bregman divergence, the result of Charikar et al. (2024) constitutes special case of this theory. More importantly, the standard loss function that is used in classifi- cation tasks, namely cross-entropy, can also be written down (KI.)
parameterized regime provably exhibits weak-to-strong gen- eralization. Shin et al. (2024) take data-centric approach, and propose that data points that contain an overlap of easy as well as hard patterns most effectively elicit generaliza- tion. Ildiz et al. (2024) recently provide a high-dimensional analysis of knowledge distillation from surrogate models. Our work adopts the geometric viewpoint of Charikar et al. (2024), which interprets the finetuning of the strong model with weak labels the strong model class (where the projection corresponds to minimizing a loss function).
projection of the weak model onto
as Bregman divergence (namely the Kullback-Leiin divergence (Kullback & Leibler, 1951)) plus an additive entropy term. This implies provable weak-to-strong gener- alization in the setting of classification (i.e. cross-entropy loss provably improves), along with a misfit-based charac- terization of the gain similar to Charikar et al. (2024).
Our work significantly generalizes Charikar et al. (2024) to more loss functions and non-convex strong model classes, providing stronger evidence that the gain in weak-to- strong generalization for many machine learning tasks typically characterized by the disagreement between the student and teacher model (an observation originally made empirically by Burns et al. (2024)). This aligns well with the general flavor of results in co-training (Blum & Mitchell, 1998) and disagreement- based generalization bounds (Dasgupta et al., 2001; Wang & Zhou, 2017: Yu et al., 2019). Par- ticularly relevant also is the vast literature on knowledge distillation (e.g., Hinton (2015); Nagarajan al. (2023)) and oiff-distillation (e.g.. Wei et al. (2020); Mobahi et al. (2020); Pareek et al. (2024)). It is worth mentioning that the work of Lang et al. (2024) draws insights from both Blum & Mitchell (1998) and Wei et al. (2020).
Our approach, however, yields non-standard recipe: over a convex class of strong models, minimize the expected KL divergence with the strong model's output as the first argument and weak labels as the second. This "reverse" " KI. is contrary to standard cross-entropy training, where the labels are the first argument. Our experiments, as well as concurrent work Yao et al. (2025a), suggest this reverse direction is better for weak-to-strong training for classifi- cation tasks. Furthermore, the reduction in loss is at least the KL divergence between the strong model and the weak model at the conclusion of wcak-to-strong training! Thus, performance gain can yet again be measured in terms of the misfit between the strong and weak models, albeit with the notion of misfit being the KI. divergence. Typically, however, the strong model class fails to be convex. In this case, we show that suitable approximation can be achieved by training a convex combination of k strong models.
Concurrent Work. While we primarily focus on relaxing convexity assumptions in theorems relating misfit and gain, concurrent works Yao et al. (2025a;b) extend results relax- ing the realizability assumption from Charikar et al. (2024) and establish lower bound on gain by the misfit. It is worth noting that the results in Yao et al. (2025h) echo our obser- vation that reverse KL divergence appears to be more suited for weak-to-strong generalization than forward KL diver- gence. Concurrent work Medvedev et al. (2025) studies the impact of early stopping on weak-to-ste weak-to-strong generalization for regression in shallow ReLU networks.
We validate our theory across synthetic as well as real-world NLP and vision datasets, and find that when the strong model is trained according to the stated recipe even with modest values of k (e.g.. 100), its performance gain is faith- fully characterized by its KL misfit with the weak model. Additionally, this behavior becomes more apparent as k increases, further validating our theory.
3. Preliminaries
2. Related Work
We begin by formally describing the weak-to-strong general- ization setup, adopting notation from Charikar et al. (2024).
Our work is complementary a growing line of works, each of which seeks to theoretically explain the phenomenon of weak-to-strong generalization via a different lens. The work of Lang et al. (2024) posits that the two crucial properties governing weak-to-strong generalization are coverage ex- pansion and pseudolabel correction. The work of Somerstep et al. (2024) formalizes weak-to-strong generalization as the problem of transferring a "latent concept" from the weak model to the strong model. Wu & Sahai (2024) show how finetuning a linear model on Gaussian features in the over-
3.1.
Generalization
The setup constitutes a "strong model" and "weak model", where the strong model is typically larger, and represen- tationally more powerful AI model than the weak model. The weak model plays the role of teacher and the strong model plays the role of student, in that the strong model is trying to learn a target function from (potentially inaccurate) evaluations made by the weak model on data. Abstractly, we
2

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
think of the strong and weak models via their representation
to have finite support. For function / of random vari- able, [x)] denotes the expectation of f fXX over Px. When no subscript is attached to E, the expectation is taken with respect to all random variables in scope. The proba- bility distribution of a random variable X is written as Px- For a pair of random variables X, Y jointly distributed, the conditional distribution of Y given X is notated as PyJx-
maps ha : X' + TR and h X - TR respectively the data domain X'. For example, lt2 could be a deep trans- former architecture, and h a shallow architecture. Suppose X' + y/ is some target function. The only signal that the strong model gets about g is through evaluations of the weak model on data. Namely, it sees a dataset labeled by fu (+)). where fw R +> y is a finetuning map that the weak model has obtained, possibly after seeing a differ- ent batch of data labeled by itself. Importandly the labels that the weak model feeds to the strong model is from a separately held-out dataset, so that the weak model does not have access to the true labels for it. The objective of the strong model then is to obtain a finetuning function Sa for itself that it can compose onto its representation hgg such that fo (+) estimates 9 better than fo (hw (-))
3.3. Convexity
We will take for granted many basic results about convex functions; Ekeland & Temam (1999) is a good reference for these. For convex function v R - I we write domb for {i E IR" : (xx) < øo). A convex function is proper if domb + 0. All convex functions in this paper will be assumed to be proper. We denote Up := intdomp. Over IR",, all convex functions with nonempty domain are continuous over U*.. When è : R" + K is strictly con- vex and Ca (U,)), Ve is a homeomorphism onto its image, and plays an important role in the theory of convex duality. When such a sb is specified, we denote x = (a)(x) for x Uv. Likewise, for S C Up. S == (v)(). The Legendre dual of such 1 is of R" + E where dx((x)) == (z, x²) - (xx)) for z E Us. and xo otherwise. It is also a strictly convex function that is C2 on Uv- = (U,)". Furthermore, 3- = v. and Ve- = (V As such, When 1o and र/" are distinguished, we call Ug the primal space and Ug- the dual space. We refer to Ve as the dual map, and x" the dual of a E Up-
For the theory in the main body of the paper, we make two assumptions to avoid measure-theoretic and functional- analytic complications and to simplify the exposition: (1) we assume all distributions have finite support, and (2) we restrict our attention to binary classification rather than c-ary classification for c 2. The main results can be generalized when (1) is relaxed; however, more involved technical ma- chinery is required (see Appendix A.3). Only minor modifi- cations are needed when (2) is relaxed (see Appendix A.2). We now set up some notation.
3.2. Notation
For n E N, we denote [n] := (1.....m.. R == RU
For a function class F and a function h, we write F h for the set (f o h For function f. we write f = c if f is constantly c. For E [0, 1]. denote p == 1 - p. All logarithms have base e. H : [0, 1] + log 2] is the binary Shannon entropy H (p)) :- -p log plogp. P. DKL.
3.4. Bregman Divergences
We generalize beyond the squared loss analyzed in Charikar et al. (2024) via Bregman divergences (Bregman, 1967).
Definition 3.1 (Bregman Divergence). Let v : R R
[0, 102 + uR is the binary KL-divergence (pl()) ==
be a strictly convex and (U,)))) Then the t ---rrgmann divergence, De: : R² x Up + R is defined
p plog? P +p +og 7. and : j0. 1]2 - Te." is the binary cross- entropy XE (g) == -p log
p log q. Note XE(pllg) (plg)) + (p).. a : IR + (0, 1) is the sigmoid function (x) == 1 and -1 is its inverse, the logit function.
Dx = +2)) - v0v) - (xx (x-y.y') v, = =x(x) + 3s (v²) -)((),,)) (x)
For a subset 5 C R", Snt denotes its interior, and 3 its closure. coS is its convex hull which is the intersection of all convex sets containing S, and coS is its closed convex hull which is the intersection of all closed convex sets containing S. Note coS coS. The convex hull of set can be expressed as the set of all k-convex combinations of points in S. with A ranging through N. We use co* S to denote all convex combinations of any k elements of S.
Here, eb is the generator of Da. Intuitively, D3 measures how much the linear approximation of 1b at y underestimates vxx). It is always nonnegative, and is 0 and only if x = y. The alternative expression for Dg also reveals an important property: D (z. 3) = D (v",, x²). Furthermore, D& is strictly convex and differentiable in its first argument.
Table 2 in Appendix A. lists some examples of Bregman divergences. While Bregman divergences are convex in their first argument, they are not necessarily convex in their second argument. The logistic cost (Table 2) is an example of this.
We use capital letters for random variables. These are instan- tiated by specifying a distribution (e.g.. X ~ Px signifies random variable X is drawn with probability distribution Px). As mentioned earlier, all distributions are assumed
3

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
3. GROSS-ENTRONE AND KL-DIVERGENCE
the dual map. Legendre dual, and Bregman divergence of V, get f = V J. V'["*] Rajs (X)(),, and (f. = (Dx 9())(()))())))) (()). Thus, the previous results about Bregman divergences also apply to expectations of Bregman divergences when | < 00. See Appendix Ap- pendix A.3 for a discussion when X' is not finite.
In binary classification tasks, we work with data X.Y Px,y, where input data X Rd and labels Y {0.1). Let F- - [0, 1] be the conditional probability function g(x) == (I)r) Given some hypothesis class F of functions Rd + (0, 1), we wish to find an f E F that min- imizes (x (FFFE((())I((XXX) (X))] Since g is fixed for a given classification task, we see that ([XE(g(X) II differs from B[DKI.(g(X)) ((X)) by the task-dependent constant (g (x()) Thus, for classification tasks, applying Breg- man divergence theory will produce corresponding results aboul cross-entropy, modulo a constant.
3.4.4, BREGMAN DIVERGENCES IN THE WILD
A long line of literature has established the value of Breg- man divergences in machine learning. Perhaps the most well-known application is to the mirror-descent algorithm, which generalizes the gradient descent algorithm to non- Euclidean geometries Gupta (2020). This generalization can lead to improved bounds on regret for online learning tasks, and unifies several optimization algorithms under one framework. Similarly, in unsupervised learning, Banerjee et al. (2005) unifies wide variety of clustering algorithms as arising from a particular choices of Bregman divergences. Furthermore, they show every exponential family for soft- clusering corresponds to an expectation-maximizaen gorithm using a Bregman divergence. In statistics, Reid & Williamson (201 I) establish a connection between Bregman divergences and f-divergences.
3.4.2. GEOMETRY OF BREGMAN DIVERGENCES
We would like to use Bregman divergences analogously to distances like the l distance. However, Bregman diver- gences are not always symmetric. The KL-divergence is a classic example of non-symmetin symmetric Bregman divergence. Thus, they are not valid distances. However, they do possess many geometric properties. We use two properties below to generalize the main result from Charikar et al. (2024). Below, &b refers to strictly convex, (U,) function.
al-
Fact 3.2 (Generalized Law of Cosines (Chen & Teboulle, 1993)). Let x. y/ E Ug. Then
In this paper, misfit-gain inequality (Corollary 4.2) for cross-entropy, and generalizing it to non-convex classes of strong learners (The- orem 4.3). However, we frame these theorems and their proofs (Appendices A.2.1 and A.2.2) through the lens of Bregman divergence theory to provide a toolbox for re- searchers looking to apply these results to more specialized settings.
primarily focused establishing
Dv 2)= Dz y) De 2) (x y,2² - y²).
To generalize the Pythagorean inequality for inner product spaces, we need the following notion
Definition 3.3. Let W C Ug.. For x E Up. de- fine the forward Bregman projection to be Pw(()) = arg min z))
3.4.5, FORWARD KL VS. REVERSE KL
Here we provide a brief intuitive explanation of the differ- ence between forward and reverse KI. divergence. We echo observations made in Kristiadi (2016); Yao et al. (2025a); readers should refer to these sources for more detailed dis- cussion.
W convex and closed then the forward projection exists and is unique. Existence and uniqueness follow from conti- nuity, boundedness of sublevel sets (Exercise in (Fawzi, 2022)), and strict convexity of D& in its first argument. The following is a known generalization of the standard f Pythagorean inequality to Bregman divergences.
Roughly, we think of forward KL as being mass-seeking: it prioritizes learning distribution that covers all possibili- ties dictated by the teacher. On the other hand, reverse KL is mode seeking it prioritizes learning a distribution that captures the most frequent behavior in the teacher. One can notice this behavior in how the two loss functions handle student that disagrees with the teacher and predicts 0% probability for an event: forward KL will be too while re- verse KL will exclude that event from the loss. Conversely if the teacher predicts 0% probability for an event and the student disagrees, then forward KL will disregard that event, reverse KL will become +00.
Fact 3.4 (Generalized Pythagorean Inequality (Dhillon, 2007)). Let W C Us be a closed, convex set. Then for all = E Up and x E W
(x) 2 (x, Pwo()) + (Pw(()))
3.4.3. EXPECTATIONS OF BREGMAN DIVERGENCES
Suppose we have random variable X ~ Px with finite sup- port X. Functions X' RR" form an xu.x'| dimensional vec- tor space that we give the L norm /E [ll [x
From strictly convex, C²,, +b : R" + H we get a con- vex functional V defined [f[] = TE (X))) Computing
In the context of weak-to took-t generalization, it is plausi-
4

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
ble that the (weak) teacher makes errors. Thus, we do not want our (strong) student be mass-seeking: it should be free to disagree with its teacher. Instead, the student should be mode-sed seeking as this is likely where most of the signal from the teacher comes from.
satisfied with very weak assumptions on th. Theorem 4.1 generalizes Theorem 1 in Charikar al. (2024), since &2 is Bregman divergence (and clearly sequentially consistent).
As corollary to Theorem 4.1, we also obtain the misfit-gain inequality for cross-entropy.
Corollary 4.2 (Cross-Entropy Misfit-Gain Inequality)- Let h RRd and h. X + Rdd be the strong and weak model representations respectively. Let g: - (0, 1) be the target function. Let F be class of functions mapping Rd + (0, 1). Let fw BRd + (0, 1) be the classifier for the weak model. If the following hold:
4. Main Results
We now in position state main results weak- to-strong generalization using Bregman divergence theory. The first result is a direct application of Fact 3.4.
Theorem 4.1 (Bregman Misfit-Gain Inequality). Let vb Rn + R.' be a proper convex function s.1. Up # O. Lel h2 xx and hw : X -R be the strong and weak learner representations respectively. Let fm RR + U be the weak model finetune layer, and g X Up be the target function. Let F be class of functions mapping
(Realizabillity) 3f. E so that g= f. hgg (Convexity) F is a convex set of functions,
then for any 0, there exists 5 > 0 such that for all Ss E F that satisfy
Rd. + Ug. If the following hold:
IE (h (Xx))
1. (Realizabillit) 3f. E E s.I. g - fo h.,
I inf IE [DEL (f(hs (X)))) 6 JET
2. (Convexity) F is a convex set of functions,
we have
3. (Sequential Consistency) For y E Un fixed, if
E (EE(((X))) S EX (o(X)) (X)))
DT(InD y) + O, then In + y,
- -E [DRL (fa (x) (h (X))) +
(2)
then for any > O, there exists 8 > 0 such that for all fs E F that satisfy
Proof Sketch. We can first rewrite Equation (2) in terms of DKL instead of XE by subtracting (x))) from both sides. Since DKL. is sequentially consistent by Pinsker's incquality, we can apply Theorem 4.1.
D
E
S inf fEF
EID (X)), (ha (X]]] +
We note that if
we have
Ss = arg min E (Dea, fEF
E (9(X), (x(xxx))
[Dy (o(X), (m)hho f(xx)x - DD (Ja(ha())) Su (m(hoo + (1)
then Equation (2) directly holds with no term. Such an Ss uniquely exists if F o h2 is closed. We refer to as the "misfit" term and E (X)](XX] f (h (X))) E (f
E (x))(
Proof Sketch. Because F is convex, we get that F
as the "gain" term.
h, is also convex. Since E Fo hn by Fact 3.4.
we have that 9proj = I'Fon, (fw o hw) uniquely ex-
In practice, F is usually not convex. For example, when learning linear probe (Burns et al., 2024) on top of h, together with the sigmoid, F is convex, but F itself is not. Nevertheless, our main observation in this case is that Theorem 4.2 still applies to coF! By Caratheodory's is theo- rem, co co x1 F,, and hence we can simply consider convex combinations of [ + 1 functions in F to represent co.F. However, | oc, this representation is not com- putationally tractable. We therefore suggest remedying this by attempting to project the weak model onto " o hgg and show that the error in the misfit- -asitt-gain inequality can still be bounded by a decreasing function in l that is independent of |. Concretely, we show that:
ists and satisfies the Bregman Pythagorean inequal- ity. Now, by continuity of both B[Dp( (mmho and (g(x) )] there exists an e1 E [ll)(h,(X)) Iproj (X)] C1, then almost satisfies the Pythagorean inequality with error Choosing 8 suffi- ciently small, we can apply the Pythagorean inequality to bound B (ff (h (X)), Speoo (X)) Sequential consistency will then make [[I (X)) - goroj( (X)] < 61- 0
s.t. if
Sequential consistency is a common assumption in the litera- ture when defining Bregman divergences (Reem et al., 2018; Bauschke & Combettes, 2003; Butnariu et al., 2003), and is
5

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Theorem 4.3. Let hws J I be as in Theorem 4.2. Let F be a class of functions mapping TRo, + (0,1). If the following hold
KL divergence between the strong and weak model output. Importantly, the strong model's s output is in the first argu- ment of the KL divergence. This is in contradistinction to the standard weak supervision objective, wherein the weak supervisor's 's output is in the first argument. However, this distinction provably leads to a performance gain, provided k not too small, and the empirical estimates are accurate. In the next section, we empirically validate this recipe.
1. (Realizabilitt) 3f. E F so that g fa o hss
2. (Regularization) F. satisfies
1/5(x)
max
E 1/5(X)
sup JEF
,E
sup FEF
< <00,
5. Experiments
then for any k E N. there exists 8 > 0 such that for all fo E co F that satisfy
We conduct experiments on synthetic data similar to Charikar et al. (2024), and also NLP and vision datasets considered originally in the work of Burns et al. (2024).
E (fsh
S inf E [DDxu(f(,s(XX))lllllc + 6, fEco+ F
5.1. Synthetic Data Experiments
we have
We follow the setup in Charikar et al. (2024) for the synthetic experiments. We assume that the target g takes the form ((h*()) for some ground-truth representation map h² and finetuning function f F. We set ht : R 1 to a randomly initialized MLP with 5 hidden layers and ReLU activations, where the hidden size is 16. We choose F to be the set of c-class logistic regression models on R6 Namely,
E [XE(g(XX]1( S E (X (X)))
O(1/Vk). (3)
-
The proof of Theorem 4.3, inspired from the work of Zeevi & Meir (1997), is more involved, and is a careful application of the probabilistic method; we defer it to the Appendix as Theorem A.2 where we prove in the multi-class setting; in particular, with classes, the error term becomes O(v /ck)..
F F= (Wx): W E R (cc-1)x16], where
" := softmax((0,
We note that both Corollary 4.2 and Theorem 4.3 make no assumption on the weak model, and only the realizability assumption on the target. Theorem 4.3 makes only a mild as- sumption on F that can be enforced by regularizing the mod- els in F (e.g., via an & penalty). We emphasize however that the inequality does not guarantee significant weak-to- strong generalization for any weak model. If fw E F. then the fa obtained by either theorems will be fw itself, and the misfit term becomes O. However, Corollary 4.2 and The- orem 4.3 allow us to quantify how much weak-to-strong generalization we should expect to see. All expectations involved can be estimated on hold-out dataset, yielding a bound on the gain in terms of empirical misfit, modulo esti- mation error. This quantification is in the loss of the learned strong model relative to the (realizable) ground truth data. Since a reasonable loss function is correlated with other error metrics like accuracy, we should expect to empirically see improvements in accuracy with increases in loss misfit. However, a relationship between accuracy and loss misfit is not theoretically guaranteed.
The marginal P of the data is (0, I); we set 100.
The class of strong and weak model repre- sentations, H. and Hw. are respectively set to the class of 8-layer and 2-layer ML.s mapping R& - R16 again with ReLU activations and hidden layer size 16. To ob- tain he and hen, we first randomly sample T = 10 maps f()), f(T., from F, and generate data {z 99 Jj=1
Pretraining.
(1) N..
for each, where N₂- = 2000. Here, every x
(1) ~ P, and
v = (h) Thereafter, the parameters of h2 and N are obtained by performing gradient descent to optimize the cross cross-entropy loss:
(i)
T N XE(,) ( 1100 (1(25)))
h4 = arg min hEHr TN,
1=1==1
for ls E {w, s).
(4)
Weak Model Finetuning. Next, we finetune the weak model M 100 fresh finetuning tasks f(1) (M)
Theorem 4.3 provides concrete recipe for weak-to-strong generalization that allows for quantitative handle on the performance gap between the weak and strong model. In this recipe, we finetune a convex combination of l functions from F on top of the strong model representation. The objective of this finetuning is the empirical mean of the
from F. To do so, we again generate {a yz 1==1 for
(0) Ny
every i E [M]. where N, = 2000, each x ~ P and / = (1' The weak model representation h that was obtained in the pretraining step is held frozen, and the parameters in the final linear layer are obtained via
6

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
gradient descent minimize the cross-entropy loss:
from gpt2 um, gpt 2 1al-largg larg and gpt For each dataset, we first finetune the linear probe of the weak model (by minimizing cross-entropy) on 50% of the training data with ground-truth labels. We then compute weak labels given by the trained model on the remaining 50% of the training data. We then optimize a convex combination of A = 100 logistic regression heads on top of the strong model to minimize the reverse objective as in Equation (6). We add an l2 regularization penalty (with coefficient 0.1) on the linear weights in the objective to help with regularization, and to better align with the requirements of Theorem 4.3. Finally, we estimate each of the terms in Equation (7) from the test data. The obtained results are shown in Figure 2.
N
( = arg min FEF
(5)
NJ
Weak-to-Strong Supervision. For each finetuning task above, we obtain data labeled by the weak supervisor as follows. For every i E [M]] we generate {af)) 8/ 1==1"
where a ~ P and 7/6)) = f (ha (a.)) This data is fed to the strong model, which goes on to learn:
arg min fIF.ILEF I. A==1
N₂
DKL
Σ 2fo)
Firstly, we see weak-to-stree -taronng generalization (in terms of the loss) for all the datasets: the test loss of the strong model is always smaller than the test loss of its weak supervisor. Next, we see a clear trend on all four datasets: as we range the strong model from gpt2-medi um to 2ptt2-larg to gpt -x1, the misfit onto the weak model increases, and concurrently, the loss on the test data decreases. In fact, for CosmosQA, Amazon Polarity and SeiQ, in addition to the test loss decreasing, we observe that the test accuracy is non-decreasing too (note again that our result does not claim anything about accuracy per se).
= Sa where 2/c =1:)==
(z(0))
(6)
Namely, we optimize over a convex combination of k lo- gistic regression heads. Importantly, observe that the order of arguments in DKL in Equation (6) above is flipped from that in Equation (5), in keeping with the recipe given by Theorem 4.3. We set k 100.
Evaluation. To evaluate that the inequality given by The- orem 4.3 holds, for each task i E [M], we estimate the three expectations in Equation (3) from a new sample of size N,. Our result says that upto an error term of O(V O(Vc/k))
We remark that our accuracies are inferior to those reported in the experiments by Burns et al. (2024) on these datasets. One reason for this is that we only ever tune the logistic regression heads of models (in both weak model training, as well as weak teak-to-strong training), whereas Burns al. (2024) allow full finetuning that includes the representation layer parameters.
E
- (X
2 E
(7)
In Figure 1, we plot the LHS on the y-axis and the RHS on the x-axis, for the experiment above performed with e = 2, 10, 50, 100. We can observe that Equation (7) is exhibited more or less with equality, for both the binary (c= 2) as well as multiclass cases (c 2). We do note that the plot gets noisier grows to 100. These plots show the same trend as for the squared loss in Charikar et al. (2024).
5.3. Vision Tasks
We next perform experiments on image classification datasets. Following Burns et al. (2024), we fix the weak supervisor to be AlexNet (Krizhevsky et al., 2012). For the strong model, we consider ResNet-50 (He et al., 2016) and VIT-B/8 (Dosovitskiy, 2020) based on DINO representations (Caron et al., 2021). Having fixed these representations, we finetune a convex combination of i 100 logistic heads top of the strong model the weakly labeled data. We consider two datasets: CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015), and obtain the same plots as we did for the NLP datasets in Figures 3a, 3b. Again, we clearly observe that as the misfit of the strong model onto the weak labels increases, both, the test loss de- creases as well as the test accuracy increases! Interestingly, we observe that our weak-to-strong accuracy on ImageNet for ViT-B/8 is better than the corresponding weak-to-strong accuracy reported for the same experiment in Table 3, Burns et al. (2024) (namely 64.2% respectively). We note that the only difference in our setup is the reak-to-strong training
5.2. NLP Tasks
Next, we consider four real-world NLP classification datasets: BoolQ (Clark et al., 2019), SciQ (Welbl et al., 2017), CosmosQA (Huang et al., 2019) and Amazon Polar- itty (McAuley & Leskovec, 2013). We work with models in the gpt family (Radford et al., 2019), where the weak model fixed to be 2pt2, and the strong model is chosen
These four of the five datasets considered in the code- base provided by OpenAl (Ecoffet et al., 2023) for their weak- to- to-strong generalization paper. We chose to not include the fifth (A (Anthropic/HH- RLHF (Bai et al., 2022; Ganguli et al., 2022)) since the results of Burs et al. (2024) on this dataset are extremely noisy (see the plot in Ecoffet et al. (2023)), e.g., no model seems to be doing hetter than random guessing.
7

---

Relating Misfit to Gain in Wcak-to-Strong Generalization Beyond the Squared Loss
2022
10.00
37446
20000
32000
3000
2000
20004
100. 1800 s000 2000 80002 caro LANA caro
100p cora cams aoau 60.2 COD0 20013
6904 1407
0,00
1900
4790
0307
6000
0.021
2.000 0207
(a) c=)=2
(b) 10
(c) 50
(d) 100
Figure I. Synthetic data experiments. The Gain and Misfit closely track each other. For c 100, we see that the correlation between misfit and gain weakens, suggested also by the O(y c/ckk) error term from Theorem A.2.
(a) CosmosQA
(b) Amazon Polarity
(c) BoolQ
SciQ
Week Model Accurwyy D B1B
Week Model Loss Acoumay
Week Model Acoumay D
Weak Mode Loos 870 Acoumay 0.124
6:111
0,2
0.50
C046
0.80
COW
6:00
1:000
0.55
6:38
6995
6012
wid 2301
lede Saio code Outo 2300
Di-iei doet
laree 240 doot Strong ouro
Stoong Model
Strong
Stoong Model
Figure 2. Weak model (gpt:2) is trained once on true labels and thereafter fixed. Each strong model is trained on the weak labels (on a held-out separate from that which weak model was trained on) random initializations of the k-convex combination of logistic regression heads; we plot the average test loss/misfit across these 10 runs, along with the standard deviations as the error bars.
of the linear probe: while Burns et al. (2024) adopt standard cross-entropy minimization on single logistic head, we em- ploy the reverse KL minimization on a convex combination of k heads, as suggested by our theory.
5.5. Enforcing the Realizability Assumption
Finally, while the plots in Figure 2 and Figure 3a, 3b do il- lustrate that the gain in performance is directly proportional to the misfit as expected, our theory would also additionally suggest that the gain should quantitatively be at least the misfit (up to an error term decreasing in k). This does not quite hold the plots-we can observe that the misfit is consistently larger than the difference between the weak model and strong model loss. One significant reason for this is that our result assumes realizability; namely, the target task should be exactly representable by the strong model. We verified that this does not actually hold in our experiments- even we train the strong models on data with ground-truth labels, we see a non-trivial test loss at the end of training. To isolate this cause of discrepancy, consider evaluating the test losses of the weak and strong models (two terms on the LHS of equation 7) with respect to the best possible strong model (that is trained on true labels) instead of the ground-truth target function. This simply ensures that the realizability assumption holds. If we evaluate the quantities thus, consolidate the numbers across all the different NLP and vision datasets, and plot them with in Figure 1, we obtain Figure 3d. Note how this plot is more aligned to the plots in Figure I, with the misfit more faithfully capturing the quantitative gain in performance.
5.4. Varying h:
Recall that with a convex combination of k logistic heads, the upper bound on the difference between misfit and gain in Theorem A.2 scales as O(vc/k); particular, as in- creases, the upper bound becomes smaller. As our next experiment, for each dataset, we fix the strong model to be the largest one (qpt2 for NLP, and VIT-B/8 for Vi- sion), and vary 1: in (1, 10, 50, 100). For each dataset, we plot the difference between misfit and gain against k; the plots are shown in Figure 4 in Appendix A.4. We observe that for all datasets but ImageNet, the difference between misfit and gain consistently decreases as i increases; fur- thermore, beyond point, increasing k does not decrease the discrepancy by much. This can also be seen in Figure 3c, which collates the plots for all datasets (except ImageNet) in single figure. We suspect that the trend does not show up for ImageNet because e = 1000, and hence dominates in the error term for the values of & that we consider.
We chose to not include ImageNet in this plot as the plot for ImageNet was significantly off scale, and was skewing the y-axis.
8

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Wiesk Model Luse 0.746, Accurnry 0.605
Model 1.52%, Accaueeyy
3.14
Lcw
3.44
Mork
sfev1
LoGaMERR
600
3.54
6.0
0.21
coloata Eog:
dis axddi
NDc 0000 2030
scc
2om
Stoory Modiel
Strong Model
(d) Consolidated misfit/gain plot across all datasets.
(a) CIFAR-10
(b) ImageNet
(c) Varying k
Figure 3. (a). (b) Weak model (AlexNet) is trained once true labels and thereafter fixed; we report numbers averaged over 10 runs of weak-to-strong training. (c) We observe that the difference between misfit and gain decreases as k increases, and also that the decrease slows down with increasing k. (d) The test loss of the weak and strong model is measured not with respect the ground truth test data, but instead with respect to the predictions of the best strong model on the test data. This is done to ensure realizability.
CosmosQA
Amazon Polarity
BoolQ
SciQ
CIFAR-10
ImageNet
Forward
Reverse
Forward
Reverse
Forward
Reverse
Forward
Reverse
Forward
Reverse
Forward
Reverse
Test Accuracy
0.6378
0.6407
0.8963
0.8984
0.6140
0.6141
0.6171
0.6254
0.8981
0.9005
0.7133
0.6986
XE(tt, strong)
0.6147
0.6141
0.2788
0,2570
0.6464
0.6476
0.6439
0,6436
0.4052
0.3361
1.4042
1.4742
Gain Misfit
0.0368
0.0372
0.0717
0.0934
0.0706
0.0694
0.0264
0.0266
0,1772
0.2447
0.5217
0.4517
0.6093
0.0613
0.3717
0.1444
0.6319
0.0980
0.6558
0.0961
1.1024
0.3185
3,9392
1.6401
Table I. Comparison of Forward XE and Reverse KL.
5.6. Comparing Forward XE and Reverse KL
tative of the gain, as confirmed by our theory. Again, this indicates a clear "directionally" in the Pythagorean inequal- ity for weak-to-strong generalization in the classification setting! Note that it is plausible that in practice, running standard forward XE(weak, strong) minimization might lead to minimizer that is close to the minimizer of the reverse D weak).
Finally, we empirically compare the performance of the for- ward and reverse KL. loss in weak-to-strong supervision. For the experiments in Section 5.2, in the weak-to-strong train- ing phase, instead of finetuning a convex combination of == 100 logistic regression heads on the reverse KL diver- gence objective (i.e., DKL (strong, weak) in Equation (6)), we instead finetune these on the forward cross-entropy loss
(i.e., XE(wcak, strong)) with the same & regularization. The latter is the more standard form of finetuning, while the former arises from our theory. Table 1 shows the compar- ison for the test accuracy of the strong model, as well the cross-entropy loss between the ground truth and the strong model. The comparison indeed quite interesting- we can see that the strong model that is finetuned on the reverse KL objective shows better final test accuracy for nearly all the datasets! Namely, we do not see significant performance degradation (in fact, we see improvement in nearly all cases) with reverse KL compared to the standard setup.
6. Conclusion and Future Work
The theory in Section 4 establishes a connection between misfit of the strong learner its weak supervisor, and the generalization gain of the strong learner over its weak teacher. Experiments in Section 5 confirm this connection. However, these results raise several questions. Firstly, the empirical results seem tighter than expected. Can the error term in Theorem 4.3 be improved, thereby more accurately explaining the empirical results? Secondly, can we use our geometric insights improve weak-to-strong generaliza- tion beyond the standard framework? Finally, while our experiments focused on linear probes (Burns et al., 2024), Theorem 4.3 applies to significantly larger classes of mod- els. How tightly does Theorem 4.3 hold in these different settings, and can the ideas from this work be applied there to encourage better aligmment?
We also computed the gain and misfit terms in the Pythagorean inequality equation 7, where for the reverse KL experiment, we compute the reverse misfit that we propose (i.e., DKL (strong, weak)), whereas in the stan- dard forward XE experiment, we compute the misfit as (E'eeak, strong) - the "natural" misfit that one might consider. We see in the standard setup that the Pythagorean inequality is completely off (gain and misfit don't quanti- tatively align), whereas the reverse misfit is more represen-
The seminal work of Burns et al. (2024) exhibited the phe- nomenon of weak-to-strong generalization. We hope our work sheds some light the mechanisms at play and pro- vides theoretically justified modifications to the framework.

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Acknowledgements
annual conference on Computational learning theory, pp. 92-100, 1998.
We thank ICML reviewers for their helpful comments. This work is supported by Moses Charikar's and Gregory Valiant's Simons Investigator Awards. This work also supported by NSF under awards 1705058 and 1934915.
Bregman, L. M. The relaxation method of finding the com- mon point of convex sets and its application to the solu- tion of problems in convex programming. USSR com- putational mathematics and mathematical physics, 7(3): 200-217, 1967.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Burns, C., Izmailov, P. Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L.. Chen, Y., Ecoffet, A.. Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 4971-5012, PMLR, 21- 27 Jul 2024. URL https: ptttt:::/pproceeeinng http///proceeeeeeee/////////s.se mmr. press / v235//235///// /bturss2b22bbb html.
References
Abramovich, S. and Persson, L.-E. Some new estimates of the 'jensen gap" Journal of Inequalities and Applications, 2016(1), February 2016. ISSN 1029-242X. doi: 10.118 6/s 6/113660-016-0985544 URL http: /tx www.doo..oooo /10 1 001111166/1666-- 6 0985-4.
Butnariu, D.. Bymee C., and Censor, Y. Redundant axioms in the definition of bregman functions. Journal of Convex Analysis, 10(1):245-255) 2544,, 2003. URL https: www. heldermann veee--eellgeee de http://aajjjaaaaaaaaaaaa 13. 13.pdf.
Amari, S.-i. Information geometry and its applications, volume 194. Springer, 2016.
Ay, N., Jost, J., Lê, H. V., and Schwachhöfer, L. Information Geometry. Springer International Publishing. 2017. ISBN 9783319564784, doi: 10. 10.10007978-3331-5566488 http:/ /tpp//xxx/// www.oooorrooooooooooooooooooooe. 1007 900000099988333333 6478-4.
URL
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in sulervipe vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-99090 966099960,000 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint w0Xi::2204.05862. 2022.
Charikar, M., Pabbaraju, C., and Shiragur, K. Quantifying the gain in weak-to-strong generalization. In Glober- son, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 126474- / 126499. Curran Associates, Inc., 2024. URL https: wtwp:pppc....p....pppp...cc.ee.e
Banerjee, A., Merugu, S., Dhillon, S., and Ghosh, Clus- tering with bregman divergences. Journal of Machine Learning Research, 6(58):1705-1749, 2005. URL http: //jmlr. org papers/ hww.l..org/ppprs//////// v6 /baner Jee html.
Bauschke, H. and Combettes, P. Construction of best breg- man approximations reflexive banach spaces. Pro- ceedings of the American Mathematical Society, 131 (12):3757-3766, April 2003. ISSN 1088-6826. doi: oi. oi.org/10. 1 (000009//00002
Chen, G. and Teboulle, M. Convergence analysis of a
proximal-like minimization algorithm using bregman functions. SIAM Journal on Optimization, 3(3):538- 543, 1993. doi: 10.1137/98 URL https:
URL htR::/ 03 0000999000000000-000000
//doi. 111111117111 1137//80302..
Bauschke, H. H., Borwein, J. M., and Combettes, P. L. Bregman monotone optimization algorithms. SIAM Jour- nal on Control and Optimization, 42(2):596-636 2003.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
doi: 10. wttp:///...111333333000000200000 /tt.:///0000000000000000
URL https:
//doi.
Dasgupta, S., Littman, M., and McAllester, D. Pac gen- eralization bounds for co-training. Advances in neural information processing systems, 14, 2001.
Blum, A. and Mitchell, T. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh
10

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Dhillon, S. Learning with bregman divergences. https: //www. utexas wwt.s/edddddinieeeeeeeeee inderr/-nndrjit/Taakk bregtut pdf 2007.
Huang, L., Le Bras, R., Bhagavatula, C., and Choi, Y. Cos- mos QA: Machine reading comprehension with contex- tual commonsense reasoning. In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Prucessing (EMNLP-IJCNLP), pp. 2391-2401, Hong Kong, China. November 2019. Association for Computational Linguistics. doi: 10. 10.186633/1/D119--224 URL 3/.
Dosovitskiy, A.
An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Ecoffet, A., Joglekar, M., Wu, Kirchner, H.. and Izmailov, P. Weak-to-strong generalization. https: //github. http:///peea///eea-tostsoto/oo 2023.
Ildiz, M. E., Gozeten, H. A., Taga, O., Mondelli, M., and Oymak, S. High-dimensional analysis of knowledge distillation: Weak-to toakknn generalization and scaling laws. arXiv preprint S00i::2410.188377 2024.
Ekeland, I. and Temam, R. Convex Analysis and Variational Problems. Classics in Applied Mathematics. Society for Industrial and Applied Mathematics, 1999. ISBN 9781611971088. URL https: hattps://bocks https///ooossoooooo.... google.e ont/bookss1d==007nnae
Ji, J., Chen, B., Lou, H., Hong, D., Zhang, B., Pan, X., Dai, J., and Yang, Y. Aligner: Achieving efficient align- ment through seak-to-strong correction. arXiv preprint arXiv:2402.02416, 2024.
Fawzi, H. Topics in convex optimisation. https::/wwrr damtp. cam. ac. uw//ssee/hh33//222-111 PT/lecturel1 pdf, 2022.
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., al. Red teaming language models reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint 20X0022209907888, 2022.
Konenkov, N. The jensen's 's gap and comparison of f- divergences. Preprint, 2024. URL http: //tp:://dxx// doi .org/10.2139/saet 406.169656,,
Kristiadi, A. KI divergence: Forward vs reverse? https: faroog//fowwrdtt
/ //agustinus www.huu...istti. www.kiiiiii.................i..t everse-k1/, 2016.
Gao, X.,
Sitharam, M., and Roitberg, A. E. Bounds on the jensen gap, and implications for mean-coneentrated distributions, 2020. URL htttpp://axxivvooo s/1712. 05267.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. 2009.
Guo, J., Chen, H., Wang, C., Han, K., Xu, C., and Wang, Y. Vision superalignment: Weak-to-strong general- ization for vision foundation models. arXiv preprint arXiv:2402.03749. 2024.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Gupta, A. Mirror descent. https: //wps. cs. cmu.e du afs cs. cmu. edu/ http///aaamcccaaasaa/aaaaasaaase 158 50- http//////////eo wwwp:/te///ttttt/////ooo 2020.
Kullback, S. and Leibler, R. A. On information and suf- ficiency. The annals of mathematical statistics, 22(1): 79-86, 1951.
He, K., Zhang. X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 770-778, 2016.
Lang, H., Sontag, D., and Vijayaraghavan, A. Theoretical analysis of weak-to-strong generalization. In Glober- son, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 46837- 46880. Curran Associates, Inc., 2024. URL https: / hrwooeeednggs.nnurips..c//speeee /aappe//DDa//iee/3388aaeee111112
Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0253: 2015.
devia- tion. Bounding jensen's gap: Elementary approaches.
Mathematics Stack Exchange. URL https:// ma th
com /
Liu, Y. and Alahi, A. Co-supervised learning: Improving weak-to-strone generalization with hierarchical mixture of experts. arXiv preprint arXiv:2402. 15505, 2024.
09. URL.tt/s//maabb..nuucccaaaaoo//g (version: 2023-06-20).

---

Relating Misfit to Gain in Wcak-to-Strong Generalization Beyond the Squared Loss
McAuley, and Leskovec, J. Hidden factors and hidden top- ies: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender 2013.
Somerstep, S., Polo, F. M., Banerjee, M., Ritov, Y., Yurochkin, M., and Sun, Y. A statistical framework for weak-to-strong generalization, 2024. URL https: //arxiv. wtwpiiiwrgg/bbooe /4002 2403..6266. 16236
systems, pp.
Medvedev, M., Lyu, K., Yu, D., Arora, S., Li, Z., and Srebro, N. Weak-to-strong generalization even in random feature networks, provably, 2025. URL https: rxiv.oor
Sun. Z., Yu, L., Shen, Y., Liu, W., Yang. Y., Welleck, S., and Gan, C. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint M:Xiv:2403.09// 094472 2024.
g/abs/2503, 02877.
Mobahi, H., Farajtabar, M., and Bartlett, P. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Ullah, H., Adil Khan, M., and Saeed, T. Determina- tion of bounds for the jensen gap and its applications. Mathematics, 9(23), 2021. ISSN 2227-7390. doi: 10.3390/mathh2333133 URL https: www. md pi com wtl.coom2222112 0223999///233////
Nagarajan, V., Menon, A. K., Bhojanapalli, S., Mobahi, H., and Kumar, S. On student-teacher deviations in dis- tillation: does it pay to disobey? Advances in Neural Information Processing Systems,
Wang, W. and Zhou, Z.-H. Theoretical foundation of
co-training and disagreement based algorithms. arXiv preprint arXiv: 1708.7440.03 2017.
2023.
OpenAL. Introducing Superalignment. https: /ppenai 2023.
Wei, C.,, Shen, K., Chen, Y., and Ma, T. Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622, 2020.
Pareek, D., Du, S. S., and Oh, S. Understanding the gains from repeated self-distillation. arXiv preprint arXiv:2407.04600, 2024.
Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L.., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.186653/1/WW11-4413
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
URL htttps//aclanthhooggg. org. 441 3/.
Reem, D., Reich, S., and De Pierro, Re-examination of bregman functions and new properties of their diver- gences. Optimization, 68(1):279-348, November 2018. ISSN 1029-4945. doi: 10.189000331994220885553999 URL http::tpp//. /xx. doi wttp.ooogg.....00//0220000 4. 2.2018.1 1543295.
Wu, D. X. and Sahai, A. Provable weak-to-strong generalization via benign overfitting. arXiv preprint arXiv:2410.04638 2024.
Yang, Y., Ma, Y., and Liu, P. Weak-to-strong reasoning.
Reid, M. D. and Williamson, R. C. Information, divergence and risk for binary experiments. Journal of Machine Learning Research, 12(22):31-8877 2011. URL http: www.llooorpppppppeee /vtpss////////// hamml...
arXiv preprint arxiv:2407.13641 2024.
Yao, W., Yang, W., Wang, Z., Lin, Y., and Liu, Revis- iting to-s-nn generalization in theory and prac- tice: Reverse kl vs. forward kl, 2025a. URL https: //arxiv. org 220...1002111.00
Rockafellar, R. Convex Analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1997. ISBN 97806910158666 URL petettss://boot googlee.cmm/booss111111111.f.
Yao, W., Yang. W., Wang, Z., Lin, Y., and Liu, Y. Under- standing the capabilities and limitations of weak-to-strong generalization, 2025b. URL https: http://axxvvvorvo abs /5502. 01 458.
Russakovsky, O., Deng. Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet I Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICCV), 115(3)211-252, 2015. doi:
Yu, X., Han, B., Yao, J., Niu, G., Tsang, I., and Sugiyama, M. How does disagreement help generalization against label corruption? In International conference on machine learning, pp. 71. 7164-7173. PMLR, 2019.
10.1007/s1 1210001----111111
Shin, C., Cooper, J., and Sala, Weak-to-strong gener- alization through the data-centric lens. arXiv preprint 2XXi:2442.003881 2024.
Zeevi, A. J. and Meir, R. Density estimation through convex combinations of densities: Approximation and estimation bounds. Neural Networks, 10(1):99-109, 1997. ISSN
12

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
0893-6080. doi: http://ooloog//o0000066009990990
J00037-8. URL htt.tpss//www.ccceenccerr.......
http://aeeceeartaaee///////20300 00378.
Zhang, E., Zhu, V., Saphra, N., Kleiman, A., Edelman, B. L., Tambe, M., Kakade, S. M., and Malach, E. Transcen- dence: Generative models can outperform the experts that train them. arXiv preprint arXiv:2406. //741, 2024.
13

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
A. Appendix
A.1. Common Bregman Divergences and their Generator Functions
Name
Generator
Divergence
Dual Map
Legendre Dual
L2 Loss
(xx) 122
Dola,y) = 1/0 - y)²
x' x
(xx)) = 1x²²)²
KL Divergence
(p)) = (p)
Do(p,q) = (x
p' = o (p)
3 (p") = log + EP')
Logistic
((x) log(1 e*)
De(a.y) (
x' (x)
4 (x") -(x*)
Itakura-Saito
(x) log x
Do(a,y) - 1o
x² 1/x
(x*) log x
Table 2. Examples different univariate generator functions and their Bregman divergences.
A.2. Generalizing the Misfit Inequality to Multiple Classes
Here expand on the modifications needed to generalize the results discussed in Section 4 to the multi-class setting. There is both moordinate-dependee and coordinate- iestiiaate-idepeddeen approach to generalizing the misfit inequality. Below we detail the coordinate-deepedden approach as it is easier to understand. Later we discuss the coordinate-indeeendent approach which is more in line with the approach taken by softmax regression.
Again we make the simplifying assumption that our input data X Px has finite support X.
A.2.I. MULTI-CLASS KL DIVERGENCE AS A BREGMAN DIVERGENCE
In the binary-classifen classifictinn setting, we learning models that output probability distribution {0, 1) given the input X. While such distribution is written with two probability values and 1 in [o. 1J. it suffices to only know one of the two probability values to determine the other. Thus we can represent our space of models on data X by the (finie-dimensional) function space [0, 1|x² This motivated framing KL-Divergence as a Bregman divergence generated by the univariate convex function (p) == log + (1 - p) log(1 - p). In binary-llassfficttin,, we tend to interpret the output of a model e [0, 1|xd as the probability of the positive class; however, this choice was largely arbitrary. This arbitrariness the coordinate-dependeen
of this formulation. However, had we chosen the output of f X + [0, to be the probability of the negative class, we would have obtained the same Bregman divergence by the symmetry of sb. The rigidity of the Bregman divergence to this arbitrariness is the coordinate- ponrdinate-iddeeendennc of the underlying theory.
In the c-class setting for > 2. positive probability distributions are represented by c-dimensional vectors p E c-1 where c-1 is the (open) e 1-dimensional probability simplex:
A 0- = (0 Σ Σm=11
Now, for p E --1, knowing 1 of the e probabillities is sufficient determine the th probability. Making the arbitrary choice of dropping the redundant e-th coordinate, we get the set
p (0,1)
Σ P {
parameterizing positive probability distributions on [c]. Thus, we can represent our space of models on data X by the dimeni-dimenssonal) function space {f X' e-1)
Using this parameterization, we can express the negative Shannon entropy as
11 11 Pi
v (p) p1 log +... Pe-1 log 1
log
1
Σ Ps
i=1
i=1
Its dual map is the logit function
Re-1 with respect to the c-th probability
1og Pi c-1
p² == :==0 (p)
pe
i=1
14

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
where abuse notation slightly for consistency with the binary class case. Then, we compute the Legendre dual of vi
c
(x'"") log
epi
where P2 := O. 20* is the log-sum-exp function, and its gradient the softmax function a. So similar to the binary case, softmax regression learns models in the dual (logit) space X' Re-1, and applies a convert the outputs primal (probability) values.
Since & is strictly convex and C for p,g q e-1, Substituting our choice of th, we get that Du (p, q) is the multi-class KL divergence:
determines Bregman divergence (p. - (p q, Ve(())
Do(p,g) = Σ ps log Pi q:
DKL(pll)),
taking Pe 1- Ei=i P, and qe :=: 1- q. Using this Bregman divergence, analogs of the results Corollary 4.2 and Theorem 4.3 in Section 4. We first include a proof of Theorem 4.1 in more detail than in the main body of the paper.
state the multi-class misfit
Theorem 4.1. Let ub IR" - - be a proper convex function s.1. Up + O. Let hi : X - RRd, and h: - Rd
be the strong and weak learner representations respectively. Let fw Rd + Up the weak model finetune layer, and g X + Us be the larget function. Let F be a class of functions mapping R.d, - Ug. If the following hold:
1. (Realizability) 3fx e F.t. g fx o. ha.
2. (Convexity) F is convex of functions,
3. (Sequential Consistency) For y Uv fixed, if Di (2 y) + 0, then 2n + y.
Then any > 0, there exists > O such that for all fa E F. that satisfy
Ex [D (f)) (X)]] s inf Ex (D fi +
feF
we have
Ex [D (o(x),, Ex
Ex [De fo(X), Ex (Do (X), +c.
(8)
Proof. First note that convexity of F implies convexity of F o h2 (fo F}: for any fi, E F, F, [0, 1], and X, have
Xʃi (h) + (1 A)12(11,(x)) = (AS
(h)))()(()) Fo h.
Since g E Foh, by Fact 3.4, we have that Opmi := Poon (fw o hw) uniquely exists and
Ex [xx(o((X),ooo Sproj Sproj (X)) & Ex (Dvl((X), (X))) Ex [Doog (X), fw (Nww (X)))
Now we approximate Oproj sufficiently well using F hg- By continuity of both Ex [Do(- ((hw and Ex (Dx(g(X), }),, there exists e1 0 s.t. Ex [llg(X) (x)) S e1, then
Ex
(he < Ex (xX),,,, fm (x))) Ex (F=(fa(h, (he (X))) +
By sequential consistency, there exists > 0 s.t. if 2, E Up are s.t. vz s 8, then || - S 6₁. Let
Puin == min Px (x) > O. Let f. F s.t. s.t.
Ex [Dp(f.(h, for (X))) inf Ex (Dp(f(h,(X)) (han P.nin.. JeF
15

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
By definition of Sproj. have that
Ex (D (u
min Ex (h
inf Ex
JEF
where the last equality follows by continuity of Ex (D fir ( (x)))). Then, applying Fact 3.4 again, this time to fa o his E F ha instead of g, we have that
Ex (D)()s()))))) Spros (X)) Ex (D. (M)m) Ex [D (X), Lnr (Nw (X))) S Pmin Ó
Then, for all x E X, (h. Srooj
Sproj (1)) < 8. Thus,
2 sup
[1Ifa - Sproy' 9pmo((x))3 (x)1
=EX
Ex [Ilsa(ha(X)) Sproj x))
Thus,
Ex
Ex
Ex (uu((a,,,()))), +
Like in the binary case, the multi-class misfit-gain inequality follows as a corollary to Theorem 4.1.
Corollary A.1 (Multi-Class Misfit-Gain Inequality). Suppose h2 - Rdd, is the strong learner hidden representation be the classifier for the weak model, and let
and w X + Rd is the weak learner representation. Let fw Rd g: X c be the larget function. Let FF be a class of functions mapping IRd, c-11 If the following hold:
1. (Realizability) 3f. F that g f- ha.
2. (Convexity) FF is convex set of finctions,
then for any > 0, there exists 8 > 0 so that for all Js E F such that
Ex
inf Ex (Pxsa 8, JEF
we have
Ex [X s Ex (l Ex
(9)
Proof. First note that by subtracting Ex [H()( from both sides of equation we get
Ex [Ki. (o(X) (x(X)) < Ex (X)) (X))) Ex
+
so it suffices to prove this latter inequality. But this inequality follows from Theorem 4.1 after noting DRL. is sequentially consistent by Pinsker's inequality.
A.2.2. PROOF OF THEOREM 4.3 FOR MULTIPLE CLASSES
Next we prove Theorem 4.3 for the multi-class setting. From this result we will recover the Theorem 4.3 for binary classification. Recall that for a set S in a vector space V, 5 is defined as
S y VEV. --
Br,,.... . on E &SSe k--1
k-1 s.t. y
Σ pess
Theorem A.2. Let he, Nw, fw. g be as in Corollary A.I. Let a class of functions mapping IRd - c-1 If the following hold:
16

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
1. (Realizabilitt) 3f. F that g I hgg
2. (Regularization) F is s.t. max
Ex [supres (ampeex/ff(()) < xc.
sElc]
Then for uny k E N, there exists 8 > so for all Js E J such that
Ex (x)() (X))) z
inf /ECO
Ex ( +
we have
Ex [XE(G(()]11((( Ex [XE[g(()]Il Ex (2((5 (10)
The proof will come down to determining how large we need to make & to get satisfactory bounds on both
Ex DKiL. (Frooj (x) (X))) Ex (7),
Ex [DKL (xX) X Sprof (X)) Ex (ff) (X))).
where Ooroj ;= (fn ). We will see the first of the two terms is controlled by the Jensen approximation gap of &0 for a Bregman divergence Du. The second term more difficult to bound from general Bregman theory. We will instead bound specifically for KL divergence, leaving it open what conditions are necessary to get a satisfactory bound for general Bregman divergences.
For random variable Z Pz, we let Z denote the empirical mean of n i..d. samples of Z. Definition A.3. Let yo Rd - R be proper, convex function. The Jensen approximation gap of ạ is
Gap(n; := sup
E [PplZ") (EET)))
2 Pyy on domp R.Eedomp
Lemma A.4. Let th RRd + R be the generator for Bregman divergence Dy Let S C Up and let w E Un. Then any e N and y E coS, we have that there exists a E co S s.t.
D0 w) < w) G))(k;
Note that in the finte-dimeessonnl case Caratheodory's 's theorem does tell us that cod+1$ coS. However, we should imagine d >> k. In the dineee-dimmesionall case, Caratheodory's 's theorem does not hold, so we need to use the approach from the lemma above. The proof of the lemma applies probabilistic method in a way similar to (Zeevi & Meir, 1997).
Proof. Let y E CoS. Then there exist n E N, 21...., 2n E S, and (PI...., .P.) E A s.t. y = Em= Piz.. Note that a probability distribution on [n]. Let 7 ~ Pz be a random variable on (zI, 2n. with Pz(zi) = Ps. Then EZ = Now observe
(Z
(k) w) (v( E (z
(ww) (z (k) (
w. v((w)) +(v) ++w))= (v w. Vv(w))
(2)
- (EZ(M) (b)
V(b(w) - ((v) -w, vv(w))
E ( (1)
- -b(
< Gap(k; v)
So there exist i E [n] and x & ES= 2g E co
Do(a, w) Dv w) & EZ , w) Do w) S Gap(k; v))
17

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
From here, we now turn to bounding the Jensen approximation gap of the negative Shannon entropy function. While there is extensive literature on bounding Jensen gaps (Abramovich & Persson, 2016; Gao al., 2020; Ullah et al., 2021; Konenkov, 2024), we find that simple method inspired by to get our desired bound.
deviation) is sufficient
Lemma A.5. Let S C A c-1 be a set S.I. == (suppes()/p)) (= is finite in all its coordinates (here Pr. := >i=1 Pi)
Let vo: c-1 - R be the negative Shannon entropy function and let ub) looo be a/ restricted to coS. Then for any l E N,
we have
2
Gap(k; / loos)
S
as
a2
8k:
8k
Proof. First note that it suffices to assume S is convex. Indeed for any E coS, note that az 2 1 for all i E [c]. since
+ is convex.
So assume S is convex. It is more convenient to consider the extension of &e to all 1R50 = fo E R 0 Vi E [c]. We call this extension vi RS + R and define as
(xx)
Σ x, log Zt.
Le c-1 A--1 be the map defined
(p Pe-11) (p1..... Pe-1,
Σ p.).
Then ووا = cl... So Gap(k; (/)) = Gap(k; T o of dgg), and it suffices to bound the Jensen approximation gap of 인 on (S).
To that end, let Z ~ Px be a random variable on (S). Since ((S) is bounded and convex, IEZ is well-defined and (SS). Let (()) be the empirical mean of k i.idd samples drawn from Pz. Note that (V² = 6i x; for i.j E [c]. where des is the Kroenecker delta. So (xx) Diz axa concave as function of æ E ((8). By Jensen's 's inequality, we have
E
Z
1
ag
(1)
2
S
(EZ
a, (EZ) (k) )²
2
2
E
(1))
|]
(EZ (k)
1
aj
E
(Z (k) y²]
(BZZ)))
2 i=1
(1)
(BZ (2)
1
+
ag Var [Z]]
2k
1=1
Since each 2, E (1/a;, 1 - 25 مi= 1/0,). Popoviciu's inequality tells us
(
Var 23 <
1
1
Σ
1
1
1
1
Σ
6
4
aj
4
f=1,j
As Z was arbitrary, we have that
2
Gap(k; (1 wls) = Gap(k:
Σo
1
Σ
c
<
1
0
8k
8k
1=1
1=1
i=1
18

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
As corollary, we bound the "functional" Jensen approximation gap: Corollary A.6. Let X Px be random variable finite set X. Let F (AC-1)2 be class of functions s.t. == Ex (upser(1/f((X)) < oo for all i E [c] (where fe == - - fi). Let vp c-1 - R be the negative Shannon entropy function and let v 1- + R be defined v == Ex [+(f())) Ler V Vleor be V restricted to coFF Then for any k E M, we have
Gap(k; v (00F) S
Pi.
8k
i=1
Proof. Note that Ex
Gapp(k; s ( 2 Gap(k; 1/ (r)) Thus,
Gap(k; (1) S Ex
sup lX)
Pe-
8k
feF
8k
To complete the proof of Theorem A.2, we need one more bound:
Lemma A.7. Let p. E A-1! Then
logp; - log gel 5 /2Dxx(pllg)
max KE|c]
min (P:.
Proof. Observe that
max log P; log 9:1 <E|c
$ max (max(1//p, 1/9:))) 1/9:)/p; -6l) iE[c]
(Mean-Value Theorem)
< max((1p;, 1/g:)
Σ
Ips qel
refe]
1=1
< max(1/p:; 1/4:) (plla) kE(c)
(Pinsker's Inequality)
/2DKL (plkapppp)) g)
minsetd (Pt. 9:)
Numerical calculations we did when 2 suggest the bound in Lemma A.7 can be improved to
2D 2Pki(pllg)
max log Pt log qil <
AELc
mingele (pt.
However, the dependence mingeld (Pi, 9:) does appear to be necessary. We leave it for future work to tighten this bound.
With these lemmas, we can now prove the main result for the multi-class setting when our strong class is not convex.
Proof of Theorem A.2 and 4.3, Similarly to the proof of Corollary A.I, it suffices to prove the inequality
Ex
Ex (XDD)))))) Ex D2a.DDD.a...11a0000)000000000110 )
C
instead.
Let P: = Ex Jsu IX)] for i [c]. By Corollary A.6 and Lemma A.4, have that for all f(cnnv) coF.
1 Σ Pi.
inf F
Ex
(X))) Ex DKL (f(cow)) (h, (X)))
+ 8k 1=1
19

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Thus,
inf Ex (DN,I//1 inf Ex (1 + 8k Pi. fEooFF
fEcoF
Let (prrj)
(Fo()oh,)) (fw hw). Since
Ex (p(wi)) (X) (hw
inf Ex (X SEcoF
have that
inf Ex (hal Ex (X) feco F
Σe Pi-
8k: 1=1
Applying Corollary A. for any <> if E Fis
Ex
(x)))
inf Ex (DKc((((,((x))))5scccccc + t.c JEco
then
Ex (Dx(((()]]lg(mm) (x)) 5 Ex (DKr) Ex [DKL. (g (p (x)]l5(((
s Ex [Ixxl
inf Ex (Ka.ff SECOK
1
Σ pi
8k
< Ex (D)l - Ex (a +et Pr.
8k
Now it remains to bound the left-hand side. First observe that, by applying to Corollary A.1 to Ss h2 instead of g, we have
Ex DEL (x) (pro)) (X)) S Ex DRL. (f
Ex (g (X)]l (X)()
Pi tc.
8k
Now we apply Lemma A.7. First note that
Ex (I (X))] - Ex [DxL(0(X)]1
$ Ex
(pri) (X)) (x..f((X]llia(hhxxxx))
< Ex
(pro)) (X) - log (,,
(ooa))
1=1
log g. (ori)) (X) - log (h.
< Ex
max ii(d
Using Lemma A.7, we then have
22DKI. (f. (paj) (X))
Ex (D Ex [DKL (0Xx Ex
min(fe. (X) (X). (proj) (X))
vE(e)
Ex
max Piy LiEle!
max pi +
4h
iElo)
3/2
max Pi kE(c)
+ E2
4k
20

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
where e2 -
0. Incorporating this estimation into
Ex
[BxxDRrnlo(X))1100000 Ex (xx 44000//0000000044444444444444444
S
+
Pi
8k
1=1
we have
Ex (XXPK(66(x))11 Ex [Dkr,o(X)]llf (h))))))))))) - Ex (D
3/2
+
Pi
+
max pi iEle
+ E2.
8ki
4k
Since are free to choose as small want, make (K).)). ensuring that the error term is (c/c/k).
Therefore, for every k E N, there exists 5 >> 0 s.t. for all fs co* F for which
Ex (xx(((((hh((Xx)))
s inf Ex (f(,, fEco
+
we have
Ex [Xx(((())]1f(()(x)))) Ex (1 D| Ex -X (Ve/k).
As noted below Lemma A.7, numerical calculations of the optimal bound for Lemma A.7 for c = 2 suggests that our
constant term in the O ( cV/kk) error term be improved from derive this bound rigorously, and determine if the decay rate of the error term in Corollary A. is asymptotically tight.
max F
to
Ps E
We leave it for future work to
A.2.3. A BRIEF NOTE ON THE
APPROACH
As noted in Appendix 1. the Bregman divergence DKL. for multiple classes is derived by arbitrarily choosing to drop the redundant e-th class. What if we had chosen to drop the i-th class instead, for i [c]? We can capture the fact that the underlying theory is invariant under such a choice by the use of smooth manifolds, specifically affine manifolds. manifold M is topological space (satisfying some additional technical properties) equipped with a collection of coordinate charts {UU, (p:) he)) where U, is open subset of M and yo: U, R is topological embedding, s.t. (Uthez is an open cover of M. For manifold M and pair of charts (1) and (U2, sp2), for open Un C U2, call the map P2 "p 11 (U1 n n2)) - (22) transition function. A manifold is smooth if all its transition functions are smooth as functions in the sense of usual multivariate calculus. A manifold is affine if all its transition functions are affine maps.
We extend the notion of convexity an affine manifold M by declaring function & M R to be convex p E M there exists a chart (U, sp) containing p such that ei o y-1 is convex as a function from (U)) C R" to RT. Because all transition functions are affine, if sb convex at p E M with respect to one chart, is convex with respect to all charts containing p. Thus, our definition of convexity does not rely on specific choice of coordinates.
One can then check that Bregman divergences can be defined in similar way that is independent of the coordinate charts. DKL will arise when use the affine charts yp: - i [c]. The calculations we did in the previous section were manipulating DKL with the specific chart &Pc-
e-1 defined by (p(()) := (Pi, Pi-1 Pt+1. ....................... Po) for each
This coordinate-indeeendeer approach is employed in Information Geometry, We refer the interested reader to (Amari, 2016; Ay et al., 2017) for a more in-depth treatment of this approach.
A.3. Generalizing beyond Input Data Distributions with Finite Support
major simplifying assumption in this work is that the input data distribution Px has finite support. When Px has infinite support or continuous, function spaces of models become subsets of needs to be taken to justify the calculations we did in the previous sections.
Banach spaces, and more care
21

---

Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
0:28
2004
8332
0.220
00222
mafll-aain
0 0322
miffl-gain
00246
0:200
0034
0:238
0811
20
80
130
20
92
100
23
60
30
100
(a) Amazon Polarity
(b) BoolQ
(c) CosmosQA
0.574
obb-10
Imagerat
1.15
0.0000
midlf midlt-gan 3ain
mitsft-gaii
0/0675
misfil-giin
0.0005
8670
0077S
100
0.0100
(d) SciQ
(c) CIFAR-10 10
(1) ImageNet
Figure 4. For datasets except ImageNet, we observe that the difference between misfit and gain decreases increases, and also that the decrease slows down with increasing k.. Since ImageNet has 1000 classes, and we only consider values of & till 100, suspect that term dominates the O( (e/kk) error.
The fundamentals of convex analysis in addition many works address the topic of Bregman divergences in infiniteddimensional spaces, with (Bauschke et al., 2003; Reem et al., 2018) being good comprehensive overviews on the topic. Below we briefly summarize the steps needed to complete the generalization.
vectors spaces can be found in (Ekeland & Temam, 1999). In
Given an arbitrary probability measure Px on X' let X ~ Px. Suppose we have strictly convex v Rd - R that is C' (U+). As described in Section 3, y generates Bregman divergence D&- In Section 3.4.3, we showed that the expectation of a Bregman divergence is itself a Bregman divergence of the convex functional [ : Ex [*[[()))] When our function space is infiiitedimmensionll we need a weaker notion of C function. The appropriate notion is that of Legendre function defined in (Bauschke et al., 2003), extending the finite-dimensional notion from (Rockafellar, 1997). Existence and uniqueness of Bregman projections then follows from weak lower- dower-semicontinuity of D and boundedness of the sublevel sets of Da. Theorem 4.1 follows in much the same way, but we need sequential consistency of Dy instead of just D Fortunately, Pinsker's inequality still establishes that Ex DKL. is sequentially consistent with respect to I.P topologies on X' + c-1 for p < oo. This then completes the generalization Corollary A. Many of the details in our proof of Theorem 4.3 in Appendix A.2.2 generalize directly due to convenient properties of convex functions. For example, we can swap expectations when passing the Jensen gap 1/ to bound on the gap of because the gap always nonnegative by Jensen's 's theorem, and Tonelli's 's theorem allows to swap the order of the expectations.
A.4. Plots for Varying
k
See Figure
22


Reviews:
[{"review_id": "gUejxpxMny", "reviewer": "Reviewer_7oRJ", "summary": "This paper generalizes the recent theoretical analysis of weak-to-strong generalization beyond squared loss regression to arbitrary Bregman divergence-based loss functions in the fixed-representation finetuning setting when the strong class is convex.\n- For classification tasks, the authors propose to minimize the expected KL divergence between the strong model’s output and the weak labels by optimizing a convex combination of $k$ logistic regression layers on top of the strong model representation. With this special classification objective, it is shown that the gain in weak-to-strong generalization is at least the KL divergence between the strong model and the weak model at the conclusion of weak-to-strong training (i.e. the misfit).\n- Empirical results on synthetic, NLP (BoolQ, SciQ, CosmosQA, Amazon Polarity), and vision datasets (CIFAR-10, ImageNet) support the theoretical predictions, demonstrating a correlation between misfit and performance gain.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reading our paper, and for your comments. We address your concerns ahead:\n> ...intuition behind key contribution not clearly explained\n\nThank you for bringing this up, we want to make sure that the ideas can be understood since they are counterintuitive! Our understanding of the differences between forward and reverse KL is based on [this nice blogpost](https://tinyurl.com/2p9zr93x). We want to note that, **after the ICML deadline**, an [independent work](https://tinyurl.com/mr2jjbr8) also highlighted the value of reverse KL divergence.\n\nRoughly we should think of forward KL as being *mass-seeking*: It prioritizes learning a distribution that covers all possibilities dictated by the teacher. On the other hand, reverse KL is *mode-seeking*: it prioritizes learning a distribution that captures the most frequent behavior in the teacher. One can notice this behavior in how the two loss functions handle a student that disagrees with the teacher and predicts 0% probability for an event: forward KL will be $+\\infty$ while reverse KL will exclude that event from the loss. Conversely if the teacher predicts 0% probability for an event and the student disagrees, then forward KL will disregard that event, reverse KL will become $+\\infty$.\n\nIn the context of weak-to-strong generalization (WTSG), it is plausible that the teacher makes errors. Thus, we *do not* want our student to be mass-seeking. The student should be free to disagree with the teacher. It should be *mode-seeking* as this is likely where most of the signal from the teacher comes from. We will include this discussion (and possibly visualizations) on the reverse KL in the paper.\n>...strong assumptions like convexity...new insights beyond Charikar et al. 2024...sample complexity, dynamics\n\nWhile the high-level framework of viewing WTSG as “projections onto a convex space” is certainly inspired from the work of Charikar et al. 2024, we want to highlight that our theory in terms of Bregman divergences is more complete, in that it recovers their results for regression as a special case, and also paves the way for other settings. Furthermore, as described in the paper, several things become more complicated while applying this theory to non-symmetric Bregman divergences. We believe that our technical contributions towards fixing these yield insights that are of interest by themselves. \n\nFirst, the fact that the misfit term in the Pythagorean inequality manifests in the reverse direction, shows that the reverse KL is a *better-suited* objective function for WTSG in classification—we show that minimizing reverse KL divergence in the WTS training procedure leads to provable and quantifiable WTSG, whereas minimizing forward KL (i.e., the standard cross-entropy) may not necessarily yield these guarantees. In fact, we also see (see Rebuttal to 93MC) that the reverse KL setup yields better WTSG experimentally when compared to the standard setup. Our result thus shows that there is a certain “directionality” to the correlation between misfit and gain when one goes beyond the squared loss.\n\nSecond, our guarantee (Theorem 4.3) for reverse KL minimization, **holds even if the function space is non-convex** (unlike the work of Charikar et al. 2024). Our solution to handle non-convexity, namely optimizing over a convex combination of $k$ functions so as to approximate the convex hull of the function space—is both practical (as evidenced by the experimental results), and also allows us to derive concrete theoretical guarantees. The error term in the guarantee becomes smaller as $k$ increases, matching the intuition that the convex hull is being approximated better. **Importantly, this solution allows us to optimize over convex combinations of functions from an arbitrary (possibly non-convex) function space, and not just linear/logistic heads!** This opens the gates for obtaining WTSG guarantees with *full-finetuning*, and not just linear probing. In particular, the same bound holds even if we finetune all the weights of a convex combination of neural networks—such freedom is not afforded in the theory of Charikar et al. 2024. Our solution requires deriving an insightful connection with existing density estimation methods, and requires non-trivial adaptations to our setting. We view this contribution to be novel, and possibly also as a useful modeling paradigm in other learning settings, especially given that our experimental numbers improve with reasonably large values of $k$.\n\nFinally, sample complexity bounds are a straightforward adaptation from Charikar et al. (2024), and we wanted to keep the paper focused on new contributions. As for learning dynamics, for our initial study, we assume that the $\\arg\\min$ of the reverse KL can be approximated well (as also suggested by our experiments); nevertheless, this is an interesting future direction.\n\n>...what's $\\mathcal{F}*$\n\nIt is the dual space of $\\mathcal{F}$ (see Line 123).", "claims_and_evidence": "Yes, claims made in the paper are well-structured, supported by clear theoretical analysis and convincing empirical evidence.", "methods_and_evaluation_criteria": "Yes, the theoretical tools used in the paper are reasonable.", "theoretical_claims": "I checked all the theoretical statements and proof sketches in the main text, but not all the proofs in the appendix. The main theoretical results in the paper seem reasonable and well-structured.", "experimental_designs_or_analyses": "I checked the experiments in the main text but not the details in the appendix. The experiments provide sufficient evidence to support the theoretical claims.", "relation_to_broader_scientific_literature": "The paper builds upon recent theoretical insights by Charikar et al. (2024), which first established a misfit-based characterization of weak-to-strong generalization specifically for regression with squared loss. By employing concepts from information geometry and Bregman divergences, the authors extend this analysis to classification tasks.", "essential_references_not_discussed": "To my knowledge, the paper discussed the essential references in the field.", "other_strengths_and_weaknesses": "Strengths:\n- The analysis is well-structured.\n- The proposed objective seems interesting (but could be better motivated).\n- The empirical evidence is extensive for a theoretical paper.\n\nWeaknesses:\n- The intuition behind the key contribution does not seem to be clearly explained in the introduction, lines 071-094. In this paragraph, the authors described a special classification objective whose corresponding misfit is connected to the gain in weak-to-strong generalization. However, the objective is described purely verbally, with neither (a sketching of) the formulation nor any intuitions why the formulation works.\n- A major concern that push me toward a negative rating is that the contributions seem marginal. \n    1. The main contribution of this work is extending the analysis in Charikar et al. (2024) on the connection between misfit and weak-to-strong generalization gain to the classification setting. The main results in the paper either come with strong assumptions like convexity or are built upon special objectives that could be better motivated.\n    2. From a high-level perspective, I appreciate the neat theoretical connection between misfit and weak-to-strong generalization gain as unveiled in Charikar et al. (2024). However, beyond a different objective and the associated theoretical tools, I was having difficulty finding new insights in this work, e.g., why weak-to-strong generalization happens in the classification setting; whether the mechanism is different from the regression setting in Charikar et al. (2024); and how to quantify the gain in weak-to-strong generalization when taking training configurations (like sample complexities, learning dynamics, and architectures) into consideration.", "other_comments_or_suggestions": "Some minor questions:\n- In the abstract, line 24, the notion of \"strong class\" is not clear in the context.\n- Line 260 (left), what's $\\mathcal{F}^*$?", "questions_for_authors": "Major questions are raised in the \"Other Strengths And Weaknesses\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FQh0hcjmJN", "reviewer": "Reviewer_93MC", "summary": "This paper generalizes the conclusion that \"performance gain correlates with misfit in weak-to-strong generalization\" from prior work on squared loss to Bregman divergence loss. It provides empirical evidence through experiments on synthetic tasks, language tasks, and vision tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reading our paper, and for your comments. We are glad that you like our work! We address your concerns ahead:\n>1...additional insights beyond confirming the correlation between misfit and gain. \n\n>2..intuitive interpretation for reverse KL...\n\nWe address both 1) and 2) in our response to Reviewer 7oRJ. Please also see our response to Reviewer fGrp for some intuition on the geometry induced by Bregman Divergences.\n\n>3...impractical aspects...If the experiments were conducted in a more standard manner, would the same trends and conclusions still hold?\n\nWe thank the reviewer for prompting us to do this comparison; we believe the obtained results lend more conceptual support to our theory. For the experiments in Section 5.2, we ran a comparison where we perform linear-probing for the strong model in the “standard” manner. That is, in the weak-to-strong training phase, instead of finetuning a convex combination of $k=100$ linear layers on the reverse KL objective, we finetune these on the forward cross-entropy objective XE(weak, strong) (i.e., the natural direction), and compare the findings to the numbers presented in Section 5.2. Note that the only difference between the settings now is the objective that the linear layers are finetuned on. The table below shows the comparison for the test accuracy of the strong model, as well the XE loss between the ground truth and the strong model.\n||cosmos_qa||amazon_polarity||boolq||sciq||CIFAR10||ImageNet||\n|---|---:|---|---:|---|---:|---|---:|---|---:|---|---:|---|\n||Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|\n|Test Accuracy|0.6378|**0.6407**| 0.8963|**0.8984**|0.6140 |**0.6141**|0.6171|**0.6254**|0.8981|**0.9005**|**0.7133**|0.6986|\n|XE(gt,strong)|0.6147|**0.6141**|0.2788|**0.2570**|**0.6464**|0.6476|0.6439|**0.6436**|0.4052|**0.3361**|**1.4042**|1.4742\n\nThe comparison is indeed quite interesting—we can see that the strong model that is finetuned on the **reverse KL objective shows better final test accuracy (i.e., better WTSG) for nearly all the datasets!** Namely, we do not really see significant performance degradation (in fact, we see improvement in nearly all cases) with reverse KL, when compared to the standard setup.\n\nMoreover, we also computed the gain and misfit terms in the Pythagorean inequality, where for the reverse KL experiment, we compute the reverse misfit that we propose (i.e., KL(strong, weak), whereas in the standard forward XE experiment, we compute the the misfit as XE(weak, strong) (i.e., the “natural” misfit that one might consider).\n\n||cosmos_qa||amazon_polarity||boolq||sciq||CIFAR10||ImageNet||\n|---|---:|---|---:|---|---:|---|---:|---|---:|---|---:|---|\n||Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|Forward KL|Reverse KL|\n|Gain|0.0368|0.0372|0.0717|0.0934|0.0706|0.0694|0.0264|0.0266|0.1772|0.2447|0.5217|0.4517|\n|Misfit|0.6093|0.0613|0.3717|0.1444|0.6319|0.0980|0.6558|0.0961|1.1024|0.3185|3.9392|1.6401|\n\nWe can clearly see that if one were to run the standard WTSG setup, the Pythagorean inequality is completely off (gain and misfit don’t quantitatively align), whereas the reverse misfit is more representative of the gain, as confirmed by our theory as well. Again, this indicates a clear “directionality” in the Pythagorean inequality for WTSG in the classification setting! We would be happy to include this discussion in the final version.\n\nIt is plausible that in practice, running standard forward XE(weak, strong) minimization might lead to a minimizer that is in fact close to the minimizer of the reverse KL(strong, weak) that we propose. It would be an interesting future direction to precisely characterize scenarios when this is the case.\n\nFinally, we want to emphasize that in our experiments, we only do weak-to-strong finetuning of the last linear layer. In contrast, the best numbers reported in the original WTSG paper by Burns et al. 2023 correspond to full finetuning (on the forward KL objective) of all the weights of the architecture, and the WTS accuracies are hence naturally better. The primary concern we point out about reverse KL minimization is the non-convexity of the objective, which can possibly cause issues for first-order methods. However, if one is finetuning all the weights in the architecture anyway, this concern is not central, since in such a case, even the standard forward cross-entropy objective becomes non-convex.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The datasets and benchmarks are appropriate and make sense for the problem.", "theoretical_claims": "I didn’t go through all the proofs in detail, but the theoretical results seem convincing to me.", "experimental_designs_or_analyses": "The experiments are sound and valid. Notably, compared to prior work (Charikar 2024), they additionally include experiments with language and vision tasks.", "relation_to_broader_scientific_literature": "This paper is particularly related to Charikar (2024). It generalizes the prior results on squared loss to Bregman divergence loss, making them applicable to classification tasks. Additionally, compared to Charikar (2024), which conducted synthetic experiments and experiments on molecular predictions, this paper includes experiments involving language and image tasks to validate its results.", "essential_references_not_discussed": "I didn't notice any.", "other_strengths_and_weaknesses": "Strengths: The theoretical results seem solid and effectively generalize prior work. Additionally, the paper includes experiments on a larger scale and in more commonly relevant domains (language and vision) compared to the initial work on misfit-gain relation. \n\nWeaknesses: My main concerns and questions relate to the practical aspects, which are outlined in the questions section.", "other_comments_or_suggestions": "I don't have other comments.", "questions_for_authors": "1. While I appreciate the theoretical contribution of the paper—particularly in how it generalizes the conclusion in Charikar (2024) from squared loss to Bregman divergence, which is a non-trivial task—I still wonder whether it provides any additional insights beyond confirming the correlation between misfit and gain.\n\n2. The authors mention that their theory regarding convex combinations of k strong models provides a \"recipe\" for minimizing the discrepancy between the weak supervisor and strong student, but “in the opposite direction”. While I understand the mathematical formulation, what is the intuitive interpretation of this being in the opposite direction? How does this impact the results differently compared to simply using the regular loss (i.e., placing the supervisor’s output as the first argument)? Would the standard approach lead to worse results and why?\n\n3. In the experiments (e.g., those in Section 5.2), the authors mention using the reverse KL divergence objective for consistency with the theory while acknowledging that this choice might negatively affect performance. Does this suggest certain impractical aspects of the theoretical framework? Specifically, does the theory require an experimental setup that is not practical in order to produce results that align with its predictions? If the experiments were conducted in a more standard manner, would the same trends and conclusions still hold?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FfuMLGn2qp", "reviewer": "Reviewer_D6HR", "summary": "The paper characterizes the gain in weak-to-strong generalization by relating it to misfit, extending the results of Charikar et al. (2024) to general Bregman divergence. This work also weakens the condition on the strong model class, which was considered convex in Charikar et al. (2024), by allowing it to be a convex combination of functions in a general function space.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reading our paper, and for your comments. We address your concerns ahead:\n>...recent references, concurrent works\n\nAs the reviewer notes, **the concurrent works listed were released after the ICML submission (and some fairly recently).** Nevertheless, we will be sure to cite these works and include some discussion in our final revision, since they do provide complementary perspectives.\n>...use of KL divergence in the opposite direction...this deviates from the standard choice...more discussion on this point\n\nThank you for bringing this up, we want to make sure that the ideas can be understood since they are counterintuitive! Our understanding of the differences between forward and reverse KL is based on [this nice blogpost](https://tinyurl.com/2p9zr93x).\n\nRoughly we should think of forward KL as being *mass-seeking*: It prioritizes learning a distribution that covers all possibilities dictated by the teacher. On the other hand, reverse KL is *mode-seeking*: it prioritizes learning a distribution that captures the most frequent behavior in the teacher. One can notice this behavior in how the two loss functions handle a student that disagrees with the teacher and predicts 0% probability for an event: forward KL will be $+\\infty$ while reverse KL will exclude that event from the loss. Conversely if the teacher predicts 0% probability for an event and the student disagrees, then forward KL will disregard that event, reverse KL will become $+\\infty$.\n\nIn the context of weak-to-strong generalization (WTSG), it is plausible that the teacher makes errors. Thus, we *do not* want our student to be mass-seeking. The student should be free to disagree with the teacher. It should be *mode-seeking* as this is likely where most of the signal from the teacher comes from. We will include this discussion (and possibly visualizations) on the reverse KL in the paper.\n\nThe fact that the misfit term in the Pythagorean inequality manifests in the reverse direction, shows that the reverse KL is in fact a *better-suited* objective function for WTSG in classification—we show that minimizing reverse KL divergence in the WTS training procedure leads to provable and quantifiable WTSG, whereas minimizing forward KL (i.e., the standard cross-entropy) may not necessarily yield these guarantees. Our result thus shows that there is a certain “directionality” to the correlation between misfit and gain when one goes beyond the squared loss. In fact, we also see (see Rebuttal to 93MC) that the reverse KL setup yields better WTSG experimentally when compared to the standard setup---to this extent, we view the use of reverse KL as a feature rather than a limitation.\n>...realizability assumption\n\nWe would like to emphasize that realizability of the ground truth is *not* a critical feature of Theorems 4.1 and 4.3. In particular, note that in these theorems, the weak model has no assumptions placed on it. Thus, we could just as easily take $g$ in Theorems 4.1/4.3 to be the optimal strong model that approximates (with respect to any metric) the possibly non-realizable target function, and we would obtain a misfit-gain inequality with respect to this $g$ (see also Section 5.5, where we do precisely this). This perspective is in line with that taken in Burns et al (2023) where they care about *performance gap recovered (PGR)*, which in some sense, does away with the realizability assumption.\n\nAs you mention, a result similar to Theorem 2 in Charikar et al (2024), which incorporates the \"distance\" of the target function from the strong class into the bound, would be interesting. This needs a little more care in the KL-divergence setting since we don’t have a triangle inequality. We left this result out of our paper to focus more on our Theorem 4.3, which generalizes the misfit-gain inequality to *non-convex* classes of strong models. However, in the reference [3] that you mention, this generalization is worked out.\n\n>...lower-bounding misfit by gain\n\nLower-bounding the misfit is tricky because we don’t have a triangle inequality for KL divergence. However, in reference [3], such a lower bound was proven, albeit at the cost of a potentially large constant. Without some regularization assumptions, we suspect it would be hard in general to control this constant.", "claims_and_evidence": "The main theorem statement is clearly presented, and both the proof in the appendix and the proof overview in the main text appear to be well-structured and understandable.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "I have checked the proof of the main results, and it appears to be correct.", "experimental_designs_or_analyses": "The experimental design and results appear to be well-structured.", "relation_to_broader_scientific_literature": "The paper extends Charikar et al. (2024) by generalizing to Bregman divergence and relaxing the convexity assumption. This contribution enhances the broader understanding of weak-to-strong generalization.", "essential_references_not_discussed": "Almost all essential references are included. However, there are some recent references and concurrent works released after the ICML submission period that are not discussed. I suggest the authors include and discuss these references in the next revision. In particular, the discussion with [2] and [3] seems crucial as they are concurrent works.\n\n[1] Medvedev, Marko, et al. \"Weak-to-Strong Generalization Even in Random Feature Networks, Provably.\" *arXiv preprint arXiv:2503.02877* (2025).\n\n[2] Yao, Wei, et al. \"Understanding the Capabilities and Limitations of Weak-to-Strong Generalization.\" *arXiv preprint arXiv:2502.01458* (2025).\n\n[3] Yao, Wei, et al. \"Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL.\" *arXiv preprint arXiv:2502.11107* (2025).", "other_strengths_and_weaknesses": "One crucial limitation of the work is the use of KL divergence in the opposite direction. As the authors noted, this deviates from the standard choice. Despite this, the paper's strength lies in the authors' effort to weaken many of the assumptions discussed in Charikar et al. (2024), such as the squared error and the convex strong model class assumption. See the following sections for my suggestions on improving both the strengths and weaknesses.", "other_comments_or_suggestions": "I acknowledge the technical challenges involved in using KL divergence in the standard direction, and the use of the opposite direction is somewhat inevitable. However, I hope the authors provide more discussion on this point, perhaps by adding a separate subsection or paragraph.", "questions_for_authors": "Can the realizability assumption be resolved? For example, is it possible to provide a similar equation using the infimum of the error between strong models and the target model?\n\nAdditionally, the main results only show that the misfit is upper-bounded by the gain in weak-to-strong generalization, while the experimental results seem to suggest a positive correlation between the two. Can we also show that the misfit is \"lower-bounded\" by the gain in weak-to-strong generalization, or is there a case where the misfit is small but the gain in weak-to-strong generalization is large?\n\nIf the authors' response successfully addresses my suggestions or questions, I am open to increasing my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "flcJHEpG3y", "reviewer": "Reviewer_fGrp", "summary": "This paper extends theoretical understanding of weak-to-strong generalization—where a strong model trained on weakly labeled data can surpass the weak model's performance—beyond regression with squared loss to general loss functions defined by Bregman divergences, including classification tasks with cross-entropy loss. It builds on prior work and generalizes their “misfit-to-gain” result by showing that for convex function classes and losses expressible via Bregman divergences, the gain in strong model performance can be quantified in terms of its disagreement (misfit) with the weak model. The paper further relaxes the convexity assumption by considering convex combinations of k strong models and shows that the misfit-gain inequality still holds up to an error term that vanishes with increasing k. Empirical validation is provided using both synthetic data and real-world NLP and vision benchmarks, demonstrating the theoretical predictions in practice.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for reading our paper, and for your comments. We are glad that you like our work! We address your concerns ahead:\n>...submission is not self-contained\n\nWhile the high-level geometrical framework of viewing weak-to-strong generalization (WTSG) as “projections onto a convex space” is certainly inspired from the work of Charikar et al. 2024 (as we also generously attribute throughout our paper), we do want to emphasize that our theory in terms of Bregman divergences is a strict generalization of the results in Charikar et al. 2024. As such, their result about regression with the squared loss can be deduced as a special case of our Theorem 4.1 (as also stated in Lines 214-216). We do also set up all the preliminaries about Bregman Divergences and their geometry (Section 3) for deriving Theorem 4.1. Nevertheless, the feedback is well-taken; we will try and include a brief discussion of the Charikar et al. 2024 result in Section 3.1.\n\n>...conclusion section\n\nWe will make sure to elaborate more on the future directions mentioned in the conclusion section.\n\n> Could you provide more intuition or visualization around the dual space projections used in your geometric arguments?\n\nThank you for bringing this up. It can indeed be difficult to visualize the geometry of KL-divergence since it can behave quite differently from our familiar Euclidean geometry. Frank Nielsen has a very [nice article](https://www.ams.org//journals/notices/201803/rnoti-p321.pdf) discussing this very topic.\n\nSince we are most familiar with Euclidean geometry, it is best to appeal to those native intuitions when understanding dual space projections. More specifically, our geometric arguments require us to generalize the notion of angles so we can discuss the Pythagorean theorem in a more general setting. Bregman divergence theory gives us such a generalization, but it comes at a critical cost: asymmetry. In Euclidean geometry, we can measure angles between any two vectors by taking an inner-product. In Bregman geometry, we can only measure angles between a primal vector and a dual vector. But once we designate one vector as primal and one as dual, much of our traditional intuition from Euclidean geometry applies.\n\nWhen technically manipulating these expressions, we like to imagine that we are in Euclidean geometry and have access to inner products. The only restriction is we cannot swap terms in inner products or in squared $L_2$ distances. If proofs go through after taking into account these restrictions, then it is very likely they will hold in the general Bregman setting. We would be happy to include some more intuition and visualization regarding this in our final revision.", "claims_and_evidence": "The central claim—that performance gain in weak-to-strong generalization can be lower bounded by a misfit term (e.g., KL divergence between strong and weak models)—is clearly stated and supported by solid theoretical analysis. The authors rigorously derive this claim using generalized Pythagorean inequality for Bregman divergences and provide multiple variants (Theorems 4.1–4.3 and Corollary 4.2) to cover different settings (ideal convex classes, approximate convex combinations).", "methods_and_evaluation_criteria": "Evaluation is carried out on relevant synthetic setups and popular benchmarks like CIFAR-10, ImageNet, BoolQ, and CosmosQA.", "theoretical_claims": "The derivations appear mathematically sound, and the assumptions are clearly stated (e.g., realizability, convexity, sequential consistency). The theoretical results properly extend known results and are consistent with the underlying convex geometry.", "experimental_designs_or_analyses": "The experimental design is solid. The synthetic setup mimics the theoretical conditions closely, and the real-world experiments provide practical relevance. Careful considerations are made in terms of controlling for realizability and analyzing the role of k.\nOne minor weakness is that the empirical loss reduction does not always match the theoretical lower bound (misfit) exactly, but the authors acknowledge this and explain it as a consequence of non-realizability or optimization difficulties.", "relation_to_broader_scientific_literature": "Not very clear. It may connect to the broader themes of co-training, disagreement-based learning.", "essential_references_not_discussed": "I am not very familiar with this field.", "other_strengths_and_weaknesses": "Slight discrepancy between theoretical and empirical gain.", "other_comments_or_suggestions": "1. The submission is not self-contained. Without reading the original paper for misfit, the reader can only understand this paper very vaguely. \n2. It is very hard to understand the content in the conclusion section. I guess it is a wrong paragraph.", "questions_for_authors": "Could you provide more intuition or visualization around the dual space projections used in your geometric arguments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses a very interesting problem using an original tool for the analysis (Bregman divergences). The reviews make the paper stand borderline so I read the paper.

Reviewer 7oRJ asks for a motivation of the tool used, which is indeed a fair question for a technical paper employing a "tangential" approach to the SOTA. The authors did not do an especially good job at answering this question, choosing to justify it by a property of the functions (the projection property) making the result (Cor 4.2) look nice in context, rather than an axiomatic reply: "why indeed should we use Bregman divergences generally ?". Fortunately, the seminal paper by Leonard J. Savage "Elicitation of personal probabilities end expectations" answers the reviewer's question: Bregman divergences have a specific analytical form desirable for properness. Another justification follows from the paper "Clustering with Bregman divergences" by Banerjee et al.: Bregman divergences fully axiomatize the best constant parameter approximation to be the population average, which makes sense in the context. An the work of Reid and Williamson (2 JMLR papers) will definitely conclude this "why" question in support of the authors' approach.

Being familiar with Bregman divergences I have particularly appreciated Th 4.1, which gives consistency to the "switching arguments" property as a core property of (some) Bregman divergences.

I also appreciated the authors' honesty in pointing to concurrent work. I have checked it and at least from a theoretical standpoint, the authors' work is superior.

Therefore, I would tend to accept the paper under the condition that the authors should definitely polish their narrative to support their approach -- it is definitely suboptimal as it stands (see above).
Decision: Accept (poster)