Limitations of refinement methods for weak to strong generalization
Authors: Seamus Somerstep
Yaacov Ritov
Mikhail Yurochkin
Subha Maity
Yuekai Sun
Venue: COLM 2025 (2025)
Keywords: Weak to strong generalization
superalignment
transfer learning
TL;DR: We study label refinement methods for weak to strong generalization.

Abstract: Standard techniques for aligning large language models (LLMs) utilize human-produced data, which could limit the capability of any aligned LLM to human level. Label refinement and weak training have emerged as promising strategies to address this *superalignment* problem. In this work, we adopt probabilistic assumptions commonly used to study label refinement and analyze whether refinement can be outperformed by alternative approaches, including computationally intractable oracle methods. We show that both weak training and label refinement suffer from irreducible error, leaving a performance gap between label refinement and the oracle. These results motivate future research into developing  alternative methods for weak to strong generalization that synthesize the practicality of label refinement or weak training and the optimality of the oracle procedure.

Published as a conference paper at COLM 2025
Limitations of refinement methods for weak to strong gener- alization
Seamus Somerstep" Ya'acov Ritov Mikhail Yurochkin
Subha Maity Yuekai Sun
University of Michigan
* IFM MBZUAI
University of Waterloo
Abstract
Standard techniques for aligning large language models (LLMs) utilize human-produced data, which could limit the capability of any aligned LLM to human level. Label refinement and weak training have emerged as promising strategies to address this superalignment problem. In this work, we adopt probabilistic assumptions commonly used to study label refine- ment and analyze whether refinement can be outperformed by alternative approaches, including computationally intractable oracle methods. We show that both weak training and label refinement suffer from irreducible error, leaving a performance gap between label refinement and the oracle. These results motivate future research into developing alternative methods for weak to strong generalization that synthesize the practicality of label refinement weak training and the optimality of the oracle procedure.
Introduction
Given the rapid advancement of LLMs, it is important ensure that they align themselves with human values. Standard techniques for doing so, such as super- vised fine-tuning and reinforcement learning from human feedback (Ouyang 2022) utilize human-produced data, which could limit the capability of any aligned LLM to human level. This has led to a continued interest in developing techniques for aligning super-human model; such goal is referred to the superalignment of large language models, a term introduced by OpenAl (Leike and Sutskever, 2023).
al.,
The same team introduced weak to strong general- ization as an analogy for superalignment (Burns et al., 2023). Weak strong generalization is meant to act as an empirical verification frame- work for superalignment techniques: small LLM acts as a human producing data while large LLM acts as the superhuman model. Two promising classes of techniques for super- alignment/ weak to strong generalization have emerged. The first directly utilizes the weak data in an alignment/tren /traiiiaannggggggii process; this includes direct weak training and bootstrapping chains of models with weak training (Burns et al., 2023). Understanding the success of this class of tech- niques is a work in progress; the authors of (Burns et al., 2023) find empirical success for them, and in some theoretical frameworks, these techniques are provably successful (Lang et al., 2024; Charikar et al., 2024; Wu and Sahai, 2024; Xue et al., 2025). On the other hand, weak train- ing can lead to deception of the strong model and reduce its capabilities in tasks such as reasoning (Yang et al., 2024b;a) (cf. Figure for a replication). Additionally, the theoretical
0.5
turr Accuracy
0.4
0.2
0.1
0.04
0.06
0.08
0.10
0.12
0:14
Weak Model Accuracy
GPT-3.5-turbo Refinement
GPT-3 5-turbo
oracle
Weak Training
Figure 1: Performance of weak-to-strong methods on GSMSK. Details are provided in Appendix B.
* "smrstep@uumic.eduu

---

Published as a conference paper at COLM 2025
frameworks proposed in (Somerstep et al., 2024; Yao et al., 2025) training the weak model outputs can actually degrade strong model capabilities. In response to this uncertainty, second class of weak to strong generalization techniques that utilize latent capabilities of the large pre-trained LLM to refine the weak labels (Somerstep et al., 2024; Yang et al., 2024b) has emerged. While the empirical results of these techniques are promising, and some theory underpins their success, it is still unknown if refinement (or weak training) is the best that we can do in weak to strong generalization settings. The authors of Yang et al. (2024b) observe that, empirically, refinement underperforms training on gold standard target data (in this context data produced by capable LLM), phenomenon that we also observe in Figure 1. Inspired by this, we seek to study if refinement (or weak training) is optimal in certain settings.
To get partial answer, we adopt a probabilistic set-up (a generalization of Somerstep et al. (2024)) used to justify refinement, and study if it (and direct weak training) optimal for weak to strong generalization. Our framework is fransfer learning framework, the source distribution (X, Y) ~ P represents data from the unaligned model while (X, Y) ~ Q represents data from aligned version of the strong model. The key challenge in weak to
strong generalization is that we only observe weak labels (Y") over Q. From the source and weak data, the goal is to obtain an estimate of the distribution Qyix-
superalignment pretrained LLM
weakly-supervised TL
Yp Xp
(super)alignment task weak teacher
Yo Xe
Yg Xo
Table Transfer Learning
Superalignment
The difference between the base and hypothetical fine-tuned version of an LLM is complex; we assume that this differnce may be attributed to latent concept shift. Latent concept shift occurs when both the source and target distributions are mixtures of distributions indexed by an unobserved latent concept variable K, with only the proportions of K changing between and Q.
Following Wu and Sahai (2024), we consider two desiderata that ensure (non-trivial) weak to strong generalization has occurred:
1. The weak to strong estimator should asymptotically converge to the target (as the number of weak target and source samples grows).
2. Estimators trained with either only the weak target data or the source data should not asymptotically converge the target task.
The second requirement lays a framework where weak to strong generalization task is not trivial. Our primary contribution summarized the following (informal) theorem.
Theorem 1.1. Under the latent concept shift framework, both source and (weak) target distributions are required to produce a consistent estimator. Additionally, the following holds:
1. Both label refinement and weak training produce inconsistent estimators of the target function.
2. There exists a deconnaolution-base procedure meets the consistency criteria.
The remaining discussion sets up the latent concept shift framework and provides results demonstrating when 1.1 holds. As further details about the deconvolution-basee procedure are deferred to Section 4, for now, we'd like to note that it's an idealized procedure and may not be practical for actual weak to strong generalization tasks (e.g. mathematical reasoning); rather, we develop it to show that there is a non-trivial gap in current techniques and demonstrate possible room for im improvement in weak to strong generalization tasks.
2

---

Published as a conference paper at COLM 2025
1.1 Related work
Transfer Learning: In transfer learning, the goal is to leverage pre-trained model or abundant source of data to improve model performance on a target task (as compared to only training on any target data). Often, the form of knowledge transfer that must occur is categorized by the difference in the source and target distributions. Generally, the three categories of transfer are covariate shift (P(X) * Q(X)) (Kpotufe and Martinet, 2018; Huang et al., 2006; Dai et al., 2007), label shift P(Y) # Q(Y) (Maity et al., 2020; Lipton et al., 2018; Zhang et al., 2013), and posterior drift (P())X) + QQ(X))) (Maity et al., 2021; Cai and Wei, 2019; Liu et al., 2020). Our work falls in the category of posterior drift; as in prior works, we must make assumptions on how the source function (Y X) and EY relate. In Cai and Wei (2019); Cai and Pu (2024); Maity et al. (2021) the difference must lie in simple function class (e.g,, a linear function of a sufficient statistic or a polynomial class). In our work, as in Alabdulmohsin et al. (2023), the posterior drift is due to distribution shift in an unobserved random variable.
Weak to Strong Ceneralizator//Suprrliinnnenn theoretical studies of weak to strong generalization (Lang et al., 2024; Charikar et al., 2024; Wu and Sahai, 2024; Somerstep al., 2024). The closest related work is Somerstep et al. (2024) which considers similar causal graph but restricts the relationship between X, Y and Y' to a mixture of linear regressions. In contrast with this work and the work in Somerstep et al. (2024), others have considered frameworks where weak training is sufficient for weak to strong generalization (Lang et al., 2024; Charikar et al., 2024; Wu and Sahai, 2024). Wu and Sahai (2024) show that weak training allows for weak to strong generalization in spiked covariance models through benign overfitting process, Charikar et al. (2024), show that weak training may lead to good generalization properties if the base and target models are close with respect to the representation layer. Lang et al. (2024); Shin et al. (2024) studies properties of the weak data, and establish desirable conditions for weak training.
Several prior works have conducted
Interest in the weak to strong generalization problem has grown on the empirical side. One line of work transfers alignment behavior induced by RLHF from a weak model to strong model by learning the distributional differences in the weak model before and after alignment (Zhu et al., 2024; Zhou et al., 2024). Other works look to expand generalization from weak training to other model properties beyond accuracy. For example, the authors of Yang al. (2024b) improve large models' mathematical reasoning with responses from smaller models while in Pawelczyk et al. (2024) trustworthiness properties such as faimesss privacy, and adversarial robustness are transferred.
Other lines of work include leveraging the temperature parameter for weak training (Zhang et al., 2024), studying deception from weak training (Yang et al., 2024a), and using a dynamic logit fusion approach for weak training (Fan et al., 2024).
2 Set Up: Latent concept transfer
Let P and Q be the data distributions in source and target domains with corresponding probability distribution function (or probability mass function) p and q. Over the source and target distributions, there are three observable random variables of interest denoted X, Y, Y'. The variable X generally denotes model input, Y is gold-standard/ huncorrrpuud model output, and Y' are proxy labels produced from a smaller model.
Goal: We wish learn the target predictive distribution q(y\x) from knowledge of p(x,y,y') and q'x,y').
To make progress, we begin with specifying the conditional dependencies of the data- generating process.
Assumption 2.1. All data drawn from P or Q is generated by the process given in Figure 2 and this process is faithful and Markov. Additionally, K is never observed in either or Q and Y is never observed in Q.
3

---

Published as a conference paper at COLM 2025
Concretely, the data generating process in the source and target specified probabilistic directed acyclic graph. The Markovian and faithfulness assumptions simply imply that conditional independences exist in the data generating process iff they exist in the graph. This factorizes the source and target distributions as
p(x,k,y,y' (()))()))) x, x,k)p(y/ 1x,,), q(x,k,,,y) ((x)q(()x)qq( x,k)q(y x,,)).
Table summarizes the analogy between transfer learning and weak-to-strong generaliza- tion. The variable X plays the role of natural language query to a small large LLM, K is a latent mechanism through which models generate an output, Y are outputs from a strong model, and Y' are outputs from a weaker model. As in other transfer learning problems, the learner has access to source and target samples, though it is key to note that the true labels are only observed over
Assumption 2.2. We assume that the learner has sampling access to the following distributions (X ~ P and samples (X, ~ Q. The latent variable never observed.
In most weak to strong generalization applications, the learner always has access to queries from some dataset of interest, and base/ paseeunalignee large LLM (e.g. a base version of Llama2- 70b). The sampling access to (X, Y) from P represents this. Likewise, access to a weak/ smmll LLM with some expertise on the target task (e.g. a base version of Llama2-7b fine-tuned on good data) is always given; this is represented by the sampling access to (X, Y') over Q- Access to Y' over P depends on the specific weak to strong generalization setting. In some cases, direct access to the base/ baase//nalggeed version of the small LLM (e.g. an unaltered version of Llamm2-7b) and observing the dif- ference between the unaligned and aligned small LLM can be used to "reconstruct" the aligned large LLM (Zhu et al., 2024; Zhou et al., 2024). In other works, namely Somerstep et al. (2024); Yang et al. (2024b), base small LLM observations physically observable data); rather, the conditional dis- tribution P(Y|X, Y") represents the outcome distribution of the base model fed with an X and weak label Y'. We primarily work in this setting.
Y'
X
not available (i.e. (X)Y")p is not
Figure 2: Latent concept transfer
In latent concept shift, the key assumption is that the shift between source and target occurs in the frequency of K. We also allow for a shift in Y' x,,, our reasoning explained below. Assumption 2.3. We assume that p(x) = (xx, and p(y|x, k) = q x, k), but p(k[x) + (k([x) k).
and p(y'l k) +
The shift from pkk[x) to (k)xx) occurs due to the shift in prior over K from P to Q. The shift between p(y' xx k) and q(y' k) represents some imperfection in the aligning process of the weak model. In our setup, recall that p(y|x, y') represents the output distribution of LLM fed with an input x and a weak label y'; is not reasonable to expect that the base LLM is aware of the exact bias present in y' This is encoded by appropriately restricting the shift between p(y'le v, k) and q(y k). Note that we assume the target version of the strong model has (conditional on k) density p(y) x); that this imperfection is absent in the hypothetical gold standard model. The final assumption we make parameterizes each of the distributions of interest.
Assumption 2.4. Let π. represent the marginal distribution of the latent concept. Then for known density functions g(-) and q(.)) parameterized by 17 and @ respectively, the source and target distributions satisfy
p(y|x) Σ <(g(x(11)) (x)0(xx00)::p(/1>> E
xfg(x(m))
(2.1)
Ex Th((x(1)))
Ex Th(
x(((((111k)
((ylx) = Σ
ply;xx,8):44 = Σ
(2.2)
Ev xgg((())))
Σx x((((1111)
4

---

Published as a conference paper at COLM 2025
Assumption 2.4 parametrizes the relationship between X, Y, Y, and K. Our parametric assumption 2.4 leaves questions such as identifiability novel, and is general enough to cover prevalent examples in the literature.
Example 2.5. Wang et al. (2024) assumes that there is set of language modeling lasks T indexed by d, and that for the d-th task it holds that y f(x, ,)),), Px, Pe. By equating the d-th task with the k-th value of the latent concept in our setup, we see that the framework of Wang et al. (2024) is covered under Assumption 2.4.
Example 2.6. Somerstep et al. (2024) assume that y= Ex T& x e, with e ~ NN, Pathak et al. (2024) show that there exists transformer architecture that matches this mixture of regression distributions.
Example 2.7. Xie et al. (2021) posit that an LLM can be represented by a hidden Markoy model (HMM) mixture. In particular, there is a set of transition matrices @ [01,..., 0gk, priors x?, re, emission parameters T, T and a state space H. such that the sequence of tokens (x, y) = (x2, x,,)) satisfies
1
p(ylx)
E
Σ T)(1), 0₂)) jl
II
II
Σ
0k)
T
NEH
f
(011x)
Σ
Σ KRN
t" plylh, ((1|x, 0k)
i Σ 1lh
t)p(/xz)) 0x)
We return to the motivation of problem set up: recall X are meant to represent queries to an LLM, while Y/Y' represent outputs of large and small language models. The latent variable represents some internal mechanism by which language models produce their responses. In each of these examples assumed that language models are essenfially emulating the distributions they trained on. In other words, are assuming that the base (strong model) is trained on a distribution specified by equation (2.1) while the target model is an LLM (hypothetically) trained on distribution specified by equation (2.2). When an LLM receives a query, it internally attempts to infer the concept most related, then samples response from the corresponding expert.
3 Weak to strong generalization strategies
In this section we discuss three possible approaches to the weak to strong generalization problem and their performance within our framework. The first two approaches, weak training and label refinement, have been previously proposed and studied (Burns et al., 2023; Somerstep et al., 2024; Yang et al., 2024b). Within our framework, we will see that each of these fails to produce estimators that satisfy the weak to strong generalization desiderata. Inspired by this, we propose a new weak strong generalization procedure, which is impractical to implement but does enjoy superior theoretical properties.
3.1 Weak Training
Burns et al. (2023) advocate for training base model, then training the base model for small number of epochs on the weak data.. Consider the regression settings of Examples 2.5 and 2.6. In our framework, this essentially corresponds to fitting estimator of the form
tt dm = arg min (inngge,,,rE)) (()
(1) En 7xfx v/) + { [E 7f
y
Here A > controls how much attention the estimator should pay to the source versus the weak target data. Fitting an estimator to a loss combining both sources of data is core component of many transfer learning techniques Maity et al. (2020); Cai and Wei (2019); unfortunately, the presence of weak supervision in the target domain prevents this from being particularly useful in our setting.
5

---

Published as a conference paper at COLM 2025
Proposition 3.1. Consider the regression setting of examples 2.5 and 2.6 with up nio - Let Ep, eạ' be the vectors of the bias in the source and weak models conditioned k. Suppose that μ(x;0) 0) is Lipschitz 0. Then holds that 0wt P o Awi 2 The where Owr The satisfies
x [[L(Vl] Σx π w (x;; mmx) 71|Ep112 (1 (11-1))lleee )) + 1/1 (
Proposition 3.1 shows that within framework, weak training does not satisfy the first criteria of weak to strong generalization (it does not produce a consistent estimator of the target function). This inconsistency within our framework is also prevalent empirically; for example, in Figure 1 GPP-3.5-Turbb does not successfully learn any mathematical reasoning ability from the weak models. While Proposition 3.1 covers the regression case, the intuition carries over to the HMM case. Both the source and weak target data are biased, and thus constructing consistent estimator by mixing the two generally not possible.
3.2 Label Refinement
Simultaneously, Somerstep et al. (2024) and Yang et al. (2024b) introduced a set of methods which feed the weakly labelled data as auxiliary information the source model and then drawing a new label for each query in the target data set. For example, in both Somerstep et al. (2024) and Yang al. (2024b) the first step of the refinement procedure is draw new labels Y that satisfy
Ex - = EyPp(Y
y(p=y/X=x))
1 Qr
(3.1)
In other words, for each query x, they collect in weakly labelled pairs (x), drawn from Qx,y' (i.e fte.ucdd by the weak model), form "auxiliary information" v y/.... Xm, y/m and then draw new label from the source model that is fed with both and x. The notation (P(()) emphasizes that the refined labels are being drawn from the source distribution.
Following Somerstep et al. (2024); Yang et al. (2024b) utilize our framework to study refined data (X, Y) with the following distribution:
X Qxi p(ğ1x) A (ve x) fy ply x, xxv)dQ(v x).
is also possible to perform label refinement by correcting one weak label time (Somerstep et al., 2024), in this case we consider refined distribution of the form 4re (y x) = ply x, x.y x)
3.2.1 Irreducible enror of label refinement
In this subsection, we ask whether for a given query x, the refined label is good stand-in for a hypothetical gold-standard label drawn from the target distribution. To begin, we work (ylxx into a more interpretable form.
Proposition 3.2. For P(k X,K} fv p(k I x,0)g(u|x,K) K)dv, which is an entry of the (conditionallon-X) X) confusion matrix of the prediction k (of true class K' from V), one can write
gre(vlx) = ()(11x and q(k[x) E9(K (x) X,K}
For the case of correcting weak label at time, can simply replace with y'.
Proposition 3.2 reveals the statistical intuition for refinement methods weak to strong generalization: The model internally updates its prior on the latent variable when fed the weak data as auxiliary information. This essentially transfers the problem of estimating (xx) from weak data to inferring the marginal of the latent variable K from pairs (X, VX/ Y'). Unfortunately, Proposition 3.2 also reveals that even refinement does not produce labels that match hypothetical gold standard labels, as if xk(xx) # q(k(x) then q ( * q(ylx).

---

Published as a conference paper at COLM 2025
Example 3.3 (ICL Refinement). Consider the HMM set-up of Example 2.7, with V IT Qrix, and taking ((k) = 1(k = k3} for simplicity. We may then write
M {(k[x) & Evp(k,,,0) R T Io
(3.2)
h
The approximate trealment of the ICL. pairs as iid is justified by the use of additional delimiter tokens in V. For simplicity, we take this as a given and refer to Xie al. (2021) for a detailed discussion.
We also see that for a given k + k", ((k)x)) > O unless the expected likelihood of the sequence of tokens x1, V1.yy...., Xm, Vm is zero under the transition matrix Oz. Thus, under general conditions, the labels 9 will still incur some bias from weighting towards the incorrect concepts, leading to irreducible error.
Example 3.4 (Weak label improvement). The authors of Somerstep et al. (2024) also propose a refinement procedure in which the source model "corrects" the weak label for each data pair (x, y'). We return to the regression selting (Example 2.5) and again assume that q(k) = 1(k = k=}. We
can model this refinement step
((k[x) = Jy- p(k x, kx)
In Appendix A we show that if we let a2(xx = -(xx then it holds that q(<(x) S p(k())e caß(x),
(x))),
We see that for x in which (xxx) is large for all k + k", the label bias can be drastically reduced. Tf, on the other hand, it turns out that A (x) is large due to the distribution shift in y'Ix, then label refinement will suffer from large bias and still have undesirable irreducible error. These examples are evident in Figure 2, where we see that label refinement improves on weak training, but does not compare an oracle proxy.
3.3 Weak to strong generalization through latent concept identification
We have seen that within the latent concept transfer framework, both weak training and label refinement produce biased estimates of the target function. The question remains if there exists procedure that produces a consistent estimate of the target function. To answer this the affirmative, we propose a two-step procedure: first identify the latent mixture components and mixture proportions in P and Q then construct the target function by borrowing mixture proportions from Q and the mixture components from P. We lay this out in Algorithm 1.
Algorithm 1 Latent Concept Identification
Require: Source data Dp {x; V/. y/} target data Do (x;,y/ maximum likelihood esti- mation procedure MLE
1: Compute (0g, 0 fMr A 1K-1 t- MLE(Dp)
2: Compute (0)))
K - MLE(DDo)
3: Compute assignment a(k) = arg min mi (( (00m, 00h0)
4: Compute final estimate q(x) Es a9 (()))1))))))) /EE 8(x0,,
The success of this strategy hinges the MLE iLE/identiiiiation step, we discuss this further in Section We briefly comment on possible relaxation on observing Y' over for the success of Algorithm 1. In particular, note that t does not play role in the final estimate; rather is only used to compute the assignments. With no weak source samples available, one could instead opt to utilize fg in the assignment step.
7

---

Published as a conference paper at COLM 2025
4 Identification and estimation under latent concept shift and weak supervision
In this section, we study the question: When are the parameters of Qyix identifiable? To formalize identifiability, consider the space of measures A(X x 3)) and the space of mea- sures on A(X x }), which we denote as A y). For any mixture density speci- fied by fly,x; x; 7), K) - Σكا " (pg((ing) Ply; x,),, we can define the mixing measure F- (1) 0, K) E - y) as the weighted sum of Dirac measures F. (,,,0) = -K T As in prior work (Aragam et al., 2018), we say that a mixture family fx:; 0) identifiable if the map
xeg((zm))
II A x R) + (² x R)
(x (T, - F
is injective. Recall our goal is learning ((ylx) = Ea ingredients to identifying qylxx) are:
E { Q(v:x,0x) The necessary
1. Identifiability of the mixture system. Note this property is dependent on the functions p(+)..) and g()-)).
matching identifiability condition between {6x and )
Our first contribution is to characterize the identifiability of our general mixture model; this result generalizes those in Nguyen et al. (2024b).
Definition 4.1. We say that the functions "((,,",) 8(:,)) .) are zero'th 'th order algebraically independent if for any collection of distinct parameters {0,}, ] the collection of functions (g(x;; n)) is linearly independent and the collection of functions in {, xx, in x, is linearly independent.
Proposition 4.2. Suppose that (p(:,r)), and g(.,)) are (zero'th th order) algebraically independent. Then the mixture family fx; 0) is identifiable.
Proposition 4.2 establishes a sufficient condition for the parameters ij, 0, 7 being identifiable from observing data (X, Y/Y) from P or Q. The algebraic independence condition ensures that distinct sets of the parameters cannot lead to the same distribution over and Q-
Example 4.3. In Example 2.5, the density of fx, e, needs to satisfy algebraic independence; in Example 2.6 it is sufficient that the { be linearly independent.
In the case of Example 2.7, discussion on when the necessary algebraic independence condition is met is more complex.
Proposition 4.4. Consider the HMM model setup of Example 2.7 and in Xie et al. (2021). Following their nolation, suppose some additional assumptions hold:
1. The state space H. is finite, the tokens x! and also lie in finite token space O. Emission sequences x² x from the mixture of hidden markov models may be arbitrarily long.
2. For value T, there exists set of tokens (o) of E EO such that p( > 0 and hh, =0
ply;h, h, 7) satisfies the algebraic independence condition T.
Then q(x,y') and p(x,y) satisfy the algebraic independence conditions in 0, and T.
Assumption 2 of Proposition 4.4 is fairly strong; in the topic model literature, the special tokens of are referred to as anchor words (Arora et al., 2012) and in hyperspectral mixing they are referred to as pure pixels (Jasmine and Pattabiraman, 2018). Anchor words play an important role in the identification of the parameters of topic models.
Unfortunately, because we do not observe X, Y ~ Q to identify q(ylx) we require more than just identifiability of the individual components. At its heart, need to match the
8

---

Published as a conference paper at COLM 2025
target prior 79 observed from Q' to the source components 1,, observed from P, keeping in mind that each of these is identifiable only up to permutation. To assure this is possible, we impose statistical distinguishabiiityy condition on p(y'l |x, k, and (xy(x,,k)
Assumption 4.5 (identifiability over permutation). Lef [K] = {1,...,K] .). For all k, [K] and
> 0 we assume: (i) arg m #xll e c+ A, and (ii) |10"" "1211 <c
4.1
Consistency of algorithm
Recall our estimator desiderata are (i) the weak to strong estimator should asymptotically converge to the target (as the number of weak and source samples grows) and (ii) an estimator using one source of data should not asymptotically converge to the target. Under assumptions 2.1 2.4 (ii) clearly holds. With no alteration, the source model P( clearly does not match Q(Y since P(k) X) + Q(k Likewise Q(Y' k) + Q( long as f% # lz. We partially establish (i) in the following proposition.
Proposition 4.6. Suppose that the MLE step satisfies the following: For some constants C > and a E [0, 1] it holds that
P(Ile 0|12 [logn//n]]*) S exp(-Cloonn)
A 10 60 lkk x " 1==1
and P(1ee 0| [logne/nol S exp(-Clogno)) E (0 1
Then, if (a(k)) denotes the parameter estimates produced by Algorithm 1 and so long as A max (Ologg//l////))) ((1ogm///// ( then holds that
[|0x(k) 0all min(np, n 2
In the case of regression (cf. Examples 2.5 and 2.6) we may not be simply interested in estimating the parameters, but actually obtaining the regression function Eo[Y|X]. Letting (xx;0)= = = 0J, we may write the function of interest and our estimate (9(x), as
((x) A Eo[Vx]] = =Σ (())()))))))) K Ev The (((()))))
49
(()) (((())
(((x;))), ((x) A
Σ K
1 (xx;; 0 (K)))
En a (x)
Proposition 4.7. Suppose that the MLE step satisfies the conditions in Proposition 4.6. Additionally, suppose that the functions log (x|n) and z 2xx(0) are Lipschitz continuous in their respective parameters. Then (up logarithmic factors) the estimator 4(y) x) from Algorithm 1 satisfies
llq(x) (xx) S min mn. $
Example 4.8. When does MLE satisfy the required convergence assumption? For the case of Examples 2.5 and 2.6, suppose that p x, corresponds to the normal density with mean u(()) and variance a(.) parametrized by x and 0. Ho et al. (2022) show that if u(-) and o(.) meet a pair of higher-order algebraic independence conditions, then the assumption on MLE holds with a 1/4. We also note that the specification in Example 2.6 meets the needed independence condition.
For the more general case, where g(.) ) is not constant in x, possible rates of a or & = 1/2 are established in Nguyen et al. (2024b;a).
4.2 Empirical Example
In order to empirically test the findings in the paper, we have adopted a toy empirical setting from Somerstep et al. (2024). The objective is to teach model a specific persona; we assume that the base model is mixture of an undesirable and a desirable persona (here the persona corresponds to the concept k). We implement Algorithm 1 by clustering data drawn from the source model using K-Means++, computing assignments using cosine similarity, and computing the final estimate by performing supervised fine-tuning on the data from the cluster that is closest to the weak data.

---

Published as a conference paper at COLM 2025
We utilize Dolly as the training set, with weak labels provided by Falcon 7B, Gemma 2B, Ilama2 7B, and Mistral 8b. GPT-4o-mini acts as the strong model. At test time, we measure the strong model's accuracy and use of the desirable persona on Tiny TruthfulQA and AinyAlpaca Eval (Maia Polo et al., 2024). Our findings are consistent with the theoretical findings in the draft. In particular,
1. Weak training biases the model by reducing the accuracy at test time (see content scores of Figure 3)
Refinement does not reduce accuracy but does leave Bias by not fully transferring the persona to the model (see style scores of Figure 3)
3. The identification procedure produces a model with no "bias", the persona is learned and the accuracy is not reduced (see both scores of each Figure 3).
05
15
Wark
Figure 3: Comparing performance of weak training, refinement and the identification procedure persona learning task IrinyIruthfuIQA (left) Alaaavlaal (right)).
Conclusion
In this paper, we have studied two popular classes of learning methods for weak-to-strong generalization: (i) weak training and (ii) label refinement within a general latent concept al. (2024) that includes popular classes of latent concept models for LLMs (Xie et al., 2021; Pathak et al., 2023; Wang et al., 2024). Under the adopted framework, it is shown that there exists weak-to-strong generalization procedure that produces estimators which asymptotically converge to the target function as the number of unaligned base model and aligned weak model samples grows. This result primarily relies on identification conditions for the aforementioned latent concept models for LLMs. We also show that both weak training and label refinement produce biased estimators of the target function, suggesting that they do not satisfy the consistency property enjoyed by the yeak-to-strong generalization algorithm we produce. While both weak training and refinement are tractable, our results suggest the need for further research into seek-tt--ttrong methodologies which enjoy good theoretical properties and remain implementable for practical language modeling problems.
transfer framework, adopted from Somerstep
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and Others. Gpt-4 technical report, 2024.
Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D' Amour, Arthur Gretton, Sanmi Koyejo, Matt J. Kusner, Stephen R. Pfohl, Olawale Salaudeen, Jessica Schrouff, and Katherine Tsai. Adapting to latent subgroup shifts via concepts and proxies. In International Conference on Artificial Intelligence and Statistics, pages 9637-9661. PMLR, 2023.
Bryon Aragam, Chen Dan, Pradeep Ravikumar, and Eric Xing. Identifiability of Nonpara- metric Mixture Models and Bayes Optimal Clustering, arXiv:1802.04399 [cs, math, stat], February 2018
Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topic models going beyond svd, 2012. URL https: //axxiv..vv////////////aa//iii http://aawwv..aabaaaaaaa
1956.
10

---

Published as a conference paper at COLM 2025
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, llya Sutskever, and Jeff Wu. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, December 2023.
T. Tony Cai and Hongming Pu. Transfer Learning for Nonparametric Regression: Non- asymptotic Minimax Analysis and Adaptive Procedure, January 2024.
T. Tony Cai and Hongji Wei. Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifier. XrXiv:1906.022 02200 [cs, math, stat], June 2019.
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to-strong generalization, 2024. URL https://arxiv http:///aaaiii.ooaa//////// horg/bbs/24055 15116.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.cm htt.:/oogg/bbb///11//// 14161
Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. Boosting for transfer learning. In Proceedings of the 24th International Conference Machine Learning, ICML '07, page 193-200, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781595937933. doi: 10.1145/1273496..
URL https://doi: org/10. 1145/1273496. 1273521.
Chenghao Fan, Zhenyi Lu, Wei Wei, Jie Tian, Xiaoye Qu, Dangyang Chen, and Yu Cheng. On giant's shoulders: Effortless weak to strong by dynamic logits fusion, 2024. URL 15480.
Nhat Ho, Chiao-Yu Yang, and Michael Jordan. Convergence rates for gaussian mixtures of experts. Journal
Journal of Machine Learning Research, 23(323):1-81, 2022. URL http://jml. org/ papers/v23/20-1 1129. 121......... html.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https: httpp:://proceedinggs neurips. wwtpcc/pppppeeeeeeeeeeeeeeeeeeee hipper/iile//pppe//26///// a21 att.ee6beaaaaeeeeeaeaaaaaaaaa1-0
Graceline Jasmine and V. Pattabiraman. Improved pure pixel identification algorithms to determine the endmembers in hyperspectral images. Computers Electrical Engineering, 71: 515-532, 2018. ISSN 0045-7906. doi: https:/ http:///w..oogooo..ooooooooeeeep URL wwtphttps::/ htttss:/www..siieeeerrrert...oot com http://ccenneeeeeeeeeee ott.e//eeeerrriieeoe////////9/1/
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL http://aaxxvvoor/abbb/s// 200o//22..............
Samory Kpotufe and Guillaume Martinet. Marginal Singularity, and the Benefits of Labels in Covariate-Shift. arXiv:1803.01833 [cs, stat], March 2018.
Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak- to-strong generalization, 2024. URL htt.:tp://aariv..rr/aaa/////// 160433....... 16043.
Jan Leike and Ilya Sutskever. Introducing superalignment. https://openai. wwtpp/eennn...miiiieeeeeeeeeeeee com/index/ index/
introducing wwweraligneeeneeneeeee 2023. Accessed: 2024-04-27.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and Correcting for Label
Shift with Black Box Predictors. In Proceedings of the 35th International Conference on Machine Learning, pages 3122-3130. PMLR, July 2018.
Ruiqi Liu, Kexuan Li, and Zuofeng Shang. A computationally efficient classification algo-
rithm in posterior drift model: Phase transition and minimax adaptivity, 2020.
11

---

Published as a conference paper at COLM 2025
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating ilms with fewer examples. arXiv preprint (#Xiv:2402.14992, 2024.
Subha Maity, Yuekai Sun, and Moulinath Banerjee. Minimax optimal approaches to the label shift problem. arXiv:2003.10443 [math, stat], April 2020.
Subha Maity, Diptavo Dutta, Jonathan Terhorst, Yuekai Sun, and Moulinath Banerjee. A linear adjustment based approach to posterior drift in transfer learning, arXiv:2111.10841 [stat], December 2021.
Huy Nguyen, Nhat Ho, and Alessandro Rinaldo. Sigmoid gating is more sample efficient than softmax gating in mixture of experts, 2024a. URL 13997.
Huy Nguyen, Nhat Ho, and Alessandro Rinaldo. On least square estimation in softmax gating mixture of experts, 2024b. URL https://arxiv,
240pb.//242..00 02952.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances Neural Information Processing Systems, October 2022.
Reese Pathak, Rajat Sen, Weihao Kong, and Abhimanyu Das. Transformers can optimally learn regression mixture models. In The Twelfth International Conference on Learning Repre- sentations, October 2023.
Reese Pathak, Rajat Sen, Weihao Kong, and Abhimanyu Das. Transformers can optimally learn regression mixture models. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL httpss:
Martin Pawelczyk, Lillian Sun, Zhenting Qi, Aounon Kumar, and Himabindu Lakkaraju. Generalizing trust: Weak-to-strong trustworthiness in language models, 2024. URL 00418.
Changho Shin, John Cooper, and Frederic Sala. Weak-to-strong generalization through the data-centric lens, 2024. URL https://arxiv. wttg/bs////a// 03881.
Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Ya' 'av Ritov, Mikhail Yurochkin, and Yuekai Sun. A transfer learning framework for weak-to-strong generalization, 2024. URL httpps://arxils wwwpaawwwww...ra/////55/555/ 16236.
Gemma Team and Others. Gemma: Open models based on gemini research and technology, 2024. URL https: http:://axvvvvvv wrtp://////////////// http:://arrii..or//bb//22222
08295.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning, 2024. URL https: http://axvvvoooooo orggabs//3/11 11916.
12

---

Published as a conference paper at COLM 2025
David X. Wu and Anant Sahai. Provable weak-to-strong generalization via benign overfit- ting, 2024. URL https://arxiv..r
20/0//41.0..00 04638..
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An Explanation of In-context Learning Implicit Bayesian Inference. arXio:2111.02080 [cs], November 2021.
Yihao Xue, Jiping Li, and Baharan Mirzasoleiman. Representations shape weak-to-strong generalization: Theoretical insights and empirical predictions, 2025. URL https://arxiv. org/abs/2502..00 00620.
Wenkai Yang, Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Zhi Gong, Yankai Lin, and Ji-Rong Wen. (ater(ficia)aaligmmmeet Strong models may deceive weak models in weak-to-strong generalization, 2024a. URL httpp:://rxxv..
https/////rxiv.oo/abb///////// wrgg /246. 11446.11433 11431.
Yuqing Yang, Yan Ma, and Pengfei Liu. Weak-to-strong reasoning, 2024b. URL https: wwtrxii.org/abss.e.
13647.
Wei Yao, Wenkai Yang, Ziqiao Wang, Yankai Lin, and Yong Liu. Understanding the capa- bilities and limitations of weak-to-strong generalization, 2025. URL https::////rtt...o abs/2502. 01458.
Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade, and Eran Malach. Transcendence: Generative models can 11741.
outperform the experts that train them, 2024. URL
Kun Zhang, Mingming Gong, and Bernhard Scholkopf. Multi-source domain adaptation: A causal view. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pages 3150-3157, Austin, Texas, January 2015. AAAI Press. ISBN 978-0-262- 51129-2.
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to- strong search: Align large language models via searching over small language models, 2024. URL https: / /ttp//aa.rggaaaaaaaaeaa /rgg 19262.
Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong preference optimization: Stealing reward from weak aligned model, 2024. URL https: // //axxiv.org//bb/////....... . 18640.
13

---

Published as a conference paper at COLM 2025
A Proofs
Proof of Proposition 4.2. Consider the set up in section 2. We need to show that if fx,, yi yi, 0, K) fxx, y; ô, K) for almost every x and y, then F(1, 0, F(îj, 0, R). Sup- posing that the hypothesis holds true, we have
K
(xg(x(11) ===1 Txg(())1)))
gg((1 fl))
((y;x,00)
(1yxx,00) x, almost everywhere
Axg(x((x))
Under the linear independence assumption on ply; x0) if K & K then there exists kt E [K] such that 0x- & 0g for all k E [K]. But this implies that gxx 1/;- = 0 which is a contradiction since 8(x(7x+) is probability density. Thus we must have K = K. Now note that because the marginal p(X) must remain the same over fxx; fj, 0 K) and fxx; K), we have that
K E
<xg(x(m))
{ =0
Thus is a violation of the linear independence requirement on (x;;)) unless we have that the collections {g(x)))) and (g(x) }jx]) are equivalent. Because only interested in identifiability over permutation of k, WLOG we may take
ggg (x) (x
Integrating over x shows that 7x = As which in turn implies that gxx])e)) ((x) so that f fjK- Now putting what we have so far together we get that Lx (x; 77kx (p(y; x(y;xxx)) for almost every x. Note that (x;; 7jk) 2 so because there can be no linear relationship among the set { 02), p(y; x, x,)),, we have that 0,0, which establishes that F- [. 8, K) = F. (f),
Lemma A.1. Suppose that 0 and 6 are distinct transition matrices for Markov process on a finite state space H. of size M. Then there exists sequence of states hig..., hi such that Po Polh; hhhh.....hhh hi, Pe [h; h.. ht. h;]
Proof. If the matrices diag[0] and diag[d"] are not equivalent, then we are finished, we can take the cycle to be of length 1. If diag(0] * diagl0"), then there exists states hi,, liz such that (h; (h,)) > O'hh;, ). Now if 0h; 1h;,) 2 00 0'(,; (h;,) we are done, so consider the case where 6 (h;, (hi)) < < (h,)) In this case, there must exist state hig distinct from hig and hi such that (hh; (22) > 8' (1t (12)) Consider the m' - 1'th step of this process,
where have constructed a sequence of states Miy.....,_.. , such that 011;,) [1 1) > 0 (ht]Mii) for <j m' that (hti hiy ,) 2 0l (hi hh and the proof is complete or there is distinct (from (he -1)mm state hm such that (hh (,wr.))) > 0' (h: [)) Finally, consider the case where have followed this process to the M'th step, in particular we are at state liser in this case there must exists hi,- such that (hi, (hi))) > (h;, (in otherwise we would have
Suppose m' -
M, then either there exists j < m' such
Σj 0hi, M < 1, the existence of such state completes the proof.
Proof. We establish the algebraic independence condition for the HMM set up. For collec- tion of transition matrices ( and prior ( πg) we may write the distribution of the pair (x,yy)
p(x,)) = ΣΣ ply h,, (4) WEW
Recall for identifiability need that if p(x,y') is the density of y') for the collection of distinct parameters (ōI....,g) and ( Ax] then p(x,y') and p(x,y') need to be
14

---

Published as a conference paper at COLM 2025
linearly independent. Note that we may write
ap(x,y) + (p(x,y))
E p (alssaaa[[[(sssaapaa( + B B1Σ p (1)),, REE
Given the assumption that there exists a vector of tokens (of,...., 0HH) such that p(o h, r²) > 0 and pl h, -²) = O, the only way for the function to be identically zero is if
Σ (x(M)x + PII_(HXX, 0; for all h H
First, we consider the case where the collections {0&} and {6x} are distinct, WLOG we may say that 8 , 8g for all k E K, and 81 & (8xhks2- WLOG, we may reference Lemma A.1 and note that there exists a cycle hi, ... N: hi, such that pı (h; ... hi,) Px (hi ... him hig for k K.
By the assumptions, with non-zero probability it may occur that x 1o ... % I' for
E N+ Letting pk _ pı (h/ ... n hi)) ) for the above to hold, it must necessarily hold that
a. < philleee 0x) II (0 p((o M [pxl my + +BI p(h Mi... 0x) plo (1 =
Ex phh
Oz) [Px] me]
for all I E N , E H
E (hilhi (px Rq.]
Note though that it is impossible for this to hold for all large I, thus if any of the transition matrices are distinct the algebraic independence condition must hold. In the case where the collection of transition matrices {êr } are identical {6k}, a similar argument will show that TTk Ag for all k (i.e.use the fact that 9% + Bg and select cycle as we did before).
Proof of Proposition 4.6. The key ingredient is that step 3 of Algorithm 1 produces the cor- rect assignments. By Assumption 4.5 for all k E [K] and some A > 0 we have that (i) arg min 00 - o > c c A (ii) 110 - -6x 112 Under the assumptions of Proposi-
tion 4.6 we have with high probability
116% - 80h 112 = 110% 8% (8%
x0 + 8%
B' 112 k
~110%' 8% 112 + || (8"" 0% 112 + 110% "11 S ( / +
and additionally for # K,, we have that
110%" e0'1 1|2
110% "
8 + 18th
0x my
8 + 0% 112
S 110 "
8 1|2 + -10% 1 2 +
11
8 1|2
k
((ogm//))))) + 110m
1
Or re-arranging the inequality
lloh x'' 1|2 2 |10% - e 112 (1 2 A ([log=/11])
so that for if A > O x(0g)1)))) it must hold true that arg 1in - 0. 12 = k.
Proof of Proposition 4.7. In the notation of Algorithm 1 we have established that
82 ((x (((K))
q(x) + Σ
EE a(K 8(x (a(k)) (xx;00
A9
a(k) = k for all E [K]
15

---

Published as a conference paper at COLM 2025
So we seek bound on So what remains is to establish the Lipschitz continuity of the function (O(x; flg(x)m))
Σ x mk)
lx
1 (x:0x)
Ex o ((() [x]me))] (x;
(x;xxx)) 0x) (d)
Ee A (x(()))))
(x 17:))
g8g(11
Triangle Inequality E x
(xx; 0k)
[E {x(((1 (xx (xx;
EE & f(
M x; 0z). Note that we can equivalently write
Ey x1
Mo x; 7t, 1,, 0) SFTMAX T18 (18(x)1)))...... log Rxg (x)g((11)))xxxxxx10)
where SFTMAX denotes the softmax function. Thus Lipschitness follows from the assump- tion that log (x(1) and (x(0g) are both Lipschitz and bounded and the fact that the softmax function composed with a set of K Lipschitz function is itself Lipschitz.
Proof. First, multiply the optimization problem by 1/1+>
" (f(x;,)) -
arg gO I+ A Σ (xx;)))) - y1)² + 1+
Σ
Then let = I4X and expand terms to get
Σ f(x 0)2 2nyif 0) (1-1))/ffffpp) + ny? + (1 (1-1)y,² i=1
arg men
Next add and subtract (mz: (1 - 7),) to complete the square, and drop the addition of any terms that are not a function of 0.
Owt = arg min Σ (x; ( + (1 1)),())
Due to the assumption that the function f(x; 0) is Lipschitz in the empirical loss satisfies uniform convergence and thus
Bm L, arg gig Exy.y -
(ny (xl (1- (1-r))))²
Which implies that
0w
L arg E (n En xx;(x;))))))))))))))))))))))){ + (1 (1- 1) E (x;
Note that we may write
Ex (1) Σ (x;(0) + (1- (11-1) 11) Σ T of ))
- + 71
7 17 seofce
By the assumption on θ f(;; ( must be within a ball of radius 12 of 17 Ex n& (xx;0)) +
(1 1) Ee (x; ) which completes the proof.
Proof of Proposition 3.2. We have
[ plylxxx (l1(/////////ll)) = / p(ylx,y') Eg(y |x (g(K)x)))))))))
= -ff
16

---

Published as a conference paper at COLM 2025
Ep(ylx,k) [ p(k)x,y) y) (x))y Ep(ylx,k) [q(K1x) (y
Calculation of Example 3.4. By Proposition 3.2 we have
q(ylx) [p(y(x()(((k1)))
q(k[x) /,, p(k
Now by Bayes' rule note that
Sy.PL p(k x, xx (x,
- S pLV (x,K))((/()x dy [xp(klz)p((/1xx)))
S p(k)x)) Jy p(k(x)p(y'' k, + ((y Now we plug in the parametric form of Assumption 2.4 to arrive at
p(y x, kx,
(A)x)) S p(k[x) x L e 2 12 1)2
+ WV' Mxxay ))?
te a (v' mxxoy 1)2 dy
e 22 (V-(0= ))?
Multiplying the numerator and denominator by e 2 (V/ -ixxee 11? defined as in the statement we have
and letting Af(x) be
2022 (V 1 1)²
(({xx) < p(k|x) x / 4f(x) (x) Mixee dy
1+e
Then we simply break the integral into the two cases: /// 1xx; < 11xx))) and ly' 0x) /4 to arrive at the final result.
B Details of Figure
Figure tests three weak to strong methods using the GSM8K (Cobbe et al., 2021) data set. Weak labels are produced by Llama-2-7B-Chat (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023), and Gemma-1.25 (Team and Others, 2024) models.
provide some expertise on the task, each weak model recieved supervised fine tuning with gold standard data, the data is produced by (Yang et al., 2024b). The strong model is GPT-3.5-Turbo-012 (Achiam al., 2024). We compare thee weak to strong methods: (i) simply training GPT--3.-Turbe on the weak data (ii) Using the ICL refinement method of Somerstep et al. (2024) (iii) an oracle method where GPT-4o produces answers to the training set that GPT-3.5-Turie is trained on. Evaluation is done on provided test set with answer key included, GPT-4o is used to judge if the given test response matches the answer key in both the reasoning and the final answer.
17


Reviews:
[{"review_id": "snBSYp07TR", "reviewer": "Reviewer_PQE1", "summary": "The paper studies methods that try to turn weak supervision into strong performance in the so-called weak-to-strong generalization setting. It places both weak training and label refinement inside a latent concept transfer framework, proves that these common approaches carry an irreducible bias, and gives a constructive though impractical two-step procedure that can close the gap under identifiable mixture assumptions. The analysis is mathematically careful and unifies several prior toy models in one general view.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is timely because many groups now rely on small helper models to align larger ones. The theoretical framework is broader than earlier papers and includes mixture of experts, hidden Markov mixtures, and regression mixtures in one shot. Proofs are detailed and the key identifiability condition is clearly spelled out.", "reasons_to_reject": "The contribution is purely theoretical and the assumptions needed for the positive result are strong, in particular anchor-word style conditions and full knowledge of weak labels in the source domain. No empirical study shows whether the bias predicted by the theory matters in realistic systems. In addition, the exposition is dense and some proofs rely on lengthy technical lemmas that may hide important edge cases.", "questions_to_authors": "Can you provide a small synthetic experiment that illustrates the latent concept identification algorithm and shows the predicted consistency gap?", "extra_scores": {}}, {"review_id": "WspcjvYIrt", "reviewer": "Reviewer_nFeH", "summary": "Authors investigate two primary methods of weak-to-strong (w2s) alignment: refinement and weak-to-strong training. The motivation starts from the performance gap between Oracle training and W2S methods. Specifically, refinement works better than weak training but worse than oracle training. Authors then show that the two methods produce biased estimators (though convergence) and label refinement has irreducible error. To this end, authors propose a theoretical method to leverage the problem.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The angle of transfer learning is interesting, which provides a new perspective in understanding weak-to-strong\n* Examples are helpful for understanding\n* Clear structure", "reasons_to_reject": "I am not an expert in theory and transfer learning, so I did not dive into the details of the theory much. Instead, I have some general concerns:\n\n* The key assumption is latent concept transfer. However, I do not quite understand why this assumption is selected to analyze the weak-to-strong alignment instead of other assumptions. For example, how good is assumption 2.4 in describing the real w2s setting?\n\n* How can the theoretical method become practical, considering g() and \\phi are unknown?", "questions_to_authors": "Typo:\n* Extra bracket in formula on page 4 (top)\n* Misalignment the formula on page 5 (bottom)", "extra_scores": {}}, {"review_id": "eLvW3QwsDn", "reviewer": "Reviewer_BFqD", "summary": "This paper develops a latent‑concept‑shift framework to analyze weak‑to‑strong generalization (WSG) techniques for aligning large language models. Within this probabilistic mixture‑of‑experts setting, the authors (i) prove that weak training and label refinement are inconsistent: each suffers irreducible bias that prevents convergence to the target distribution; (ii) derive information‑theoretic lower bounds that expose a performance gap between these methods and an oracle; and (iii) construct an identification‑based two‑stage procedure (Algorithm 1) that is provably consistent whenever mixture components are identifiable and the weak/strong mixtures are statistically distinguishable. They further supply sufficient algebraic‑independence conditions covering regression and HMM instantiations, and replicate recent empirical results on GSM8K to illustrate the theory.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Recasts WSG as a transfer‑learning problem with latent concept drift, unifying several prior toy models under one general graphical framework.\n2. Shows there is a statistically consistent solution (Algorithm 1) and precisely states the identifiability requirements under which it works.", "reasons_to_reject": "1. Algorithm 1 requires full‑batch MLE over latent mixtures (often NP‑hard) and presumes the number of concepts K is known; no scalable approximation or empirical test is provided.\n2. Key identifiability conditions (anchor tokens, unique mixture components, observed weak labels in the source domain) may rarely hold for real LLMs. Results may therefore overstate attainable guarantees.\n3. Apart from a single GSM8K replication, there is no quantitative study showing the proposed estimator working in practice, nor any ablation on assumption violations.\n4. Empirical or theoretical contrasts with more recent WSG variants (e.g., dynamic logit fusion, preference optimisation) are missing.", "questions_to_authors": "Some proofs (Appendix A) rely on lengthy algebraic‑independence arguments that could be condensed; the paper would benefit from a running toy example to ground intuition.", "extra_scores": {}}, {"review_id": "cyLCFlKvER", "reviewer": "Reviewer_PyrR", "summary": "The paper is on weak to strong generalization, where a smaller model (weak) provides data to train a larger model (strong). The authors argue that existing methods like label refinement and weak training have irreducible errors and propose a deconvolution-based method that might perform better but isn't practical yet. their proposed method only highlights a theoretical performance gap.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has a potential theoretical and analytical contribution to weak-to-strong generalization in LLM alignment. This ** may ** trigger some insights for practical explorations along this direction. However, have to note that this as more of analytical contributions, as additional data points for researcher to examine. The assumptions of the theoretical aspect are in wave-of-hand style.", "reasons_to_reject": "[1] limited practical impact and insufficient empirical validation. \n[2] the empirical evaluation is narrowly focused on GSM8K with sparse experimental details \n[3] only incremental analytical value", "questions_to_authors": "Propositions 4.1 and 4.3 rely on algebraic independence and anchor words. How frequently do these conditions hold in practice for modern LLMs? Have you empirically validated these assumptions (e.g., in HMMs or regression settings), or are they primarily theoretical?\n\nThe latent concept shift framework assumes shifts only occur in KK and Y′Y′, while XX and Y∣X,KY∣X,K remain unchanged. How do you justify this assumption for real-world LLM alignment, where shifts in XX or Y∣XY∣X (e.g., due to time, topic, distribution shifts, or human preferences etc) might also occur?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper provides a valuable theoretical contribution to the problem of weak-to-strong generalization. The reviewers were in consensus about the paper's novelty and the importance of its theoretical perspective. The primary weakness identified by reviewers was the initial lack of empirical validation for the theoretical claims. The authors addressed this during the rebuttal by adding a toy experiment that illustrates the predicted performance gaps and shows their proposed algorithm working in a controlled setting.

For the camera-ready version, I would recommend that the authors add the new experiments from the rebuttal into the main paper, as they strengthen the work.

Overall, the paper's novel framework and insightful analysis of a critical problem in LLM alignment make it a solid contribution to the conference.
Decision: Accept