Does Weak-to-strong Generalization Happen under Spurious Correlations?
Authors: Chenruo Liu
Yijun Dong
Qi Lei
Venue: ICLR 2026 Poster (2026)
Keywords: Weak-to-Strong Generalization
Spurious Correlation
Primary area: learning theory
TL;DR: A unified theoretical and algorithmic study exploring when W2S happens under spurious correlation and how to improve upon failures.

Abstract: We initiate a unified theoretical and algorithmic study of a key problem in weak-to-strong (W2S) generalization: when fine-tuning a strong pre-trained student with pseudolabels from a weaker teacher on a downstream task with spurious correlations, does W2S happen, and how to improve it upon failures? We consider two sources of spurious correlations caused by group imbalance: (i) a weak teacher fine-tuned on group-imbalanced labeled data with a minority group of fraction $\eta_\ell$, and (ii) a group-imbalanced unlabeled set pseudolabeled by the teacher with a minority group of fraction $\eta_u$. Theoretically, a precise characterization of W2S gain at the proportional asymptotic limit shows that W2S always happens with sufficient pseudolabels when $\eta_u = \eta_\ell$ but may fail when $\eta_u \ne \eta_\ell$, where W2S gain diminishes as $(\eta_u - \eta_\ell)^2$ increases. Our theory is corroborated by extensive experiments on various spurious correlation benchmarks and teacher-student pairs. To boost W2S performance upon failures, we further propose a simple, effective algorithmic remedy that retrains the strong student on its high-confidence data subset after W2S fine-tuning. Our algorithm is group-label-free and achieves consistent, substantial improvements over vanilla W2S fine-tuning.

Published as a conference paper at ICLR 2026
DOES WEAK-TO-STRONG GENERALIZATION HAPPEN UNDER SPURIOUS CORRELATIONS?
Chenruo Liu; Yijun Dong, Qi Lei New York University New York, NY, USA (c17758, yd1 319, q1 @nyu. edu
ABSTRACT
We initiate a unified theoretical and algorithmic study of a key problem in weak- to-strong (W2S) generalization: when fine-tuning strong pre-trained student with pseudolabels from weaker teacher on downstream task with spurious correlations, does W2S happen, and how to improve upon failures? We consider two sources of spurious correlations caused by group imbalance: (i) weak teacher fine-tuned on group-imbalancee labeled data with a minority group of fraction lle, and (ii) group-inbalanced unlabeled pseudolabeled by the teacher with a minority group of fraction Y/u- Theoretically, a precise characterization of W2S gain at the proportional asymptotic limit shows that W2S always happens with sufficient pseudolabels when Ta = Hh but may fail when The # Te. where W2S gain diminishes (Th - ' increases. Our theory is corroborated by extensive experiments on various spurious correlation benchmarks and sacher-studen pairs. To boost W2S performance upon failures, we further propose simple. effective algorithmic remedy that retrains the strong student on its high-confidence data subset after W2S fine-tuning. Our algorithm is group-label-fi hauplaaeel-free and achieves consistent, substantial improvements over vanilla W2S fine-tuning.
INTRODUCTION
Traditional learning paradigms like supervised learning and knowledge distillation (Hinton et al. 2015) learn from training data generated by strong teachers, e.g., human experts. In contrast, contem- porary foundation models encode encyclopedic knowledge through astronomical-scal pre-training, thereby achieving comparable or even superior performance human experts in various domains via light post-training adaptation like fine-tuning (Brown et al./ 2220 Achiam et al. (2023). This motivates the question on superalignment (Leike & Sutskever| 2023): can models with superhuman intelligence learn from weaker human supervision? Weak-to-strom (W2S) generalization (Burns) ett al. 2024) provides an encouraging answer for this question: a strong pre-trained student fine-tuned with pseudolabels generated by weaker teacher can often outperform its teacher.
Since the first introduction of W2S by Burns
2024), its mechanism has been extensively studied
empirically (Guo et al.] [024 Liu & Alahi [2244]]] Guo & Yang 2024 Yang et al., 2024 2025 Goel
al. 2025), and theoretically (Lang et al.] 2024 Charikar et al. 2024 Wu Sahai 2025 lkdiz et al.
2025 Mulgund & Pabbaraju 2025 Dong al.)
2025 Medvedev et al. 2025). While existing works |
on W2S generally assume access to clean downstream data, in practice, both the weak teacher and
the unlabeled data for weak supervision often carry systematic biases, such spurious correlations
tied to demographic or acquisition factors (Arjovsky et al. 2019 Sagawa et al. 2020).
This challenge is especially relevant in the very settings that motivate W2S: a student broadly pre- trained on general data is fine-tuned for a specialized task where labeled samples are scarce and imperfect. In medicine, labels may be biased toward certain patient groups (Gupta et al., 2016) or imaging devices (Zech et al. 2018); in law, datasets may overrepresent particular jurisdictions or case types (Chalkidis et al. 2022); ) in autonomous driving, sensor data may be skewed toward specific weather or traffic conditions (Liu et al. 2024). For these specialized downstream tasks, one usually cannot interfere with the data acquisition process, nor obtain additional balanced data. It is therefore
"Equal contribution.

---

Published as a conference paper at ICLR 2026
erucial to understand whether W2S can remain effective under spurious correlations caused by group imbalance- when succeeds, when it fails, and how its procedure can be improved.
Our contributions. We initiate a systematic study of W2S under spurious correlations, providing (i) theoretical analysis that answers the "when" question comprehensively by characterizing the impact of spurious correlations on W2S precisely in the proportional asymptotic limit, as well as (ii) simple, effective remedy for the failure of W2S under spurious correlations inspired by our theory, toward answering the "how" question. Concretely, our contributions are as follows.
A theory of W2S under spurious correlations. In Section 2 we conduct a systematic analysis in the ridgeless regression setting with zero approximation error, where W2S happens due to different estimation emrors (i.e., efficiency in utilizing data). At the proportional asymptotic limit, we provide precise characterizations for the generalization errors of both teacher and student. Consider using weak teacher fine-tuned on labeled samples with a minority fraction ME to pseudolabel N unlabeled samples with a minority fraction The for W2S fine-tuning. We show that (i) W2S always happens with sufficiently large N when T) Th, and improves when the teacher and student have distinct representations; whereas (ii) when He o 'lu W2S can fail even with N + oc, and W2S gain tends to diminish as (M, T/ increases. Our theory is validated with extensive experiments on synthetic regression problems and real classification tasks (Section 3
An algorithmic enhancement for W2S when o Y/u- In Section 4 we propose simple, effective algorithm that retrains the strong student its high-confidence data subset after W2S fine-tuning via the generalized cross-entropy loss (Zhang & Sabuncu 2018). Our method requires no group annotations and improves W2S when the gap between 7) and me large. We conduct extensive experiments on assorted spurious correlation benchmarks (e.g., Waterbirds (Sagawa et al. 2020), BFFHQ (Lee et al., (02T), and ImageNet-9 (Xiao et al.) 20200) across 10 different teacher-student model pairs. The results show that our algorithm achieves consistent and substantial gains over vanilla W2S.
1.1 RELATED WORKS
W2S generalization. Empirically, many methods have been developed validate/enhance W2S
across various vision and natural language modeling tasks, including adjustable loss functions (Guo
et al. 2024), multi-teacher algorithms (Liu & Alahi 2024), data refinement strategies (Guo & Yang
2024
Yang et al. 2024), and the use of weak models for data filtering (Li et al. 2024). Notably, Jeon
2025) specifically focuses on improving W2S performance under distribution shifts. Theoretical
work on W2S is also rapidly expanding, offering various mechanistic explanations from first principles, including the perspectives of neighborhood expansion (Lang al.) 2024). data overlap density (Shin
et al. 2025), transfer learning (Somerstep et al., 2024). teacher-student disagreement (Charikar et al.
2024 Mulgund & Pabbaraju. 2025] Yao et al. 2025 Xu et al., (025), benign overfitting (Wu &
Sahai 2025 Xue et al.) 20255) knowledge distillation (Ildiz et al. 2025), low intrinsic dimension
of fine-tuning (Aghajanyan et al. 2021 Dong et al. 2025), regularization (Medvedev et al.. 2025
Moniri & Hassani 2025), ) and feature learning with different inductive biases (Oh et al.] 2025). However, theoretical understanding of W2S under distribution shift, especially in the presence of spurious correlations, remains limited.
Group robustness in knowledge distillation. When transferring knowledge from a strong teacher to weaker student, knowledge distillation (Hinton et al. group performance (Lukasik et al., 2021] Vilouras et al. 2023 Wang et al.) 2023 Lee & Lee 2023
2015) has been shown to harm the minority
Kenfack et al. 2024)) To address this issue, different methods have been proposed, including adaptive
mixing weights and per-class margins (Lukasik et al., 202T), distributionally robust optimization
(Wang et al. 2023 Vilouras et al. )023)_ last-layer transplantation (Lee & Lee 2023), and gradient-
based reweighting (Kenfack et al. 2024 M Our work differs from these approaches in three key
aspects: (a) W2S generalization, where a weak teacher supervises stronger student, is fundamentally distinct from classical knowledge distillation, (b) we explicitly consider the impact of mismatched minority group proportions between teacher and student, and (c) our method for improving W2S performance does not require any auxiliary information such as group annotations.
Group rohustness under spurious correlations in supervised learning has been extensively studied and is out of scope of this work. We defer more discussions Appendix B
2

---

Published as a conference paper at ICLR 2026
2 A THEORY OF W2S UNDER SPURIOUS CORRELATION
Notations. For any P.q P.q N. p e 9, let Stiefel(p, q) = (Q Rexę Q ) be the Stiefel manifold. @ B Rmpxng denotes the Kronecker product of A E [mxx and B RPx; when 9, let [A; B] E (m=+))x be the vertical stack; when horizontal stack. For any w E Rd and E [d] or I C [d]. let w; and [w]z denote the i-th entry and the subvector of w indexed by I. For any A E RmX and i [m] E [n]. let Aij denote the
p, let [A, B| E Rmx((+() be the
(i, entry; [A] L,: E _2 denotes the i-th row; [A],, E R" denotes the j-th column; and index
subsets I _ [m]. J S [n]] pick the corresponding submatrices.
2.1 PROBLEM SETUP: REGRESSION UNDER SPURIOUS CORRELATION
Downstream task. Consider a downstream regression task characterized by a distribution D(1) X x x 9 + [0, 1] where X' is the input space, y R is the label space, and 9 {0, contains group labels (i.e., 1 for minority and O for majority). The fraction of the minority group in the population is controlled by r [0, 2 such that Pr[g = 1]
Prlg = Pr(g= Definition (Regression under spurious correlations). Let Dx be the marginal distribution of x E X; Dxig be the conditional distribution of x given gi and Dylx be the conditional distribution of y given x satisfying y = fx) + E for unknown f+ X + R and ii.d. label noise e ~ N og) independent of x. Consider two feature maps: (i) the core feature z X - Rdd determines the label y through z(x) N (Oa., Id,) and f.(x) z(x) B. for fixed G_ E W2 while (ii) the group feature& & X + R2 (2 < p < oo) determines the group label g through E(x) ~ Nggu, p)) for fixed jag E RP with fiieenionnindeeenen
[1uee12, 1.
Here, z(x) encodes the core information for predicting y that is invariant across groups, typically rich in semantics and therefore hard to learn thigh-dimensionall while {(x) is latent feature controlling which group x belongs to, typically simpler to represent and therefore now-dimensional.
Classify Cow vs. camel: 2(x) ~ 11)
Platonic representation: Pre-trained models encode concrete objects similarly
dg intrinsic dimensions
Ss
Dr. D3 model sires dp dg
y- (((x)) Br, a.)
Weak teacher (y(x) Up4,00
Foreground in 2(x)
Weak teacher & Strong student
z(x) Rd
U RM
Weak teacher encodes
20x ② 500
Minority Pris = I
5lx) less efficiently:
T'3(0) R
p,
7 E Stiefel(p. Pr
Group feature (p 4 d.y. (ix) 1g ~ (ggn, ofI,)
Majority Prlk 0J l
Strong student encodes ((x) more efficiently: Stiefel(p.Ps
Strong student (x(x) = Ugh.00
ST, R
U3 R al) ST(())
The abstract notion of group encoded distinctly by different pre-training
Teacher-student similarity: ==
Background z(x)) @ 5(x)
Figure 1: Visualization of the theoretical setup in Definitions I |and through Example
Weak vs. strong models. We consider two pre-trained models that provide reasonably high- quality features for the downstream task: a weak teacher model fr X' - lR and strong student model fs X + I. Adapting the setting in Dong et al. (2025). we model fine-tuning in the kernel regime (Jacot et al. 2 Malladi et al. 2023) with low intrinsic dimensions (Aghajanyan al., 2021). particular, we consider learning overparametrized linear layers Qr E RRP and Os E D2 over high-dimensional pre-trained representations fo X' RDr and yss : X - IR Ds respectively. When fine-tuning lies in the kernel regime, ST, Pss correspond to the gradients of the tunable parameters in Jr, fs at the pre-trained initialization, respectively, where Dr. Ds stand for the large tunable parameter counts. The difference between pr, ips that separates the weak and strong models on the downstream task with spurious correlations is pivotal in this setting:
Definition (Weak vs. strong models). (i) The weak teacher representation fr heavily entangles the core and group features: there exists UT E Stiefel (Dr DDrdd) dr) (dr & Dr) such that (x((x)
3

---

Published as a conference paper at ICLR 2026
Uror (x) and Φr(x) = z(x) @ w(x). where w(x) [1; T [(x)] R (2 & Pr & p)
for a fixed T E Stiefel(p. pr - 1) that projects E(x) to a lower dimension (i.e.. Φr (x)
zxx & (T [(x)) E ). We note that dr = prd Let fT = ' Me E BTT
(i) A strong student representation ups partially disentangles the core and group features: there exists Us E Stiefel ds) (ds < D2s) such that (ps(x) = U ( and (x) = z(x) @ (x(x,, where (x(x== [1:S E(x)] E IR (2 & ps & for a fixed S Stiefel(p. ps - 1) that projects E(x) to a much lower dimension, ps < p (i.e., (x( = [x(x); z(x) @ (S E(x)) E Rdds ). We
note that ds
psd=. Let μs = S μ4g E R
Definition 2 formalizes the intuitions that compared to pr, the stronger pe (i) represents the infor- mation required for the downstream task more efficiently (ds & dr) and (ii) partially disentangles the core and group features, bringing robustness spurious correlations. Notice that with z(x) prepending in both (x) and (x), the teacher and student both have zero approximation error (i.e.. both pre-trained models are expressive enough for the downstream task), and W2S happens due to different estimation errors (i.e., the student is more sample efficient than its teacher).
Example We take the well-known analogy of classifying cows (often in pastures) vs. camels (often in deserts) (Arjovsky et al. 2019) as an example (see Figure Z With z(x) encoding the foreground of cows/camels, {(x) represents whether the buckground is typical or not, while z(x) 8 w w(x) and 2xx) & (x) correspond to the representations of background from the weak and strong models.
While the Platonic representation hypothesis (Huu et al. 2024) suggests that different pre-trained models tend to represent similar concrete objects similarly (with the same z(x)), different model capacities can lead to distinct representations of "typical" group in E(x). For instance, strong model that has learned the natural habitat of cows/camels during pre-training can encode typical samples those with their respective backgrounds, leading whereas a weaker model without such knowledge have to rely on more complicated mechanisms to represent typical samples (e.g.. counting), resulting in more complex, higher- jigher-dimeesional w(x).
simple, low-dimensional (xx);
Analogous to Dong al. 20255) a key quantity that controls W2S gain is the similarity between the weak teacher and strong student representations, fT and yss, as formalized in Definition 3
Definition 3 (Teacher-student similarity). Under Definition
2 we define a similarity matrix
T E ()x 1), Notice that
|=|
< ps and 2 1.
= measures the similarity of group features extracted by fr, (Ps, PP,, e.g.,
EB + 0 means w w(x) and
xb( (x) are orthogonal, while ||8| + Ps - means w and (x) are highly aligned.
W2S fine-tuning pipeline.
We consider two training sets with i.i.d. samples: (i) small labeled
training set 5 [(x,, ūe) l [n]) ~ D(ne)" that is privately available only to the weak teacher, so:
and (ii) a large unlabeled training set S₂ = (x i E [N]} from S = {(x; v.) | E [N]} ~ D(
with hidden labels that is privately available only the strong student, Pps, where TE Ta E 10, :.
The W2S fine-tuning pipeline consists of two stages: (i) Supervised fine-tuning (SFT) of fr(-)-
(-) Or on S via ridgeless regression: assuming > dr.
Or argmin 110122 s.f. 0 E argmmnn
1
Σ (p)((x)) 0' 0)),
DER
P'ERPT n
(ii) W2S fine-tuning of f.()) Lss (.) 0g S, labeled by fo via ridgeless regression:
0- argmin 0ER"s
|011 s.l. 0 E argmin
(0 0'
B'ERPS
Following Burns al. (024), in this W2S fine-tuning pipeline, we assume the weak teacher after SFT is fixed and not trainable, accessible in the W2S fine-tuning stage only through inference. Moreover, the labeled training set, S, is only accessible in the first, SFT stage to the weak teacher, whereas the unlabeled set S, is only accessible in the second, W2S fine-tuning stage to the strong student.
FFrr both w(x) and (xx). the first entry 1 effectively prepends the core feature z(x) in pr (x((x) and (x)... which is essential to ensure that both teacher and student have negligible approximation error. Intuitively, pre-trained models have sufficient expressivity to learn the downstream task over population.
4

---

Published as a conference paper at ICLR 2026
Remark 1 (Why ridgeless regression provides sufficient regularization"). We note that under Defini- tion 2 where both şr and (x) are constrained in low-dimensional subspaces, Range Ur) and
Range( U Us). ridgeless regression provides nearly optimal regularization to avoid overfitting (Wu &
Xu
2020 Hastie et al. 2022), which is essential for W2S generalization (Burns et al. 2024). When Pr x) and (x(() are concentrated (in contrast to contrained) in low-dimensional subspaces with
taily evenly distributed in the orthogonal complement, explicit regularization (Moniri & Hassani
2 Dong al.) 2025 or early stopping (Burns et al. 2024 Medvedev et al. (002) becomes necessary to prevent the student from overfitting to noisy teacher labels. Nevertheless, analogous to
Dong et al. (2025), extending our ridgeless analysis ridge regression does not alter our key insights on spurious correlations. Therefore, we focus on the ridgeless setting for clarity of exposition.
The generalization performance is evaluated over a test distribution D(n+)) for some 7) E [0, 1]: with the test risk R. (f) == ((xxy))>D... (t))) (x] y)²), we consider the excess risk
ER,, (f) :- Rr, (f) Rm (f.) R, (f)
In particular, 7/- corresponds to the average test risk; 7/: = 0 corresponds to the majority test risk; and 1/2- = corresponds to the minority test risk.
2.2 W2S GENERALIZATION UNDER SPURIOUS CORRELATION
With the problem setup, we are ready to present our main theorems regarding the effect of spurious correlations on W2S generalization. First, to characterize the excess risks of fr and fs (and thereby the W2S generalization gain) precisely, we push the problem the proportional asymptotic limit:
Assumption 1 (Proportional asymptotic limit). We consider n, + oo with dz/n + % pT. (i.e., n > dr), d + v. E (0.P3') (i.e., N > ds), whereas 2 & ps & pr & p are fixed.
We highlight that in practice, the unlabeled samples are typically much more affordable than the labeled ones, leading to 1/= < 7z. Now, we characterize the excess risks of the weak teacher after SFT and the strong student after W2S tie-ttning, respectively, in Theorems I [ 2 Theorem (SFT of weak teacher (Appendix(D.)) Under Assumption
7 satisfies
E [ER (fr) 5 og To
(1 ne) pT
of
D from label noise
Vi from spurious correlations satisfies
Theorem 2 (W2S, formally in Theorem
3
Under Assumption
②
E D "(n)) [ER (fs) L o3
PTAS))
+
11 T) uer + (m The B
(v=))
of
(0)
v(
<
VL when Nu = ne Es < 1
where PrAs =1 |E| E [L.ps)] is the effective group feature dimension learned by the strong student from the weak teacher controlled by the similarity between yr and yps in encoding group features (see Definition 3 less similar teacher-studens pairs enjoy lower pras: V and v are
generalization errors of is and fr from noisy labels; (1) and ! are generalization errors of fs and fr induced by spurious correlations, Ma; Me + q6 and the higher-order term Es, formalized in Theorem 3 becomes negligible when V2- < 1.
worth noting that the proportional asymptotic limit (Assumption D assumed in Theorems I and 2 can be relaxed to incorporate finite-sample cases via standard edge fluctuation analysis (see e.g.. Hastie et al. (0022); Cheng & Montanari (2024h). We omit such extensions here since they do
not bring additional insights to Theorems and 2
As a special case, without spurious correlations (T = Th = 7/2 or μ4 = 0p)). Theorems and 2 exactly recover the results in Dong et al. (2025) at the proportional asymptotic limit: E E|ER,,, (Fz - y Y-Pr
and EE ER, (fs)] -oyy Y + (vz)), where with a small 1/3 < 1, the W2S gain is larger when the teacher and student representations are less aligned (i.e., lower PrAS). Meanwhile, Theorems and 2 together reveal insights regarding the effect of spurious correlations on the W2S gain,
1
RR := ED(n)" " [ER,, (fr)) - ED(n.) x "Pm)) [ER,,. (fs)),
as discussed in Remark 2 where W2S generalization happens whenever AR, >
5

---

Published as a conference paper at ICLR 2026
(Does W2S happen under spurious correlations?). Theorems 1 and provide mixed answer to this question conditioned on various factors, including the teacher-studens similarity, the separation between groups, and the choice of Thu for given m as summarized below:
Remark 2
(a) W2S happens whenever nue = The and v is small, e.g., when || = 0 and vs < 1, T. > 0 optimized The тe (Figure 2}. We highlight that when The Te. in addition the W2S gain from
variance reduction D V = PT - PTAS 2 O, the strong student improves upon its teacher
in handling spurious correlations: V -VS" ((T) - -me)² (1 1 >
|E
where the gain increases as the teacher-studens similarity decreases.
(b) For fixed E, assume pe + Eps. when V, < 1, the optimal 1uu that maximizes W2S gain is nh wellr 12 (-(nntneell Eug | e.g., when nee = 2. Th = when ||pullla < l ll2.
p Sus 12
nu nei with Euslle
0, nh tendy increase with || and deviate from ne when
|ssll ~ | (Figure 3 left).
(c) W2S gain increases as the
similarity |>| decreases (Figure right).
(d) W2S may not happen if The # TJ, even when Uz < 1 and | = 0, e.g., when de = 0.4 but 0, W2S does not happen if the majority and minority groups are well 12.5(pT - 1). More generally, for | 0, y increases proportionally to (Na - mm)),, and thus AR, diminishes as the gap increases.
The = 0.1, with ||
separated: 2 < for any Vs if ||/T|//02
In Appendix F
further discuss implications of Theorems aand the fairness of W2S.
2.3 SYNTHETIC EXPERIMENTS
By
V/
Up 0.04
Theoretical Empirical
fy 0.5 0.5
0.5
1.0
= 0.1 == 0.5
n n= 0.1
n==
1.0
0.9
1.2
goin
0.0
1.1
0.7
0.50
0.6
0.6
0.300
0.9
62
-0.10
0.0
Figure 2: W2S gains across different combinations of ne and 7e- Each panel shows theoretical (solid lines) and empirical (circles) results for W2S gain as function of Tas across different Vs values. Here we fix Ju,, pss. E, and d2 with || = 10.0, || = 0.1, |E| = 0.1ps. Vertical dashed lines indicate the theoretical optimal 77 values that maximize W2S gain.
17 0.1, different || fixed ||
0.1ps
0.1, different ||E| fixed ||sa|l 0.1
0.625
0.600
W25 gain
0.575
0.550
0C -034
0.525
0.500
0/ -1013
-0.10
-0.10
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
||ea|| - 0.1
||A:11
||| - 0.10s
||||| - 0.2ps
||sl1 =
|| = 10
| 0.4p,
||i|| = 0.8ps
Figure 3: Impact of μı and E on W2S gain. Both panels show theoretical (solid lines) and empirical (circles) results for W2S gain as a function of nn. Fixed parameters: me = 0.1, T = 0.5, V2 0.04, μy 10.0. Left: varying ||l with fixed | f 0.1ps. Right: varying | with fixed pss = 0.1. Dashed lines indicate the theoretical optimal n7, values that maximize W2S gain.
II practice, is typically fixed and known (e.g., given weak teacher fine-tuned on the Waterbirds training set), while 7/u can he controlled by the practitioner when collecting unlabeled data for W2S fine-tuning.

---

Published as a conference paper at ICLR 2026
Figures and validate the theory in Section 2.2 through synthetic Gaussian experiments, with fixed d, = 2048 in all experiments. We begin by examining how varying 7u affects W2S gains under different values of m/. As shown in Figure 2 when | F is small (a distinet teacher-student pair), W2S gains are maximized at Tu R Tee for both balanced (7) = 0.5) and highly spurious (n) = 0.1) unlabeled data. This holds for both the average test risk and the minority test risk, consistent with 2 a). Moreover, the magnitude of the W2S gain decreases as Vs increases, reflecting the role of &s in Theorem 2 Figure left shows that as || increases so that |ااكl becomes non- negligible compared to | l12. the optimal value n. gradually shifts away from Me. This indicates that in some special cases 7h, may not lie near ne. consistent with Remark 2b). Figure 3 right
Remark
illustrates that the W2S gain decreases as the teacher-student similarity |=] F increases, consistent with Remark Cc).
REAL-WORLD EVALUATION
Now extend our theoretical understanding of W2S under spurious correlation to real-world tasks. We first leverage the theoretical framework to interpret our findings on how spurious correlations affect W2S performance across real-world benchmarks.
3.1 MODEL AND DATASET SETUP
Pre-trained models. Our weak teachers and strong students are drawn from diverse set of pre-trained vision backbones that differ in architecture and training paradigm. Specifically, we consider ResNet- 18 (8 (He et al. 2016), CLIP ViT-B/32 (Clipb32) (Radford et al. (02I). ConvNeXI-L (ConvNeXt) (Liu et al. 2022 DINOv2 ViT-L/I4 (DINOv2) (Oquab et al.] (023), and MAE VIT-B/16 (MAE) (He et al. 2022). For each experiment on a given dataset, we include all teacher-student pairs whose relative strength (measured by accuracy) remains stable when we vary parameters including ne. M/u N, n. We freeze all backbone parameters, view the pre-trained feature for the teacher and the student as yP and yss. and only finetune the classification head.
Datasets. From both theoretical and practical perspectives, effective W2S requires that the pre- trained weak teacher and strong student have learned feature representations that are useful for the downstream task. Therefore, we evaluate W2S performance on widely used spurious correlation benchmarks that are relatively close to the pre-training distribution. These include Waterbirds (Sagawa al. 2020). BFFHQ (Lee et al., 202T). and ImageNet-9 (Xiao et al.) 2020). which contain spurious corre lations between background and bird labels, age and gender labels, and background and object labels, respectively. We further provide a self- generaede dataset, BG-COCO, by creating spurious correlations between cats/dogs from COCO (Lin et al.] 2014) and indoor/outdoor scenes from Places (Zhou et al. 2017). We note that in all four datasets, the spurious correlation arises from highly imbalanced group proportions between the majority and minority groups. We denote the minority group proportion in the original training set of each dataset as 1J., which equals 0.05, 0.005, O, and 0.05 for Waterbirds, BFFHQ, ImageNet-9, and BG-COCO, respectively. Detailed configurations of the dataset splits are provided in Appendix E.
3.2 INTERPRETING W2S UNDER SPURIOUS CORRELATIONS
We investigate how the proportion of the minority group in the unlabeled data affects W2S per- formance when the labeled data are either group-balanced or group-imbalanced. Specifically, we fix ?) 0.5 and т No. respectively, and vary 7/, while recording the change in W2S gain Figure
4 presents the average W2S gain across all teacher student pairs on all four datasets. More comprehensive results are provided in Appendix E.2
Our results show that, for both average accuracy (7): 0.5) and worst group accuracy (7: 1), increasing the minority proportion the unlabeled data improves W2S performance when the weak teacher is free of spurious correlation (7 0.5). Moreover, when the weak teacher itself encodes spurious correlation (77 = To),, the W2S gain is consistently positive across all four datasets at 7/ = 7/a but surprisingly decreases for more balanced data as 7/h increases from 7. 0.5. Overall, W2S gain is negatively affected as the gap between 7/,, and n) increases. These observations echo theory and synthetic experiments (see Theorems [ and 2 Remark [ and Figure 2), showing that our theoretical findings on regression extends natually to broader, real-world classification problems.
For classification tasks, W2S gain refers the improvement in test accuracy achieved by the strong student after W2S fine-tuning over its weak teacher.
7

---

Published as a conference paper at ICLR 2026
Waterbirds
BFFHQ
BG-COCO
ImageNet-9
0.5
7:
0.06
0.06
Average Accuracy
0.02
0.02
0.00
0.00
-0.02
0.04
0.04
Worst Group Accuracy
0.10
0.05
0.00
0.05
-0.05
0.00
-0.10
0.00
0.05
0.10
0.15
0.20
0.1
0.2
0.3
0.4
0.5
Figure 4: Average W2S gain across all teacher-student pairs as a function of The on all four datasets. Top row: average accuracy; bottom row: worst group accuracy. Left column fixes 7) = 0.5; right column fixes п No. For т 0.5, curves plotted over shared Thu interval aligned across datasets (bounded by minority group sample availability) to enable direct comparability. For TJ YJo- each dataset is plotted from its 7/ (0.05, 0.005, 0.05, and 0 for Waterbirds, BFFHQ, BG-COCO, and ImageNet-9, respectively) up to 0.5. ImageNet-9 does not have a clearly defined worst group and therefore omitted from the bottom panels.
ENHANCED- W2S METHOD
introduce simple retraining method based on student confidence and generalized cross-entropy to strengthen W2S under spurious correlations. We show that this approach remarkably outperforms vanilla W2S across multiple datasets and large pre-trained backbones, without requiring any group annotations.
Inspired by the theory and observations in Sections 2 and
Method. Both our theoretical results and empirical findings indicate that W2S gain is noticeably reduced when there is large discrepancy between the minority proportions of the unlabeled data and the labeled data. Therefore, we propose a simple method that requires no group label annotations and capable of improving W2S gain in two particularly important settings: are heavily affected by spurious correlation while the unlabeled data are free of it (Π) = !Jo- 7he = 0.5), and the other where the unlabeled data suffer from spurious correlation while the labeled data balanced (n) = 0.5, The = To))
where the labeled data
Formally, let the unlabeled data be S [xx, ÿ:) [N]), where fi is the pseudolabel given by the weak teacher x; E Sz. Our method enhances W2S gain by retraining the strong student after W2S fine-tuning, based on two components: (i) selecting fraction p e (0, of S consisting of the samples with the highest student confidence (equivalently, the lowest entropy), and (ii) applying the generalized cross-entropy (GCE) loss (Zhang & Sabuneu, 2018) with parameter q (0, 1] to each (x;, ūs) in this subset:
1 Pj, (x:)
LxcE(x,, (::9)
q
where Pg. (x;) is the softmax probability that the student assigns to the pseudolabel d for x,-
In both settings (Ne No. The = 0.5 and 0.5, no). selecting a high-confidence subset of the student's 's predictions filters for samples where all relevant features are clearly expressed and effectively used during prediction, thus preventing the strong student from over-relying any single (potentially spurious) feature. Moreover, unlike the CE loss which imposes overly strong penalties on high-confidence but incorrect pseudolabels, applying the GCE loss the selected subset mitigates
8

---

Published as a conference paper at ICLR 2026
the negative impact of pseudolabel noise from the weak teacher More importantly, for the
Ne = 7o. 7u 0.5, confidence-basee -base selection further provides a crucial benefit. As shown in Appendix E the high-confidence subset tends filter out larger fraction of minority samples
to effectively reduce the new Y/u during retraining. This observation aligns with our theoretical prediction that when f/ lJo. decreasing 1/μ, from 0.5 leads to improved W2S gain.
Teacher-Student pair
Dataset
T/
7h
DINOv2 ConNNeXi
DINOV2 Cliph32
DINOv2 ResNeti8
DINOv2 MAF
ConvNeXt Cliph32
ConNNeXt ResNer18
ConvNeXt MAF
Cliph32 ResNet18
Clipb32 MAF
ResNet18
MAF
0.5 Vo
6.60
11.29
7.34
16.68
3.79
2.05
6.28
2.07
0.77
Waterbirds
To
0.5
7.19
13.86
11.73
11.62
2.85
2,02
4.33
1.32
14.54
0.5 %
6.85
2.75
8.42
4,93
4,05
6.54
5.12
BFFHQ
7/ 0.5
3.92
8.53
2,02
4.56
2.09
2.06
-1.37
BG-COCO
0.5 1/
5.38
13.40
12.88
24.01
9.82
6.49
15.25
3.39
12.43
2.05
%o
0.5
10.21
16.99
12.25
-3.52
3.41
1.21
-3.07
3.48
0.31
3,70
IeageNet-99
0.5
1/
6.03
7.45
24.11
4.74
5.30
18.49
4.22
21.73
17.98
0.5
8.21
11:28
22.00
3.77
1.81
10.50
4.51
23.24
15.76
Table 1: Relative improvement of Enhanced- W22 over vanilla W2S (%, measured by average accuracy) across all datasets and teacher-student pairs. Each entry reports the mean improvement over all N, n combinations. For each model pair in the table header, the assignment weak teacher and strong student depends the dataset. We report for each dataset only those model pairs whose relative strength relationship remains consistent across different (me, no) settings within that dataset.
Weak Teacher
W25
Enhanced W25
(Clipb32 DINOV2)
Waterbirds (Cliph32 DIN
(DINOV2 Clipb32)
N=4000
N= 7216
N16060
0.5
Worst Group fccuracy
0.8
0.6
0.7
0.5
0.5
8.7
0.6
0.5
0.4
3000
1500
2000
2500
3000
35300
4000
1000
2000
3000
4000
5000
1000
1500
2000
2500
3000
n= 3000
n=1000
p=3000
Group Accuracy
0.90
0.925
0.500
0.85
0.5
Worst Grost
0.85
0.80
0.875
0.75
0.825
0.800
3000
4000
3000
4000
Figure 5: Comparison of Enhanced- W2S and original W2S for the (Clipb32, DINOv2) pair on
BG-COCO, Waterbirds, and BFFHQ. Top worst group accuracy with π Mas 7u- = 0.5 (fixed N, varying n).. Bottom row: worst group accuracy with 17 = 0.5, Thu 7o (fixed n, varying N).
Main results. We evaluate our Enhanced- W2S method across all four datasets. Table D reports the relative improvement of Enhanced-Wi W2S over vanilla W2S for each teacher-studen pair. Figure 5 further visualizes the performance Enhanced- W2S versus vanilla W2S for representative model pair. Overall, for both average accuracy and worst group accuracy, Enhanced- -25 achieves consistent and substantial improvements over vanilla W2S under both (Me, Th.) settings. Specifically, Table
shows that 67 out of 70 model pairs exhibit a positive gain (measured by average accuracy), with the
analogous to original use in Zhang Sabuncul (2018) for handling noisy labels. Different from the setting studied in Nam et al. (2020). where GCE loss on ground truth labeled datasets with spurious correlations was observed to amplify bias. in our method GCE loss applied to pscudolabeled dataset restricted high-confidence subset, and thus serves a fundamentally different role.
"n our Euhanced-VV W2S method, the of GCE

---

Published as a conference paper at ICLR 2026
mean relative improvements across all pairs reaching 7.02% (Waterbirds), 4.32% (BFFHQ), 7.50% (BG-COCO), and 11.73% (ImageNet-9). In addition, the relative improvement of Enhanced- W2S in terms of worst group accuracy across all pairs is 21.14% (Waterbirds), 3.81% 1% (BFFHQ), and 7.73% (BG-COCO). Further details are provided in Appendix E.3
Waterbinds
BFFIQ
BG-COCO
ImageNet-9
Metric
Comparison
(x (x107))
(me, No)
(Te, nu)
(me, 1w)
(Ve,
(0.5, No)
(%o. 0.3)
(0.5, Ma))
(9a, 0.5)
(0.5, No)
(Ne 0.5)
(0.5,
(Mo, 0.5)
Enhunced
Average
1.00
3,51
3.41
0.52
0.51
Enhunced W2S
0)
2.39
3.79
1.46
Enhanced W2S
Average
2.79
4.84
2.77
2.69
4.97
3.37
4,51
5.08
Enhanced W25 (p
Worst
3.57
8.58
2.75
3.73
6,26
5.44
Table 2: Ablation across four datasets: improvements of Enhanced-W2S over two baselines, using only the CE loss (i.e., the q 0 limit of GCE) and using all unlabeled data (p 1), in terms of either average accuracy or worst group accuracy. For each dataset, improvements are computed as the mean across all model pairs whose relative strength relationship remains consistent under different (me, Bu) settings. ImageNet-9 has no well-defined worst group, so those entries are omitted.
Ablation study. We conduct controlled ablations examine the contribution of the two key components of our Enhanced- methods, namely the use of the GCE loss and confidence-based selection. Specifically, we conduct two separate ablations: (i) replacing the GCE loss with the standard CE loss, and (ii) replacing confidence- based selection with using the full unlabeled dataset. We then retrain the model under each variant and compare the results with the original Enhanced- W22S method. Table
Ssoowss shows that under both (Me, Mh)) settings, the GCE loss and confidene--asse selection each play positive role in improving W2S gain. When ?) T). 7)u 0.5, the impact of using CE loss consistently smaller than that of using the full unlabeled dataset; whereas under π 0.5, Th Too the effects of the two ablations are more comparable. This suggests that filtering out minority group samples via high-confidence selection plays more critical role in the former setting, while in the latter setting the contributions of GCE and tigh-coofidence selection are comparable.
5 CONCLUSIONS, DISCUSSIONS, AND FUTURE DIRECTIONS
In this work, we start with a theoretical framework that models W2S generalization under spurious correlations induced by group imbalance. Within this framework, precisely characterize how different factors, such as the proportions of minority groups in labeled and unlabeled data and the teacher-student similarity, affect W2S, which is validated through extensive synthetic experiments and on diverse real-world tasks. Inspired by our analysis, we proposed Enhanced- W22, a confidence- based retraining algorithm that does not require any group labels and substantially improves W2S gains when the labeled or unlabeled data are highly group-imbalanced. The effectiveness of this approach is demonstrated across assorted real-world datasets.
It is important to emphasize that spurious correlations in W2S constitute a critical issue that deserves closer attention. Beyond standard benchmarks, such correlations can pose substantial risks: in socially sensitive domains they may reflect demographic biases, and in safety-critical applications they can degrade reliability under rare but high-stakes conditions. While our experiments focus on public computer vision benchmarks, the mechanisms we analyze are broadly relevant. Our algorithm provides the first attempt to improve W2S in this setting, and we hope this work will inspire further efforts toward more reliable and efficient W2S methods.
Meanwhile, from a more technical perspective, another exciting future direction is to investigate W2S generation (with without spurious correlation) beyond the kernel regime by taking the training dynamics of the teacher and student models, conditioned on their pre-trained initializations, into consideration.
ACKNOWLEDGMENTS
The authors would like thank Jason D. Lee and Denny Wu for insightful discussions. YD acknowledges support of the NYU Courant Instructorship. QL acknowledges support of NSF DMS-2523382 and DOE Office of Science under Award #DE-SC0024721.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319-7328, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paze Invariant risk minimization. arXiv preprint arXiv:1907.0289) 2019.
Simone Bombari and Marco Mondelli. Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-paramtttriizaionn. arXiv preprint arXiv:2502.01347, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerr. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: eliciting strong capabilities with weak supervision. In Proceedings of the 41st International 2024.
Conference on Machine Learning, pp.
Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Felix Schwemer, and Anders Søgaard. Fairlex: multilingual benchmark for evaluating fairness in legal text processing. arXiv preprint arXiv:2203.0722 2022.
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to-strong generalization. Advances in neural information processing systems, 37:126474-12649, 2024.
Chen Cheng and Andrea Montanari. Dimension free ridge regression. The Annals of Statistics, 52(6): 2879-2912, 2024.
Yihe Deng, Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. Advances in neural information processing systems, 36:1390-1402, 2023.
Yijun Dong. Kevin Miller, Qi Lei, and Rachel Ward. Cluster-aware cuni-supervieed learning: relational knowledge distillation provably learns clustering. Advances in Neural Information Processing Systems, 36, 2024.
Yijun Dong, Yicheng Li, Yunai Li, Jason Lee, and Qi Lei. Discrepancies are virtue: Weak-to-strong generalization through lens of intrinsic dimension. In Forty-second International Conference Machine Learning. PMLR, 2025.
Shashwat Goel, Joschka Strüber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines ai oversight. In Forty-second International Conference on Machine Learning, 2025.
Jianyuan Guo, Hanting Chen, Chengeheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vision superalignment: Weak-to-strong generalization for vision foundation models. arXiv preprint arXiv:2402.03749, 2024.
Yue Guo and Yi Yang. Improving seak-to-strong generalization with reliability-aware alignment. arXiv preprint arXiv:2406.1900 19032, 2024.
Alpana K Gupta, Mausumi Bharadwaj. and Ravi Mehrotra. Skin cancer concerns in people of color: risk factors and prevention. Asian Pacific journal of cancer prevention: APJCP, 17(2))52577, 2016.
11

---

Published as a conference paper at ICLR 2026
Yujin Han and Difan Zou. Improving group robustness spurious correlation requires preciser group inference. arXiv preprint arXiv:2404.13811 2024.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 103.0053,,, 2015.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint urXiv:2405.07985 2024.
Muhammed Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. In ICLR, 2025.
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35: 38516-38532, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Myeongho Jeon, Jan Sobotka, Suhwan Choi, and Maria Brbié, Weak-to-strong generalization under distribution shifts. arXiv preprint arXiv:2510.21332. 2025.
Patrik Kenfack, Ulrich Aïvodji, and Samira Ebrahimi Kahou. Adaptive group robust ensemble knowledge distillation. arXiv preprint WxXi:22411114988 2024.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International conference on machine learning, pp. 5815-5826. PMLR, 2021.
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robustness with fewer annotations. Advances in Neural Information Processing Systems, 36, 2024.
Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to-strong generalization. In The Thirty-eighth Annual Conference on Neural Informatinn Processing Systems, 2024.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of statistics, pp- 1302-1338, 2000.
Jiwoon Lee and Jacho Lee. Debiased distillation by transplanting the last layer. arXiv preprint 111.1 2023,
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123-25133, 2021.
Jan Leike and Ilya Sutskever. Introducing superalignment, July 2023. Accessed: 2025-09-24.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang. Ning Cheng, and Tianyi Zhou. Superfittering: Weak-to to- to-strong data filtering for fast instruction-tune -tnning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14255-14273, 2024.
12

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chenruo Liu, Hongjun Liu, Zeyu Lai, Yiqiu Shen, Chen Zhao, and Qi Lei. Superclass-guided repre- sentation disentanglement for spurious correlation mitigation. arXiv preprint arXiv:2508..855 2025a.
Chenruo Liu, Kenan Tang, Yao Qin, and Qi Lei. Bridging distribution shift and ai safety: Conceptual and methodological synergies. arXiv preprint wtw::2255.222299 2025b.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Lydia T Liu, Max Simchowitz, and Moritz Hardt. The implicit fairness criterion of unconstrained learning. In International Conference Machine Learning, pp. 4051 PMLR, 2019.
Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, and Alois C Knoll. A survey on autonomous driving datasets: Statistics, annotation quality, and future outlook. IEEE Transactions on Intelligent Vehicles, 2024.
Yuejiang Liu and Alexandre Alahi. Co-supervised learning: Improving with hierarchical mixture of experts. arXiv preprint arXiv:2402.15505, 2024.
generalization
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 1197-111886 2022.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. arXiv preprint arXiv:2106. 10494, 2021.
Fan Ma, Deyu Meng, Xuanyi Dong, and Yi Yang. Self-paced multi-view co-training. Journal of Machine Learning Research, 21(57): 2020.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. kernel-based view of language model fine-tuning. In International Conference Machine Learning, pp- 23610-23641. PMLR, 2023.
Marko Medvedev, Kaifeng Lyu, Dingli Yu, Sanjeev Arora, Zhiyuan Li, and Nathan Srebro. Weak-to- strong generalization even in random feature networks, provably. In Forty-second International Conference Machine Learning. PMLR, 2025.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):13333 2021.
Behrad Moniri and Hamed Hassani. On the mechanisms of weak-to-strong generalization: A theoretical perspective. arXiv preprint 3rXiv:2505.18346 2025.
Abhiject Mulgund and Chirag Pabbaraju. Relating misfit to gain in weak-to-strong generalization beyond the squared loss. In Forty-second International Conference on Machine Learning, 2025.
Vaishnavh Nagarajan, Aditya K Menon, Srinadh Bhojanapalli, Hossein Mobahi, and Sanjiv Ku- mar. On student-teacher deviations in distillation: does pay to disobey? Advances in Neural Information Processing Systems, (6:5596166000 2023.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De- De- biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-2066 20684. 2020.
Junsoo Oh, Jerry Song, and Chulhee Yun. From linear to nonlinear: Provable weak-to-strone tt-sonn generalization through feature learning. In High-dimensional Learning Dynamics 2025, 2025.
13

---

Published as a conference paper at ICLR 2026
Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? Advances in Neural Information Processing Systems, 36: 11037-11048, 2023.
Luca Oneto and Silvia Chiappa. Faimnss in machine learning. In Recent trends in learning from data: Tutorials from the inns big data and deep learning conference (innsbdd(2019), pp. 155-196. Springer, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX0v:2304.07993, 2023.
Hoang Phan, Andrew Gordon Wilson, and Qi Lei. Controllable prompt tuning for balancing group distributional robustness. arXiv preprint arXiv:2403.02695, 2024.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference on machine learning, pp. 5142-5155 PMLR, 2019.
Aahlad Puli, Lily Zhang, Eric Oermann, and Rajesh Ranganath. Out-ff-distribution generaliza- tion in the presence of nuisance-induced spurious correlations. arXiv preprint arXiv:210.005500, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PmLR, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. International Conference on Learning Representations, 2020.
Jessica Schrouff, Alexis Bellot, Amal Rannen- Triki, Alan Malck, Isabela Albuquerque, Arthur Gretton, Alexander D' A'mour, and Silvia Chiappa. Mind the graph when balancing data for fairness or robustness. Advances in Neural Information Processing Systems, 37:29913-299944 2024.
Changho Shin, John Cooper, and Frederic Sala. Weak-to-strong generalization through the data- centric lens. In The Thirteenth International Conference Learning Representations, 2025.
Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Yaacov Ritov, Mikhail Yurochkin, and Yuekai Sun. A statistical framework for weak-to-strong generalization. In ICML 2024 Next Generation of Al Safety Workshop, 2024.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34: 6906-6919, 2021.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12(4):389-434 434, 2012.
Victor Veitch, Alexander D' Amour, Steve Vadlowsky, and Jacob Fisenstein. Counterfactual invariance to spurious correlations: why and how to pass stress tests. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pp. 16196-16208, 2021.
Konstantinos Vilouras, Xiao Liu, Pedro Sanchez, Alison Q'NNeil, and Sotirios A Tsaftaris. Group distributionally robust knowledge distillation. In International Workshop on Machine Learning in Medical Imaging, pp. 234-242. Springer, 2023.
Serena Wang, Harikrishna Narasimhan, Yichen Zhou, Sara Hooker, Michal Lukasik, and Aditya Kr- ishna Menon. Robust distillation for clast-class performance: on the interplay between teacher and student objectives. Uncertainty in Artificial Intelligence, pp. 2237-2247. PMLR, 2023.
Tao Wen, Zihan Wang, Quan Zhang, and Qi Lei. Elastic representation: Mitigating spurious correlations for group robustness. arXiv preprint arXiv:2502.09850. 2025.
14

---

Published as a conference paper at ICLR 2026
Robert Williamson and Aditya Menon. Fairesss risk measures. In International conference machine learning, Pp. 9786-6797, PMLR, 2019.
David Xing Wu and Anant Sahai. Provable weak-to-stron, generalization via benign overfitting. In
The Thirleenth International Conference on Learning Reprexentations, 2025.
Denny Wu and Ji Xu. On the optimal weighted regularization in overparameterized linear regression. Advances in Neural Information Processing Systems, 33:101 12 12-10 2020.
Kai Xiao, Logan Engstrom, Andrew llyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. ArXiv preprint arXiv:2006. 09994, 2020.
Qizhe Xie, Minh- Than-Than Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10687-10698, 2020.
Gengze Xu, Wei Yao, Ziqiao Wang, and Yong Liu. On the emergence of weak-to-strong generalization: A bias-variance perspective. arXiv preprint arXiv:2505.2433 2025.
Yihao Xue, Jiping Li, and Baharan Mirzasoleiman. Representations shape weak-to-strong generaliza- tion: Theoretical insights and empirical predictions. In Forty-second International Conference on Machine Learning, 2025.
Wenkai Yang. Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Gong Zhi, Yankai Lin, and Ji- Rong Wen. Super (ficia))-liggmment: Strong models may deceive weak models in weak-to-strong generalization. In The Thirteenth International Conference on Learning Representations, 2025.
Yuging Yang, Yan Ma, and Pengfei Liu. Weak-to-strong reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8350-8367, 2024.
Wei Yao, Wenkai Yang, Ziqiao Wang, Yankai Lin, and Yong Liu. Understanding the capabilities and limitations of weak-to-strong generalization. In Scaling Self- leifovnpgoiing Foundation Models without Human Supervision, 2025.
Sriram Yenamandra, Pratik Ramesh, Viraj Prabhu, and Judy Hoffman. Facts: First amplify correla- tions and then slice to discover bias. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4794-4804, 2023.
Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pre- trained language model with weak supervision: A contrastive-regularized self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1063-1077. 2021.
John Zech, Marcus Badgeley, Manway Liu, Anthony B Costa, Joseph Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11)x :eleeee 1002683, 2018
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint 2t0.::200002223000 2020.
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness spurious correlations. arXiv preprint arXiv:2203.01517, 2022.
Min-Ling Zhang and Zhi-Hua Zhou. Cotrade: Confident co-training with data editing. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), (1(6))16122180 1626, 2011.
Zhilu Zhang and Mert Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torrala. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
15

---

Published as a conference paper at ICLR 2026
Appendices
Usage of Large Language Models
16
Additional Related Works
16
Additional Notations
17
D Proofs for Section 2.2
17
DII SFT of Weak Teacher
17
D.2 W2S Fine-tuning of Strong Student
21
Additional Experimental Details
26
Dataset Statistics
26
Results for Interpreting W2S under Spurious Correlations
................................
Results for Enhanced W2S
28
Group Fairness in W2S Generalization
30
A USAGE OF LARGE LANGUAGE MODELS
Large language models were used in limited manner to (i) search for related literature, (ii) check grammar/phrasing, and (iii) make stylistic adjustments.
B ADDITIONAL RELATED WORKS
Knowledge distillation. Knowledge distillation (KD) HHinton et al.
2015) is closely related to W2S
but with the roles reversed: KD transfers knowledge from larger teacher model to smaller student model. A series of works has analyzed when and why distilled student generalizes (Phuong &
Lampert 2019 Stanton et al. 2021 Ojha et al. 2023 Nagarajan et al. 2023 Dong et al. 2024 Iidiz
et al. 2025). Analytically, W 2S departs from traditional KD because "weak" vs. "strong" is defined
relative to pretraining, so W2S is naturally studied as fine-tuning on pscudolabeled data.
Group robustness and spurious correlation. Extensive efforts have been devoted to mitigating spurious correlation for robust and safe generalization unseen test domains (Arjovsky et al. 2019
Sagawa et al.] 2020 Krueger et al.] 2021 Deng et al., 2023 Phan et al. 2024; Wen et al. 2025
Liu et al. 2025b Bombari & Mondelli 2025). Among these studies, a subset of work specifically targets spurious correlation arising from group imbalance. When group labels are available, canonical approaches include reweighting minority groups (Sagawa et al., 2020). downsampling majority groups (Deng et al. 2023), distributionally robust optimization (Sagawa et al. 2020; Zhang et al. 2020), and progressive data expansion (Deng al.) 2023). Since obtaining group annotations in training data can be costly or even infeasible, alternative strategies aim to identify biased samples
without explicit group supervision (Nam et al. 2020) Liu et al., [202] Zhang et al., 2022] Yenamandral
Han & Zou 2024), leverage auxiliary signals such as knowledge of spurious attributes (Puli et al. 2021), class annotations (LaBonte et al. 2024), and superclass-leva information (Liu
al. 2023
al. 2025a).
Multi-round retraining and confidence-based selection. Multi-round retraining and confidence- based data selection are widely adopted ideas that have been leveraged, independently. to improve model performance for W2S generalization (Burns al. 2024 Liu & Alahi (2024), mitigate spurious correlation (Liu et al. 2021 [Nam et al.) (2000), and in the broader literature on self-training (Xie et al.
2020 Yu et al. 2021 and co-training /Zhang & Zhou 2011 Ma et al. 2020). Our theory provides
16

---

Published as a conference paper at ICLR 2026
principled motivation to combine these practical techniques, bringing an effective algorithmic remedy for the failures W2S under spurious correlation.
C ADDITIONAL NOTATIONS
e; is the i-th canonical basis conformable dimension. We adapt the standard big-O notations for functions of multiple variables: for two functions f.g
For any n N, let [n] = {1, 2..., n).
N + R30, f(n) (n)(n) means that there exists a constant C' > such that f() & Cg(n) for
all n E N²: f(n) = (g)(n)) means that there exists a constant с >> 0 such that f(n) > cg(n) for
all E No; f(n) (g)(n) means that f() (g)(n) and f(n) {2(g(n)): f(n) (()(n) means that limmin, n. fnn)/g(n) = 0. For a scalar quantity f(n) 3 O depending on n E N,
f() = (r))(n)) means that for any 5 10, there exists constant C(6) 0, independent of
and a natural number (6) E N such that Pr(f(n) & (())g(n) > 5 for all n > N(6).
D PROOFS FOR SECTION 2.2
We first observe that when n > dr, can be equivalently written as 0r Ur where
" B argmin E (xT()))
7.) 2 can be equivalently written as As = UsAs where
(5)
BER n
i=1
Analogously,
Bg argmin BERds
Σ (os(x,) fr(x;))² N
(6)
D.1 SFT WEAK TEACHER
We start by considering the population-optimal linear predictor over the weak teacher representation,
Φr (r(:): as n + 0o, 5 converges to
3 argmin (x winii-DD.nnniiiilioe (d)r) B --
BeRdr
Lemma (Population SFT of weak teacher). When supervisedly fine-tuned over the population, the weak teacher from satisfies (x) = (x 37 z(x) B, fr.
Proof of Lemmal
7 Notice that 7 admits closed-form solution
BT
E-1
Hor Mor Eør.m (xxx)lor(x))))()]]] MorT
RE (
Since z(x) and w(x) are independent, have
Egr. me =ED((n))
(z(x) 8 w(x)) (z(x) & x(
=ED(n)) [x(x)z(x) [ @ Ep(m) [w(x)((x))
(8)
-Ia, @ Cr(H)),
where
Cr(ne) =
Mi -eIp--1 Th H
= diag O +
1
[1 neut), (9)
neET
neter
whose inverse can be computed via block matrix inversion as
1 [ -0g MEt
0g 1
ne
Cr(ne)-==
(10)
dE Ipr-1
Meanwhile, by the independence of (xx) and w w(x), have
p -ee =Ep(m) (x(x) @ (x))x((x) B. + c)] (ED(ne) (z(x)z(x) 7 B.)) @ ED(ne) [w(x)]
=B. &
17

---

Published as a conference paper at ICLR 2026
Therefore, the population-optimal linear predictor over @r is given by
MENTT] )) = B. @ (Cr(ne) (
Bi = =x 1
(Id. & CC((n))
Br &
1 + og 2 1 gg neter
- - ne
=Bw 8
= G. & e1,
c pr-1
neper
where e₁ PPr the first canonical basis.
While the f = S. achieves the optimal population risk sentation (dr d=pr > dz) and the entangled features of drr make the finite-sample generalization challenging. especially under spurious correlations, as we will show next. Theorem (SFT of weak teacher (Appendix D.I.. Under Assumption 7
the inefficient repre-
satisfies
Em (fr)) S og
(m ne) pTl
of
V from label noise
ViU from spurious correlations
Proof of Theorem 7 For a small labeled set = -(xx,, F1 E [n]] ~ D(ne),, the SFT in admits closed-form solution
Br = )T) 11
(11)
where T (xr((x)), ... (x,) E Raxdr and y 1i .n) E R" Since Lemma
shows that the population-optimal linear predictor over oT is fx)()) = or(x) B7 = S. (x). we
have y fr + where é NO In). Therefore, we observe that
Br Bf = = T...
Since the excess risk over the test distribution D(n:) given by
ERg, (fr) =ED( (ffr fx(xx)))) = ED (lxr(() (Br -Br-37)) 37))11 || 37 Eorm"
where Eer. E()) (x)) T1, let Σn, $ Tr Rd be the sample covariance matrix of Φr (x) over Dx(TE). we have
E B5-D(n) ERR (fr))
(55-Dmn)" Eør. in (Br B7) (Br - )
= tr Σ Exr B5wD(nn [
(12)
o tr EE Enm1
n
Recall (x(x) = z(x) 8 w(x) and notice that for any E [0, 1].
Epp](o(()) EEpon Ep(y) [x(]] 0ar
while the derivation of 8 suggests that for any 17 [0. 1].
Σ Esr. ED (á)xx) (x) I Id, Cr(n).
(13)
However, we notice that (x)) is not multivariate Gaussian due to the non- non-Gaussianity of products of independent Gaussian variables and the dependence of entries in z(x) & w(x). Therefore, Σ, cannot be directly computed using inverse Wishart. Instead, we leverage the concentration Σ, in the proportional asymptotic limit (Assumption T.. In particular, Lemmal 2 and (33) implies that as d, n + oo with dz/n- 72 (0.PT")
ir B.rr
F or 72 ir (Cr(n))C((n)))
(14)
18

---

Published as a conference paper at ICLR 2026
Leveraging the derivation of (
and 100.
observe that
tr (Cr(m)Cr(m))-) =p + o mmē | 11 - 20 m p + o 2 pT 12
pe 2
(15)
PT + og (m - ne) 11 pr + (m - me
and therefore,
14 becomes
o
30z (
Epnm
d2 n
pr + (n ne)² | 112
n
of
| 112
pr + (1): no)2 2 7s -0y
0
Plugging the above into 12 completes the proof.
Lemma 2. For fixed pr 2, let C Rerxxe be any fixed symmetric matrix with ||C||2 co.
Recall Σn 1 -.= or (x)))(()))))) where X, ~ Dx(1))) i.i.d. for all E [n]. As d2, n - 0o with
dz/n + 72 E (0.PT))
tr (Id, @ C) E, 7/2 tr (CCr((n)))) n
Proof of Lemma 2 We observe that (x)))) (x) = (z(x)z(xx (w((x)((x)) ") and the sample covariance matrix En can be partitioned into d2 x d2 blocks of size x pr:
E, (4.0
where E.A.0
Σ (x,,) (X,)) RPTXPT,
k.EEd,1
where for any x E X, (x) is the k-th entry of zxx). Notice that I Σ Cr where dg.1 is the Kronecker delta since z(x) and w(x) are independent, and E Exzx(x)2((x)) 8x,1 given zxx) ~ N(O,,, I.
Off-diagonal blocks are negligible. For any k, E [da] with + I, we define pr x pr self-adjoint matrices,
Y (k.))
24 x x
(x;)) - Cr(n)) and
R (k.))
2x ( CT
where recall from the derivation of that Cr ED (w(x) J.. Since (x)x (k.1)
SK, have E
E
E [
(k.()
Oprxpr for is + I, and
(k.))
R (k.))
E E(K = 0prxpr
Let L, 4vlog(n and consider the event
max [|w [E(M]
E, = max iE(A)
| < 12
< 12
First, for z(xi) ~ N N It,), the union bound and Gaussian tail bound imply that
Ln
Ln /2
Pr max
|z(x:)ll
Pr
max kE(d,)
|Ex (x,)]
(E(n)
V2
iE(n],
I
2d
<2nd, exp
nn 2 onn-11
4
Meanwhile, we observe that for g: ~ N0Oppp 1))
1|
2
(xxx))11 = +
2n
+ +20 11:
19

---

Published as a conference paper at ICLR 2026
while the Laurent- Massentt x² tail bound (Laurent & Massart 2000) implies that
Pr g: pr + 2/ (pT 1)t 21
< <e', Vt 0.
Then, for fixed and finite pr and ||rl|| with a sufficiently large n, there exists a constant dn > 1/8 such that applying the union bound with an L2 yields that
Pr
max iEIm
[ 2 > E
< nxxp (-anba) = d(n
Applying the union bound again, get Pr(E o(n-1) for sufficiently large n. Meanwhile, conditioned on En, we have for any i E [n]]
L
L
|z( (w,
( 11
R
(k.1)
|2(xx; )
11
||Cr(n))l
2
n
which implies that
E
(2222
Em
E
Y
(k,))
2
E
<
n
L
L7
n
11
R
R
(2
En
E
R
(R.1)
2
S n
L1
11
Then, applying the matrix Bernstein inequality (rropp 2012
Theorem 1.6) to Ei= Y
(k.0)
and
R (k.)) and union bound over all k, / E [da] off-diagonal blocks with h: + yields that
log(n)
log(n)
Pr
max
Σ (2.1)
Pr
E + Pr
max light
(A:)) -
En
Aggh
n
2 d = o(1)
<o(n + 2prm
(m))
and therefore, the off-diagonal blocks are negligible:
(V
log(n)
max A
(k.1)
Or
Diagonal blocks are concentrated.
Consider the k-th diagonal block (k. for any fixed & E [dg]:
(k.))
(x,) w (X.)
+
Σ (Z,) -
- (xx)
n
1=1
where we denote Cr Dim w(x))w(X.)) Let
1 Σ
8k
(x,)2
(16)
1=1
Then.
(..A)) sk. Cr ((C)) (n)) + CG
-
Σ (2x (2,) 1) (w(X,) (xx Cr()))
where both terms are sums of independent random matrices with zero mean. Leveraging the same argument as for the off-diagonal blocks using the same event En, the matrix Bernstein inequal- ity (Tropp, 2012 Theorem 1.6), and a union bound over all k E [dz], we have for sufficiently large n,
max ((k.A) Sk (Te) =0r (Vlogtin) k kE|d,]
(17)
n
20

---

Published as a conference paper at ICLR 2026
Also, the x² concentration (Laurent & Massart 2000) implies that for any fixed (0,1/2), as d2, with d2/n 722 (0, (0,PT').
Pr
min 3k < - ke(d=)
< d. Pr < 1 < exp (-O(ne²)) =o())
(18)
Pr
8k
1 l
< dr Pr sk 1+ & d. exp (- o(1)
[E(d=>]
so that all 8 's are close to 1 with high probability.
Concentration of E. Let diag (s) CT())). Sd. CT()))) be the block block-diagonal matrix with k-th diagonal block 8g (g)(()) for all l E [dz]: and E, = Σ, Dn be the fluctuations around
D., Since (Tt) positive definite, (T8) implies that || ll2 < oc with high probability for sufficiently large Then, the resolvent identity implies that
E- (D,, + -,,) D E,,
ED,
(19)
In particular, the block matrix inversion formula implies that for any k [dz], the k-th diagonal block of Σi,! denoted as (Σ, ((e,) E R xpr concentrated around the k-th diagonal block of D (s% (T+)))
())))())) (s) & |Enll2- Oz (V (log(n)
(20)
Concentration of the trace. Finally, notice that the trace of interest, ! ir (Iss @ C)D de-
pends only on the diagonal blocks Σn, Then, 20) implics that
1 ir (Ja, 8 C)
1 = Σ ur (C(E,)) k=1
(V llg(n)
ir (CCr(()))) + Or +0p
Sk
n
de
d: tr (CCr(())-)) + or(1).
d
n
Since (ns*) are independent and x distributed, for any fixed n > 2, E[s weak law of large numbers implies that as dz, n - Dc.
Then, the
n-2
P
7 71
d=
8k
2
n n0000
Putting everything together with dz/n + d
E (0, ") completes the proof.
D.2 W2S FINE-TUNING OF STRONG STUDENT
Theorem 3 (W2S fine-tuning strong student (formal restatement of Theorem 2h). Under As- sumption d, N do with d=/n 72 E (0, (0,PT") and dz/N Vz (0.p3))
21

---

Published as a conference paper at ICLR 2026
fs(x) fs(xx) +s(x)] Bs from (2) satisfies
Bs,,NDon.)" ""
ER (fs))
- ogy
PTAS
+
Vg
[17
me me + (ne Tu Eus
V V when nh me
1lsl ps + (n - nn)
Vs (pT -Prss)
o
Es = 0(v)) & 1 negligible when Vs &
E [s. quantifies the effective dimension of group features learned by the strong student from the weak teacher.
where PTAS = 1++ -
Notice that in Theorem y + + is dominant and will be small if T, S. are nearly orthogonal
(i.e., |||| 0) and Ta Rs Mi: whereas Eg tends to be much smaller than V + v(1), especially since unlabeled data is usually abundant compared to labeled data (i.e., V/ < %=).
Proof of Theorems 2 ladd3 3
We first introduce some helpful notions for the proof. Recall Φs(x) (x))6((()))(())) for any E [0, 1] and observe that
2xx) 8 (x). Let
Eøs.n = Id, & Cs(n),
Cs(n) = ED (xx)
nHs ppIps- + " Hssu
(21)
mus
The block matrix inversion formula implies that
Cs(n)-1= 1 0e 2 usl1
nps
5
(22)
0g 1
0 Ps
Meanwhile, the cross covariance of the student-teacher representations under D(1) is given by
Eds.dr.n -D (ds (x) ] EDO ((((x) @ (xx)) w
=ED(n (x())(() 1 @ ED(M (x)()) Ia, 2 A(r).
where
[ [1 TTX
(n)) -ED()) [x)())((x)]] = ED(w)
(23)
nus T nht
RPT.
(24)
of p
Close-form solution and population-optimal predictor of W2S fine-tuning. Given the equiva- lence between (2) and (6., we consider the latter throughout the proof. Adapting the notion from the
proof of Theorem given the labeled set (x,, Ti) E [n]] ~ D(ne)" and the unlabeled set
((x; 9:) E [] (x,),)) with unknown yi 's we denote
T (x)())) ), or or (xx))]
Baxdr, Φ = (xs(x))) ... (xx]] E IR Nxds.
y M T.
R
Φ ( (x1), ... (xN)) E Nxdr
Then, since n > dr and N > ds by Assumption I (6) admits a unique closed-form solution
Bs ( 1 PT where Br (T) T
22

---

Published as a conference paper at ICLR 2026
from 11 Recall from Lemmal that the population-optimal linear predictor over Φ1 in Lemma is fi (x) (x) f = f. with 3T G. @e1. Conditioned on ff (x), the population-optimal
linear predictor over Øs is given by
g ED(n)) [0s(x)os(x)) J D())) [xx(x)xr(x)] B7
[LI, & Cs(nu)) (Ld, A (7
=(Ia, @ (Cs(NT)) (T))))) (3. & 01)
=B. &
01) = B. & C1:
which implies that (x) (x) 33 = (x), i.e., strong student W2S fine-tuned with pseu- dolabels from the Bayes-optimal weak teacher over the population is also Bayes-optimal. Therefore, the student estimator in (6) difters from B by
Bs - B3 = = 4s
4 (3T
-7 pr)
= $s)
+
$ r
- 1
and the estimation error of W2S fine-tuning given by ER v. (fs) = (.E..mm)(ffsss f. (x)) ED ((os(x) T (Bs 3) 113s - 33 11 1
Then, conditioned on y and ss Pr. the excess risk can be expressed as
Ez
ER.,,
(Js)
4r. ss 4T
E2
④ 25)
4 4T D TT)
4 7
Pr.
Ls LT
(25)
Eggin
= tr
E Egg.n
($f$5) -1 $5.
o
7 Ds (I5 $3)
Concentration of sample covariance matrices.
Define the sample (cross) covariance matrices
ΣS.N = N
$5 Ds.
ΣST.N = N D Dr.
Σr.-
$
Then, taking the expectation of 25 over S and Sn yields
E3 S.s. ER (fs) tr Es..
Σ S.T.N SgN
SNN Eoson Eg, E.
Es [ ))
(26)
At the proportional asymptotic limit, Lemma below shows that
tr Es. ΣJ S S.N Us me E Ds. ST E3 (
E 7/ tr (CT,s CT,s(mn nh CT( Mo (PT Pras) tr (s(n)))
Leveraging 21
22 and
23
have
(C) = (A( Cs(nn) Cs(()) (A (n))))
=1 | 11(Tre m + (л
o²
[1 ne)per + (n
nu E 2
=PrAs+ +
while an analogous derivation as in 15) implies that
tr = ps + (n - nn)) llsl
o
23

---

Published as a conference paper at ICLR 2026
Overall, plugging everything back (26) yields gg [E (fs))
3= pras 11 nolee (m
T/ 17,)) J 12
og
Il 12
+ (pr - -Prns)
ps + (Th na)2
of
Lemma 3. In the proof of Theorem 2 at the proportional asymptotic limit,
Ir [Es, ΣT
E mosm S.N S..T,N Eg (
S.N.N S.N
3 72 tr (Cr,s Th (Te) T= (pr pras) tr (s(mn)))s(m)))))
where CT na) E RerXPT is defined as
Cr,s( nn) = (m)) CCs
A(t))
Proof of Lemmal 3 The proof mostly follows the same argument as in Lemma with the key difference being a careful treatment of the off-diagonal blocks in the sample (cross) covariance matrices Es.N and EsTr.N. which are still small but with an additional non-negligible higher-order moment in the proportional asymptotic limit.
Following the proof of Lemma "Separation of block diagonals and off-diagonals", we first partition
Es,N and ΣST,N into d₂ x d2 blocks:
(k.)) de ΣS,T.N
(k..) d. S.T.N 1.1=1
Σs,N
S.N
k1==1'
where (A.))) S.N E Rpsxpss and (k.
S.T.N
RppSXpT are given by
N
(k. S,N
Σ
N
t=1
00.0
(x:)a((
(xxx)((xx)))
S,7,N
N
==1
We denote
Σ 2 (x)) for all k, E [dz),
8k1 N
and observe that for & + 1, E[sa]] = 0, and for k = 1, E[ssA] 1. We therefore observe and denote that
ESNN Ia, @ Cs(nw), Ds. Ep(n.) [STT,N] Ld, @ A(pr))
D = Ep(n.)
(27)
We further define the reminder fluctuation matrices around Ds and D..T:
Es = ŠS -Ds;
EST E.ST. DS.T,
(28)
where
(1.1) Cs(nu) SNN
(1.2)
(1.d.)
SNN
S.N
Es-
E
d,
(2.1)
S.N
(2.2)
S,N
Cs(nw)
(2.d.
S,N
k,t=1
(d..1) S,N
(d. 2)
(dd, Cn(Nn)
S.N
S.N
S (1,1)
S.T N A(ma)
(1.2)
ESTN
S.T.N
E (k.1) d
STT.N
(2.1)
(2,2)
A
S.T.N
(2,d.)
Es.
STT
kl1=1
()
S (d.22)
(dd_,d))
A(T))
S.T.N
S.T.N
S.T.N
(29)
24

---

Published as a conference paper at ICLR 2026
Again, following the proof of Lemma
the resolvent identity implies that
1=1 S,N = DDs + Es)
D D Eg D5 + Es)
(30)
Ds DD EgDr D
+ +Ds E
EssDD'EsSDD Es (Ds + Es)
Then, since E. E[
dg xds and E E xx we have
Es.
T S.T.N N S.N 9s.h
5-1
S-1 3112
S,N S..T.N
-E [ ST E
(Ds + Es) (La & (Ds Es) Es) + Es.T)]
-D D5 LL. Cs(
Ds RN
+ E [ D (La & Cs D5
(=: Rs.))
(31)
E SD. E& [La Cs(n:)) Dssh) Ds Ds.r]
(=: Rsss)
D5. D2 D (La & Cs(n)) ES. (=: R.S,.T
E [ 15 & Cs(n)) EsD DsTT (=: R.S..s)
where |Rwll2 = (1) for sufficiently large N; Es and Es,r are averages over N iid. random matrices with d, x de independent blocks. Therefore, when taking expectation for the second moments of Eş and Es.r, the off-diagonal blocks in RS.T, Rs,s, RS.S.T, RS,,, E Iddr vanish due independence, and only the diagonal blocks remain, which are i.i.d. across k E [dz]. Notice that (77 implies that
D'' (Ia, & Cs(nn))D DsTT Ia, @ Cr,s(M Ba).
Also, recall that the fourth moment of any Gaussian random vector g ~N(00, Ia) satisfies for any fixed matrix M E Rdxdd
E
ggg )o = (d+ 2)
E [[gg]]
M
gg )]
tr(M))a M M
(32)
Define function Rdrxdr + R as
1 ir
((A)
AA
Then, have
1 Ir Es. DJ E S
E on Σz 1 ES.T. S.T.N
Eg T
(Ia, @ Cr,s n.)) + g (R + g (Rs,s) - g(Rsss))) ((RT,, + op(1),
where given the It, Cr,s (Th, nh) structure, Lemma 2 then implies that under the proportional asymptotic limit,
g (L. @ (r)s)))))))) 5. Ya tr (CT,s nue CT(e))")) Let M Cs(m))sss
RPsxPs and M' [M] M. 2:ps' Recall from (31 that R.s
E T s D (Ia, & Cs(m)) EsT]] and 15 along with the proof Lemma imply that
(RsT)) P 7=Uz pr tr (M) + tr (M' + Cs (mh - 7/)3 |
o
for some constant Cs > 0 independent of dz, N. Analogously, we have
g (Rs,s) P 7=V=
prAstr(M)
(M'E + Cs
(1/
m1)
| 112
of
o²
((R,,s))) 3, NzV; Pras (rnstr(M) + tr (M' E) Cs
2
(nu me 2
of
o²
,(RR,r,,)) (((R,,,))).
25

---

Published as a conference paper at ICLR 2026
Overall, at the proportional asymptotic limit,
tr Es. ST. Ss, Los S5,N E S.T 3g (TTT)))
- + 7a tr (T) + TaPs -PT tr (P (Cs(m))
E ADDITIONAL EXPERIMENTAL DETAILS
E.1 DATASET STATISTICS
In this work, we construct three distinct splits for each of the four datasets, Waterbirds (Sagawa et 2020), BFFHQ (Lee et al. 202I), ImageNet-9 (Xiao et al. 2020), and BG-COCO. Specifically, each dataset is partitioned into a group-imbalanced training set D1, a group-balanced training set D2, and group- balanced test set Dz. The minority group proportion in D1, D2, and D3 is fJo, 0.5, and 0.5, respectively. Across different real-world experiments (μ) and 1hh)) as well as the sample sizes (n and N). We first summarize the dataset statistics for each benchmark, and then describe how D to D3 are utilized in different experimental setups.
the paper, we vary the group proportions
Waterbirds statistics. The Waterbirds (Sagawa correlations between natural backgrounds and bird labels, with fo = 0.05. Table reports the detailed group distributions across D1 to D3- Following Sagawa et al. (2020), we supplement additional samples for the minority groups (waterbird, land) and (landbird, water) in the same manner as the original dataset, due the limited size of the raw data.
2020) dataset designed capture spurious
Split
(waterbird, water)
(waterbird, land)
(landbird, water)
(landbird, land)
Total
D1
1,057
56
184
3,498
4,795
D2
1,804
1,804
1,804
1,804
7,216
D3
451
451
451
1,804
Table 3: Dataset statistics for Waterbirds, Each column corresponds to group, and the last column gives the total sample count.
BFFHQ statistics. The BFFHQ (Lee et al., 2021 dataset is designed to capture spurious correla- tions between age and gender labels, with n. 0.005. reblor] the detailed group distributions across Dj to Dz. Due to the limited size the minority groups in the raw data, our splits are constructed from de-duplicated samples across multiple BFFHQ subsets.
Split
(young, female)
(young, male)
(old, female)
(old, male)
Total
D1
9,552
9,552
19,200
D2
790
790
790
790
3,160
D3
198
198
198
198
792
Table 4: Dataset statistics for BFFHQ. Each column corresponds to a group, and the last column gives the total sample count.
ImageNet-9 statistics. The ImageNet-9 (Xiao et al. 2020) dataset is designed to capture spurious correlations between object and background labels. Different from Waterbirds and BFFHQ. ImageNet- 9 is 9-class classification task over categories dog, bird, wheeled vehicle, reptile, carnivore, insect, musical instrument, primate, and fish. The original dataset provides two variants, mixed-same and mixed-rand. In the mixed-same version, each image background is replaced with a background from an image of the same class, thus preserving spurious correlations; in the mixed-rand version, the background is randomized and contains no information about the true label. These two variants correspond to minority group proportions of and 0.5, respectively. Table 5 reports the dataset statisties across D to Dz. Based on this table, set n. O. Note that ImageNet-9 does not have well-defined group structure under either the mixed-sana same or mixed-rand settings. Therefore, we do report worst-group accuracy for this dataset.
26

---

Published as a conference paper at ICLR 2026
Split
mixed-same
mixed-rand
Total
Per-class
D1
4,050
0
4,050
450
D2
3,240
3,240
360
D3
810
810
90
Table 5: Dataset statisties for ImageNet-9, Within each split, the nine classes have identical counts.
BG-COCO statisties. The BG-COCO dataset is self-generated benchmark designed to capture spurious correlations between cats/dogs from COCO (Lin et al. 2014) and indoor/outdoor scenes from Places (Zhou et al. 2017). Specifically, we define the outoorr split living room, dining room (indoor) and park (outdoor). By construction, cats are aligned with indoor scenes and dogs with outdoor scenes. Table 66 reports the detailed group distributions across D to Dz. Based on this table, we set l/o 0.05.
Split
(cat, indoor)
(cat, outdoor)
(dog, indoor)
(dog, outdoor)
Total
D1 1,900
100
100
1,900
4,000
D2
1,000
1,000
1,000
1,000
4,000
D3
250
250
250
250
1,000
Table 6: Dataset statistics for BG-COCO. Each column corresponds to group, and the last column gives the total sample count.
Across all four datasets, construct training and evaluation splits as follows. When either ne or Yu fixed 7o. samples are drawn from Dj with the desired size (for unlabeled data) or n (for labeled data). When either T/e or yh is fixed 0.5, balanced samples instead drawn from Dz. Several experiments involve fixing Y) while varying Tu- In this setting, if necessary, we keep the labeled data unchanged and supplement the unlabeled data with additional samples independently drawn from D or D2, while ensuring that the total unlabeled sample size N remains constant across different Tue The balanced dataset D3 is reserved for testing, and when required, we further split 20% of D3 as separate validation set.
E.2
RESULTS FOR INTERPRETING W2S UNDER SPURIOUS CORRELATIONS
# of model pairs with increased W2S gain
Dataset
Average accuracy
Worst group accuracy
Waterbirds BFFIIQ
10/10
10/10
10/10
9/10
BG-COCO
9/10
10/10
ImageNet-9
7/10
Table 7: Proportion of teacher-student pairs that exhibit increase in W2S gain as increases from to the maximum feasible value of The (Waterbirds: 0.5, BFFHQ: 0.23, BG-COCO: 0.5, ImageNet-9: 0.4) when = 0.5, summarized across all datasets. ImageNet-9 has no well-defined worst group, so only average accuracy is reported.
In Section 3.2 we primarily presented how the average W2S gain across all teacher-student pairs varies with increasing 7u on each dataset. Here, we further provide results for individual model pairs. Specifically, and group-imbalanced (r)= = nh) settings on selected datasets. Table summarizes, for TJe 0.5, the proportion of model pairs that exhibit an increase in W2S gain 7h, increases from 0 across all datasets. Table summarizes, for n = no. the proportion of model pairs that exhibit a decrease in W2S gain 77u increases from no across all datasets. These results further validate theoretical analysis in Section [2.2] which predicts that in most cases the larger the gap between n,, and m)s the smaller the resulting W2S gain.
compares the difference in W2S gain between the group-balanced (ne = 0.5)
27

---

Published as a conference paper at ICLR 2026
Average
Worst-groupp
Waterbirds Nt 0.5
0.05
0.00
Weak Strong
ConvNeXt DINOv2
ResNet118 DINOv2
ResMetts ConuNeXt
Clipb32 DINOv2
Clipb332 ConoNext
Clinb32 ResNet18
MAE
MAE ConvNeXt
MAE RedNat18
MAE
0.00
-002
-0.04
ImageNet-9
-0.06
nr no
DINOv2 ConvNeXt
ResNet16
ResNet18
Clipb32
Clipb32
Strong
ConvNeXt
DINOV2
ConvNeXt
DINOV2
ResNet18
ConvNeXt
DINOv2
ResNet11
Clipb32
Figure 6: Top: On the Waterbirds dataset, the change in W2S gain (value at Thu = 0.5 minus value at ?) = 0) across all macher-student pairs with fixed ?) 0.5. Bottom: On the ImageNet-9 dataset, the change in W2S gain (value at l/u 0.5 minus value at To) across all teacher sttuent pairs with fixed 7 = Mo- ImageNet-9 does have clearly defined worst group and therefore omitted from the bottom panel.
# of model pairs with decreased W2S gain
Dataset
Average accuracy
Worst group accuracy
Waterbirds
7/10
8/10
BFFHQ
8/10
7/10
BG-COCO
8/10
7/10
ImageNet-9
9/10
Table 8: Proportion of teacher-student pairs that exhibit decrease in W2S gain 17, increases from To to 0.5 when 1/2 = n), summarized across all datasets. ImageNet-9 has no well-defined worst group, so only average accuracy reported.
E.3 RESULTS FOR ENHANCED W2S
Model training. Enhanced- W2S improves upon vanilla W2S by retraining the strong student after the initial W2S fine-tuning. First, we select fraction E (0, 1] of S consisting of those samples for which the student exhibits the lowest prediction entropy. Second, we apply the GCE loss LGCE(x; ds: 9) with parameter q (0. 1] to each selected sample (x;, 9:). We tune the hyperparameters by grid search over e (0.2, 0.4, 0.6. 0.8. 1.0) and g E {0, 0.2, 0.7). where q = corresponds to the CE loss (i.e., the - 0 limit of GCE). To avoid a trivial overlap with the vanilla (1,0) is excluded from the Enhanced- --anncddd--SS search space. the case of (me, Th.. (1), 0.5), we further restrict the subset ratio E (0.2, 0.4, 0.6) to emphasize the role of high-confidence subsets in filtering for the majority group. Each run of Enhanced- W2S is repeated with multiple random seeds, and the reported results obtained by averaging across seeds.
W2S baseline, (p. q)
Role of confidence-based selection. When (M), 7m) = (1. 0.5), Figure 7 shows that samples with high student confidence (i.e., low predictive entropy) after W2S fine-tuning are almost exclusively drawn from the majority group, and furthermore are nearly always assigned the correct pseudolabels by both the weak teacher and the strong student. At the same time, Theorem 2 predicts that reducing nu from 0.5 directly increases the W2S gain. These two observations together suggest that confidence- based selection provides significant benefits for improving W2S performance in the setting (M), nh) (no, 0.5).
28

---

Published as a conference paper at ICLR 2026
Waterbirds
BFFHQ
BQ-COCD
Clipb32 Canvleeii
Conviilii - Clipb32
Clip32 Convteett
Majafty
Wajoriy
Mojorty
Minorty
Winrfity
Densty
3.5
02
5.4
0%
0.1
0.0
2.5
0.2
23
Connect
Cornect
Correct
Weok Correct, Styang Whong
Wesik Coreet Strong Wrong
Weak Coreet Strong Weog
Whok Wennng Strong Cor
Weak Weong, Strong Corrcc
Weak Whong Strong Conect
Woon
Whong
Both Whnn
Entropy
Entropy
Entrpyy
Waterbirds
BFFHQ
86 COCO
Clip322 DINOV2
ClipB32
Clip322 DINOV2
Majotty
Mayeftty Group
10-
Majoriy Group
Minentty
Venerity Group
H nonty Group
20-
0.3
022
22
23
0.4
05
0.6
6.
0.0
02
0.2
0.3
6.4
25
2.6
077
0.2
02
0.4
05
0.6
07
Correct
Correct
Correct
Conect Strong Woog
Strong Wrong Weong, String Coreec
Weak Conect. Strong Werg
Weac Whon Strong Corme
30-
West Strong Cormer
Wrong
Whong
Whong
Emissy
Entrapy
Emir5d
Figure 7: Student confidence on unlabeled data as stacked density plots of predictive entropy (Π = Yoo 7a 0.5). Fach panel shows the student's predictive entropy (after W2S fine-tuning), visualized as two stacked density plots: (top) split by group (majority vs. minority) and (bottom) split by prediction correctness of the weak teacher and the strong student. Columns correspond to datasets (Waterbirds, BFFHQ, BG-COCO). Rows correspond to model pairs: (ConvNeXt, Clipb32) and (Clipb32, DINOv2).
Mean relative improvement (%)
Dataset
7/u
Average Accuracy
Worst Group Accuracy
Waterbirds
0.5
%)
6.32
10.12
To
0.5
7.72
32.15
BFFHQ
0.5
No
5.52
4.06
Yo
0.5
3.12
3.57
BG-COCO
0.5
11o
10,51
11.76
%o
0.5
4.50
3.71
ImageNet-9
0.5
Vo
12.23
Vo
0.5
11.23
Table 9: Mean relative improvement (%) of Enhanced- Sahancce-W22 over vanilla W2S, averaged across selected reacher-student pairs, for both average accuracy and worst group accuracy. For each dataset, we select all model pairs whose relative strength relationship remains consistent across different (me, Bh) settings.
Mean relative gains. Table summarizes the mean relative improvement of Enhanced -W2 over vanilla W2S, averaged across all teacher-student pairs. Consistent with the main text, our method achieves clear gains under both average accuracy and worst group accuracy. On the Waterbirds dataset,
29

---

Published as a conference paper at ICLR 2026
further compare the performance of Enhanced-W W2S with the auxiliary confidence loss proposed in (Burns et al., (024). which was also designed to improve the generalization ability of W2s. Specifi- cally, we perform a grid search over the auxiliary confidence loss weight {0.2, 0.4, 0.6, 0.8). and Table 10 reports, for each (Mt, Tu) configuration, the mean relative improvement of Enhanced- W25 minus the mean relative improvement obtained with the auxiliary confidence loss. Our method yields larger gains in all cases, confirming that it is motivated by our theoretical analysis (see Section 4 and specifically tailored to address W2S under spurious correlation.
Difference in mean relative improvement (%)
1/e
Average Accuracy
Worst Group Accuracy
0.5
1
5.22
3.88
0.5
5.90
3.41
Table 10: Difference in mean relative improvement (%) Waterbirds, computed as Enhanced-W?7 minus the auxiliary confidence loss baseline, averaged across selected teacher-student pairs for cach (h) configuration.
F GROUP FAIRNESS IN W2S GENERALIZATION
Ensuring that algorithmic decisions do not exhibit systematic bias against certain attributes (e.g.. race, gender, age) has long been central objective in fair machine learning (Liu et al., 2019: Oneto & Chiappa 2020 Mchrabi et al. 2021). At the same time, when the data contains spurious correlations caused by group imbalance, unfairness across different groups is likely to arise, as the model tends to rely on spurious features when making predictions (Izmailov et al. (222)). This lack of group faimess particularly concerning when groups are defined by sensitive attributes. Therefore, a line of work on mitigating spurious correlation has explicitly targeted group robustness, and the evaluation metrics adopted this literature (e.g., worst group accuracy) can be interpreted as measure of fairness. In parallel, several works have more directly studied the relationship between spurious correlations and formal notions of group fairness /Veitch et al. 2021: Schrouff' et al. (024).
In this section, extend our analysis of W2S under spurious correlation to incorporate the notion of group fairness. Under W2S, the minority group proportions in both the labeled dataset (ne) and the unlabeled dataset (1)w) jointly influence the extent to which the strong student preserves group-level parity after the W2S process.
we define the group risk disparity of Definition (Group risk disparity). Under Definitions 7 Mnnd? 2
the strong student after W2S fine-tuning as
(fs) = |BD(n.) (DDn))) [ERo(fs)] - ED(n D(ne)" [E)](s))]
(33)
where (Ro(fs) and (fs()s) denote the excess risks of the student on the majority (1) = 0) and minority (Mt = 1) groups, respectively.
In Definition 4 we quantify the group fairness through the absolute difference between the student's excess risk on the majority group and the minority group. It is important to note that our definition of group risk disparity is directly aligned with the notion of perfect fairness (also referred to as risk
parity) in the group-fairness literature (Williamson & Menon, 2019 Liu et al. (025b). In particular, the condition Agrp = O is equivalent to achieving perfect fairness (risk parity).
Corollary (Group risk disparity of W2S). Under Definitions and [and assumption risk disparity of the strong student after W2S fine-tuning satisfies
7 the group
A (fs)
P, 0 of
2( 7a TT - (1m) (1Eus|1 + Pris) us))
Corollary follows directly from the precise asymptotic characterization of the strong student excess risk in Theorem 3 providing precise quantification of the group risk disparity the proportional asymptotic limit.
We outline several key insights from Corollary
below:
(a) Low teacher-student similarity (PrAS = 1) brings robustness of group fairness (fr)(s)) to teacher bias ne < 0.5, where W2S is fair if the unlabeled training set is balanced f/. 0.5.
30

---

Published as a conference paper at ICLR 2026
Notably, when Pras = 1 (i.e., | 0), (fp((f) becomes independent of and is only affected by 7 through the (1 217,] factor. When T/ = 0.5 further, we have 21/u, 0, and
therefore A prp (fs) 4 i.e. the strong student from W2S fine-tuning fair, even though biased weak teacher fine-tuned with f/r 7/, can still hurt the generalization of the strong student.
(b) Low teacher-student similarity (PrAS 1) induces group fairness of W2S, grpp + O, as U2 - O. While PTAS = (i.e., | P
0) alone does not guarantee fairness, it provides
A
B (1 27 U2 (PT PTAS) M (1 27
where we have Agp (fs) E O as d2, n, N - if v/ O, i.c., when N is large enough compared to dz, low teacher-student similarity induces fairness of W2S.
(c) For high teacher-student similarity (PTAS > ps), group fairness of the student (gp((s) influenced by the fairness of the teacher T)s, with the dependence determined by Eps. In
particular, when PrAS > ps so that |nt is non-negligible, W2S fair (i.e., grp(fs) (fs) 3 0) when
Eus l
vs (pr - PI |
fair Th + (1 27)a) Me
Eps
assuming it falls the range n E [0, 0.5). while the group fairness gets worse (i.e., Agrp (fs) increases) as ne deviates from r/c fair. Notably,
(1) if mz < 0, the group fairness gets worse as my increases, best when 17 = 0; while
(2) if m > 0.5 the group faimess gets worse ne decreases, best when = 0.5.
31


Reviews:
[{"review_id": "gOVCuY2Lpq", "reviewer": "Reviewer_c6kM", "summary": "This paper extends the results of Dong et al. (2025) to the setting where there are spurious correlations induced by group imbalance in the training data distribution. The paper derives excess risk bounds for (1) a teacher model trained on a small number of labeled examples (Theorem 1) with a representation that doesn't well-separate the spurious and non-spurious features, and (2) a student model trained on an unlabeled data distribution pseudolabeled by the teacher. The bounds indicate that the weak-to-strong gains are largest when the percentage of the \"minority group\" data is very close in the teacher and student's training data distributions. Based on this observation, it proposes to do two rounds of weakly-supervised training, where in the second round the student retrains on the subset of examples where it was confident in its label.", "questions": "- If we're doing multiple rounds of training, why not co-train the teacher?\n- Why not use the labeled data to train the student?\n- \"For each experiment on a given dataset, we include all teacher—student pairs whose relative strength (measured by accuracy) remains stable when we vary parameters including $\\eta_l$, $\\eta_u$, $N$, or $n$” ← What does this mean?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Testable theory for the magnitude of weak-to-strong generalization in the presence of group imbalance / spurious correlations\n- Theoretical motivation for retraining on subset of data where the student is most confident\n- The above approach outperforms \"Vanilla\" W2S generalization on several datasets and for several teacher/student pairs.", "weaknesses": "- The analysis seems to rely on the fact that the teacher model is an ERM on the labeled data. In this case, why doesn’t the student also use the small labeled set instead of (or in addition to) using the pseudolabeled data? How would the student's performance bound compare in that case? I’m sure the analysis can be fixed to incur some additive error term that measures the teacher’s excess risk over the ERM, but the same point applies\n\n- Not enough discussion of (or motivation for---why are we interested in W2S in the context of spurious correlations?) the incremental novelty over Dong et al. The paper seems like a targeted extension of Dong et al. to the spurious-correlation regime, but that regime is not well-motivated.\n\n- The confidence loss / retraining idea is similar to the confidence loss in Burns et al and very similar to the iterative confident retraining in COSINE [1], and the idea of confident data selection in weak supervision is old (see e.g., [2] and many others). The experiments are missing \"3rd-party\" baselines like the Burns confidence loss, and the paper is missing contextualization of this idea in the long literature on weak supervision, self-training, and co-training.\n\nMinor Presentation comments:\n- Assumption 2 takes a long time to parse. A figure here might be helpful?\n- Section 2.3 can probably be cut or moved to the appendix…we might not need experimental validation of the theory in the exact setting of the Theorem. It would be more interesting to argue why some of the assumptions of the theorem are practically relevant, or to bring some of the content from Appendix E into the main text\n\n[1] Yu et al. \"Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach\"\n\n[2] Zhang et al. \"CoTrade: Confident co-training with data editing\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qy9O8TuE4B", "reviewer": "Reviewer_caiF", "summary": "The paper considers a weak-to-strong generalization setup with ridgeless regression under spurious correlations with teacher and student models having fixed sets of features that entangle the core and group features differently. They show that in the appropriate limit of sample sizes and core feature dimension, weak-to-strong generalization happens (i) whenever the spurious correlations are the same and there is enough teacher labeled samples, (ii) when there is a mismatch of spurious correlations between the teacher and the student the the W2S gain decreases with this difference, and (iii) algorithmic improvement to standard W2S training pipeline that improves the W2S student performance when there are spurious correlations.", "questions": "1. Is there a choice of parameters in Assumption 2 (i.e. T and S) under which we can have a theoretical characterization of $p_{T\\and S}$?\n2. Why is the asymptotic limit taking in this exact form? Is it mostly for the simplicity of analysis of the excess risk in ridgeless regression or is there any intuition behind? Could get a similar result for finite sample size and feature dimensions?\n3. Is there a natural agnostic definition of weak and strong that implies Assumption 2?\n4. Do you think that the enhanced W2S training would improve performance for the original weak-to-strong setup in Burns et al 2023 (and is it even applicable)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main significance of the paper is:\n1. Consider regression with spurious correlation in the context of W2S is an interesting way to capture different pre-trainings and capacities of student and teacher models\n2. Provides crisp characterization of weak-to-strong generalizations in regression with spurious correlations in the appropriate limit of sample sizes and feature dimensions in terms of mismatch in level of spurious correlation, similarity between student and teacher features, and the number of samples\n3. Alternative way of performing W2S training when the spurious correlations don’t align with retraining the student model on high confidence points which improves over the vanilla W2S.\n\nThe idea of considering spurious correlation is original. The writing is clear and the mathematical arguments seem sound.", "weaknesses": "1. The main theorem, Theorem 3, does not formally establish weak-to-strong generalizations. It only gives a sufficient condition under which weak-to-strong generalization happens in this case in terms of $p_{T S}$. It would be more interesting to find natural choices of $\\phi_S$ and $\\phi_T$ for which we can understand $p_{T  S}$ and therefore W2S improvement.\n2. Here the W2S improvement, like in other ridgeless regression setups, comes “artifically” from the fact that the teacher is trained with too little samples and thus is not optimally. The student is able to  improve over the teacher because it has sufficient samples, and so the effect of student having better features than the teacher is confounded with the effect of the teacher being suboptimal. \n3. The paper asks whether W2S still happens in data with spurious correlation, but the definition of what makes a teacher “weak” and a student “strong” (Assumption 2) is based on the spurious correlations, i.e. that the weak model entangles the core and group features and the strong model doesn’t. The result could be further strengthened with a more agnostic way of defining weak and strong which implies Assumption 2. That way, we could use this setup to get insights on the mechanisms underpinning W2S generalization.\n4. It is unclear how broadly applicable algorithmic insights are to other non-vision related W2S setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tya4IOkMVp", "reviewer": "Reviewer_ZDqv", "summary": "This paper investigates weak-to-strong generalization in the presence of spurious correlations arising from group imbalance. The authors provide a theoretical analysis demonstrating that the disparity in group imbalance between the weak and strong model training datasets is the key factor controlling the weak-to-strong gain. Their analysis shows that weak-to-strong generalization can fail when the group imbalance distributions differ significantly between the two phases. Furthermore, the paper proposes a novel method called Enhanced-W2S, which is empirically shown to achieve superior weak-to-strong generalization on spurious correlation benchmarks.", "questions": "* In main results, the terms $\\mathcal{V}_T^{(0)}, \\mathcal{V}_T^{(1)},  \\mathcal{V}_S^{(0)}, \\mathcal{V}_S^{(1)}$ appear. What is the precise meaning of these terms? It seems this term might relate to variance (according to line 228), but there are no explicit discussions on it. It would be helpful if the authors could explicitly discuss the meaning of each term right after the statement of the main theorem.\n* In the draft, there is a discussion on why the proposed method can be beneficial in the spurious correlation setting. However, the connection between this intuition and the theoretical insights is vague for me. Could you discuss this point in more detail?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work addresses the role of spurious correlation in weak-to-strong generalization, a topic that has not been sufficiently explored in prior literature.\n* The theoretical framework and results are clearly stated, and the paper provides detailed, insightful discussions on these findings.\n* Beyond the theoretical analysis, the authors propose a novel method, Enhanced-W2S, as an effective remedy. The effectiveness of this method is well-supported by extensive experiments on various spurious correlation benchmarks and architectures.", "weaknesses": "* The theoretical framework, particularly the formal statement of Assumption 2, can be hard to follow at first glance. I believe that adding a more intuitive, illustrative description can significantly improve reader understanding.\n* Theoretical discussions are provided before explaining some technical terms (See Questions).\n* The connection between theoretical insights and the intuition behind the proposed method is not clearly discussed (See Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RsxsvTrskh", "reviewer": "Reviewer_rtLS", "summary": "This paper presents a unified theoretical and algorithmic investigation into the phenomenon of Weak-to-Strong (W2S) generalization under the presence of spurious correlations caused by group imbalance. The authors model two key sources of imbalance: the minority fraction in the labeled data ($\\eta_l$) used to train the teacher, and the minority fraction in the unlabeled data ($\\eta_u$) used for W2S fine-tuning.\n\nThe main theoretical contribution, derived within the proportional asymptotic limit of a linear regression setting, precisely characterizes the W2S gain. The theory predicts that W2S always occurs when $\\eta_l = \\eta_u$, but critically, W2S generalization can fail when $\\eta_l \\neq \\eta_u$, with the gain diminishing as the squared difference $(\\eta_u - \\eta_l)^2$ increases.\n\nBased on these findings, the paper proposes Enhanced-W2S, a simple yet effective algorithmic remedy. This method involves post-W2S retraining of the strong student on its high-confidence data subset using the Generalized Cross-Entropy (GCE) loss. The method is entirely group-label-free. Extensive experiments across four spurious correlation benchmarks and ten different teacher-student pairs confirm the theoretical predictions and demonstrate consistent, substantial gains achieved by Enhanced-W2S over vanilla W2S fine-tuning.", "questions": "- I infer that achieving parity between the minority group proportions in the labeled ($\\eta_l$) and unlabeled ($\\eta_u$) datasets is crucial for W2SG. Theoretically, does the primary W2S gain come from the minority or the majority group? In the optimal gain scenario ($\\eta_u = \\eta_l$), which group primarily contributes to the remaining excess risk?\n- Strictly speaking, is data imbalance the sole source of spurious correlation in the setting analyzed by this paper?\n- How does group robustness affect fairness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a highly critical, yet previously unexplored, aspect of W2S generalization: its robustness to real-world biases like spurious correlations. The idea attracts me a lot and I'd like to see the core idea published in the conference. To my best knowledge, this issue has not been investigated in previous W2SG papers, both empirically and theoretically.\n2. The key theoretical conclusion that W2S gain increases as the teacher-student similarity decreases is intuitive and aligns with teacher-student disagreement literature (Charikar et al., 2024; Mulgund & Pabbaraju, 2025; Yao et al., 2025; Xu et al., 2025).\n3. To my knowledge, the paper includes a thorough review of relevant literature concerning W2S generalization and group robustness.\n4. The result appears correct since it can recover Dong et al. (2025), though I have not verified the proof details.", "weaknesses": "## Weaknesses\n1. The paper is generally well-written, but it could be significantly improved by providing more explicit and intuitive explanations for every mathematical choice and definition (e.g., the precise geometric meaning of $\\left\\\\|\\Xi\\right\\\\|_F^2$ and the role of $p_T$ and $p_S$) alongside the formal derivations. This would greatly enhance the accessibility and understanding of the theoretical model.\n2. The paper's theory successfully demonstrates that under certain ideal conditions, such as when the minority group proportions in the labeled and pseudo-labeled data match ($\\eta_l = \\eta_u$), W2SG is more likely to occur. However, the conclusion doesn't seem to fully elucidate the precise conditions under which W2SG is guaranteed to succeed or fail in the non-ideal scenario where data imbalance persists (i.e., $\\eta_l \\neq \\eta_u$). The theory notes that W2S may fail when $\\eta_u \\ne \\eta_l$. For instance, when $\\eta_l \\neq \\eta_u$, the W2S gain tends to diminish but is not universally negative, suggesting there may be boundary conditions or trade-offs that are not fully explored by the analysis on $(\\eta_u - \\eta_l)^2$ alone. \n3. The detailed excess risk characterizations rely on the ridgeless linear regression setting and a specific, low-intrinsic-dimension kernel assumption.\n4. The Enhanced-W2S method introduces two new hyperparameters ($p$ for subset fraction and $q$ for GCE loss). While tuning ranges are provided, the paper does not thoroughly explore the robustness of the optimal $(p, q)$ pair across different $\\eta_l, \\eta_u$ settings or the effort required to tune them.\n\nMinor: Line 270: The text above Figure 2 should be revised.\n\n## Future Work\nSince the paper quantifies the error, I strongly suggest the authors separately quantify the excess risk for the minority group and the majority group in the theoretical analysis. The current theoretical results (Theorems 1 and 2) characterize the average excess risk $ER_{\\eta_t}(f)$ across a test distribution $\\mathcal{D}(\\eta_t)$, but presenting the excess risk specifically for the majority ($\\eta_t=0$) and minority ($\\eta_t=1$) test groups would provide deeper insight into how spurious correlation affects group performance.\n\nThis work touches closely upon the concept of fairness in machine learning, as spurious correlation fundamentally impacts group robustness. The data imbalance investigated here—specifically the role of the minority group fractions $\\eta_l$ and $\\eta_u$—bears a striking resemblance to the problem of base rate differences discussed in the fairness literature [1]. Given this relationship, I recommend including a discussion on this intersection. Specifically, exploring the connection between the W2S failure mechanism (when $\\eta_l \\neq \\eta_u$) and established fairness criteria could be a valuable direction for future work in W2S.  If you have time, and can derive the difference in errors between the two groups from your current proof to build a similar theory of fairness (maybe in the appendix), this paper would then connect W2S to both robustness and fairness. This would make it much more interesting, and I'd be able to offer you my stronger support. Please don't hesitate to correct me if I've misunderstood anything. \n\n[1] The Implicit Fairness Criterion of Unconstrained Learning. ICML 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)