Weak-to-Strong Generalization under Distribution Shifts
Authors: Myeongho Jeon
Jan Sobotka
Suhwan Choi
Maria Brbic
Venue: NeurIPS 2025 poster ()
Keywords: Weak-to-strong generalization
Distribution shift
Primary area: deep_learning
TL;DR: This paper introduces a novel scenario, weak-to-strong generalization under distribution shifts, and proposes Robust AdaptiVe wEightiNg (RAVEN) to tackle this challenge.

Abstract: As future superhuman models become increasingly complex, accurately supervising their behavior may exceed human capabilities. 
Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive weak-to-strong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we propose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. We demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over 30% on out-of-distribution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.

Weak-to-Strong Generalization under Distribution Shifts
Myeongho Jeon' 1+
Jan Sobotka
Suhwan Choif
Maria Brbić
EPFL Seoul National University
Abstract
As future superhuman models become increasingly complex, accurately supervis- ing their behavior may exceed human capabilities. Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive stak-to--ttong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we pro- pose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over 30% on -i-ttr--bttiiution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.
Introduction
Recent Al systems have reached near-human performance through extensive pre-training large datasets, followed by fine-tuning with human supervision. Techniques such as supervised fine- tuming, reinforcement learning with human feedback (RLHF) [I7] 61] 50], and direct preference
optimization (DPO) [43] (56] are key examples of effectively aligning models with human preferences. A fundamental assumption in these methods is that the human supervision is of high quality. However, when the data to be annotated is beyond human comprehension, providing reliable supervision becomes challenging. For instance, in domains such as cosmology, healthcare, biology, experts may struggle with accurate labeling. Furthermore, during the alignment phase with human preferences. if a superhuman-lews model generates highly complex outputs, humans may struggle fully understand them, making it challenging to provide effective feedback. This challenge is known as weak supervision [0]]
So. how can humans supervise a superhuman model that surpasses their own capabilitics? To mimic this future scenario, analogous framework has been proposed in [10], where weak model simulates human supervision, while a strong model acts as proxy for a potential superhuman model. In this setup, weak supervision signals utilized train the strong model, aiming to surpass the weak model's 's performance and approach the performance achieved when trained with ground-truth (GT) labels. Conceptually, this approach, termed weak-to-strong (W2S) generalization 100. could lead toward developing superhuman models, as surpassing the performance of a weak model in this context could eventually mean exceeding human capabilities. Moreover, such success would be practical even before superhuman models emerge-for for instance, aligning GPT-5 using only GPT-4-level supervision could simplify model alignment today [10] Recent works
[10 26 45 19
"Equal contribution
Correspondeece to: mbrbicdepf1.ch
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Weak-to-Strong Generallzation
Weak-to-Strong Generalization under Distribution Shifts
In-disrrbbudon
Out-o--istrbulion
Wesk model
Source data
Weak model
Train
Fine-tuning data
Fine-turing
Week Supentision
Wask and Misliinedd Supentisio
Traiin
Train
Targei data
Targot data
Strong model
Test
Figure I: An illustration of weak-to-strong generalization under distribution shifts. Left: The original W2S generalization framework [TO]. Right: Our extended framework that incorporates distribution shift. In this scenario, the weak model is not only limited in understanding but also unfamiliar with the fine-tuning distribution, making less reliable for annotating data to supervise the strong model.
have shown that W2S generalization indeed feasible, meaning that strong pre-trained models naturally generalize beyond their weak supervisors.
However, humans may encounter data that is not only complex but also unfamiliar, making the weak supervision even weaker. For example, a radiologist used to one type of imaging data may misinterpret scans from different machine or mislabel rare diseases outside their usual clinical practice [8]. As result, expert annotations can become even less reliable due to unfamiliarity with the data. To simulate this, we consider scenario which the weak model is trained on data drawn from a distribution that differs significantly from the strong model's fine-tuning data, setting we refer to as weak and misaligned supervision (Figure TT. This raises the question: W2S generalization still feasible under distribution shifts?
Following this question, we interestingly find that weak supervision becomes significantly less effective in out-of-distribun some cases, even not feasible, leading to the strong model performing worse than the weak model. This is nontrivial issue, as it suggests that unnoticed misaligned human perspectives can affect the annotation process and potentially reduce performance. These findings motivate the need for a robust W2S generalization framework that remains effective under distribution shifts.
(OOD) settings compared to in-distribution (InD) settings and, in
Here, we propose Robust AdaptiVe wEightNg (RAVEN robust W2S generalization framework in which the strong model dynamically learns to combine the outputs from an ensemble of weak annotators. Under distribution shift, weak models exhibit higher variance performance compared to the InD setting. RAVEN mitigates this by learning to assign higher weights to more reliable weak models. Specifically. the supervision weights assigned to the weak models are updated iteratively and jointly trained with the strong model's 's parameters. This approach effectively handles high variance of weak models by allowing the strong model to prioritize the most favorable weak model for the fine- tuning data distribution. Additionally, to guide the strong model to learn how to choose the reliable weak model the early stages training, RAVEN introduces easy-sample guided initialization, which trains the strong model exclusively on samples where the weak models consistently provide the same predictions.
We evaluate RAVEN on image classification, text classification, and preference alignment in text generation tasks. RAVEN achieves 55% improvement image classification, 57% improvement in text classification, and a 33% improvement in preference alignment compared to the best alternative baselines for each task. Remarkably, although the information about the performance of weak models is unknown to the strong model, we observe that the strong model typically assigns the highest weight the best-performing weak model without any additional guidance.
PProject website with code: https://brbiccaa.eptll http://bbbbbbbbbeoo.c.ccccc//ssp wtppo//sstaaoooettteraaaaaa

---

2 Problem statement
Weak-to-strong generalization. In W2S generalization setting [0], a strong (large) pretrained model is fine-tuned using labels generated by weak (small) model, denoted by fa and fwr respectively. Given input space X and label space Y, the data is divided into source P'sreX, , Y), fine-tuning
P Pruning (X, Y), and target data Pirg Prgg(, [X, Y). The following procedure is then evaluated: (i) Generate
a weak supervisor: fw is trained on Perc with ground-trume (GT) labels, denoted by f (ii) Train strong model with weak supervision: fa is trained on Puuning using weak pseudo-labels generated by fare. denoted by frsendo, (iii) Train strong model with GT ceiling: fa is trained on P Ptaning with GT, denoted by fg" The performance gap recovered (PGR) on Perg is then calculated follows:
PGR: == Acc(Ptro S f2seedo) Acc( Pegg's for
Acc( Ptrg: f Ace( Pargi fwre
where Acc(A; A: B) denotes the accuracy of model B on data The goal of W2S generalization is to achieve PGR close 1 by making the target model frseudo approximate l indicating that the model reaches ground-truth-level performance even when trained with weak supervision. In [TO), the assumption is that Peres Ptanings and Pirg are drawn from the same underlying distribution.
Weak-to-strong generalization under distribution shifts. We extend the concept of W2S generaliza- tion by considering distribution shifts between Pare and P Pruning. We define distribution shift through the generalization gap: A Rf) = Reaning (f) (f), where R(f) = [ (CC())p((((( (f(((x), y)] the expected risk, f denotes the model, and is the loss function. A large AR(f) indicates substantial distribution shift from Pare to Pluning. which can make weak supervision misaligned This conceptualizes the case where humans annotate leading weak and misaligned supervision. This could, for example, happen with medical data that are both highly specialized and cross-inss institutional. In this setting, PGR, as defined in Eq. ( evaluates how effectively weak supervision under distribution shift be leveraged to enhance the capabilities of the strong model.
undderrstadd and unfamiliar domains,
Observation. To evaluate the robustness of naive W2S generalization under distribution shifts, we analyze changes PGR that happen in OOD scenarios. Specifically, introduce a shift from IMAGENET to IMAGENET-C, treat- ing 19 corruption types distinct domains. For comparison, we establish both InD and OOD setups. In both cases, we use the IMAGENET training set as Percs while the validation set from IMAGENET serves as Praning and Ptrg for the InD scenario, and IMAGENET-C serves as Pruning and Pary for the OOD scenario.
No-shift
Saturate 32.68
Brightness 28.26
Elastic
Gaussian blur
Defocus blur
JPEG
Pixelate
Contrast
1.43j
Spatter
F0.09
Snow +0.19
H0.42
Speckle
+0.69
Motion blur
#1.10
Gaussian noise
11663
Shot noise
1.70
Impulse noise
1.85
Zoom blur
2216
Frost
3.00
Glass blur
8.39
We find that OOD generalization rarely lever- ages the strong model's capabilities as much as when no distribution shift is present (Figure Surprisingly, on 11 out of 19 IMAGENET-C cor- ruption cases, the W2S generalization is even infeasible, i.e., the W2S performance is worse than that of the weak model. This shows the need for an approach that achieves robust W2S generalization under distribution shifts.
10 0
10
20
30
40
50
60
70
PGR
Figure 2: Performance variation with distribution shifts. The Cyan bar indicates positive PGR of InD, gray represents positive PGR for OOD, and red de- notes negative PGR. PGR is reported as percent- age. Detailed results can be found in Appendix B
3 Preliminary
generalization. Inspired by the concept that multiple experts can collectively provide effective supervision even if each individual expert is insufficient on its weakS-to-stront framework [19] utilizes multiple weak models for training. A straightforward approach to ensemble
A more detailed formal definition of
distribution shift is provided in Appendix

---

the weak models is to use a weighted sum to combine their predictions, followed by calculating the loss:
M
Cassemble (Os, W) = Lce(fo(r), Σ (i)
Juz (x)),
(2)
where W : (w(+) Jim denotes pre-defined weights set in ensembling (e.g., the average) M weak models and Lcu denotes cross-entropy loss. fw. denotes i-th weak model and Os represents the parameters of the classifier in the strong model fa-
4 RAVEN: Robust adaptive weighting approach
In W2S generalization under distribution shift, the challenge is that weak models are trained on data distributions that different from the strong model's fine-tuning data distribution, making them even less reliable. Thus, leveraging an ensemble of weak models in Eq. (2) becomes an effective strategy. However, compared the InD setting, weak models have substantially higher performance variance OOD scenarios (Figure 3 and their effectiveness varies considerably across domains (Appendix H.3). Similarly, in analogous real-world scenario, some human annotators may adapt better than others to unfamiliar domains. Consequently, it is crucial to identify more reliable weak models within the ensemble. This motivates the need for dynamic selection mechanism that selects the most suitable weak model(s) for given finettunig dataset P Ptaning.
Motivated by this, we propose RAVEN, robust W2S generalization framework that dynamically learns how to combine different weak models. To improve ensemble performance in OOD settings, RAVEN incorporates two key components: (i) adaptive weighting, and (ii) easy-sample guided initialization.
WWilCCam
Camelyon17
fMoW
33
60
Accuracy
30
50
27
40
70
24
60
0OD
InD
0OD
InD
000
InD
Figure 3: Across different datasets, we observe substantially higher performance variance in OOD scenarios than in InD scenarios. The variance computed using 20 weak models that were initialized with different random seeds and trained on datasets from the WILDS benchmark [35).
Adaptive weighting. Among the weak models, some are more reliable than others. The key idea in RAVEN is to prioritize these models' predictions by assigning different weights to their weak supervisions during the strong model's s training. We begin by training M weak models with different random seeds, which represent human annotators with diverse backgrounds. This approach is inspired by the findings of [40 37,, which demonstrate that training individual networks with random initialization often sufficient achieve diversity in practice. In addition achieving diversity via different random seeds, we also explore using different architectures and data sources when training weak models (Section 5.3). Leveraging these diverse weak models, introduce adaptive weighting loss for the W2S training:
Endaptation (Os, θw) LeE fd(x),
Σ
( pare (x)
Σ g -0 =1 g(0) Ow-
where Ow == M represents the weights used to linearly combine the outputs of the weak models. Importantly, the weak models themselves are fixed, while the ensembling weights (Ø(*) J,M are fully trainable. At each step, these weights are adjusted, leading to a change in how the weak models are utilized throughout training. With this objective, we train a linear classifier parameterized by Oş on top of the frozen pre-trained backbone of the strong model. The backhone generates robust representations, serving effective anchor to train the linear classifier using the weak models.

---

guided initialization. To minimize the objective, the strong model may resort to shorteutting by assigning excessive weight to the weak model most similar to its initial classifier. This behavior is especially problematic during early training when the strong model's is classifier is still under-optimized and performs poorly. To address this, we implement warm-up strategy for the strong model's classifier using easy samples- on which all the weak models agree, ie., give the same predictions. During this phase, the weights g(3) E Gw are fixed to 1/M. Afterward, we enable adaptive weighting across all samples, transitioning from Lensembte with easy samples addppttionn for all the samples. The concept of initializing training with easy samples has proven effective for learning with noisy labels [5]. This approach aligns with W2S generalization, as weak supervision often involves noisy labels resulting from incorrect predictions.
Optimization procedure. After the easy-sample guided initialization, the loss for adaptive weighting addattaioon is optimized by alternating updates to Oş and Ow at each step, as follows:
O := arg min Cadaptation Oss, Ow) O arg min Cadaptation (0), Ow).
(4)
Ow
This alternating optimization effectively balances the trade-off between optimizing the strong model and leveraging weak models. The overall RAVEN training procedure is summarized in Algorithm
Algorithm Robust Adaptive Weighting (RAVEN)
Require: Fine-tuning dataset D Druning- weak models { (f Ti=1, pretrained strong backbone Ensure: Trained linear classifier Og
M
1: Easy-sample guided initialization:
2: Identify Deasy = {ar Draning arg maxi PErC (I:) [k] arg max; Say (x)) Vi
3: Fix ((x) M and define Ow (0(0) J= with >
-= ((()
M
4: Warm up Og on Deasy by minimizing Censemble
5: Adaptive weighting on the full dataset:
6: while not converged do
7: Update weights: Ow +- arg minew Cadupeation (Os, Ow), x E Druning
8: Update strong model: Og +- arg minlss Ludiptation (t,, Ow); x E Dtaning
9: end while
10: return Og
Remark 1. We observe that, in most cases, the strong model assigns the highest weight max; (0(.) MM to the best-performing weak model for Porg by the end of training, without requiring additional guidance. This behavior enhances robustness, as weak models often exhibit significant variation in OOD performance, and there is a strong positive correlation between weak and W2S performance. This capability is particularly advantagcous because it allows for the automatic identification and utilization of the best weak model in terms of Ptrg. even when the GT labels for the target data unknown, and the best-performing weak model is therefore unclear. We provide a theoretical analysis of this property of RAVEN in Appendix C along with detailed quantitative results in Section 5.3 Appendix H.4 and Appendix H.5
i=1
Experiments
We evaluate RAVEN on image classification, text classification, and preference alignment in text generation tasks. Classification training involves two stages: pre-training and fine- tme-tuning. While pre-training is performed a self-supervised or unsupervised manner [14] 12, 555, fine-tuning typically relies on human annotations. In our approach, we focus on fine-tuning by replacing human annotations with predictions of weak models. For text generation tasks, pre-training, supervised fine- tuning, and human preference alignment are the conventional learning phases for foundation models. In this work, we specifically focus on alignment, substituting human feedback with preference predictions from weak models.
5

---

5.1 Experimental setup
Image classification. For OOD setting, we use WWLLDCAM [3], CAMELYON17 [39], and FMoW [8] as benchmarks to evaluate our framework. In these datasets, the domain is defined by the location of the camera, the hospital, and the time, respectively. For each dataset, the training set is used as the source data Parcs 70% of the OOD validation set is randomly selected as fine-tuning data Puning 10% is reserved as the validation set for hyperparameter tuning, and the remaining 20% is designated as the target data Ptrg- For the InD scenario, adopt the same approach as [TO], which utilized IMAGENET. Consistent with the setup in [IO], we use AlexNet as the weak model and DINO ViT8/B as the strong model.
RAVEN is compared to several W2S approaches, including Naive weak-to-strong (Naive) 10 weaks-
to-strong with uniform weights (Ens) (Eq. ). Auxiliary confidence loss (Conf) [0), Bootstrapping
(Boots) [O]. Vision superalignment (V-sup) [26]. Bayesian weakS -t-ssrrog (Bayes) [19], and
Co-supervised learning (Co-sup) 45
Text classification. We employ AMAZON-WILDD 35 MEDMCQA 51 and MEDQA 31 to
evaluate RAVEN for text classification. AMAZON-WIHE both domain and subpopulation shifts. In this dataset, domains correspond to individual reviewers. For the InD W2S scenario, the InD validation data is used in place of the OOD validation set. We designate 1lmma3.2--1 (24), Qwen2.5-0.51 (65) as the weak models and Llama-3. 1-BB, Qwen2.5- 7B, Qwen2.5-14P as the strong models. For the medical benchmarks, we adopt MEDMCQA as the source dataset and MEDQA as the fine-ture tuning and target datasets. These originate from different examination systems from India and the U.S. resulting in natural domain shift between them. For these medical benchmarks, we use Qwen-2.5-0.5B as a weak model and Meditron-22 7B and Meditron 70B [16] as strong models.
sentiment analysis dataset, exhibits
Preference alignment. While Burns et al. highlighted the importance of alignment in W2S, their focus was solely on maximizing preference prediction accuracy rather than exploring alignment itself. Cui et al. [99] investigated the alignment phase but limited their focus to slot filling tasks. In contrast, we assess preference alignment in the context of text generation on HH-RLHF 4, OPENAI SUMMARIZE FROM (EEDBACK(6]], and HUMAN-LIKE IT datasets.
we create a distribution shift by using Helpfulness samples for Pare. randomly sampled Harmlessness samples for Planing. and 1,000 Harmlessness samples for Pergi for InD setting, we use only Harmlessness samples. In the second setup, we use OPENAI SUMMARIZE FROM FEEDBACK as Pare and HUMAN-LIKE DPO as Praning. To align the strong model with Ptaning and the preference predictions of the weak models (i.e., feedback), we employ DPO [56]. To integrate RAVEN into alignment tasks, we propose novel objective, DPO-R (Appendix E..2p. We use Qwen2.5-0.5B 2. and Qwen2.5-7B as the weak and strong models, respectively. Following Rafailov et al. (56), we use the GPT-4o win rate (WR) as our evaluation metric (Appendix (E.4). and compare RAVEN to Naive, Ens, and Bayes baselines (details in Appendix E.. Conf and V-sup baselines are not applicable due to the misalignment of outputs between the weak and strong models, and Boots and Co-sup baselines are unsuitable due to task differences.
For
Experimental details. For each experiment, we perform a grid search to select the learning rate (including its initial value and decay schedule) and the number of training iterations based on validation loss. To determine the duration of the easy-sample guided initialization phase, we conduct a grid search over (10%, 20%, 50%} of the total iterations. Additional implementation details for the method and evaluation protocol are provided in the Appendix D
5.2
Experimental results
Image classification. Results on the image classification task show that RAVEN consistently outperforms all the baselines in the W2S generalization under distribution shift scenario (Table I RAVEN significantly enhances robustness in the OOD setting, while also surpassing other baselines
'We adopt the domain generalization method outlined in Section 5.2 of 45 for robust W2S experiments, rather than their original approach, as it is better suited to the robust W2S scenario. Further details can be found Appendix DD.
"MEDMCQA from the All India Institute of Medical Sciences, and MEDQA from the U.S. National Medical Board Examination.

---

Table 1: Image classification results. We report the average performance across 10 experiments, with PGR calculated as described in Appendix D.6) Weak-to-Strong Generalization refers to using a single weak model, whereas WeakS-to-Strone highlight the best score in red and the second-best score in bold. Model(3) refers to the use of three weak models. For a fair comparison with Co-sup, which utilizes 7 weak models for IWILDCAM, FMOW and IMAGENET, and 5 for CAMELYON we report the RAVEN(>3) performance achieved using the same number of weak models as Co-sup. * indicates that the implementation code was created by us. The standard deviations are reported in Appendix G.I
Generalization denotes ensemble- -bsed methods. We
Weak model
Weaa--toSttonng Generalization
WoutS-do-Mrung Geverolization
Strong wvdde!
AlexNet
Naive
Coaf
Boots
V-sup
Enst3y
Bayes(y
RAVENE)
Co-sapt-33 31
RAVEN(
DINO VIT-B/8
Robust
Went-iin-Srong Generaliation (Ouass-ssittdhhiinn)
IWILDCAM
Ancuracy
PGR
43,83
46
4.21
47.97
6.52
48.76
8.08
47.92
6.40
40
9.73
49.96
10.90
52,79
16.25
49.83
10.00
58.46
22.14
94.76
CAMELYON17
Accuracy PGR
66,90
67.83
2.47
68,90
3,90
70.84
12.24
67.85
2.51
72.18
17.13
68.74
5.13
21.46
73.67
70.00
14.13
73.45
21.48
97.95
FMOW
Accuncy PGR
26.18
-1.26
25.68
24.75
-4.02
25.97
-0.38
25.57
-1.60
27.54
4.31
25,80
-0.07
29,46
10.36
28.70
13.34
31.85
14,06
99.72
Avg.
Accuncy PGR
45.64
46 78
1,50
47.21
2,80
48.53
6.65
47.11
2.44
40.70
10.39
47.78
2.77
51.54
16,09
49.51
12:47
53.23
19.27
$1,20
Weak to Strong Generalization Oh distrbutioo))
IMAGENET
PGR
$4,00
66
68.26
67.11
68.35
68.60
74.49
in InD W2S performance. Generally, ensemble- based methods outperform those relying on single weak model; nevertheless, RAVEN is far more effective than other ensemble-brie based approaches, achieving 55% average improvement in PGR and 3.7% average improvement in accuracy over the best alternative ensemble approach Ens. In contrast to other baselines that do not use domain information, Co-sup utilizes domain information by training each weak model on specific group of domains. Despite that, RAVEN achieves a 54% improvement in PGR compared to Co-sup, even without utilizing any domain knowledge.
Text classification. Experiments on the text classification tasks (Table show that RAVEN consis- tently outperforms all baselines. In the OOD setting, RAVEN achieves 57% average improvement in PGR and 1.4% average improvement in accuracy compared to the best alternative baseline. In the InD setting, RAVEN yields 27% average improvement in PGR over the best bascline. We further observe that Bayes [9], despite leveraging multiple weak models, underperforms in this context and even fall below the performance of individual weak models. Note that the OOD setting involves more fine-tunne munittning instances than the InD setting, which contributes to the higher PGR observed for OOD tasks.
Table 2: Text classification results. We conduct experiments three times and report the average performance on the OOD and InD settings. We highlight the best score in red and the second-best best score in bold. We three weak models for all the WeakS-to-SSiee -t-Stong Generalization methods. The standard deviations are reported in Section G.2
Neuk model
Seet--t-Sttring Generdlization
WoorS-- Gonerolization
Strong mudel
Naive
Conf
Boots
Visup
Ens(3y*
Bayest3j*
Cu-sup RAVEN(3)
AMAZON WILDS (Dar of distribution)
Llama-3.--18 + Llamm-8.1-88
Accurccy
PGR
68.17
68.601
10.88
68.19
0.50
68.59
10.63
67.8M
-7.36
65.45
6.95
-26.70
67.03
44.77
69.95
74.56
71.14
72:15
Qwen2.5-0.58 - QWen2.5-7B
Accurscy PCR
66.34
67.69
30.35
67.12
17.58
67.79
32.61
67.26
20.74
67.64
29.15
-3.01
64.48
28.32
57.60
85,88
70.15
70.78
Qwen2.5-0 5-0 Qwen2.5- 14B
Accummccy
PGR
66, 34
49.70
70.97
46.00
70.82
39.20
70.00
41.30
70.20
44.80
70.52
778.00
59.07
49.50
70.98
$3.60
71.34
75.66
MEDMCQA
MEDQA (Out of distributioo)
Qwen2.5-0.5p Meditren-71
Accursey
25.70 70
26 00
25.6
26.50
27.60
17.90
30.50
1.60
2.60
52.60
23.20
42.60
Qwen2.5-0.58 Moditcon-08
Accuracy PGR
27.57
17.50
27.54
17.30
27.32
15. 20
14 60
26
3.60
.45.00
20.90
-4.50
27.73
19.00
36.37
AMAZIN-WWi DS (In-diatribution)
Llama-3.2-18 Llamm-3.1..... 3-B-
Accurncy PGR
69.74
70.25
50.5
70.02
17.36
20.00
70006
70.02
17 68
70.33
37.26
-217.99
66.29
59.97
14.32
44,00
70.44
71.33
Qwen2.5 5-0.50 - Qwen2.5-76
Acarancy PGR
67.71
53.59
68.61
27.37
68.18
68
35.97
68. 10
23.07
68.55 50
46.14
262.76
63.23
68.56
50.05
71.95
68.94
69:42
7

---

Preference alignment. In the text gener- ation preference alignment task, RAVEN consistently achieves the best performance across both InD and OOD settings. As shown in Table 3 it surpasses the strongest alternative baselines, achieving average im- provements of 2.7% and 32.8% in the OOD settings, and 1.2% and 25.6% in the InD setting (WR and PGR, respectively). Note again that the OOD setting involves three times more fine-tuning instances than the InD setting, resulting in higher performance.
Table Preference alignment results.
Weak
Naive
Eis(3)
Bayes(3)*
RAVEN(3)
Strong
HELPPULNESS HARMLLSSSNSS. (Out of distribution)
WR
59.82
61.38
62.73
61,41
64,04
66.98
PGR
21.79
40.64
22.21
58.94
SUMMARIZATION
(Ont of distribution)
WR PGR
57.50
67.86
68.09
68,18
70.37
83.00
40.63
41.53
41.88
50.47
HARMLESSNESS
distribution)
WR PGR
59.82
61.49
61.43
62.83 42.04
63,60
66.98
23.32
22.49
52.79
5.3 Further analysis
We conduct additional analyses of RAVEN on the image classification task within the robust W2S scenario. Following the setup in Table] D we conduct ten experiments and report their average unless stated otherwise.
Ablation study. We conduct ablation studies to evaluate the impact of ensembling, easy-sample- guided initialization, and adaptive weighting in- troduced in RAVEN, We incrementally incorpo- rate each component and evaluate performance. As shown in Table 4 all components effectively contribute L'VE''s performance. validating our design choices and their individual impor- tance.
Table 4: Ablation study. The values represent the averages across all datasets. Detailed per-dataset results are available in Appendix H.6
Ensemble
/
Easy-sample guided init.
Adaptive weighting
Accuracy
46.78
49.76
50.31
51.37
51.97
PGR
1.80
10.39
10.67
14.28
16.09
Training scheduling. RAVEN first trains only the Os parameters on easy samples using Lensemble. and then updates both Oş and the ensembling weights Ow on the entire dataset using Ladaptation We further explore how different strategies for using easy samples and for applying static vs. adaptive weighting affect performance. As shown in Table our strategy of easy-sample guided initialization with adaptive weighting achieves the best performance, confirming the effectiveness our approach. We suggest that applying static weights to easy samples during the early stages discourages the strong model from relying on shortcuts. These shortcuts occur when the strong model learns weak signal combination that simply mimics its own initial suboptimal predictions, easy way minimize the cross-entropy loss without genuine learning. The detailed results for each dataset are provided in Table 13 in Appendix H.
Table 5: Sample and weight scheduling. All is naive ensemble model, Easy trains solely on easy samples, and Easy-All begins with easy samples before incorporating all samples with static weighting. Easy-All+AW uses adaptive weighting from the start without initial static weighting.
Metrie
All
Easy
Easy-All
Easy-All+AW
RAVEN
Accuracy
49.76
50.10
50,51
50.86
51.97
PGR
10.39
11.13
10.67
12.85
16.09
Increased weak model diversity. To enhance the diversity of weak models, we conduct additional experiments using (i) different weak model architectures and (ii) different data sources for training the weak models. For different architectures, we use AlexNet, ResNet18, and SqueezeNet, weak models to supervise the strong model DINO V/T-B/8. Although these models are trained on the same data source, their architectural differences result in learning diverse features. For different data sources, we follow the setup in Liu and Alahi (5), which constructs distinct data splits based on sub-domains. Specifically, we train seven AlexNet models on different "camera location" domains for (WILDCAM, five AlexNet models on different "hospital" domains for CAMELYON 17, and seven AlexNet models on different "time" domains for FMOW. As shown in Table 6 RAVEN achieves an 81% % PGR improvement in the setting with different architectures and 24% PGR improvement in

---

Table 6: Performance comparison across different weak model configurations. Reported values averaged over all datasets, with detailed results for each dataset provided in Appendix H.2
Weak model
Weak-to-Strong Generalization
WeakS-to-Streng Generalization
Strong model
Naive
Conf
Boots
V-sup
Ens
Bayes
Co-sup
RAVEN
DINO VIT-B/8
Differentt Architectares of Weak Models
Accuracy PGR
42.04
42.80
41.18
44.21
41.87
43.40
44.23
42.45
46.03
84.43
1.78
-1.72
3.15
-0.26
3.28
5.33
-0.31
9.65
Different Dato Sources for Training Weak Models
Accuracy
47.40
46.34
46.92
47.33
46.46
49.40
50.13
47.43
51.17
84.63
PGR
-2.08
-0.50
0.93
-1.75
5.46
7.76
1.05
10.49
the setting with different data sources, compared to the best alternative baseline Bayes. These results confirm that RAVEN is effective at weak-to- to-sono generalization with weak models of both low and substantial diversity.
How do the weights Ow evolve over iterations? One would expect that the performance of a weak model evaluated on the test set of the strong model (Ptrg) correlates with its W2S performance. Consequently, it would be desirable for RAVEN to assign higher weights to weak models with better Perg performance. To investigate this, we begin by examining the correlation between the performance of weak models on Pirg and their W2S performance and indeed observe high correlation, highlighting the importance of selecting the optimal weak model (Figure 4
We next aim to understand whether the weights Ow that the strong model assigns to weak models in RAVEN agree well with the actual Parg performance of the weak models. Notably, we find that the strong model predominantly assigns the highest weight to the best-performing weak model for Pirg- despite not having access information about the weak models' performance as evaluated against GT (Figure 4 Right). This indicates that the strong model can identify high-quality annotations from multiple annotators without any guidance, which is a remarkable feat. Additional graphs with more weak models and classifier initializations can be found in Appendix H.4 and Appendix H.5
IWidCCa
80
Camelyon17
Weak Model
Strong acc
58
Best
Second Best
Worst
10
75
IWVidCam
Comelyon17
1MoW
49
70
0.9
40
Weightt
33
0.4
.00
40
50
60
05
70
'NoW
Combined Summmary
0.0
Datasot
1.0
inkecam
cameyynn17
0.8
27
0.6
25
40
0.4
6.2
20
40)
5000
10000
10000
10000
Weak Model Acc
Weak Model Acc
Iterattin
Iteratiin
Iteration
Figure 4: Left: Correlation between weak model accuracy Acc(Pirgi fot) and its W2S accuracy Acc(Pirg) frsendo) ). Right: The weights Ow assigned to weak models across two runs of RAVEN.
Number of weak models M. We further analyze the performance across varying numbers of weak models. As the number of weak models increases, we observe that the performance of RAVEN im-
K
Accuracy
& 8 16
proves (Figure 5). However, the gains gradually diminish after including a large number of weak models, indicating saturation effect.
20
Number of Models
Figure Effeec of the number of weak models.

---

Other results. In addition to the results above, further experiments are provided in the Appendix, supporting our choices and assumptions on the proposed framework. Specifically, we present analysis of: RAVEN's performance on IMAGENET-C (Appendix [B), performance across weight configurations (Appendix (H.8), comparison between the best weak model and adaptive weighting (Appendix (H.7). exploration of strong model's identification of the best weak model (Appendix(H..) scaling analysis using diverse (weak, strong) model pairs (Appendix H.llp. and qualitative preference-alignmeent results (Appendix H.
Related work
Weak-to-strong (W2S) generalization. W2S generalization framework was first introduced in [10]. In this work, the authors evaluated various strategies beyond naive generalization, includ- ing confidence loss, bootstrapping, and unsupervised fine-tuning, to enhance performance. They demonstrated the feasibility of developing superhuman models, finding that was further supported
by subsequent studies 67 64). Theoretical insights behind this observation were investigated in
To enhance W2S generalization performance, (26) extended the concept of confidence loss introduced in [10), developing adaptive confidence loss, which dynamically adjusts the balance between learning from weak model confidence. Multiple weak models have been employed using the AdaBoost technique []] within the framework of hierarchical mixtures of experts [45], and estimating Bayesian-basee confidence loss [9]. Sang al. [57] leveraged debates among multiple models to enhance strong models. Within this ensemble framework, we propose a method for identifying reliable weak supervision, which proves especially effective in our
1K
38
28
60
23
66
novel scenario involving distribution shifts.
Scalable oversight. Scalable oversight B
aims to enhance human supervision in novel and challenging environments. To this end, models can be used to evaluate other models [29] [88 33 32 or decompose complex problems into simpler subproblems [4] 44 By contrast, W2S generalization
explores how to effectively leverage unreliable human supervision itself to train models that can ultimately surpass human performance.
Learning under distribution shift. A trained model often performs poorly when it faces query data whose distribution is significantly different from the training data [36] 62 21 49]. Learning under distribution shifts aims to make the model robust in this scenario. This challenge has been observed across domains such healthcare, autonomous driving, and facial recognition, where models fail to generalize across hospitals, lighting conditions, or demographic subgroups due to distribution mismatches [35 53 2 20 9 A range of methods has been proposed address this issue, and their
effectiveness is actively evaluated through diverse benchmarks 25 63 30
In the context of existing literature, our novel scenario can be viewed as an intersection of weak-to- strong generalization and learning under distribution shift.
7 Concluding remarks
Limitations. The weak models used in RAVEN differ only in their random seeds. It is unclear whether this alone can fully capture the diversity of human annotators, particularly in fields where domain expertise is crucial. Another aspect that calls for further investigation is the choice of adaptive weights (Appendix [H8).. While we found that model-wise weights outperform the (model, sample)-wise variant, the former is inherently subset of the latter (where model-wise weights are the same across all samples). Future work could explore ways to address the optimization challenges of (model, sample) weighting, potentially leading to an even stronger RAVEN variant.
Conclusion. We extend the concept of W2S generalization by explicitly modeling distribution shifts between the source and fine-tuning datasets. We demonstrate that naive W2S generalization often becomes intractable in such scenarios, and conventional approaches fail to adequately address this issue, falling short of robustness requirements. To address this challenge, we present RAVEN, novel framework that enables robust W2S generalization.
We would like to thank Artyom Gadetsky, Fabian Gröger, Maxim Kodryan, Ramon Vinas Torné, Shuo Wen, Siba Smarak Panigrahi, and Yulun Jiang for their valuable discussions
Acknowledgments.
10

---

regarding our work. We gratefully acknowledge the support of the Swiss National Science Foundation (SNSF) starting grant TMSG12 226252/1, SNSF grant IC0010 231922. the Swiss AI Initiative and the CIFAR Multiscale Human Catalyst. Myeongho Jeon was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [RS- RRS-2024-00333669, N10250156). Suhwan Choi was supported by 1) the Starting growth Technological R&D Program (RS-2024-00506994) funded by the Ministry of SMEs and Startups (MSS, Korea), 2) Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024 (Project Name: Development of K-POP artist-centered video editing solution: customized multimodal Al model and generative asset, Project Number: RS-2024-003994433 Contribution Rate: 50%). 3) Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City, and 4) Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) [NO. 20--021--111111111 1343, Artificial Intelligence Graduate School Program (Seoul National University)). Jan Sobotka was supported by the Bakala Foundation during his studies at EPFL.
References
[1] Aakriti Agrawal, Mucong Ding, Zora Che, Chenghao Deng, Anirudh Satheesh, John Langford, and Furong Huang. EnsemW2S: Can an Ensemble of LLMs be Leveraged to Obtain a Stronger LLM? In Advances in Neural Information Processing Systems, Safe Generative Al Workshop, 2024.
[2] Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a Pose: Neural Networks are Easily Fooled by Strange Poses of Familiar Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[3] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. arXiv preprint arXiv: 06565,6555500 2016.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, 7ac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. Computing Research Repository, 2022.
[5] Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iWildCam 2021 Compe- tition Dataset. arXiv preprint arXiv:2105.034449 2021.
[6] Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, et al. Mcasuring Progress Scalable Oversight for Large Language Models. arXiv preprint arXiv:2211.03540, 2022.
[7] Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39, 1952.
[8] Adrian Brady, Risteárd Ó Laoide, Peter McCarthy, and Ronan McDermott. Discrepancy and Error in Radiology: Concepts, Causes and Consequences. The Ulster Medical Journal, 81(1), 2012.
[9] Joy Buolamwini and Timnit Gebru. Gender Shades: Intersectional Accuracy Disparities in Com- mercial Gender Classification. In Conference on Fairness, Accountability and Transparency, 2018.
[10] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. In International Conference on Machine Learning, 2024.
[11] Ethem YagIz Çalik and Talha Rüzgar Akkuş. Enhancing Human-Like Responses Large Language Models. arXiv preprint arXiv:2501.050032, 2025.

---

[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Supervised Vision Transformers. In International Conference Computer Vision, 2021.
[13] Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the Gain in Weak-to- Strong Generalization. In Advances in Neural Information Processing Systems, 2024.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Machine Learning, 2020.
[15] Xinlei Chen and Abhinav Gupta. Webly Supervised Learning of Convolutional Networks. In International Conference on Computer Vision, 2015.
[16] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini. Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. arXiv preprint arXiv:2311.1607, 2023.
[17] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, 2017.
[18] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional Map of the World. In Conference on Computer Vision and Pattern Recognition, 2018.
[19] Ziyun Cui, Ziyang Zhang, Wen Wu, Guangzhi Sun, and Chao Zhang. Bayesian WeakS- to-Strong from Text Classification to Generation. In International Conference on Learning Representations, 2025.
[20] Dengxin Dai and Lue Van Gool. Dark Model Adaptation: Semantic Image Segmentation from Daytime Nighttime. In International Conference Intelligent Transportation Systems, 2018.
[21] Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal Misgeneralization in Deep Reinforcement Learning. In International Conference on Machine Learning, 2022.
[22] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow: From Reward Modeling Online RLHF. Transactions on Machine Learning Research, 2024.
[23] Yijun Dong, Yicheng Li, Yunai Li, Jason D Lee, and Qi Lei. Discrepancies are Virtue: Weak-to- Strong Generalization through Lens of Intrinsic Dimension. In International Conference on Machine Learning, 2025.
[24] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama Herd of Models. arXiv preprint arXiv:2407.21783, 2024.
[25] Ishaan Gulrajani and David Lopez-Paz. In Search of Lost Domain Generalization. In Interna- tional Conference on Learning Representations, 2021.
[26] Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vision Superalignment: -t-Sttonn Generalization for Vision Foundation Models. arXiv preprint 402iv:2402.03749, 2024.
[27] Edward Hu. yelong shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022.
[28] Muhammed Emrullah Iidiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws. In International Conference on Learning Representations, 2025.
[29] Geoffrey Irving, Paul Christiano, and Dario Amodei. Al Safety via Debate. arXiv preprint arXiv.1805.00899 2018
[30] Myeongho Jeon, Suhwan Choi, Hyoje Lee, and Teresa Yeo. An Analysis of Model Robustness across Concurrent Distribution Shifts. Transactions Machine Learning Research, 2025.
12

---

[31] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease Does this Patient Have? Large-Scurl Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences, 11(44), 2021.
[32] Zachary Kenton, Noah Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah Goodman, et al. On Scalable Oversight with Weak LLMs Judging Strong LLMs. Advances in Neural Information Processing Systems, 2024.
[33] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers. In International Conference on Machine Learning, 2024.
[34] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
[35] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasumaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A Benchmark of in-the-Wiie Distribution Shifts. In International Conference on Machine Learning, 2021.
[36] David Krueger, Tegan Maharaj, and Jan Leike. Hidden Incentives for Auto-Inducee Distribu- tional Shift. arXiv preprint arXiv:2009.09) 2020.
[37] Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems, 2017.
[38] Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical Analysis of Weak-to- Strong Generalization. In Advances in Neural Information Processing Systems, 2024.
[39] Byungjae Lee and Kyunghyun Paeng. A Robust and Effective Approach Towards Accurate Metastasis Detection and pN-stage Classification in Breast Cancer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018.
[40] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. arXiv preprint wrXiv:1511.063/ 2015.
[41] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable Agent Alignment via Reward Modeling: Research Direction. arXiv preprint 1..77.1811.07871, 2018
[42] Bo Li, Peng Qi. Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou, Trustworthy AI: From Principles to Practices. ACM Computing Surveys, 55(9), 2023.
[43] Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment. In Conference on Empirical Methods in Natural Language Processing, 2024.
[44] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, 1lya Sutskever, and Karl Cobbe. Let's Verify Step by Step. In International Conference on Learning Representations, 2023.
[45] Yuejiang Liu and Alexandre Alahi. Co-Supervised Learning: Improving Yeak-to-Strong Generalization with Hierarchical Mixture of Experts. arXiv preprint arXiv:2402. 15505, 2024.
[46] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations, 2017.
[47] Nya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference Learning Representations, 2019.
[48] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is Being Transferred in Transfer Learning? In Advances in Neural Information Processing Systems, 2020.
[49] Richard Ngo, Lawrence Chan, and Sören Mindermann. The Alignment Problem from a Deep Learning Perspective. In International Conference on Learning Representations, 2024.
13

---

[50] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, 2022.
[51] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: A Large- Scale Multi-Subject Multi- Dataset for Medical Domain Question Answering. In Confer- ence on Health, Inference, and Learning, 2022.
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, 2019.
[53] Christian S Perone, Pedro Ballester, Rodrigo C Barros, and Julien Cohen-Adad. Unsupervised Domain Adaptation for Medical Imaging Segmentation with Self-Ensembling, Neurolmage, 194, 2019.
[54] Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In Advances in Neural Information Processing Systems, 2019.
[55] Alec Radford. Improving Language Understanding by Generative Pre-Training, 2018.
[56]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In Advances in Neural Information Processing Systems, 2024.
[57] Jitao Sang, Yuhang Wang, Jing Zhang, Yanxu Zhu, Chao Kong, Junhong Ye, Shuyu Wei, and Jinlin Xiao. Improving Weak-to-Stroon Generalization with Scalable Oversight and Ensemble Learning. arXiv preprint arXiv:2402.0066 2024.
(58) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-Critiquing Models for Assisting Human Evaluators. Computing Research Repository, 2022.
[59] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential Deep Learning to Quantify Classification Uncertainty. In Advances in Neural Information Processing Systems, 2018.
[60] Changho Shin, John Cooper, and Frederic Sala. Weak-to- ttrong Generalization Through the Pata-Centric Lens. In International Conference Learning Representations, 2025.
[61] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to Summarize with Human Feedback. In Advances in Neural Information Processing Systems, 2020.
[62] Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhat- tacharya, and Jeff Bilmes. An Effective Baseline for Robustness to Distributional Shift. In International Conference on Machine Learning and Applications, 2021.
[63] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishna- murthy Dj Dvijotham, and Ali Taylan Cemgil. A Fine-Grained Analysis on Distribution Shift. In International Conference on Learning Representations, 2022.
[64] David Xing Wu and Anant Sahai. Provable Weak-to-Stroogg Generalization via Benign Overfit- ting. In International Conference on Learning Representations, 2025.
[65] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 Technical Report. arXiv preprint 1.xi::2412.1115.. 2024.
[66] Wei Yao. Wenkai Yang, Ziqiao Wang. Yankai Lin, and Yong Liu. Understanding the Capabilities and Limitations of Weak-to-Strong Generalization. Computing Research Repository, 2025.
[67] Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin Edelman, Milind Tambe, Sham Kakade, and Eran Malach. Transcendence: Generative Models Can Outperform the Experts That Train Them. In Advances in Neural Information Processing Systems, 2024.
14

---

[68] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. LlamaFactory: Unified Efficient Tuning of 100+ Language Models. In Annual Meeting of the Association for Computational Linguistics, 2024.
15

---

A Definition of distribution shift
To define distribution shift between some two distributions Pat: and P Punning we use the generalization gap: A R(f) Ruminng] (f) ( [)), where R(f) = P( y)] is the expected risk, denotes the classifier, and L is the loss function (e.g,, cross-entropy). A large AR(() indicates that the model struggles generalize from distribution Pse distribution miinngg showing the presence of significant distribution shift.
Another way quantify distribution shift is through divergence metric D Puniing), where D can be any suitable measure such as Maximum Mean Discrepancy (following (54)), KL-diveegennc,, or optimal transport. When ((Psre Puuiing) ) I and R4 is small enough, regard the distributions as approximately aligned, ie.,, Parc z Pumiing Conversely, we say that Parc ş6 Puuning and consider the shift to be significant potentiall misleading the model's performance inference- when
((Prre Puniin)) >
B Motivating experiment
Without fine-tuning, For image classification tasks, Burns et [0] used an IMAGENET TrrGtane--rettrinned AlexNet as the weak model without additional training. To replicate their setup, we compute the mean and standard deviation of the pre-trained AlexNet's performance over five random splits the IMAGENET validation set into Praming and Ptrg. and over five random splits of the IMAGENET- FT- dataset into Praming and Ptrg- The results of this experiment are provided in Table 7 and, combined with the setup described below, shown in Figure 2
With fine-tuning. A more common practice, however, to evaluate models trained with different random seeds while keeping the dataset fixed. We also adopt this approach, examining how PGR changes under distribution shifts when using five different weak models. Specifically, using different random seeds, we reinitialize the classification head of the IMAGENET-pretraann for epochs. This approach is particularly relevant in the weakS-to tt-ssoonn generalization framework (ensemble-based methods), where multiple distinct weak models are required. Using these fine-tuned models, we compare RAVEN naive W2S generalization on the IMAGENET-C dataset. shown in Table 8 despite severe synthetic domain shifts, the performance of RAVEN exceeds that of the weak model (human in the future) in most cases, outperforming the naive W2S generalization method. For this experiment,
AlexNet and train it
the same AlexNet weak models as reported in Table I and designate DINO ResNet50 as the strong model.
Finally, the results for the naive W2S generalization method shown in Figure 2 are the average over these two setups, effectively balancing clarity of presentation, adherence previous work [10, and common practices used in the field.
16

---

Table 7: Result of Motivating Experiment: without fine-tuning of weak models. For the weak and strong models, we report the accuracy, while for the W2S fine-tuning (AlexNet - DINO ResNet50), present the PGR. The values represent the mean and standard deviation over five runs.
Corruption Type
AlexNet
DINO ResNet50
AlexNet DINO ResNet50 (PGR)
No-shift
56.26 0.56
64.00 1 0.39
74.01 + 5.67
Saturate
15.72 0.43
59.68 0.43
11.43 + 1.83
Brightness
42.70 0.37
58.62 0.69
39.59 1.02
Elastic
40.18 18 0.28
62.64 + 0.30
19.56 # 2.34
JPEG
37.66 0.25
53.00 1: 0.79
9.63 4: 2.38
Gaussian
13.64
5.87 d:
Pixelate
29.06 0.42
49.44
0.47
5.50 .L 0.66
Deforus Blur
11.54 1: 0.15
39.32 1: 0.71
5.48 :t: 0.80
Spatter
23.08
0.30
48.30
0.30
3.00 + 0.76
Contrast
9.18 + 0.25
52.01
2.66 +: 0.63
Fog
13.86 0.36
47.31 + 0.31
1.91 # 0.28
Motion Blur
16.00 0.39
38.76 1: 0.19
1.57 ::: 1.92
Snow
12.32 0.25
39.54 0.44
1.56 :: 1.37
Zoom Blur
19.22
39.91
1.04 d: 1.79
Spockle Noise
8,58 +: 0.08
41.91 + 0.27
-0.21 # 0.54
Gaussian Noise
5.00 + 0.17
40.06 + 0.37
-1.03 + 0.67
Noise
5.22 + 0.16
39.64
-1.45 + 0.46
Frost
10.68 0.13
36.90 0.37
-1.69 :: 1.11
Impulse Noise
4.18 4: 0.19
37.12 1. 0.27
- -1.76 de 0.58
Glass Blur
10.68 0.13
30.92 0.41
-9.00 1. 0.80
17

---

Table 8: Performance comparison IMAGENET-C C for the Motivating Experiment: with fine-tuning of weak models. We conduct five experiments and report the results as the mean and standard deviation. Negative PGRs are highlighted in red.
Corruption
Metrie
AlexNet
Naive
RAVEN
Dino ResNet50
Accuracy
4131
47.66
+ 0.39
Saturate
PGR
20.91
1.42
Accuracy
40.64 0.36
43.72 + 0.50
45.67 + 0,06
58.90 + 0.50
Brightness
PGR
16.93 - 0.74
27.63 + 0.62
Accuracy
38.02
0.49
37.82 0.38
38.83 + 0.06
52.90 + 0.70
Elastic
PGR
-1.34 0.78
5.07 2.07
Accuracy
35.71 = 0.48
35.02 + 0.35
36.27 + 0.06
53.16 + 0.79
JPEG
PGR
44.42 = 0.36
2.94 +: 0.96
Accuracy
10.86 = 0.32
10.92 0.42
11.57 +:: 0.35
41.84 +: 0.35
Gaussion blur
PGR
0.22 + 0.57
2.40 + 0.88
Accuracy
26.46
0.43
26.12 0.70
27.47 1: 0.47
49.72 +: 0.46
Pixelate
PGR
-1.48 d: 0.55
4.54 1: 1.38
Accuracy
1: 0.27
0.34
0.50
Defocus blu
PGR
0.57 + 0.34
1.62 1.30
Accuracy
22.74
21.90
22.97 +
+ 0.46
Spatter
PUR
Accuracy
8.40 + 0.19
8.46 + 0.21
8.97 + 0.06
52.40 + 0.47
Contrast
PGR
0.20 0.12
1.34 0.24
Accuracy
13.92 0.48
14.04 0.55
14.17 + 0.06
47.64 + 0.02
PGR
-2.75 = 0.34
0.33 = 0.06
Accuracy
13.01 0.21
12.06 + 0.21
13.10 + 0.10
39.18 + 0.18
Motion blur
PGR
-3.78
0.30
0.34 + 0.60
Accuracy
12.08 = 0.22
11.58 4: 0.16
12.27 + 0.29
40.34 +: 0.38
Snow
PGR
-1.93 = 0.26
0.35 + 0.95
Accuracy
16.02
0.35
14.66
0.28
16.30 1: 0.39
40.64 :: 0.46
Zoom blur
PGR
5.36 :: 0.42
1.30 1: 1.56
Accuracy
1: 0.36
1: 0.37
0.36
Speckle noise
PGR
-1.15 = 0.25
0.10 0.25
Accuracy
4.68 + 0.26
3.90 +
+ 0.00
Gaussion noise
PGR
-2.23
Accuracy
4.76 + 0.26
4.12 : 0.16
5.03 4: 0.06
40.10 +: 0.00
Shol moise
PGR
-1.95 0.14
0.54 0.14
Accuracy
10.50 0.41
9.28 + 0.66
9.90 + 0.35
36.88 + 0.12
PGR
4.27 1: 0.42
11.36 di 0.21
Accuracy
4.00 +: 0.20
3.40 4: 0.07
4.23 4: 0.06
38.28 4: 0.38
Impulse noise
PGR
1.00 = 0.10
0.49 4: 0.34
Accuracy
9.56 + 0.36
7.82 + 0.33
8.27 +: 0.25
31.80 4: 0.35
Glass blur
PGR
-7.78 =: 0.17
- -5.79 +: 0.61
18

---

C Theoretical analysis of how RAVEN operates
To understand why RAVEN often identifies the most suitable weak model for the target data distribu- tion, we begin with the following assumption.
Assumption 1. The ensemble of weak models tends to make predictions similar to those of the best-performing weak model:
5 (x) zo arg max (R)),
arg kE(T....Cy
where
E (z)
Fie (z),
and (x) denotes the best-performing weak model. While the ensemble prediction is often close to that of the best model, assume that f is more accurate and confident.
Let fa denote the strong model, and define the adaptation loss as:
Lddaptation (0s,ww) == LCE (f(()), Σ g(6) sang (x)
s.t. Σ g(i =1, 8(0)
Ow.
(6)
This loss is optimized by alternating the following two steps:
Step 1. Optimize O with θw fixed.
When weights are uniform (i.e., g(0) 1.). the strong model learns to mimic average behavior of weak models:
f(
psc (x).
Step Optimize Ow with Os fixed.
We can rewrite the objective as:
LCE(Ow) = Σ g
(8)
where
C; = - E
aw (1:) log fo (x)
(9)
Assumption 2. Each weak model is highly confident in predicting single class:
Jan (x) = (c,..., ? C).,..,), c < 1,
(10)
with the largest value - index
by == arg max fw. [b)]
&
Then,
Σ
M. (x) log [x) log (x()))
(11)
and the objective simplifies to:
C2 f.
(12)
By Assumption 1 and Eq, (3), we infer that the best weak model f*( most closely matches the predictions of (a) and therefore receives the highest weight- consistent with empirical findings. Notably, this conclusion still holds even when only the best weak model satisfies Assumption 2, since the best weak model yields the smallest C, As result, RAVEN naturally prioritizes the weak models that generalize well to the target distribution.
Through this iterative optimization, the strong model fs becomes increasingly aligned with the best weak model and further improves by leveraging this implicit supervision.
19

---

D Experimental details
D.1 Adaptive weighting
We initialize the adaptive weights g(0) E @w uniformly, ie., 060) = M for E (1. After the easy-sample guided initialization phase, during which ((0) are fixed and only the strong model parameters Oy are trained, we start optimizing the adaptive weights as well. More specifically, for each minibatch of the fine-tuning data, first generate and combine soft labels from the M weak models using the latest Ow (Eq. 3). Then, using these pseudo labels, we calculate gradients of Ladaptation Os, Ow) with respect to O and Ow. and perform single update step of both using the Adam optimizer (4 and SGD, respectively. After each update of Ow. we clip and normalize the adaptive weights to sum up as:
(a ,)
((O)
(13)
2/=1 max
where 10 e are then set as ((0) for the next minibatch of the fine-tuning process.
For text classification, we derive the weak supervision Vson (a)) (i.e., pseudo-label) by computing weighted sum of weak model logits Vow using adaptive weights, followed by softmax operation for stabilization:
(c) exp (yew (ar)
(x)
(14)
Σ Nmm
exp (Haw (x))
(k)
where
Vaw (2)
(()) føre (z).
(15)
Above, (x) represents the logits of the i-th weak model trained on source data, K is the number of label classes, and с denotes a specific class label. Consequently, yson(()) serves as the weak supervision.
hyperparameters, including those of the baselines, are found using a grid search based on the validation loss. The hyperparameter search includes the number of fine-tuning epochs, the learning rate, and the method-specific hyperparameters (e.g., the learning rate of the adaptive weights and the length of the easy-sample guided initialization phase). Building on the publicly available code from Burns et al. [10], we use the cosine annealing schedule [46] while optimizing the strong model parameters Os. but keep the learning rate of the adaptive weights Ow fixed. We perform hyperparameter search the range shown Table
Table 9: Hyperparameter configurations for RAVEN.
Task
Learning rate for θg
Learning rate for θw
Easy-sample guided period
Epoch
Image Classification
(1e-6, 1e-5, 1e-4, 1e-3)
(1e-6, 1e-5, 1e-4, 1e-3) [le-5, 1e-4, 1e:3)
(10%, 20%, 50%}
(30, 50, 80, 100) (10,20)
Text Classification
(1e-6, 1e-5, le-4)
{20%, 50%)
Preference Alignment
(le-6, 56-6, le-5, 5e-5)
[le-5, le-4, 1e-3)
50%)
(2)
D.2
Weak models for ensemble-based methods
The weak models used in ensemble-based methods are trained using different seeds and hyperparam- eters found through a search over the number of epochs, learning rate, and weight decay. For image classification experiments, we train weak models with the Adam optimizer [34], the cross-entres cross-entropy loss function, early stopping, and multiplicative (factor 0.96) and the cosine learning rate schedule for IWILDCAM and FMOW, respectively. For CAMELYON17 and IMAGENET, we employ SGD with momentum of 0.9 and fixed learning rate. Our selection of optimizers and hyperparameters is based on the original work introducing the WILDS benchmark datasets (35 with adjustments that we found to improve the performance of our specific model architectures.
20

---

With this setup, all weak models achieve around the same InD validation accuracy on their respective datasets (+22)). Since the W2S fine-tuning step of the IMAGENET experiments employs its validation dataset, we use 10% of the IMAGENET training set the validation set for weak model training. Importantly, the case of IMAGENET, we start from a pre-trained AlexNet model loaded from the PyTorch library [52] with randomly reinitialized classification head to achieve diversity among the weak models (same setup as for Table 8)
For text classification, we train weak models using the Adam optimizer 3 the cross-entropy loss function, and cosine learning rate schedule. We initialize with pre-trained weak language models (Qwen2.5-0.55) LL.ma-- - 3.n--BBBBBBB). ) from Hugging Face. In the AMAZON-WILL WILDS OOD setting, because the OOD validation and test sets are OOD with respect each other, we only use the OOD validation set, splitting it into fine-tuning (70%), test (20%), and validation (10%) subsets. For the InD setting, we use the original AMAZON- WILDS InD validation and test sets.
For reward modeling, we train weak models using A (47 optimizer and cosine learning rate schedule. The source data is split into training (90%) and validation (10%) subsets. For further details on reward modeling, please refer to Section E.I
We report the final hyperparameter values for training the weak models in Table 10
Table 10: Hyperparameter configurations for training the weak models.
Learning rate
Weight decay
Epochs
Image Classificatin
IWILDCAM
3e-5
8e-2
60
CAMELYON17
le-3
le-2
30
FMOW
8e-4
5e:3
150
IMAGENET
3e
8e-5
20
Teat Classification
AMAZON-WILDS
le-5
0
10
Preference Alignment
HH-RLHF
lc-6
lc-3
To balance the statistical power of repeated W2S experiments and computational costs, reuse the weak models in the ensembles. More specifically, for each experiment & E {1...., .........0) 10), we use the pretrained weak models k. + 1, & + 2 for the Ens(3), Bayes(3) and RAVEN(3) methods reported in The same idea is applied for RAVEN(>3) and our analysis of the number of weak models Figure
Table and Table 2]
D.3 Baselines for image and text classification
We use the original implementation of the auxiliary confidence loss from Burns et al. 10 and the Vision superalignment (adaptive confidence loss) from Guo
[26]. We implement the bootstrapping method [10] with DINO ResNet50 as the intermediate (medium-strong) model for image classification and
for text classification. This intermediate model is first fine-tuned using the weak supervision, and then its pseudo labels for the fine-tuning dataset (P (Ptuning) are used to fine-tune the final strong model. In image classification, AlexNet serves as the weak model, and DINO ViT8/B as the strong model. In text classification, weak supervision is performed within the same model family: Llama-3.2 2B supervises Llama-3.1-8P and Qwen2.5- 0.5B supervises Qwen2.5-7B. Fine-tuning in both steps is done using the same setup in all the other reported W2S experiments. For the Bayesian weakS-to- tookk---t--toonggg baseline [9], we follow the authors' formulation while implementing the method using the code for evidential deep learning from Sensoy et al. [59]. See Section D.5 for details on how we adapt Co-supervised learning [45] for our setup.
"napss//hugggnggfce..oo www.eeaaeccoeeeeeeeeeeeeeeaeaaee aat.ee/eeeecccoaaaaaaaaaaaaaaaee 1aaaeeaaaaaamaammaamm3.3333233
21

---

D.4 Baselines for preference alignment
Cui et al. [9 implemented DPO for Bayesian weakS-to-strong by using the log probability P Pf weak models proxy for preference. More specifically, their preference reward Phayes for a given text y computed as the weighted sum of the log probabilities from M trained weak models:
M
Tbayes (1)
(.)(()
(16)
==1
Strong models are trained on target preference data using the DPO objective with and as defined Eq. 22 The chosen text Ye is the text with the highest preference reward, while the rejected text yr is the one with the lowest preference reward:
Ve
arg max Tbayes(U), Y1 1
arg min Tbayes 1=1,2....
(17)
In HH-RLHF 4), 2, where the two available text choices are the ground-truth chosen and rejected texts. The original work [19] assigned different weights X; to different weak model families. However, since we use the same type of weak models, we set X = 1
M
We note that in this Bayesian weakS-to-strong approach [19), the weak models were not trained on preference data. In contrast, we train our weak models on source preference data using DPO, allowing them to better capture preference-baie --sed signals. Additionally, the weak models are trained within the same hyperparameter space as the strong models, as detailed in Section E.3
D.5 Co-supervised learning
For DOMAINNET 48 Liu and Alahi 45 designed two-level structure of specialized weak
supervisors using sub-domain labels from DomainNet. At the first tier, the problem domain is divided into two groups of sub-domains: "clip'. "quick', 'sketch' and 'info', 'paint', 'real', At the second level, each supervisor is dedicated to a specific sub-domain.
Building on the official implementation, we adapt this approach of Co-supervised learning to our OOD scenario as follows. First, we use the domain information from 1W WILDCAM, CAMELYON17. FMoW, and AMAZON- WII.DS to create multiple distinet weak models that are trained on subsets of domains that form hierarchy. In the fine-tuning process, the strong model is first supervised by the weak model trained on all domains. Then, at each subsequent supervision level, it is further fine-tuned with weak supervision coming from the weak models, where each was trained on half of the domains of the supervision level before, effectively learning from more specialized weak models. For all these supervision levels with multiple weak models, we select the weak supervision label for each data point that agrees the most with the strong model's predictions from the previous supervision level.
D.6 Calculating PGR for ensemble-based methods
For an ensemble- method with M weak models, we compute the PGR in the following way:
PGR = Acc( Acc(Pirgi Acc(PergiSpseuue fpseudo)
1 DM'
i- Acc( Porgg fare)
(18)
M Σ. Acc(P Ptrgi 12.) - Acc(Prrog ferc)
where f2' i (1..... .M} refers to the strong model fine-tuned on Ptuning using the GT labels and the same random seed as the i-th weak model før used in its training. We report this PGR along with the average accuracy of weak models and strong models trained with GT labels in Table [ and Table 2
E
Configurations for preference alignment experiments
We perform preference alignment two phases: reward modeling for the weak models and preference optimization for the strong model. First, we train the weak models using HEL.PULLNESS samples from HH-RLHF 4 to predict preference rewards. The trained weak models then generate preference signals for target HARMLESSNESS samples, which are used to align the strong model via DPO (56).
22

---

E.1 Reward modeling
The preference signal is commonly modeled using the (rw((1))))) 21) (x v2)),
Bradley- Terry model 7 50 4
exp (rw v1))
p(y1 y2 12)
(19)
exp (r.( + exp (Tw y2))
where represents input prompt, 1/1 and y2 responses to 2, Tw is the weak model as the preference reward predictor, and a denotes the sigmoid function. For given input prompt æ with ground-truth chosen response Ye and rejected response Yr, we train weak models by minimizing the following loss function:
LRM (z, Vc. Ve. = log (rw Uc) Tw (2, Ur)).
(20)
After the weak models trained for reward modeling, use their predicted preference rewards for a pair of responses and 1/2 to determine the chosen and rejected responses for aligning the strong model on new data:
Ve = arg max Tir (3,, v), = arg min Iw (w, y)-
(21)
y
Our implementation builds upon the reward modeling framework implemented by Dong et al. [22]
E.2 DPO-R: Direct Preference Optimization for RAVEN
Unlike the cross-entropy loss that is used for classification tasks, adaptive weights cannot be updated with direct preference optimization (DPO) loss since they are not involved in the loss calculation. To address this, we modify the DPO loss, leading our DPO-R formulation. The original objective function of DPO is defined as follows:
Tolye x)
To (a x)) ( x)) ]
CoPo(TO) )e)) == -B. logo
log
3 log
(22)
(ye x))
where x represents prompt, is the chosen response and y, is the rejected response. ma denotes model being trained with DPO, while नee refers to the model my before undergoing DPO training.
Direct preference optimization with adaptive weighting. In RAVEN, adaptive weighting with multiple weak models (i.e., ensemble) can be formulated as follows:
Tens (I, y)
gl0 (w, (z.v),
(23)
where rw: is the i-th reward model, implemented weak model our setting. While CPO relies on a single parameter 3, we introduce two parameters. Be and 3r, which represent the weights for the chosen and rejected log-likelihoods, respectively. By incorporating the Bradley-Terry model 7 into the DPO loss, we derive new loss tailored for RAVEN, referred to the DPO-R loss, defined as:
To(Ue 2x) (ye x))
To ( x))
LDPO-R (Tes Teet) -E -E -E(z.9....> log o (B log
.Br log
(24)
Teet yrr xx)
Be plye Vr 1z), Br p(yr (r > Bc 1²),
(25)
exp (rene v1))
ply1 x 82 12) exp (rens 31)) + exp (rens 122)
(26)
E.3 Training details for DPO
Strong models are trained using LoRA (27 with a rank of 8 and scaling parameter 16. We use the AdamW [4 optimizer with cosine learning rate scheduling. The LoRA target modules are v_proj, gate_proj. dwwwdwwwwwooooooee s_proj. up_proj. q-proj, and k_proj. For 3, we perform grid search over the values (0.1, 0.2, 0.5). Our implementation for DPO experiments is based on Zheng et al. (68). Hyperparameter search spaces outlined in Table
23

---

E.4 Evaluation metric for preference alignment: GPT-40 Win Rate
For evaluation in preference aligmment experiments. we use the GPT-4o win rate metric, following the prompts utilized by Rafailov al. [56]. All alignment experiments are conducted with GPT-4c
with the response order randomized for each evaluation. The prompt used is shown below.
For the following query to a chatbot, which response is more <"harmless" "human-like">?
Query: <the user query?
Response A:
<either the test method or baseline>
Response B: <the other response>
FIRST provide a one-sentence comparison the two responses and explain which you feel is more <"harmless" or "human-like">, SECOND, on new line, state only "A" or "B" to indicate which response is more <"harmless" "human-like">>
Your response should use the format:
Comparison: <one-sentence comparison and explanation> More <"harmless" or "himan-like>>; <"A" or "B">
We compute WR using selected preference samples: HH-RLHF and the SFT model output for OPENAΣ SUMMARIZE FROM FEEDBACK and HUMAN-LIKE DPO. WR is calculated by comparing the outputs of DPO-trained models with pre-DPO models.
F
Computational resources
F1
Image classification
Our image classification experiments used cluster of 8 NVIDIA GeForce RTX 3090 GPUs, but each individual run required only single GPU and less than 22 GB of VRAM. The most computationally demanding part training the weak models, which each required most one day the single GPU. The subsequent W2S experiment, where we used the pre-trained weak models and pre-collected strong model embeddings, took
one hour. Hyperparameter search for image classification, as reported in Table 9 took between one and two days.
F.2
Text classification and direct preference alignment
Our text-classification and DPO experiments used cluster of NVIDIA H100 GPUs, and each individual run employed the GPUs in parallel. A single text-classification training run completed in roughly one hour. A single DPO training required about five hours per the same eight- GPU setup. Hyperparameter search for each baseline, as reported in Table took one-two days for text classification and about five days for DPO.
G Experimental results
G.1 Image classification
We present the image classification results in Table along with standard deviations calculated based on the setup described in Section D.2

---

Table 11: Image classification results. We report the average performance across 10 experiments, with PGR calculated as described in Section D.6] Weak-to-Strong Generalization refers to using a single weak model, whereas WeakS-to-Streee highlight the best score in red and the second-best score in bold. Model(3) refers to the use of three weak models. For a fair comparison with Co-sup, which utilizes 7 weak models for IWILDCAM, FMOW and IMAGENET, and 5 for CAMELYONT 7, we report the RAVEN(>3) performance achieved using the same number of weak models as Co-sup. * indicates that the implementation code was created by us.
Generalization denotes ensemble-basee based methods. We
Weok model
Wiaak-tr-Stronng Geueralization
WookS-do-Siong Goneralizaion
Strong model
AlecNet
Naive
Coef
Rois
V-up
Fins())
Bayoos3)*
RAVEN(S) Cosppp):) RAVEN
DINO VIT-R8
Robort
Woot-tt-Sinog Geserofigation (Ou--f-dismlunioo))
IWILDCAM
Accuncy
+6.05
+1.30 18
45.82
47,97
-2.42
-2.31
48.76
+-0.96
47.90
+3.04
49:38
49
+3.48
+3300
$1.79
+-245
49.83
94.76
+0.21
PCR
6.40
9.73
10.00
16.25
10.09
CAMELYON17
Accuracy
15.16
12:41
67.83
-2.67
-2.51
1.2.56
13.25
72.18
41.58
68:74
73,67
12.81
13.18
70.00
1.1.77
1.015
97.93
PGR
2.47
5.90
12.24
2.5L
17.13
5.13
21.46
14.13
21.48
FMOW
Accuracy
26.18
12.01
11.42
25.68
-1.32
24.75
25.97
-1.44
11.48
25.57
11.56
27.54
11.52
25:89
29.46
13.07
15.
28.70
1.2.52
31.85
11.04
59.72
PCR
-1.26
-4.02
-0.38
-1.60
4.31
40.67
10.05
13.24
14.06
Avg.
Accuracy PGR
45.64
46.78
1.00
47,21
2.80
48.53
6.65
47.11
2.44
49.70
10.39
47.78
2.77
$1.54
16.09
69.51
12:47
53.23
19.27
81.20
Weok Strong Genernlitation Uh dismitation)
IMAGENETT
Accuracy
+0.80
$4,90
+0 10
66.85
=0.12
68.26
=0.08
65.72
+0..0
67.88
+:1.20
67.11
+0006
49.99 99.98
+0.11
57, 27.90
+:1.24
68.60
+00.34
68.35
+0.19
74.49
PGR
60.97
68.12
55.39
65.87
62.02
-23,99
56.35
69.54
68.60
G.2 Text classification
We report the text classification results in Table 12 accompanied by standard deviations computed according to the setup outlined in Section D.2
Table 12:
Text classification results. We conduct experiments three times and report their average. We highlight the best score in red and the second-bes, score in bold. We use three weak models for all the WeakS-to-Stree ttSSoon Generalization methods.
Wesk
Weal-tt- Strong Geursliztion
Weot Soi-Ston Gearraligtion
Strong
Conf
Voup
Bayea
Co-sup))
RAVEN(1)
Weak Streng Ginnrraliation (Qur-sf-iitrünullee))
Lluma-3.2-B 3.2-1B Llama-3.1-80 IBB
Accunacy
68.17
0.02
#0.10
68.60
4:0.18
68 19
68.59
69.95
71.14
#0.02
72.15
-18.20
44.77
Qwm22.50 5R
5:7R
Accunacy
+0.6
66.34
67.69
+0.21
67.12
67.26
64.44
67.60
70.15
70.78
17.58
28.32
Woul-tosSimog Generalicaaion ((odrersbunin))
1.mm--1 1.1mm-3.1118 1-88
Accuracy
69.74
+.03
70.02
=0:00
70.02
69.97
70.44
+..06
71.33
17.56
-217.80
14.32
67.71
63:23
68.04
Qwar2,5-0555 5R Qwen?. 5-7R
Accuracy
+0.1
27.37
35,97
23,07
262.76
50.05
71,95
H Further analysis
H.1
Training scheduling
Table 5 presents the results aggregated over all datasets. The corresponding detailed results for each dataset are provided in Table 13
H.2 Increased weak model diversity
Table presents the results aggregated over all datasets. The corresponding detailed results for each dataset are provided in Table [4
25

---

Table 13: Sample and weight scheduling. All is a naive ensemble model, Easy trains solely on easy samples, and Easy-All begins with easy samples before incorporating all samples with static weighting. Easy-All+AW uses adaptive weighting from the start without initial static weighting.
Dataset
Metric
All
Easy
Easy-All
Easy-All+A
RAVEN
Accuracy
49.56
50.27
49.75
50.30
52.79
(WILDCAM
PGR
9.73
11.12
10.11
11.17
16.25
Accuracy
72.18
71.66
74.05
75.06
73.67
CAMELYON17
PGR
17.13
15.55
17:24
24.04
21.95
Accuracy
27.54
28.38
27.72
29.46
FMoW
PGR
4.31
6.73
4.67
3.34
10.06
Accuracy
49.76
51.97
Avg.
Table 14: Performance comparison across datasets and configurations. Each result represents the average over 10 runs. We highlight the best score in red and the second-best in bold.
Weak model
Weak-to-Stroni Generalization
WeakS-to-Sttong Generalization
Strong model
Naive
Coní
Boots
V-sup
Ens
Bayes
Co-sup
RAVEN
DINO VIT-B/8
Different Architectures of Weak Models
IWILDCAM
Accuracy
47,93
48.90
45,20
49,30
47,00
49,60
50.20
49.80
49.90
94,83
Accuracy PGR
59.91
60.50
61.00
62.57
60.67
62.00
63.40
63.80
63.90
97.80
CAMELYON17
1.5
2.9
7.0
2.0
5.5
-2.7
10.5
FMoW
Accuracy PGR
18.27
24.30
1.7
-2.2
-0.5
-0.8
0.8
2.0
-2.9
14.2
Avg.
Accuracy PGR
42.04
42,80
41.18
44.21
41.87
44.23
42.45
46.03
1.78
-1.72
3.15
-0.26
3.28
5.33
-0.31
9.65
Different Data Sources for Training Weok Models
Aucuracy PGR
53.80
56.00
IWILDCAM
-13.2
-8.6
-10.5
-12,7
3.4
5.4
-15.7
5.4
CAMELYON17
Aucuracy 67.04
68.98
69.58
71.10
69.10
69.00
70.90
70.50
71.50
97.8
PGR
6.3
8.2
13.2
6.7
6.4
12.5
11.2
14.5
Accuracy PGR
21.35
21.61
20.89
21.40
21.64
24.00
23.50
24.40
26.00
61.5
FMOW
0.66
0.13
0.73
5.36
7.60
11.58
Accuracy
47,40
46.34
46.92
47.33
46.46
49.40
50.13
47.43
$1.17
84,63
PGR
-2.08
-0.50
0.93
-1.75
5.46
7.76
10.49
H.3
Performance variation among weak models on multiple different domains
When utilize the domain information available in the WILDs datasets (not used by RAVEN itself), we find that different weak models generalize well to different OOD domains (6). motivating selection mechanism that adapts the particular (unknown) domain subset in fine-tuning data.
26

---

Recurwcy
0.
0.00
25
0:22
0.48
31
0:39
6:12
0:45
34
C:1
31
0:32
0.00
30
28
0:20
20
0.41
0.79
1.20
24
INlidCam OOD per-domain accuracies
1.00
22
0:21
18
0.24
0:11
13
12
0:12
10
D35
D:09
0.07
871
0.79
CJT
0:12
0.00
0:44
2-022
Figure 6: Performance variation of the weak models on different domains. Heatmap of per-domain accuracies of weak models trained with different random seeds, evaluated on the IWILDCAM fine- tuning dataset.
27

---

H.4 Weights Uw with different weak models
Figure Figure 8 and Figure illustrate the evolution of weights assigned the weak models throughout the learning iterations. Each graph represents three distinct weak models.
1.0
68
0.6
6.-
0.6-
6.6-
6.3335
6-5-
0.1
5.6-
62-
0.2
02
0.0
60
0.5
00
0.3323
23046
20000
5200
10060
5000
10000
5202
12000
5400
10000
0.333
022250
62-
0.2
011
0.2
0,33200
50000
1,0000
5500
:0000
5000
10000
3307
18359
5000
10000
Figure 7: Weights Ow over iterations for IWILDCAM. The x-axis and y-axis represent iterations and weights, respectively. The darker the line, the higher the OOD performance of the weak model corresponding to that adaptive weight.
0.18
04
0.35
0.3
03
0.2
0.30
0.45
0.
0.35
0.34
04-
0.32
12030
Figure 8: Weights Ww over iterations for CAMELYON 17. The configurations are identical those in Figure
0.-
0.-
02
0.6
2000
9230
2002
6302
2303
10000
1500
0.4
7500
32
16660
Figure 9: Weights Ow over iterations for FMOW. The configurations are identical to those in Figure
28

---

H.S
Weights Uw with different initialization of strong model's classifier
Figure 10 Figure I1 and Figure T2 show how the weights assigned to weak models evolve over learning iterations. Each graph depicts the weight progression for the same weak models but with different initializations of the strong model'ss classification head (different seeds).
0.8
0.8
0.4
0.4
0.1
10000
20200
10002
20200
10000
20000
10000
20000
10000
20100
0.8
0.8
0.8
0.0
0.2
100002
Figure 10: Weights Ow over iterations for IWILDCAM. The configurations are identical to those in Figure
0.38
0.45
5.36
0.36
0.32
0.36
0.00
0:34
0.34
0.34
0.35
0.32
0.30
0.39
0.32
0.32
0.30
0.25
0.36
0.350
3.350
0.329
9:325
0:2
5.59
10000
5200
5000
10000
1.0000
5000
10000
Figure 11: Weights Ow over iterations for CAMELYON17. The configurations are identical to those Figure
0.6
9.8
3.5
0.8
0.8
0.4
0.
4300
5030
4203
0010
2000
2000
4300
1.0
1.0
0/
3.4
3.4
8.4
0A
2000
6300
6030
2030
4303
6030
2030
2003
6400
2000
4010
6000
2000
6300
6030
Figure 12: Weights Ow over iterations for FMOW. The configurations are identical those in Figure
29

---

H.6
Ablation study
Figure
13
provides the results of the ablation study of RAVEN for all datasets reported in Table
Table 15: Ablation study on sub-components.
Ensemble
Easy-sumple guided init.
Adaptive weighting
Acc
46.82
49.56
49.75
52.07
52.79
IWILDCAM
PGR
4.21
9.73
10.11
14.79
16,25
Acc
67.83
72.18
74.05
72.28
73.67
CAMELYON17
PGR
2.47
17.13
17.24
17.32
21.95
Acc
25.68
27.54
27.72
29.77
29.46
FMOW
PGR
-1.26
4.31
4.67
10.74
10.06
Accuracy
46.78
49.76
50.51
51.37
51.97
Avg.
PGR
1.80
10.39
10.67
14.28
16.09
H.7 Best weak model vs. adaptive weighting
In practice, identifying the best weak model for Pruning and Ptry is not feasible (GT is not available). However, under the hypothetical assumption that it is possible, we explored whether RAVEN performs effectively compared to a scenario where the strong model uses only the best weak model. As shown in Table [6] we compare RAVEN against this idealized baseline. While the strong model often gravitates toward the best weak model during the later stages of training, RAVEN achieves better performance in W2S generalization. This improvement can be attributed to the strong model's ability to not only identify the best weak model but also effectively utilize diverse signals during the early stages of training. Interestingly. RAVEN exceeds the to-soooo- generalization baseline (Ens) even when it does not rely on the best-performing weak model, demonstrating its ability to discover favorable linear combinations of weak models.
Table 16: Comparison between the best weak model baseline and RAVEN.
iWiLDCAM
CAMELYON17
#MoW
Avg.
Best
RAVEN
Best
RAVEN
Best
RAVEN
Best
RAVEN
Acc PGR
53.18
52.79
73.23
73.67
28.58
29.46
$1.66
51.97
16.93
16.25
20.61
21.95
7.44
10.06
15.00
16.09
H.8
The choice of weights Ow
RAVEN, we set Ow E . as model-wise weights. We compare this approach to (model, sample)- wise adaptive weights Ow E MRN,,, where N denotes the number of instances in the dataset. Intuitively, the (model, sample)-wise approach allows the strong model to adaptively weight the weak models differently for each sample, providing more fine-gr finegrrined version of adaptive weighting. However, as shown in Table [7] model-wise weighting outperforms the (model, sample)-wise variant. We suggest that this is due the increased difficulty of optimizing the latter.
H.9 Linear non-linear combinations of weak models
We evaluate RAVEN using both linear combination of weak models with θw and non-linear variant, where the weights are determined by an MLP followed by softmax (referred as Non-linear RAVEN). Both approaches achieve comparable performance, as shown in Table [8] However, the linear formulation we adopt substantially more parameter-efficient the non-linear version requires embedding dimension x number of weak models parameters (e.g., 768 x 3 for DINO ViT-B/8 with three weak models), whereas the linear version requires only as many parameters as the number of weak models (e.g., 3).
30

---

the highest weight (()) by the end of training, and 'miss' otherwise. Interestingly, we observe that the standard deviation of the weak models' accuracy for target data is significantly higher in the miss Table 17: Weights Ww. Model, Sample denotes (model, sample)- weights while Model represents our model-wise weights (RAVEN).
Model, Sample
Accuracy
49.28
52.79
(WILDCAM
CAMELYON
Accuracy
73.50
73.67
PGR
21.39
21.95
Accurcy
27.56
29.46
FMoW
PGR
4.37
10.06
Accuracy
50.11
51.97
PGR
11.73
16.09
Table 18: Performance comparison between linear and non-linear combinations of weak models. Bold values indicate the better- perterrpeffoming RAVEN variant for each dataset.
Dataset
Weak
Non-linear RAVEN Linear RAVEN
Strong
IWILDCAM
44.8
54.9
54.6
94.8
CAMELYON17
65.2
69.9
69.5
97.9
FMOW
28.0
33.8
34.2
60.9
H.10 Hit and missed cases
We define 'hit' as the case where the strong model identifies the best weak model by assigning
case compared the hit case (Table 19.. Note that there is missed case for FMoW.
Table 19: Comparison between hit and missed cases.
Dataset
Std. (Hit)
Std. (Miss)
IWILDCAM
7.11
9.82
CAMELYON17
3.38
6.91
Avg.
5.24
8.36
H.11 Scaling analysis
We conduct additional experiments both image and text classification tasks to analyze the scaling behavior of RAVEN. More specifically, for the vision tasks, we designate SqueezeNet and ResNet18 88 the weak models, and DINO ViT-S/8 and DINOy2 ViT-L/14 Distilled as the strong models. For NLP, we use GPT2, Qwen-2.5-0.55B and Llama-3.2-1B as the weak models, and Qwen-2.5-7B and Llama-3.1-8B the strong models. We run scaling experiments all classification tasks and report the average PGR in Table 20 different scales.
As can be seen, RAVEN consistently outperforms all baselines across
H.12 Examples of preference-aligned results
For the lethal poison scenario, ChatGPTT-40 considers only RAVEN and the GT-aligned strong model to be safer than the Chosen answer in terms of harmlessness. As Figure 13]grss, RAVEN in this scenario provides detailed and precise warning about the dangers using poison.
31

---

Table 20:
Average PGR in scaling experiments. We highlight the best score red and the second-best
score in bold. We use 3 weak models for Ens and RAVEN, and weak models for Co-sup.
tt-uronn Generalization
WeakSSto-Strrong Generalization
Naive
Conf
V-sup
Ens
Bayes
Co-sup
RAVEN
GPT2 Qwen2.5-7B
29.53
29.38
29.23
29.07
-167.60
-69.89
29.82
GPT2 Qwen2.5-0.50
11.04
10.46
-337.10
-77.34
GPT2 + Llama3.1-8
16.66
15.09
12.66
14.00
-166.70
14.32
20.97
GPT2 - 1lama3.2-1B
3.26
2.84
3.42
15.42
-109.19
-51.69
15.90
ResNet18 DINO ViT-S/8
20.24
26.31
27.45
-$3.40
9.47
31.47
ResNet18 - DINOv2 VIT-L/14 Distil
27.39
27.54
28.77
24.23
6.28
11.61
29.82
SqueezeNet DINO ViT-S/8
14.22
17.72
16.11
16.85
1.81
1.53
18.79
SqueezeNet DINOv2 VIT-I/114 Distil
14.26
22.29
18.03
16.24
12.92
0.99
22.93
Harmlessness
Dasasset
User
Chosen
Rejected
What's a comutn place and lethal poison that could hands un?
I'm afraid I don't understadd your question. Could repeat
mean, like, something like posticide that people could get poisoned by aecidentally?
Loss (vs, Chosee)
Loss (vs. Chosze)
Loss (sx Chowe))
Hoyes(33
common place to find a lethal poisos is in household cleaning product. One exmple is bleach, inhulad
A comtatonn place find lethal poison in housebold cleaming pruduct, One example bleach, which highly rosic ingested inhuled large amounts. It's important to handle and store such products safely and follow all safety instructions provided the manufacturer.
common place find lethal poissn kitchan, of lethal poisos that you could get your hands arsenic.
which be highly large aasounts. It's amportant to handle and store such products safely and fotlow all safety instructions provided by the manufacturer.
Chosen)
Fas(3)
Wis (ve Rejactad)
Post-DPO
Win Rejected)
RAVENE))
common place to lethal paison houschold cleaning pruduct. One example is bleach. which can be highly vosic ingested or infuled in largo importent hundle products
chemistry or pharmaceutica facility, However, it is imporiant mote that hindling and using poisons dangerous should unly dune by trained professionals. recommended atterpt use poisons for any resson.
I'm scrry. but I carner prowide with that information.
follow safatyy instructions.
Figure 13: Qualitative results of preference alignment. Loss (vs. Chosen) denotes the case where ChatGPT-44 selects the Chosen sample over the model- response, while Win (vs. Rejected) represents the opposite scenario.
32

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We discussed the importance of addressing distribution shift in weak-to-strong generalization in both the abstract and the introduction (section Ip.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of work in the Concluding remark (section 7).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
This work is supported by empirical findings and thorough justification.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present the experimental details in Experimental Setup dsubsection 5.1) and Experimental Details (Appendix D). and Configurations for Preference Alignment Experiments (Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
34

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is is publicly available at https://githhub.coo///iii
raven
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We detail the experimental settings the Experimental Setup (subsection 5.1) and provide additional information in Experimental Details (Appendix D) and Configurations for Preference Alignment Experiments (Appendix E).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
We report the results of the main script along with standard deviations in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This is specified in Computational Resource (Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have carefully reviewed and adhered to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work offers a new perspective on understanding the conventional weak-to- strong generalization scenario, concept relevant future superhuman models, and proposes a method to address a novel problem. The evaluation is conducted in an analogy- based setting and therefore does not pose negative societal impacts when applied in practice.
Justification:
Guidelines:
36

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have specified the referenced assets in the experimental details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
37

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code and corresponding assets publicly available at htttps://githbb com commmmlbio-epffrrvvee
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
38

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
39


Reviews:
[{"review_id": "ZaWI3geu5j", "reviewer": "Reviewer_ZFQR", "summary": "This paper introduces RAVEN, a robust framework for Weak-to-Strong (W2S) Generalization under distribution shifts. W2S generalization is the paradigm where a strong model is trained using supervision from weaker models. The authors observe that naive W2S methods fail under distribution shifts, where the weak model’s training and fine-tuning data distributions differ. RAVEN addresses this by dynamically learning optimal combinations of multiple weak models through adaptive weighting and easy-sample guided initialization. The method is evaluated across image classification, text classification, and preference alignment tasks, showing significant improvements over state-of-the-art baselines, even in out-of-distribution (OOD) settings.", "questions": "Suggestions:\n\n1. The current setup varies only random seeds, which doesn’t capture real-world annotator diversity. Using weak models with different architectures, data sources, or noise types would better simulate practical supervision scenarios and strengthen the method’s robustness.\n\n2. The paper lacks theoretical analysis explaining RAVEN’s behavior under distribution shifts. Adding convergence guarantees or formal justification for adaptive weighting would make the approach more rigorous and broadly credible.\n\n3. The current tasks focus on standard classification and narrow preference alignment. Evaluating on more complex domains like medical diagnosis or reasoning tasks would better demonstrate RAVEN’s generalizability.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nFirst, the paper addresses a pressing challenge in scalable model supervision: how to train models when high-quality supervision is unavailable or misaligned. By explicitly modeling distribution shifts—a common but under-explored failure mode in W2S settings—it moves the field toward more realistic and robust solutions. Second, the proposed RAVEN framework is methodologically sound, incorporating adaptive weighting and a principled training schedule that includes easy-sample guided initialization. These innovations allow the model to automatically prioritize reliable weak supervision, leading to statistically significant improvements across multiple tasks. Extensive experiments and ablations strongly support the claims, showing consistent gains over diverse and strong baselines.\n\nWeaknesses\nOne limitation is that the diversity among weak models in RAVEN is achieved only via different random seeds. This simplification might not reflect real-world annotator variance, especially in domains like healthcare or law where supervision quality is tied to domain-specific expertise. Moreover, while the adaptive weighting strategy is powerful, the paper acknowledges challenges in optimizing finer-grained model-sample-specific weights, leaving room for improvement in dynamic supervision. Lastly, the preference alignment experiment, though promising, is narrowly scoped and may not generalize to other types of alignment tasks beyond \"helpfulness\" and \"harmlessness\" datasets.", "was_revised": true, "final_justification": "The author addressed the concerns in the review, but this does not change the paper's overall impact. I will maintain the same score.", "rebuttal": "We thank the reviewer for thoughtful review and valuable feedback. We are grateful to the reviewer for recognizing that our work addresses a pressing challenge, weak-to-strong generalization under distribution shifts, that will help move the field toward more realistic and robust solutions. We also appreciate the reviewer’s acknowledgement that RAVEN is methodologically sound, demonstrates strong performance and that our extensive experiments and ablations strongly support our claims. \n\n---\n**Weakness 1, Suggestion 1.** *One limitation is that the diversity among weak models in RAVEN is achieved only via different random seeds. Using weak models with different architectures, data sources, or noise types would better simulate practical supervision scenarios and strengthen the method’s robustness.*\n\n**Response.** Thank you for this important suggestion. In response, we conducted experiments using (1) different architectures and (2) different data sources to enhance the diversity of weak models. For **different architectures**, we used AlexNet, ResNet18, and SqueezeNet, as weak models to supervise the strong model DINO ViT-B/8. Although these models are trained on the same data source, their architectural differences result in learning diverse features. For **different data sources**, we followed the setup in [1], which constructs distinct data splits based on sub-domains. Specifically, we trained seven AlexNet models using different “camera location” domain labels for iWildCam, five AlexNet models using different “hospital” domain labels for Camelyon17, and seven AlexNet models using different “time” domain labels for fMoW.\n\nRAVEN achieves an 81% PGR improvement in the different architectures setting and a 24% PGR improvement in the different data sources setting, compared to the best alternative baselines Bayes. These results demonstrate that RAVEN enables effective weak-to-strong generalization both when weak models have relatively low diversity and when they exhibit substantial diversity. We will include these results in the final version of our manuscript.\n- Different architectures\n||W-Avg|Naive|Conf|Boots|V-sup|Bayes|Ens|Co-sup|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|\n|iWildCam|47.9|48.9|45.2|49.3|47.0|50.2|49.6|49.8|49.9|94.8|\n|PGR||2.1|-5.8|2.9|-2.0|4.8|3.6|4.6|4.2||\n|Camelyon17|59.9|60.5|61.0|62.6|60.7|63.4|62.0|63.8|63.9|97.8|\n|PGR||2.9|2.9|7.0|2.0|9.2|5.5|-2.7|10.5||\n|fMoW|18.3|19.0|17.3|18.1|17.9|19.1|18.6|13.8|24.3|60.7|\n|PGR||1.5|-2.2|-0.5|-0.8|2.0|0.8|-2.9|14.2||\n|avg.Acc|42.0|42.8|41.2|44.2|41.9|44.6|43.4|42.5|**46.0**|84.4|\n|avg.PGR||1.8|-1.7|3.1|-0.3|5.3|3.3|-0.3|**9.6**||\n- Different data sources\n||Alex|Naive|Conf|Boots|V-sup|Bayes|Ens|Co-sup|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|\n|iWildCam|47.2|48.4|50.3|49.5|48.6|56.0|55.2|47.4|56.0|94.6|\n|PGR||2.5|6.5|4.8|3.0|18.5|16.8|0.4|18.5||\n|Camelyon17|67.0|69.0|69.6|71.1|69.1|70.9|69.0|70.5|71.5|97.8|\n|PGR||6.3|8.2|13.2|6.7|12.5|6.4|11.2|14.5||\n|fMoW|21.3|21.6|20.9|21.4|21.6|23.5|24.0|24.4|26.0|61.5|\n|PGR||0.7|-1.2|0.1|0.7|5.4|6.6|7.6|11.6||\n|avg.Acc|45.2|46.3|46.9|47.3|46.5|50.1|49.4|47.4|**51.2**|84.6|\n|avg.PGR||3.2|4.5|6.0|3.5|12.1|9.9|6.4|**14.9**||\n> [1] Liu et al. \"Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture of experts.\" arXiv 2024.  \n\n---\n**Weakness 2.** *While the adaptive weighting strategy is powerful, the paper acknowledges challenges in optimizing finer-grained model-sample-specific weights, leaving room for improvement in dynamic supervision.*\n\n**Response.** **Model-wise weighting (RAVEN) is both more effective and computationally efficient than (model, sample)-wise weighting.** As demonstrated in our experiments (Appendix G.6), RAVEN consistently outperforms the (model, sample)-wise approach. The latter also presents practical limitations, as the number of weights grows with the number of samples—posing scalability challenges, particularly when training strong models on large datasets. Nevertheless, as you noted and we acknowledged in the limitations section, improving the optimization of (model, sample)-wise weighting is a promising direction for future work.\n\n---\n**Suggestion 2.** *The paper lacks theoretical analysis explaining RAVEN’s behavior under distribution shifts. Adding convergence guarantees or formal justification for adaptive weighting would make the approach more rigorous and broadly credible.*\n\n**Response.** To understand why RAVEN often identifies the most suitable weak model for the target data distribution, we begin with the following assumption.\n\n*Assumption 1.* The ensemble of weak models tends to make predictions similar to those of the best-performing weak model:\n\n$ \\arg\\max_{k \\in \\{1, \\dots, C\\}} f_{\\sum}(x)[k] \\approx \\arg\\max_{k} f^*(x)[k],\\quad$ (1)\n\nwhere  $ f_{\\sum}(x) := \\sum_{i=1}^M f_{w_i}^{\\text{src}}(x) $,  and $ f^* (x) $ denotes the best-performing weak model. While the ensemble prediction is often close to that of the best model, we assume that $ f^* $ is more accurate and confident.\n\nLet $ f_s $ denote the strong model, and define the adaptation loss as:\n\n$ L_{\\text{adaptation}}(\\Theta_S, \\Theta_W) := \nL_{\\text{CE}} \\left(f_s(x), \\sum_{i=1}^{M} \\theta^{(i)} f_{w_i}^{\\text{src}}(x)\n\\right) \\quad \\text{subject to } \\sum_{i=1}^{M} \\theta^{(i)} = 1, \\ \\theta^{(i)} \\in \\Theta_W.\\quad$ (2)\n\nThis loss is optimized by alternating the following two steps:\n\n**Step 1.** *Optimize* $\\Theta_S $ *with* $ \\Theta_W$ *fixed.*\n\nWhen weights are uniform (i.e., $\\theta^{(i)} = \\frac{1}{M}$), the strong model learns to mimic the average behavior of weak models:\n\n$f_s(x) \\approx \\frac{1}{M} \\sum_{i=1}^M f_{w_i}^{\\text{src}}(x).\\quad$ (3)\n\n**Step 2.** *Optimize* $\\Theta_W$ *with* $\\Theta_S$ *fixed.*\n\nWe rewrite the objective as:\n\n$L_{\\text{CE}}(\\Theta_W)=\\sum_{i=1}^{M} \\theta^{(i)} C_i $,  where  $C_i := -𝔼_{x \\sim D}$ $\\left[ \\sum_{k=1}^C f_{w_i}^{\\text{src}}(x)[k] \\log f_s(x)[k]\\right].\\quad$ (4)\n\n*Assumption 2.* Each weak model is highly confident in predicting a single class:\n\n$f_{w_i}^{\\text{src}}(x)=(\\epsilon,\\dots,1-\\overline{\\epsilon},\\dots,\\epsilon),\\quad \\epsilon \\ll1,\\quad$ (5)\n\nwith the largest value $1-\\overline{\\epsilon}$ at index  $\\hat{k}ᵢ:= \\arg\\max_k f_{w_i}^{\\text{src}}(x)[k] $. \n\nThen,\n\n$\\sum_{k=1}^C f_{w_i}^{\\text{src}}(x)[k]\\log f_s(x)[k]\\approx\\log f_s(x)[\\hat{k}_i]\\quad$ (6)\n\nand the objective simplifies to:\n\n$C_i \\approx - 𝔼_{x \\sim D} \\left[ \\log f_s(x)[\\hat{k}_i] \\right].\\quad$ (7)\n\nBy *Ass.* 1 and Eq. (3), we infer that the best weak model $f^*(x)$ most closely matches the $f_s(x)$ predictions and therefore receives the highest weight—consistent with empirical findings. Notably, this conclusion still holds even when only the best weak model satisfies *Ass.* 2, since the best weak model yields the smallest $C_i$. As a result, RAVEN naturally prioritizes the weak models that generalize well to the target distribution.\n\nThrough this iterative optimization, the strong model $f_s$ becomes increasingly aligned with the best weak model and further improves by leveraging this implicit supervision.\n\n---\n**Weakness 3, Suggestion 3.** *The current tasks focus on standard classification and narrow preference alignment. Evaluating on more complex domains like medical diagnosis or reasoning tasks would better demonstrate RAVEN’s generalizability.*\n\n**Response.** Following this suggestion, we show benefits of RAVEN on the (1) medical diagnosis task, and (2) on the additional preference alignment task.\n\nFor **medical domain**, we use three Qwen-2.5-0.5B as the weak models and Meditron-7B and Meditron-70B [2], open-source LLMs adapted to a medical domain, as the strong models. We use MedMCQA [3] (sourced from the All **India** Institute of Medical Sciences) as the source data and MedQA [4] (sourced from **the U.S.** National Medical Board Examination) as the fine-tuning and the target data. Since the datasets come from different countries, there is a domain shift between the source and target. RAVEN achieves a 39% PGR improvement in Qwen-2.5-0.5B → Meditron-7B and an 8% PGR improvement in Qwen-2.5-0.5B → Meditron-70B, compared to the best alternative baselines ACD and Naive, for each task. These results show that RAVEN remains effective even in the complex domain of medical diagnosis.\n- Qwen-2.5-0.5B → Meditron-7B\n||Weak|Naive|Conf|Boot|ACD|Ens|Bayes|CSL|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Acc|25.7|26.0|26.3|25.8|25.7|25.7|24.7|25.3|**26.5**|27.6|\n|PGR|-|17.9|30.5|5.3|1.6|-2.6|-52.6|-23.2|**42.6**|-|\n- Qwen-2.5-0.5B → Meditron-70B\n||Weak|Naive|Conf|Boot|ACD|Ens|Bayes|CSL|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Acc|25.70|27.57|27.54|27.32|27.26|26.08|20.90|25.22|**27.73**|36.37|\n|PGR|–|17.5|17.2|15.2|14.6|3.6|–45.0|–4.5|**19.0**|–|\n\nIn addition, to cover a **broader range of alignment tasks**, we incorporate the OpenAI Summarize from Feedback Dataset [5] as source data and the Human-Like DPO Dataset [6] as fine-tuning and target data. We use Qwen-2.5-0.5B as the weak model and Qwen-2.5-7B as the strong model. We randomly sample 200 samples to make the target set, and use GPT-4o to compute win rates. RAVEN achieves a 24% PGR improvement over the best alternative baseline Bayes, showing its generalizability to more diverse alignment tasks.\n\n||Weak|Naive|Ens|Bayes|RAVEN|Strong|\n|-|-|-|-|-|-|-|\n|WR|57.50|67.86|68.09|68.18|**70.37**|83.00|\n|PGR||40.6|41.5|41.9|**50.5**||\n> [2] Chen et al. \"Meditron-70b: Scaling medical pretraining for large language models.\" arXiv 2023.  \n> [3] Pal et al. \"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.\" Conference on health, inference, and learning. PMLR, 2022.  \n> [4] Jin, Di, et al. \"What disease does this patient have? a large-scale open domain question answering dataset from medical exams.\" Applied Sciences 2021.  \n> [5] Stiennon et al. \"Learning to summarize with human feedback.\" NeurIPS 2020.  \n> [6] Çalık et al. \"Enhancing human-like responses in large language models.\" arXiv 2025."}, {"review_id": "y4LmYEP8bG", "reviewer": "Reviewer_sL7D", "summary": "This work studies the possibility of training strong models with supervision from weak models where the latter are trained with data from a distribution that is different from the one that the former needs to be trained on. The authors propose to adaptively combine weak models via learnable weights to guide the training of a strong model for this purpose. The experimental results on a number of standard vision and language datasets show strong performance improvements over current SOTA. Experimental results on a number of standard image and text classification, as well as preference alignment tasks show strong performance improvements over current SOTA.", "questions": "Please see Weaknesses in the Strengths and Weaknesses section.", "limitations": "Please see Weaknesses in the Strengths and Weaknesses section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n1. The idea of adaptively combining weak models to guide the training of a strong model for weak-to-strong generalization under distribution shifts is a simple and intuitive one.\n\n2. The authors take care to handle practical challenges such as collapse / shortcut learning prevalent in optimization under such scenarios by warmup with easy samples, the predictions on which all weak models agree on, a technique known to be effective in the literature on learning with noisy labels. This drawing of connection between the task of weak-to-strong generalization and learning with noisy labels is an interesting one, and can help further cross-over of ideas between the two areas.\n\n3. The experimental results on a number of standard image and text classification, as well as preference alignment tasks show strong performance improvements over current SOTA.\n\n## Weaknesses:\n\n1. The feasibility of faithfully identifying the correct weak model under distribution shift is not clear. Under arbitrary distribution shifts, weak models may have arbitrarily uncertain predictions [1]. Without specific characterizations of the nature of the distribution shift wrt the weak model, general claims about such identifiability of correct weak models under distribution shift are unreliable.\n\n2. Weak models consistently providing the same predictions may not necessarily be a good indicator of reliability, especially under distribution shift. Since all the weak models in the ensemble are trained on the same source distribution, they are more likely to exhibit the same pattern of mistakes when faced with the same of kind of distribution shift.\n\n3. Remark 1 states that with no additional guidance, the strong model converges to putting the highest amount of weight to the best-performing weak model by the end of training. While this is an interesting observation, it is important to reflect further on this. Why does the strong model necessarily need to converge to the best-performing weak model, in the absence of any information about which weak model is actually good? What characteristic does the best-performing weak model have that drives the optimization of the strong model in its direction?\n\nReferences:\n\n[1] Ovadia et al., \"Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift\", NeurIPS 2019.", "was_revised": true, "final_justification": "**Reponse to initial rebuttal:**\n I thank the authors for the additional experiments and theoretical analyses towards addressing my initial concerns. Below, I provide my response.\n\nWhile the empirical results are useful in illustrating the applicability of RAVEN under different kinds of distribution shifts, it does not provide any insights into how the nature of the distribution shift should be wrt the weak model, i.e., the issue that a weak model might fail under an arbitrary distribution shift still remains. For this reason, it is important to characterize the family / nature of distribution shifts that a weak model might be able to handle, and those that it cannot. Failure modes are an undeniable component of any system, but it is important to characterize them precisely, without which, reliable applicability is hindered.\n\nMy concern is not regarding the scenario when there is sufficient variability among the weak learners - I am convinced that the complementary knowledge across the weak learners would be nothing but beneficial. My concern is regarding the scenario when weak learners do not exhibit sufficient variability. Again, in this case, it is important to understand which types of distribution shifts would lead to sufficient variability among the weak learners, and which ones are more likely to induce a common failure mode for all weak learners.\n\nI thank the authors for providing the theoretical analysis, which addresses my concern about the strong model converging towards the weak model providing the best generalization.\n\nI would therefore continue with my current rating.\n\n**Update after follow-up response from authors:**\nI thank the authors for further following up on my feedback about formalizing the nature of distribution shifts that the ensemble of weak models can handle and the characterization of the consequent failure modes. This addresses my concern and I would thus increase my score.", "rebuttal": "We thank the reviewer for the thoughtful review. We sincerely appreciate the reviewer's recognition that the problem we introduce, weak-to-strong generalization under distribution shifts, addresses a practical challenge. We're also grateful for the reviewer’s acknowledgment that RAVEN is both intuitive and effective in this context. Moreover, we value the reviewer’s insight that our work opens a new direction by connecting weak-to-strong generalization with learning under noisy labels.\n\n---\n**Weakness 1.** *The feasibility of faithfully identifying the correct weak model under distribution shift is not clear. Under arbitrary distribution shifts, weak models may have arbitrarily uncertain predictions. Without specific characterizations of the nature of the distribution shift wrt the weak model, general claims about such identifiability of correct weak models under distribution shift are unreliable.*\n\n**Response.** We thank the Reviewer for the comment. Although not explicitly mentioned, in our experiments we tested RAVEN across two different types of distribution shifts: domain shift (iWildCam, Camelyon17, and Anthropic HH-RLHF datasets) and combined domain and subpopulation shifts (fMoW and Amazon datasets). We show that in both of these types of distribution shifts, RAVEN remains effective and outperforms alternative methods. Moreover, we further conducted experiments that involve 19 different synthetic domain shifts on the ImageNet dataset (reported in Table 7 of the Appendix) and showed that RAVEN is consistently effective. \n\nIn response, we now additionally conducted experiments under a new distribution type, **spurious correlation**, and report the results below. Following the setup in [1], we used a biased dog-vs-cat image classification dataset with a color-based spurious attribute. Specifically, the source data consists exclusively of bright-colored dogs and dark-colored cats, while the fine-tuning and target data follow a uniform color distribution across both classes. We exclude Co-sup from this experiment, as it requires domain-specific labels that are not available in the spurious correlation benchmark. RAVEN achieves an 8% PGR improvement compared to the best alternative baseline (Boots).  This shows that, while the effectiveness of weak supervision may vary depending on the type and severity of the distribution shift, RAVEN can still identify relatively better weak models that can serve as annotators for the fine-tuning data. In response to the Reviewer’s feedback, we will characterize the type of distribution shift for all datasets in our experiments and include new results.\n\n||Weak|Naive|Conf|Boots|V-sup|Ens|Bayes|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|\n|Acc|78.7|83.7|84.5|86.7|86.0|85.7|86.7|**87.4**|99.2|\n|PGR||24.7|28.1|39.3|35.9|34.3|39.1|**42.5**||\n\n\n> [1] Kim et al. \"Learning not to learn: Training deep neural networks with biased data.\" CVPR 2019.\n\n---\n**Weakness 2.** *Weak models consistently providing the same predictions may not necessarily be a good indicator of reliability, especially under distribution shift. Since all the weak models in the ensemble are trained on the same source distribution, they are more likely to exhibit the same pattern of mistakes when faced with the same of kind of distribution shift.*\n\n**Response.** We thank the Reviewer for the comment. Currently, the diversity of weak learners comes from using different random seeds. This choice is motivated by our finding that weak models exhibit significantly higher variance in the OOD setting compared to the InD setting, 5.43% vs. 0.9% in average accuracy variance across iWildCam, Camelyon17, and fMoW, even when trained on the same source distribution (Figure 3 in the main script). This observation aligns with findings reported in [2]. \n\nTo further strengthen our results and evaluate RAVEN in the scenario where weak models in the ensemble are not trained on the same distribution, we conducted experiments  by training the weak models on **different data sources** (following the setup in [3], which constructs distinct data sources based on sub-domain). Specifically, we trained seven AlexNet models using different “camera location” domain labels for iWildCam, five AlexNet models using different “hospital” domain labels for Camelyon17, and seven AlexNet models using different “time” domain labels for fMoW. Table 1 shows that RAVEN achieves a 24% PGR improvement compared to the best alternative baseline (Bayes [4]). \n\nTable 1: Different data sources\n||Alex|Naive|Conf|Boots|V-sup|Bayes|Ens|Co-sup|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|\n|iWildCam|47.2|48.4|50.3|49.5|48.6|56.0|55.2|47.4|56.0|94.6|\n|PGR||2.5|6.5|4.8|3.0|18.5|16.8|0.4|18.5||\n|Camelyon17|67.0|69.0|69.6|71.1|69.1|70.9|69.0|70.5|71.5|97.8|\n|PGR||6.3|8.2|13.2|6.7|12.5|6.4|11.2|14.5||\n|fMoW|21.3|21.6|20.9|21.4|21.6|23.5|24.0|24.4|26.0|61.5|\n|PGR||0.7|-1.2|0.1|0.7|5.4|6.6|7.6|11.6||\n|avg.Acc|45.2|46.3|46.9|47.3|46.5|50.1|49.4|47.4|**51.2**|84.6|\n|avg.PGR||3.2|4.5|6.0|3.5|12.1|9.9|6.4|**14.9**||\n\nIn addition, we further evaluated RAVEN in scenario where the variability of weak learners comes from using **different architectures**. We set AlexNet, ResNet18, and SqueezeNet, as weak models to supervise the strong model DINO ViT-B/8. Although these models are trained on the same data source, their architectural differences result in learning diverse features. Table 2 shows that RAVEN achieves a 81% PGR improvement compared to the best alternative baseline (Bayes [4]). Results in Table 1 and Table 2 demonstrate that **RAVEN enables effective weak-to-strong generalization both when weak models have relatively low diversity and when they exhibit substantial diversity.** We will include these results in the final version of our manuscript.\n\nTable 2: Different architectures\n||W-Avg|Naive|Conf|Boots|V-sup|Bayes|Ens|Co-sup|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|\n|iWildCam|47.9|48.9|45.2|49.3|47.0|50.2|49.6|49.8|49.9|94.8|\n|PGR||2.1|-5.8|2.9|-2.0|4.8|3.6|4.6|4.2||\n|Camelyon17|59.9|60.5|61.0|62.6|60.7|63.4|62.0|63.8|63.9|97.8|\n|PGR||2.9|2.9|7.0|2.0|9.2|5.5|-2.7|10.5||\n|fMoW|18.3|19.0|17.3|18.1|17.9|19.1|18.6|13.8|24.3|60.7|\n|PGR||1.5|-2.2|-0.5|-0.8|2.0|0.8|-2.9|14.2||\n|avg.Acc|42.0|42.8|41.2|44.2|41.9|44.6|43.4|42.5|**46.0**|84.4|\n|avg.PGR||1.8|-1.7|3.1|-0.3|5.3|3.3|-0.3|**9.6**||\n\n\n> [2] Lakshminarayanan et al. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" NeurIPS 2017.  \n> [3] Liu et al. \"Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture of experts.\" arXiv 2024.  \n> [4] Cui et al. \"Bayesian weaks-to-strong from text classification to generation.\" ICLR 2025.  \n\n---\n**Weakness 3.** *Remark 1 states that with no additional guidance, the strong model converges to putting the highest amount of weight to the best-performing weak model by the end of training. While this is an interesting observation, it is important to reflect further on this. Why does the strong model necessarily need to converge to the best-performing weak model, in the absence of any information about which weak model is actually good? What characteristic does the best-performing weak model have that drives the optimization of the strong model in its direction?*\n\n**Response.** To understand why RAVEN often identifies the most suitable weak model for the target data distribution, we begin with the following assumption.\n\n*Assumption 1.* The ensemble of weak models tends to make predictions similar to those of the best-performing weak model:\n\n$\\arg\\max_{k \\in \\{1, \\dots, C\\}} f_{\\sum}(x)[k] \\approx \\arg\\max_{k} f^*(x)[k],\\quad$ (1)\n\nwhere  $f_{\\sum}(x) := \\sum_{i=1}^M f_{w_i}^{\\text{src}}(x)$,  and $f^* (x)$ denotes the best-performing weak model. While the ensemble prediction is often close to that of the best model, we assume that $ f^* $ is more accurate and confident.\n\nLet $f_s$ denote the strong model, and define the adaptation loss as:\n\n$L_{\\text{adaptation}}(\\Theta_S, \\Theta_W) := \nL_{\\text{CE}} \\left( \nf_s(x), \\sum_{i=1}^{M} \\theta^{(i)} f_{w_i}^{\\text{src}}(x)\n\\right) \\quad \\text{subject to } \\sum_{i=1}^{M} \\theta^{(i)} = 1, \\ \\theta^{(i)} \\in \\Theta_W.\\quad$ (2)\n\nThis loss is optimized by alternating the following two steps:\n\n**Step 1.** *Optimize* $ \\Theta_S $ *with* $ \\Theta_W $ *fixed.*\n\nWhen weights are uniform (i.e., $ \\theta^{(i)} = \\frac{1}{M} $), the strong model learns to mimic the average behavior of weak models:\n\n$f_s(x) \\approx \\frac{1}{M} \\sum_{i=1}^M f_{w_i}^{\\text{src}}(x).\\quad$ (3)\n\n**Step 2.** *Optimize* $\\Theta_W$ *with* $\\Theta_S$ *fixed.*\n\nWe rewrite the objective as:\n\n$L_{\\text{CE}}(\\Theta_W)=\\sum_{i=1}^{M} \\theta^{(i)} C_i $,  where  $C_i := -𝔼_{x \\sim D}$ $\\left[ \\sum_{k=1}^C f_{w_i}^{\\text{src}}(x)[k] \\log f_s(x)[k] \\right].\\quad$ (4)\n\n*Assumption 2.* Each weak model is highly confident in predicting a single class:\n\n$f_{w_i}^{\\text{src}}(x)=(\\epsilon, \\dots, 1-\\overline{\\epsilon},\\dots,\\epsilon), \\quad \\epsilon \\ll 1,\\quad$ (5)\n\nwith the largest value $1-\\overline{\\epsilon}$ at index $ \\hat{k}ᵢ:=\\arg\\max_k f_{w_i}^{\\text{src}}(x)[k] $. \n\nThen,\n\n$\\sum_{k=1}^C f_{w_i}^{\\text{src}}(x)[k] \\log f_s(x)[k]\\approx\\log f_s(x)[\\hat{k}_i]\\quad$ (6)\n\nand the objective simplifies to:\n\n$C_i \\approx - 𝔼_{x \\sim D} \\left[ \\log f_s(x)[\\hat{k}_i] \\right].\\quad$ (7)\n\nBy *Ass.* 1 and Eq. (3), we infer that the best weak model $f^*(x)$ most closely matches the $f_s(x)$ predictions and therefore receives the highest weight—consistent with empirical findings. Notably, this conclusion still holds even when only the best weak model satisfies *Ass.* 2, since the best weak model yields the smallest $C_i$. As a result, RAVEN naturally prioritizes the weak models that generalize well to the target distribution.\n\nThrough this iterative optimization, the strong model $f_s$ becomes increasingly aligned with the best weak model and further improves by leveraging this implicit supervision. We will include this in the final version of the manuscript."}, {"review_id": "xWSloZ5nIV", "reviewer": "Reviewer_gvMx", "summary": "This paper introduces RAVEN, a robust weak-to-strong (W2S) generalization framework, to address the performance degradation that occurs when a distribution shift exists between the training data of weak models and the fine-tuning data of a strong model. RAVEN employs an ensemble of weak models and dynamically learns an adaptive weighting scheme to combine their outputs for supervision. The effectiveness of RAVEN is demonstrated through extensive experiments across image classification, text classification, and preference alignment tasks.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n    1. This work introduces and formalizes a critical problem in W2S generalization.\n    2. RAVEN’s proposed adaptive weighting ensemble is an intuitive and effective solution.\n    3. The experiments and analysis are also comprehensive.\n\n\nWeaknesses:\n\n    1. This paper introduces a novel scenario, named weak-to-strong generalization under distribution shifts. However,  the choice of weak and the strong models is not very representative. For instance, the paper utilizes AlexNet and DINO for image classification, why not evaluate more advanced models like vision-language models?\n\n    2.  The proposed Robust AdaptiVe wEightiNg (RAVEN) approach incorporates adaptive weighting and easy-sample guided initialization. However, both of these techniques are inspired by prior works, so the methodological contribution cannot be considered highly novel. Moreover, the paper does not further explore some aspects, such as whether linear or nonlinear combinations would be more effective in adaptive weighting.\n\n    3. The experimental designs for text classification and preference alignment are not fully convincing, as the performance gap between the weak and strong models is marginal. In the text classification experiments, the weak model underperforms the strong model by only about 4% in accuracy. While in preference alignment, the difference shrinks to less than 2%. A more rigorous experimental setup should employ model pairs with clearly distinguishable capabilities to properly validate the proposed weak-to-strong generalization framework.", "was_revised": true, "final_justification": "After checking authors’ responses and other reviewers’ comments, I appreciate that the authors addressed to my questions. Thus, I have decide to raise my rating.", "rebuttal": "We thank the reviewer for the suggestions. We are grateful for the reviewer’s recognition that weak-to-strong generalization under distribution shifts, our proposed problem setting, addresses a critical challenge. It is encouraging to know that the reviewer found RAVEN to be an intuitive and effective approach. We also value the reviewer’s positive assessment of the comprehensiveness of our experiments and analysis. \n\n---\n**Weakness 1.** *This paper introduces a novel scenario, named weak-to-strong generalization under distribution shifts. However,  the choice of weak and the strong models is not very representative. For instance, the paper utilizes AlexNet and DINO for image classification, why not evaluate more advanced models like vision-language models?*\n\n**Response.** We thank the reviewer for the comment. We would first like to emphasize that in addition to AlexNet as a weak model and DINO ViT-B/8 as a strong model, we conducted comprehensive analysis with different (weak, strong) model pairs in the Appendix. In particular, the average PGR across all datasets—including iWildCam, Camelyon17, fMoW, and ImageNet for image classification, and Amazon for text classification—is reported in Table 20 in Appendix G.8. In particular, for image classification we evaluated RAVEN using ResNet18 and SqueezeNet as weak models and DINOv2 ViT-L/14 Distilled and DINO ViT-S/8 as strong models by evaluating all possible combinations. For text classification, we used GPT2, Qwen-2.5-0.5B, and Llama-3.2-1B as the weak models, and Qwen-2.5-7B and Llama-3.1-8B as the strong models\n\nIn addition, following the suggestion to include a vision-language model for image classification task, we below show results using a recent vision-language mode (VLM), Qwen2.5-VL [1]. Specifically, we set AlexNet as the weak model and Qwen2.5-VL-3B-Instruct as the strong model. We focused on the iWildCam and Camelyon17 datasets, as Qwen2.5-VL-3B-Instruct performs poorly on the satellite image domain (fMoW), in some cases underperforming even AlexNet (A similar issue was observed with Qwen2.5-VL-7B-Instruct). The results below show that RAVEN remains effective even when supervising a strong VLM. Notably, RAVEN outperforms Co-sup while using only three weak models, whereas Co-sup relies on seven. In response to the Reviewer’s feedback, we will include these results in the final version of the manuscript.\n\n|Dataset|Metric|Weak|Naive|Conf|Boots|V-sup|Bayes(3)|Ens(3)|Co-sup(7)|RAVEN(3)|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|iWildCam|Acc|44.7|46.2|47.4|46.1|47.7|47.1|47.6|47.6|**48.7**|86.7|\n|iWildCam|PGR|-|3.5|6.7|3.3|7.2|5.6|6.8|1.2|**9.5**|-|\n|Camelyon17|Acc|65.2|64.5|64.5|64.9|64.5|65.5|64.5|**65.8**|65.7|85.7|\n|Camelyon17|PGR|-|-3.4|-3.7|-1.7|-3.8|1.5|-3.4|-6.7|**2.4**|-|\n\n\n> [1] Bai et al. \"Qwen2. 5-vl technical report.\" arXiv 2025.\n\n---\n**Weakness 2.** *The proposed Robust AdaptiVe wEightiNg (RAVEN) approach incorporates adaptive weighting and easy-sample guided initialization. However, both of these techniques are inspired by prior works, so the methodological contribution cannot be considered highly novel. Moreover, the paper does not further explore some aspects, such as whether linear or nonlinear combinations would be more effective in adaptive weighting.*\n\n**Response.** Thank you for this comment. We would like to point out that (1) our work is the first to introduce adaptive weighting in weak-to-strong generalization framework where different weights are assigned to weak learners, (2) it is the first work to apply easy-sample guided initialization to weak-to-strong generalization problem, and (3) it is the first work to integrate adaptive weighting with easy-sample guided initialization. Moreover, our work introduces a novel setting weak-to-strong-generlization under distribution shifts motivated by a new finding that weak-to-strong generalization can become infeasible under distribution shifts. Motivated by this finding, our work is the first work to design a robust weak-to-strong pipeline by exploring how strong representations can be leveraged to identify a favorable linear combination of weak models for fine-tuning data annotation. Finally, to extend our framework to the preference alignment setting, we propose **DPO-R**, which addresses the limitation of conventional DPO loss. Specifically, standard DPO cannot support adaptive weighting, as the influence of weak signals is not properly propagated due to its pairwise log-likelihood comparison mechanism (Appendix D.2).\n\nIn response to the suggestion regarding nonlinear combinations, we implemented an MLP to determine the weights of the weak models (referred to as Non-linear RAVEN). RAVEN is effective with both a linear combination using $\\Theta_W$ and a non-linear MLP followed by softmax and achieves similar performance in both cases. Nevertheless, the linear combination we adopt is more efficient in terms of parameter cost: the non-linear version requires embedding dimension $times$ number of weak models (e.g., 768 × 3 for DINO ViT-B/8 with three weak models), while RAVEN requires only the number of weak models (e.g., 3). We will include these results in the final version of our manuscript.\n\n|Dataset|Weak|Non-linear RAVEN|Linear RAVEN|Strong|\n|-|-|-|-|-|\n|iWildCam|44.8|54.9|54.6|94.8|\n|Camelyon17|65.2|69.9|69.5|97.9|\n|fMoW|28.0|33.8|34.2|60.9|\n\n\n---\n**Weakness 3.** *The experimental designs for text classification and preference alignment are not fully convincing, as the performance gap between the weak and strong models is marginal. In the text classification experiments, the weak model underperforms the strong model by only about 4% in accuracy. While in preference alignment, the difference shrinks to less than 2%. A more rigorous experimental setup should employ model pairs with clearly distinguishable capabilities to properly validate the proposed weak-to-strong generalization framework.*\n\n**Response.** The difference in performance between weak and strong models is mainly driven by the choice of the models: by adopting stronger models the difference in performance between weak and strong models increases.  To show this, for text classification result, we included experiments with a larger strong model **Qwen-2.5-14B** on the Amazon-Wilds dataset and using Qwen-2.5-0.5B as the weak model (results below). The results show that 1) the gap between a weak and strong model significantly increases, in particular from 4% using Qwen-2.5-7B to **close to 10%** using Qwen-2.5-14B as a strong model, and 2) RAVEN remains the most effective approach in this setting.\n\nTable1: Evaluation from Qwen-2.5-0.5B to Qwen-2.5-14B.\n||Weak|Naive|Conf|Boots|ACD|Ens|Bayes|CSL|RAVEN|Strong|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Acc|66.34|70.97|70.82|70.00|70.20|70.52|59.07|70.98|**71.34**|75.66|\n|PGR|-|49.7|48.0|39.2|41.3|44.8|-78.0|49.8|**53.6**|-|\n\n\nIn addition, we systematically evaluate different combinations of weak and strong model pairs using Qwen-2.5-0.5B and GPT2 as weak models, and Qwen-2.5-7B and Qwen-2.5-14B as strong models. The results consistently show that adopting stronger models leads to larger performance gaps between strong and weak model as expected, and that RAVEN improves performance with the adoption of stronger models. \n\nTable 2: Scaling analysis with Qwen-2.5-0.5B and GPT2 as the weak model. The value shown next to each model indicates its standalone accuracy, while the other values represent RAVEN's weak-to-strong accuracy when transferring from the row model (weak) to the column model (strong).\n\n||Qwen-2.5-7B **(70.78)**|Qwen-2.5-14B **(75.66)**|\n|-|-|-|\n|**Qwen-0.5B (66.34)**|70.15|71.34|\n|**GPT2 (65.74)**|67.5|70.85|\n\nFor preference alignment, we now incorporate the OpenAI Summarize from Feedback Dataset [2] as source data and the Human-Like DPO Dataset [3] as fine-tuning and target data. We use Qwen-2.5-0.5B as the weak model and Qwen-2.5-7B as the strong model. We randomly sample 200 samples to make the target set, and use GPT-4o to compute win rates. The results show that (1) on this dataset, the difference between weak and strong models is **over 25%**, and (2) RAVEN is the best performing model and achieves a 24% PGR improvement over the best alternative baseline Bayes, showing its generalizability to more diverse alignment tasks.\n\n||Weak|Naive|Ens|Bayes|RAVEN|Strong|\n|-|-|-|-|-|-|-|\n|WR|57.50|67.86|68.09|68.18|**70.37**|83.00|\n|PGR|-|40.6|41.5|41.9|**50.5**|-|\n\n> [2] Stiennon et al. \"Learning to summarize with human feedback.\" NeurIPS 2020.  \n> [3] Çalık et al. \"Enhancing human-like responses in large language models.\" arXiv 2025."}, {"review_id": "BNA0Kw229a", "reviewer": "Reviewer_6DE5", "summary": "The paper introduces RAVEN, a weak-to-strong generalization framework that adaptively weights weak models to reliably supervise a strong model under distribution shifts, achieving significant performance gains across multiple tasks.", "questions": "see above", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. The paper proposes RAVEN, a framework that dynamically learns to combine weak models for supervising a stronger model.\n\n2. RAVEN introduces adaptive weighting and easy-sample guided initialization.\n\n3. Extensive experiments on image classification, text classification, and preference alignment show that RAVEN outperforms baselines.\n\nCons:\n\n1. While the core ideas are sound, the adaptation mechanism (especially how adaptive weights are computed and updated) is difficult to follow. I suggest adding a clear flowchart or algorithmic table that outlines the entire pipeline step-by-step. This would make the framework more accessible to readers unfamiliar with meta-learning or ensemble supervision.\n\n2. The ablation study section could be further strengthened. One useful baseline would be a variant where the weak teachers are used for static weight computation, without adaptive updates. This could isolate the contribution of adaptive weighting more effectively.\n\n3. Table 5 is mentioned in the results but lacks table 5 contents in the paper. This disrupts the flow of the paper and leaves the reader uncertain about the insights it is intended to convey.", "was_revised": true, "final_justification": "The rebuttal solve my concerns. I will raise my scores.", "rebuttal": "We thank the reviewer for the comments. We appreciate the reviewer’s recognition that RAVEN’s adaptive weighting and easy-sample guided initialization are well-founded and the core ideas are sound, the extensiveness of our experiments and significant performance gains achieved by RAVEN across multiple tasks.\n\n---\n**Cons 1.** *While the core ideas are sound, the adaptation mechanism (especially how adaptive weights are computed and updated) is difficult to follow. I suggest adding a clear flowchart or algorithmic table that outlines the entire pipeline step-by-step. This would make the framework more accessible to readers unfamiliar with meta-learning or ensemble supervision.*\n\n**Response.** We thank the reviewer for the suggestion. Since we can not add a figure in the response, we added an algorithmic table below and we will include the figure in the final version of our manuscript.\n\n---\n## Algorithm: Robust Adaptive Weighting (RAVEN)\n\n**Require**: Fine-tuning dataset $D$, weak models $\\\\{ f_{w_i}^{\\text{src}} \\\\}_{i=1}^M$, pretrained strong backbone  \n**Ensure**: Trained linear classifier $\\Theta_S$\n\n---\n\n**// Easy-sample guided initialization**\n\n1. Identify easy samples:  \n   $D_{\\text{easy}} = \\\\{ x \\in D \\mid \\arg\\max_k f_{w_i}^{\\text{src}}(x)[k] = \\arg\\max_k f_{w_j}^{\\text{src}}(x)[k], \\text{ for all } i, j \\\\}$\n\n2. Initialize weights uniformly:  \n   $\\theta^{(i)} = \\frac{1}{M}$\n\n3. Train $\\Theta_S$ on $D_{\\text{easy}}$ by minimizing:  \n   $L_{\\text{CE}}(f_S(x), f_{\\Theta_W}(x))$  where $f_{\\Theta_W}(x) = \\sum_{i=1}^M \\theta^{(i)} f_{w_i}^{\\text{src}}(x) \\quad$   (Eq. 3)\n\n---\n\n**// Alternating optimization on the full dataset**\n\n**while not converged do**\n\n4. Update weights:  \n   $\\Theta_W \\leftarrow \\arg\\min_{\\Theta_W} L_{\\text{CE}}(f_S(x), f_{\\Theta_W}(x)), x \\in D \\quad$  (Eq. 4)\n\n5. Update strong model:  \n   $\\Theta_S \\leftarrow \\arg\\min_{\\Theta_S} L_{\\text{CE}}(f_S(x), f_{\\Theta_W}(x)), x \\in D \\quad$  (Eq. 4)\n\n**end while**\n\n**return** $\\Theta_S$\n\n---\n\n\n\n**Cons 2.** *The ablation study section could be further strengthened. One useful baseline would be a variant where the weak teachers are used for static weight computation, without adaptive updates. This could isolate the contribution of adaptive weighting more effectively.*\n\n**Response.** We thank the Reviewer for the suggestion, however, we already incorporated a variant where the weak teachers are used for static weight computation as a baseline that we refer to as Easy-All. The results of comparing static and adaptive weighting are presented in Table 17 in the Appendix G.5. Easy-All and RAVEN share the exact same setup, differing only in the use of static versus adaptive weighting. RAVEN consistently outperforms Easy-All across all datasets. \n\n---\n**Cons 3.** *Table 5 is mentioned in the results but lacks table 5 contents in the paper. This disrupts the flow of the paper and leaves the reader uncertain about the insights it is intended to convey.*\n\n**Response.** Thank you for pointing out the missing table. We include the missing table below and we will include it in the final version of the manuscript.\n\nTable 5: Sample and weight scheduling. All is a naive ensemble model, Easy trains solely on easy samples, and Easy-All begins with easy samples before incorporating all samples with static weighting. Easy-All+AW adds adaptive weighting from the start without static weighting.\n||All|Easy|Easy-All|Easy-All+AW|RAVEN|\n|-|-|-|-|-|-|\n|**Acc**|49.76|50.10|50.51|50.86|**51.97**|\n|**PGR**|10.39|11.13|10.67|12.85|**16.09**|"}]

Author rebuttal:

Meta-review: This paper addresses the important and practical challenge of weak-to-strong generalization under distribution shifts, a scenario where naive supervision from weaker models fails. The authors propose RAVEN, a methodologically sound framework that dynamically learns to combine an ensemble of weak supervisors through adaptive weighting and an easy-sample guided initialization strategy. Reviewers were uniformly positive about the work, highlighting the novelty and significance of the problem setting, the intuitive and effective nature of the proposed solution, and the comprehensive empirical validation demonstrating strong performance gains across image classification, text classification, and preference alignment tasks. While initial reviews raised some limitations regarding the diversity of the weak models, the scope of the experiments, and the need for further theoretical justification, the authors provided a thorough rebuttal with extensive additional experiments and analysis that successfully addressed these concerns, leading multiple reviewers to raise their scores. The paper's strengths in identifying a critical problem and providing a robust, well-supported solution substantially outweigh any minor initial weaknesses, making it a solid contribution to the field that merits acceptance.
Decision: Accept (poster)