Weak to Strong Generalization for Large Language Models with Multi-capabilities
Authors: Yucheng Zhou
Jianbing Shen
Yu Cheng
Venue: ICLR 2025 Poster (2025)
Keywords: Weak to Strong Generalization
Large Language Model
Superalignment
Primary area: foundation or frontier models, including LLMs

Abstract: As large language models (LLMs) grow in sophistication, some of their capabilities surpass human abilities, making it essential to ensure their alignment with human values and intentions, i.e., Superalignment. This superalignment challenge is particularly critical for complex tasks, as annotations provided by humans, as weak supervisors, may be overly simplistic, incomplete, or incorrect. Previous work has demonstrated the potential of training a strong model using the weak dataset generated by a weak model as weak supervision. However, these studies have been limited to a single capability. In this work, we conduct extensive experiments to investigate weak to strong generalization for LLMs with multi-capabilities. The experiments reveal that different capabilities tend to remain relatively independent in this generalization, and the effectiveness of weak supervision is significantly impacted by the quality and diversity of the weak datasets. Moreover, the self-bootstrapping of the strong model leads to performance degradation due to its overconfidence and the limited diversity of its generated dataset. To address these issues, we proposed a novel training framework using reward models to select valuable data, thereby providing weak supervision for strong model training. In addition, we propose a two-stage training method on both weak and selected datasets to train the strong model. Experimental results demonstrate our method significantly improves the weak to strong generalization with multi-capabilities.

Published as a conference paper at ICLR 2025
WEAK TO STRONG GENERALIZATION FOR LARGE LANGUAGE MODELS WITH MULTI-CAPABILITIES
Yucheng Z Jianbing Shen'r, Yu Cheng
SKL-IOTSC, CIS, University of Macau, 'The Chinese University of Hong Kong yucheng. zhoufconnect. um www.ummuu..... edu. mo jianbingsheneum edw.. edd,mo, chengyuêcse.
ABSTRACT
As large language models (LLMs) grow in sophistication, some of their capabilities surpass human abilities, making it essential to ensure their alignment with human values and intentions, i.e., Superalignment. This superalignment challenge is particularly critical for complex tasks, as annotations provided by humans, as weak supervisors, may be overly simplistic, incomplete, or incorrect. Previous work has demonstrated the potential of training a strong model using the weak dataset generated by weak model as weak supervision. However, these studies have been limited single capability.
this work, we conduct extensive experiments investigate weak to strong generalization for LLMs with multi-capabilities. The experiments reveal that different capabilities tend to remain relatively independent in this generalization, and the effectiveness of weak supervision is significantly impacted by the quality and diversity of the weak datasets. Moreover, the self- bootstrapping of the strong model leads performance degradation due its overconfidence and the limited diversity of its generated dataset. To address these issues, we proposed novel training framework using reward models to select valuable data, thereby providing weak supervision for strong model training. In addition, we propose two-stage training method on both weak and selected datasets to train the strong model. Experimental results demonstrate our method significantly improves the weak to strong generalization with multi-capabilities.
INTRODUCTION
With large language models (LLMs) continuing grow in strength and sophistication, some of their capabilities surpass human abilities, e.g., text summarization (Pu et al.] 2023), predicting neuroscience results (Luo al. 2024), etc. Consequently, Buuns al. (2023) introduce the concept of "superalignment" which seeks to guarantee that these superhuman models' capabilities remain aligned with human intentions and values when understanding and executing tasks.
As LLMS become increasingly powerful, they can handle more complex tasks, often exhibiting reasoning abilities that exceed those of humans. This progression underscores the challenge of providing sufficient labeled data for training LLMs, especially for intricate tasks. For such complex tasks, humans may only be able offer simple, incomplete, or even erroneous annotations. Despite this, L.LMs are expected to understand human intent to solve these tasks effectively. Some works (Burns et al., 2023 (aambashizze et al. 2024) investigate the weak to strong generalization setting, which involves training strong model using data generated by weak model. In this setting, the weak model is analogous to humans, providing weak supervision to guide the training the strong model, similar to how humans teach superhuman model. However, previous work (Burns et al. 2023 Liu & Alahi 2024 Gambashidze et al. 2024) has only demonstrated weak to strong generalization in single capabilities. A strong model typically necessitates range of capabilities, and single weak model
"Corresponding Author. This work was supported by the National Natural Science Foundation of China (No. 624B2002), the Science and Technology Development Fund of Macau SAR (FDCT) under grants 0102/2023/RIA2 and 001/2024/SKT. the Jiangyin Hi-tech Industrial Development Zone under the Taihu Innovation Scheme 1BP2025-0000--KK-11OSSS and the University of Macau SRG2002-0002-OOTSSC grant.

---

Published as a conference paper at ICLR 2025
may lack the capabilities required. Thus, weak-to- -o-kknn generalization across multiple capabilities is essential, enabling strong model to acquire diverse abilities from different weak models.
In this work, conduct extensive experiments to investigate the multi nulti-capabilities weak to strong generalization. Firstly, we validate the weak to strong generalization for LLMs with multiple capabil- ities. Subsequently, we analyze the interactions between these capabilities during the generalization process, aiming to identify the factors influencing performance. Preliminary experiments reveal that the different capabilities exhibit tendency remain relatively independent, irrespective of the strength of the generalization. Moreover, the quality of the generated data significantly impacts the effectiveness weak supervision. Upon delving deeper into the weak strong generalization, we discover that strong models do not strictly adhere to weak supervision, i.c., weak models. Further analysis indicates that strong models tend to exhibit overconfidence in certain knowledge bility, which also leads to degradation in performance through its self- seff-bootstrapping The analysis demonstrates that data generated by weak models tends be more diverse for the strong model, which is more beneficial for the weak to strong generalization.
capa-
To improve weak to strong generalization with multi-capabilities, we propose a novel training framework that employs reward models to select valuable data as weak supervision for strong model training. Firstly, generating weak datasets by weak models serves as weak supervision. Subsequently, the strong model trained weak datasets relabels the weak datasets to obtain strong datasets. Furthermore, according to previous findings of inconsistency between weak models and strong models, we divide the weak datasets and strong datasets into three parts based on consistency: consistent datasets, inconsistent weak datasets, and inconsistent strong datasets. In previous findings, weak datasets exhibit greater diversity relative the strong model and are better beneficial for weak to strong generalization. Consequently, we train a reward model to identify correct datasets that differ in distribution from the strong datasets. In addition, we then propose a two-stage training method, enabling the strong model to learn from both the weak dataset and the selected datasets, thereby achieving superior nulti-capabilities weak to strong generalization.
In this study, our contributions are as follows:
We conducted extensive experiments to investigate weak to strong generalization with multi- capabilities. The analysis results provide insights, including the relative independence of different capabilities and weak datasets more diverse compared with strong datasets.
We propose a novel training framework for strong model training, consisting of a reward model for selecting valuable weak supervision data, and two-stage training for the strong model on weak datasets and selected datasets.
Our method significantly enhances weak to strong generalization for LLMs with malti-capabilities. Further analysis demonstrates that the reward model can select more accurate data that is also more divergent from the data distributions generated by strong models.
2 BACKGROUND AND NOTATION
Weak to strong generalization in large language models refers to a weak model guiding a strong model training, an analogy setting of human- -ugided superhuman model training (Burns et al., 2023). Specifically, the weak model generates a weak dataset to provide weak supervision for the strong model's training. However, the data generated by the weak model may contain errors and be incomplete. Consequently, the strong model needs to infer the correct task intentions from the imperfect guidance provided by the weak model. Following Burns al. (2023), the weak to strong generalization involves two different training sets with a single capability i, namely D. and DÍ. In this setting, the dataset De is annotated by humans, while D. is an unannotated dataset.
Firstly, the weak model M (ww) is trained on the weak model M(ww) This ensures that the model acquires a certain capability i, similar to human understanding, This training process can be expressed as follows:
dataset D resulting in the trained
Σ
min Ou
(a.))D?
where L denotes the loss function, mm. represents the weak model parameters, and (ax y) are the labeled samples from the dataset Dg.
2

---

Published as a conference paper at ICLR 2025
Subsequently, the trained weak model is used to label the unannotated dataset DJ resulting in the dataset DV. Since the weak model's capability i may be limited, its annotations on D' could be incomplete and noisy. The annotation process by the weak model can be described as follows:
DJ = (xx, ()) x DJ D Jom., (x))
where i is the parameters of trained weak model, and label is predicted by trained weak model.
In weak to strong generalization, the strong model M. learns the capability by training on the dataset DJ annotated by the weak model. The training process for the strong model as follows:
min
Σ
fo., (x))
(2,))
where 0g., represents the parameters of the strong model, and L denotes the loss function for training the strong model. The objective is to minimize L by adjusting the parameters 0,,s so that the strong model can effectively learn from the annotations generated by the weak model.
Related Work. Recent advancements have been marked by the development of large language models (LLMs), e.g., GPT-4 (OpenAl 2023), Gemini (Anil et al., 2023), LLaMA (Touvron et al.) 2023), Qwen Bai et al. 2023). Due to some capabilities of LLMs surpassing those of humans (Pu et al. 2023 Luo et al. 2004). significant focus has been on weak to strong generalization. Studies like (Togg et al. 2024] Li et al. 2024a Sun et al. 2024) have shown how this approach optimizes capabilities through methods like self-reinforeemmen and data filtering. addition, training data selection methods, both efficiency-base (Xie et al.) 2023 Zhou et al. 2023) and quality-based (Wettig et al., 2024] Yu et al. 2023)). have been developed to enhance LLM performance by focusing on computational efficiency and data quality. The full version can be found in Appendix
3 ANALYSIS ON MULTI-CAPABILITTE. WEAK TO STRONG GENERALIZATION
3.1 MULTI-CAPABILITTE FOR LLM
Numerous studies are currently investigating the various capabilities of LLMs (Chen et al. 2024
Zhao et al. 2024). including mathematics (Li al. 2024b), temporal reasoning (Zhou et al. 2019). planning (Logeswaran et al. 2022). and more. Unlike previous research that primarily focused on a single ability (Burns et al. 2023), our study aims explore whether weak to strong generalization can occur across multiple capabilities in LLMs. To explore the models' performance in weak to strong generalization settings, we employ eight datasets that span various skills, such as GSMSK (Cobbe
et al. 202T) for mathematical abilities, MC-TACO (Zhou et al., 2019) for temporal reasoning, SCAN (Lake & Baroni (2088) for planning ability, CREAK (Onoe al., [202] for fact-checking and commonsense reasoning ability. ECQA (Aggarwal et al. (021) for explainable commonsense
reasoning, e-SNLI (Camburu et al. (018)) for logical reasoning ability, OpenBookQA (Mihaylov
et al. 2018) for fact reasoning, and SciQ (Welbl et al. 2017) for science-related abilities.
3.2 CAN TRAINING MODEL WITH
FROM WEAK SUPERVISION?
Weak to Strong Performance
To investigate whether LLMs can learn multiple capabil-
0.75
ities from multiple weak supervision, we train series of LLMs with different parameter scales (i.e., 0.5B, 1.8B,
Performane
0.70
0.65
4B. 7B) on datasets annotated for these capabilities. The average performance of these LLMs in multiple capa- bilities is shown in Figure (Left). From the figure, we can observe that larger LLMs, i.e., with more parame- ters, exhibit better average performance, confirming a positive correlation between model size and capability in these LLMs. Based on the premise that these models, ordered from smallest largest, exhibit correspond- ing increase in capability, we conduct weak to strong generalization experiments these models. Following
0.6
0.60
Weak
0.55
WTS
0.50
70
0.50
1.00
40
Model
Figure 1: Left: Performance of different- scale models on multi-capabilities tasks; Right: Performance of strong model with single- capability and multi-capabilities weak strong generalization.
3

---

Published as a conference paper at ICLR 2025
Burns et al. (2023). ) we employ weaker models to generate datasets that served weak supervision for stronger model training, i.c., 0,5B, 1.8B, and 4B models provide weak supervision for 188B, 4B, and 7B models, respectively. The results, shown in Figure [ (Right), reveal that the average performance of the weaker models consistently underperforms compared to the "WTSSS" (averaging performance across models with single-capability weak to strong generalization and "WTS" (averaging performance of model with multi-capabilities weak to strong generalization). The performance between single-capabilty and multi-capabilities generalization is comparable. In addition, we can observe that the gains from weak to strong generalization are more pronounced when the model size smaller due its weaker capability. Further details, i.e., the performance of each capability for Figure] 1 can be found in Appendix D
different datasets)
3.3 How DOES WEAK MODEL GENERATED DATA IMPACTS WEAK TO STRONG GENERALIZATION WITH MULTI- MULTI-CAPABILII CAPABILITIEE??
To further explore the factors impacting multi multi-capabilities weak to strong generalization, we train strong models datasets generated by weak models of varying model sizes. As shown in Figure 2 the quality of data generated by weak models significantly impacts the performance of weak to strong generalization. Higher-quality data generated by weak models with greater model sizes improve the performance of stronger models, highlighting the importance of improving the data quality produced by these weak models. Beyond the direct impact of the generated data, we also analyze the effect of dataset compositions with different capabilities on strong models. In our experiment, randomly weaken certain capabilities in weak supervision. Specifically, we replace the training data for given capabilities with weak data generated by smaller-size weak model corresponding to that capability. As shown in Fi we observe that the performance of other capabilities in the strong model is not significantly affected. It demonstrates that weak data quality is relatively independent across different capabilities. To further validate this observation, we randomly remove some capabilities to investigate any potential synergistic or inhibitory interactions among capabilities. The results, presented in Figure f confirm that removing certain capabilities does not compromise strong model performance in others, supporting the independence of data quality across various capabilities. More results of and Figure 4 can be found in Appendix E
Perfrmance
0.70
0.65
0.60
Wrak Model
Strong
0.55
Strong Model (4B)
Strong Model
0.58
1.88 Wea Model
4B
Figure 2: Performance of strong model trained on different weak supervision.
strong models on each capability for Figure 2 Figure
3
0.8
0.5
0.6
0.2
0.2
Weecen Capebiity Rest Capebility
Weecen Capebiity Rest Eapability
1.56
(a) GSM8K
(b) BCQA e-SNLI
(a) w/o GSMSk
(b) w/o ECQA & c-SNLI
Figure 3: Results under weaken some capabilities.
Figure 4: Results under remove some capabilities.
3.4 HAVE CONSISTENCY BETWERN STRONG MODEL AND WEAK SUPERVISION?
To deepen our understanding of the performance improvements exhibited by strong models in weak to strong generalization, we analyze the consistency of prediction results between strong and weak models. Firstly, weak models generate datasets for training strong models, which are then re-annotated these datasets by the trained strong models. As shown in Figure 5 the majority of predictions by the strong and weak models were consistent, denoted as "True Con." and "False Con.". This indicates that strong models predominantly learn from the knowledge provided by weak models. However, it observed that strong models correctly predict more samples compared to weak models, i.c., "WTS True" versus "Weak True". This suggests that the essence of weak to strong generalization is not
4

---

Published as a conference paper at ICLR 2025
0.8
0.6
0.5
0.4
Number
0.4
0.2
0.1
0.2
0.0
0.0
0.0
Thtt Con
WTS Tur
False Incer
Fabe Con.
True Cen-
WTS Tiue
False Ineen
Fabi Can.
True Con
WTS Thue
Falss Incon
Fake Con
Weak
Wenk
Waak
(a) 0.5B to 1.8B
(b) 1.8B to 4B
(c) 4B to 7
Figure 5: Prediction results of strong model (WTS) and weak supervision (Weak). "True Con and "False Con." indicate cases where both models give the same correct or incorrect answers, respectively. "WTS True" means the strong model is correct and the weak model is incorrect, while "Weak True" is the opposite. "Falsc Incon." indicates both models incorrect but with different answers.
0.8
0.5
0.4
0.6
Number
Number
0.3
0..4
0.2
0.2
0.0
0.0
0.0
Con
wr5-2 True
False incon
faise Con.
Con
W75-2 True
WTS Fun
False Incon,
faise Con.
Tru Con.
WTS:2 True
WTS Fue
Falss Incon.
False Con.
WTS
(a) 0.5B to 1.8B
(b) 1.8B to 4B
(c) 4B to 7B
Figure 6: Prediction results of strong model (WTS-2) 2) and its supervision from strong model (WTS).
merely the strong model replicating the weak model but rather learning the task's intent and rules from the datasets. Nevertheless, the strong models incorrectly answer "Weak True" samples correctly predicted by the weak models, highlighting a potential overconfidence issue in strong models that prevent them from learning some correct samples identified by the weak models. The details of each capability consistency between strong model and weak supervision can found Appendix
3.5
CAN IMPROVE STRONG MODEL WITH
Based on the above analysis, we can observe that strong model trained on datasets generated by weak models can achieve better performance. This raises a new question: "Can strong model continuously improve itself through self-bootstrapping."" To explore its answer, relabel the datasets using the trained strong model (WTS) and then trained a second version of the strong model (WTS-2) on the relabeled datasets. We compared the predictions of the trained strong model and the second version for consistency. As shown in Figure [6] we found that the second version correctly labeled fewer samples than the original strong model, i.e.. "WTS-2 True" "WTS True". This indicates that fails to enhance the performance of weak to strong generalization. As mentioned in Sec. 3.4 the strong model tends to be overconfident, leading data collapse into particular distribution during self-bootstrapping Similar conclusions have been observed in other studies (Alemobammad et al. 2023, Shumailov al. 2023 Seddik et 2024). More details of each capability can be found in Appendix G
3.6 HOw THE WEAK DATASETS IMPACT THE WEAK TO STRONG GENERALIZATION PERFORMANCE?
Given that self- uuffbboostrapping the strong model does not yield improved results, we focus on how datasets generated by the weak model impact the weak to strong generalization. Firstly, we train the
5

---

Published as a conference paper at ICLR 2025
strong model on the clean weak datasets that include only the correct sample in datasets gen- erated by the weak model. Subsequently, we
Weak
0.75
0.75
Strong Data
Performance
Performance
Mixed Data
introduce varying proportions of noise (i.e., in- correct samples from the same weak dataset) to the clean datasets for strong model training. As shown in Figure (Left), the strong model performs best on clean datasets, which indicates that training on accurate weak data yields better results, emphasizing the importance of filtering out erroneous samples in weak datasets. In ad- dition, as shown in Figure (Right), we use the strong model trained on the relabeled data from the strong model, termed "Strong Data", to train a second version of the strong model. We ob- serve that its performance is inferior to the strong model trained on "Weak Data", further confirming the failure of model self-bootstrapping. Moreover, to mitigate the distributional collapse caused by the overconfidence of the strong model, we combine the data labeled by both the weak model and the trained strong model for training the strong model, "Mixed Data" in the figure. Although this approach improves performance compared "Strong Data", it still does not surpass the performance achieved with "Weak Data". This underscores the significance of weak data for training strong models. The performance on each capability can be found in Appendix H
0.70
0.70
Clean
0.65
0.25
0.65
0.5
0.60
0.75
1.0
0.60
0.58
1.80
4B
0.58
1.88
4B
Weak Model
Weak Model
Figure 7: Left: Performance of strong model trained on clean samples and different proportions of noise samples of weak datasets; Right: Performance of strong model trained on weak, strong, and combination datascts.
MULTI-CAPABILTTISS WEAK TO STRONG GENERALIZATION WITH REWARD MODELS
As mentioned before, the significance of weak datasets is shown in Figure 7 which shows that clean weak datasets can provide better supervision for the strong model. A key factor in improving the performance of weak to strong generalization is the accuracy of the weak datasets. Moreover, weak datasets provide greater gain than strong datasets for the strong model. This difference arises from the data distributions of weak datasets and strong datasets. Since strong datasets align more closely with the internal knowledge the strong model, this knowledge can be more easily activated when using weak datasets to train strong model. Self- seff-bootstrapping strong model on its generated data gradually increases the model's 's confidence in its internal knowledge, leading to potential collapse and overconfidence during prediction. In contrast, weak datasets present more diverse data distribution for the strong model. Our method improves the strong model by increasing the accuracy of weak datasets and selecting more diverse weak data for the strong model.
4.1 THE GOAL OF WEAK DATA SELECTION
As shown in Sec. we first use the labeled dataset Dj to train a weak model M (1) and then leverage the trained weak model (I(")) to predict the answer for the unlabeled dataset D to obtain labeled weak dataset D. Subsequently, we train strong model M with the weak dataset Dj. The trained strong model M (s) is employed to relabel the dataset D to obtain the strong dataset D..
Let Pos (x) and Poy (I)) represent the data distributions of datasets Dj and DJ, respectively. Our goal is to identify subset DJ C D such that the accuracy of samples in D) is high and the distribution Pro (x) differs significantly from Przg (z), ic..
max D
E 5 Accuracy(=r) subject to (Pp || Pby)
is high.
(4)
where IDxL denotes the Leibler divergence between the distributions PBy(e) and Poaaa and the divergence can be defined as:
Poy Po (z)
DEL (Ppę Ppp))= =Σ Ppp (x) (z) log
(5)

---

Published as a conference paper at ICLR 2025
This measure ensures that the selected subset D not only maintains high accuracy but also introduces sufficient diversity to prevent a collapse of the strong model.
4.2 TRAINING REWARD MODELS
To obtain subset DJ for strong model training, train reward model differentiate between data distributions. Following Figure we first divide the data into three parts: consistent dataset D from weak dataset D. and strong dataset DJ, and inconsistent data, with Dł representing inconsistent data in D and Dj representing inconsistent data in DJ. We define the consistent and inconsistent datasets as follows:
D. = (x
x E Dj nx E D
D = (o Dj fx e DK D.}
x E Dt D.}
(6) In the setting of weak strong generalization, do not know which samples in DJ and D correct. Therefore, we choose Dj as positive samples because they include data where both the weak model and the strong model agree, indicating higher confidence (as shown in Figure In contrast, we select Dj as negative samples due to their low confidence. Additionally, this data includes "WTS True" samples from Figure which we know from Sec. 3.5 to be less diverse from strong model. We aim to ensure that the distribution of the training data has high accuracy and differs more from Dj to prevent collapse. The objective function for the reward model can be defined as:
max R
ED [R(x]]
E [R()]]
where R(x) denotes the reward assigned to a sample x by the reward model R. This ensures that the reward model favors data points where the weak model's predictions are consistent with the strong model's 's predictions (i.c., higher accuracy than the inconsistent part) while penalizing inconsistent
ones the strong dataset. The proof of the reward model can be found in Appendix C
4.3 REWARD MODEL FOR DATA SELECTION
To obtain better datasets from strong model training, we employ the trained reward model R to filter the weak dataset from each capability to collect selected dataset DP. The reward model distinguishes each sample æ from weak datasets based on its accuracy and its contribution to diversity. Specifically, the sample, classified as correct by the reward model, exhibits significant divergence from the strong model's data distribution in the dataset D In the weak dataset D/, each sample is evaluated by the reward model. The reward model, R(x), is a binary classifier that determines whether a sample is beneficial for training the strong model by considering both accuracy and diversity. Samples are filtered based on the reward model classification as follows:
D = {a R(x) = 1}, E D.
(8)
The selected samples have higher accuracy and introduce sufficient diversity for strong model training.
4.4 TRAIN STRONG MODEL WITH SELECTED DATA
The strong model (+) is trained in two stages to leverage the benefits of both the weak datasets D = (D).... D' } and selected weak datasets D (D) ... DD, }. where N is the number of capabilities. Firstly, the strong model is trained using the weak datasets D.. This initial training phase serves as warm-up, allowing the model to learn from a broader range of weak data, even though it may not be entirely accurate comprehensive.
M Warm-up(M (D))
After the warm-up phase, the model trained using the selected weak dataset D'. This phase focuses on the higher accuracy, and diverse samples selected by the reward model, ensuring that the strong model refines its predictions and avoids collapse.
M M Train(M,) .D")
(10)
7

---

Published as a conference paper at ICLR 2025
By employing this two-stage training approach, the strong model benefits from initial broad exposure to weak data, followed by focused training on higher accuracy and diverse samples. This method enhances the model's generalization capabilities, leveraging the advantages of weak data to improve overall performance while mitigating risks of overconfidence and collapse.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Capti-Capabilitie Task. For the multi-capabilities task, we utilize a range of datasets encompass- ing various capabilities to comprehensively evaluate the models' performance under weak to strong generalization settings. The tasks were selected to cover diverse capabilities, such as reasoning, comprehension, robot planning, math skills, etc. Details of the datasets are provided in Appendix [ (aoloiing/Brrss et al. (202)). each capability's 's dataset is split into a labeled training set, an unlabeled training set, and test set to simulate the weak to strong generalization scenario. The weak models were trained on the annotated datasets, and their prediction on the unannotated datasets served as the weak supervised dataset for training the strong models.
Experimental Details. In the experiments, we utilized a series of Qwen- models (Bai et al. 2023) with varying parameters. specifically 0.5B, 1.8B, 4B, and 7B. The reward models are initialized from the strong model, i.e., Qwen-1.5, and they maintain the same parameters. To ensure a fair comparison, we followed the experimental setup from [Burns et (2023), conducting all experiments with 2 epochs and batch size of 40. The optimizer used was Adam (Kingma & Ba 2015) with learning rate of le-5, Weight decay was set at 0.01, and a cosine learning rate decay strategy was employed. During inference, the models utilized greedy decoding strategy. The performance refers to accuracy, where a correct prediction exactly matches the ground truth answer. For weak to strong generalization, we use the performance gap recovered (PGR) metric (Burns al.) (023) measure the weak to strong generalization performance. All experiments are conducted on NVIDIA A100 80G GPUs
5.2 RESULTS AND DISCUSSION
As shown in Figure for LLMs with multi capabi-aaaaiilities.. Our method, denoted as "Ours", is compared against the multi- capabilities weak to strong generalization approach, i.e., "WTS", and the performance of strong models trained on datasets D with human annotations is used as a performance ceiling. The average performance of the models is plotted against the size of the strong models (1.8B, 4B, and 7B parameters) in Figure 8 In addition, we show the performance gap recovered of our method and baseline, i.c., "WTS", in Figure The results of each capability can be found in Appendix [
present the experimental results the weak to strong generalization
Analysis of Performance. As shown in Figure 8 the "WTS"" shows a gradual improvement in performance as the size of the strong model increases from 1.8B to 7B parameters. However, its relatively modest performance improvement indicates that weak to strong generalization benefits from strong model capac- ities but still faces limitations. Our method, "Ours", significantly outperforms the "WTS" across all strong model sizes. For the
0.85
WTS-1.86
WTS-4B
0.80
Ours-0.58
Average Performance
0.85
Qurs-48
0.75
Strong Model
0.70
same weak model, the performance gap between "Ours" and "WTS" increases as the strong model size increases, suggesting that our approach scales more effectively with stronger models. The performance of the strong model trained on clean datasets is a performance ceiling. While our method does not completely bridge the gap to the performance ceiling, it achieves a substan- tially closer performance than the "WTS" method, especially for the models with larger sizes. Our method exhibits a notable with models with
0.55
1.88
40 Strong Model
70
Figure 8: Performance of weak strong generalization. In "Ours-xB" and "WTS-xB", the "x" indicates the size of the weak model used.
advantage, indicating its effectiveness smaller sizes or when the model faces relatively harder tasks. For the 7B model, method achieves performance comparable
8

---

Published as a conference paper at ICLR 2025
to the "WTS" model. This is because the model already performs at high level across various capabilities, leaving little room for improvement. In addition, the abundance of correct samples means the reward model's data filtering impact is likely diminished.
Analysis of Performance Gap Recovered. In Figure 9 we alyze the performance gap recovered of the baseline "WTS" and our method. This figure shows how much of the performance gap recovered between various methods and the performance ceiling as the model size increases. The "WTS-00.5" the weak model of 0.5B model, recovers approximately 30% of the performance, while our method recovers around 50%. This indicates that our method is more effective at recovering performance, even with weaker models guiding the strong model. As the model size in- creases to 7B, the performance gap recovered for both methods slightly decreases. However, our method still outperforms the "WTS" approach. This reduction in recovery at larger model sizes is due to the increased difficulty in further improving already high-performing models and the diminishing impact of reward model filtering as the proportion of correct samples increases.
0.5
Performance Gap Recovered
0.4
0.3
0.2
WTS-0.58
WTS-48
0.58
0.1
Ours-1.08
Ours-4B
1.00
70
Strong Model
Figure 9: Performance gap recov- ered of weak to strong generalization.
5.3 ANALYSIS
Deep Dive into Weak Data. To validate the effectiveness of using weak datasets Do in the first stage of training, we compared the performance of models trained with first-stage weak dataset D against those trained directly on the selected weak dataset D without first-stage training. The comparison spanned models of varying sizes, including 0.5B. 1.8B, and 4B, with corresponding strong models at 1.8B, 4B, and 7B. As shown in Figure 1O models trained using weak datasets D in the first stage consistently outperform those that skip this stage and train directly on D', This demonstrates that even though weak datasets De may not be entirely accurate or comprehensive, they still provide broader range of weak data that benefits model training.
Performncce
Ours
Durs (wil first stage)
0.9B
1.58 Weak Model
4B
Performance of model with and without first stage training on weak datascts Do.
Figure 10:
Impact Reward Model. To analyze how our method achieves performance gains through the selection of weak data, we further examined the accuracy of the selected dataset. As shown in Fig- ure we present the accuracy of datasets provided by different models, including the weak model, strong model, and our reward model. The reward model consistently outperforms both the weak and strong models across different weak model sizes (0.5B, 1.8B, and 4B). Specifically, the accuracy of the reward model increases significantly as the weak model size grows. This indicates that the reward model and the weak model are correlated, as the weak model provides positive training samples for the reward model, as men- tioned Sec. 4.2 As the capability of the weak model improves, it provides more accurate positive training samples. However, the per- formance gains from the reward model diminish as the weak model's performance improves because the task becomes more challenging. In addition, in Figure| 11 we also show the weak dataset retention rate of our reward model, represented by the gray bars. The figure shows as the model size increases, the reward model not only improves accuracy but also retains a higher proportion of useful data.
Weak
WTS
Retertion Rate
Accuracy
Model
Figure 11: Accuracy of datascts from reward model, weak model, and strong model.
Deep Dive into Reward Model. In Figure II we have demonstrated that the accuracy of the data selected by the reward model is higher than that of the original weak data. Thus, we aim to observe whether the distribution of this data differs significantly from the distribution of the strong data, specifically whether the reward model can select more data labeled "Weak True". Based

---

Published as a conference paper at ICLR 2025
0.8
Weak Reward
0.6
Weak Reward
Weak
Rerward
0.5
Number
0.3
Numbbe
0.4
Number
0.4
0.2
0.1
0.2
0.0
0.0
0.0
Thut Con
WTS Tur
Weak True
False incer
True Cen-
WTS Tiue
False Ireen
Fabie Can.
True Con
WT5 Thue
Falss Incon
Fake Con
Fabe
Wenk
Wank
(a) 0.5B to 1.8B
(b) 1.8B to 4B
(c) 4B to 7
Figure 12: Performance of reward model (Reward) on (in)consistent part between weak model (Weak) and strong model.
we conduct a distribution analysis of the data selected by the reward model, with the results shown in Figure [2 Firstly, we observe that the reward model retains a higher proportion of "Weak True" data among the inconsistency between the weak model and strong model predictions, i.e., "Strong True", "Weak True", and "False Incon". It demonstrates that the reward model not only improves accuracy but also maintains data distribution distinct from the strong model's predictions. Furthermore, as the weak model with greater size, the proportion of "True Con." increases, leading to a slight improvement by the reward model, as shown in Figure g More details of the reward model can be found in Appendix
on Figure 5
Analysis Across Model Scales. To further analyze whether our approach remains effective with larger models, approaching more realistic settings, we expanded the range of weak models to include 0.5B, 1.8B, 4B, 7B and 32B, with their corresponding strong models set at 1.8B, 4B, 7B, 32B and 72B, respectively. As shown in Fig- ure 13 our method consistently outperforms the "WTS" approach across all strong model sizes. Figure 14] shows that method con- sistently maintains performance recovery rate of over 30%, even as model sizes grow, particularly for weak models of 7B and 32B and strong models of 32B and 72B. Moreover, the results demonstrate that the scale difference between weak and strong models grows (e.g., weak models from 4B to 32B with their corresponding strong models from 7B to 72B), the performance recovery rate becomes even more significant.
Ours WTS
90
Performance
80
70
60
0.58
1.88
4B
7B
32B
Figure 13: Performance of weak to strong generalization under larger size models.
CONCLUSION
0.5
Ours WTS
In this study, we have explored the concept of multi-capabilities weak to strong generalization in large language models (LLMs). Our research validates that LLMS can indeed generalize from weak su- pervision across multiple capabilities, shedding light on the relative independence of these capabilities during the generalization process. Through extensive experiments. we confirmed that the diversity and quality of data generated by weak models are crucial factors influenc- ing the effectiveness of weak to strong generalization. Specifically, weak models tend to produce more diverse datasets for the strong models, counteracting the performance degradation caused by the overconfidence of strong models. We introduced a novel training framework that incorporates reward models to select high-quality weak supervision data, facilitating superior generalization for strong models. Our proposed two-stage training method enables the strong model to learn effectively from both weak and selected datasets, addressing the inconsistencies between weak and strong datasets.
Performance
0.3
0.2
0.58
1.88
4B
7B
32B
Weak Model
Performance gap re- covery of weak to strong general- ization under larger size models.
Figure 14:
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for commonsenseqa: New dataset and models. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 3050-3065. Association for Computational Linguistics, 2021, doi: 10. 10.11863//1/20211. ACI-LLONG.238 URL http::ss//dbb..or//0111665/1///1
238
Meta. Llama 3 model card, 2024. URL UBLLhttps.//iithbb.commmmm 1laeetttalllaa/aaaaa lama
blob/ ma i CARDDLLCARDD....COEEE md
Sima Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go MAD. CoRR, abs/2307.01850, 2023. doi: 10448550ARXXV.2330.0112
URL
https: //tp:.://o....
48550/ / 2307. 01850
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittiiesrr, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/23 1805, 2023, doi: 10.48550/ARXIV2212..2 18055 htRt:ttts:://dt orw.:0...gg1.... 48550/ http://wwvvv........... 2312. 11805
Al Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CORR, abs/2309. 16609, 2023. doi: 10. 48550/ARXIV.2309 16609. URL https:://oo/ http::///ooooog///..../ wrw/10...0000 48550 / 2309. 16609
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike. llya Sutskever, and Jeff Wu. teak-to- strong generalization: Eliciting strong capabilities with weak supervision. CoRR, abs/2312.00399 2023. doi: 1044850/ARXIV.2322.033 2312.09399
URL hntpps://dot www.boo..oor...... 48 48550//x1555551x111
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Nat- ural language inference with natural language explanations. In Samy Bengio, Hanna M. Wal- lach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. neurips. wwtpppppee//////aaaaa/aaae html
2018 URL
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. Pea-bench: Evaluating multimodal large language models chain. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 1086- 104. Association for Computational Linguisties, 2024. doi: 10.
in perreppion-coogniiinn-ccc
ACL.64. URL https: /oi .oa.iioorggoo wrt./o.gggoo.oooo.ooo 18653/
v1 21/2224.22 f findi www.inniingsal.....m 64
11

---

Published as a conference paper at ICLR 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.141653 2021. URL
htttss://rrat wtt.aaiwi..gg..eeeeee...ee httpg//bb///1gggg 14168
Alexander Gambashidze, Aleksandr Dadukin, Maksim Golyadkin, Maria Razzhivina, and Ilya Makarov. Weak-to-strong 3d object detection with x-ray distillation. CoRR, abs/2404.00679, 2024.
doi: 10448550/ARXXX12244400000
00679
Jianyuan Guo, Hanting Chen, Chengcheng Wang. Kai Han, Chang Xu, and Yunhe Wang. Vision super- generalization for vision foundation models. CoRR, abs/2402.03749, 2024. doi: 10445550ARXX.222200057777 URL https://doi. /00/10..0.0.00 48550//2555/// 2402. 03749
alignment:
Jiaming Ji. Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient aligmment through weak-to-strong correction. CoRR, abs/2402.02416, 2024. doi: 1044550/ARXIV240002444 URL httpt:t//oi.....
10. 25..4850/aaae arw.v..iivvvmm 2tt.:xiiw2222.222.000 02416
Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ame Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William EI Sayed. Mistral 7b. CoRR, abw/2310.006225. 2023, doi: 10. BtLnttps:: doi. org/ 10. 48550 2310. 06825
HS550/ARXIV.2310006222
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. May 7-9, 2015, Conference Track Proceedings, 2015. URL httt?
7/arxiv. wwwp:///.rg//b/////ooooooooabs/ 1412. 6980
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequenee recurrent networks. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, ICML. 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learn- ing Research, pp. 2879-2888. PMLR, 2018. URL sttp::/ppocceeeee mlr...g.mmm..................mmm mip.prsss// press/ v80/lake 18.. html
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning, CoRR, abs/2402.00530, 2024a. doi: 10.48550/ARXXV.2400.0005 48550/arXiv 2402 00530
URL
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of lims as mathematical problem solvers. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 2961-2984 Association for Computational Linguistics, 2024b. doi: 110.0///11122AAAALLLLLOO acl loo.gn..... long 163
URL httl:tttaa//ci..rrgeee 10tp:///1111133/133000033
Yuejiang Liu and Alexandre Alahi. Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture experts. CoRR, abs/2402.15505, 2024. doi: 10.48550/ARXIV..400..
15505. http:t/ //t.pp:://oi..oo /oo. http://..........././.. 48550/ arwiv. 2402 15505
Lajanugen Logeswaran, Yao Fu, Moontae Lee, and Honglak Lee. Few-shot subgoal planning with language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 5493 5506. Association for Computational Linguisties, 2022. doi: 10.18653/V1/2022..M MLLLIM-MIIL...M 402.
URL https: //ooi.//oo.... org 1ww.11...6661111//
naacl - ma 402
12

---

Published as a conference paper at ICLR 2025
Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K. Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O. Cohen, Valentina Borghesani, Anton Pashkov, Daniele Marinazzo, Jonathan Nicholas, Alessandro Salatiello, Ilia Sucholutsky, Pasquale Minervini, Sepehr Razavi, Roberta Rocca, Elkhan Yusifov, Tereza Okalova, Nianlong Gu, Martin Ferianc, Mikail Khona, Kaustubh R. Patil, Pui-Shee Lee, Rui Mata, Nicholas E. Myers, Jennifer K. Bizley, Sebastian Musslick, Isil Poyraz Bilgin, Guiomar Niso, Justin M. Ales, Michael Gaebler, N. Apurva Ratan Murty, Leyla Loued- -oued-KKenissii Anna Behler, Chloe M. Hall, Jessica Dafflon, Sherry Dongqi Bao, and Bradley C. Love. Large language models surpass human experts in predicting neuroscience results. CoRR, abs/2403.032300 2024, doi: 4445500/ARXX.224000222222 URL httos://dol 2403 03230
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hoekenmaier, and Jun't 'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 2381-2399 Association for Computational Linguistics, 2018. doi: 10.18653/V1/D118-1222 URL.
https://doi.. org/ 10 11w11886//// vl1 c18 1260
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A. Raffel. Scaling data-constraing language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. http: 9d8 5444444455aaaaee0777777775544543
nips cC wwwpppppppppppppppptppppp.pp...p firi__fillee files/ twwes//pppeeee////2e22 3/hash/
html
Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, and Greg Durrett. CREAK: A dataset for commonsense reasoning over entity knowledge. Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
OpenAL GPT-4 techhncal report. Co absBB/22/b//30 doi: 304500000000000403333333.0000.
URL http://pp.//..oor////4/////////2
Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead. CoRR, abs/2309.09558.
2023. doi: 1048550ARXIV.2300.0555 URL
2309. 09558
Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Piere Youssef, and Merouane Debbah. How bad is training synthetic data? statistical analysis of language model collapse. arXiv preprint ar arXiv:2404.05090, 2024.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross J. Anderson. The curse of recursion: Training on generated data makes models forget. CORR, abs/2305. 17493, 2023. doi: 10.4855WARXIV.230...O 23C5. 17493
URL https:: //o::///ooi.ie 4855000000/00000/2330..
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Michael Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.). Eiigtt AAAI Conference Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver. Canada, pp. 18990 18998, AAAI Press, 2024. doi: 10. 10.1669/AAAL V3811. URL httphhtpp///oooo.o org. 10t.:/10....0./00 aaa v38117 .38.17.2916655
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hane generalization: Scalable alignment beyond human supervision. CoRR, abs/2403.09472. 2024. doi: 1048550/ARXXVV2403300477 URL
48550/arXi 2403.
09472] 24430
13

---

Published as a conference paper at ICLR 2025
Yongqi Tong, Sizhe Wang, Dawei Li, Yifan Wang. Simeng Han, Zi Lin, Chengsong Huang, Jiaxin Huang, and Jingbo Shang. Optimizing language model's reasoning abilities with weak supervision. 2024.
arXiv preprint
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR. //tp/2222..19911, 2023, doi: 10.48550/ARX1.2230.11999
URL https
4 2302 13971
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsoureing multiple choice science questions. In Leon Derezynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User- generatedd Text. NUT@ EMNLP 2017. Copenhagen, Denmark, September 7, 2017, pp. 90-106.000000000000 Association for Computational Linguistics, 2017. doi: 10.18653/V1/W17-4411 URL htt.ps://doi.orre 1ttp://....66611111
4413
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. CoRR, abs/2402.09739, 2024. doi: 00.48550/ARXIV22402. 09739, URL https: /tw.//oi..oo /1..0////........../...//. 48544/555//00 www.aa.ivvvvv... 09739
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: selecting influential data for targeted instruction uning. CoRR, abs/ 044333, 2024. doi: 10. 08850/ARXIV.2402.04333 /ttppt/ss://doi 0tt.::o.rr1111..000.0.0 48550/550//00 arwiv. 240..xivv...2.... 04333
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA. December 10 16. http///ppee///22222
u cc/paper 2023, 2023. URL htttp::/papers..t.t.// nww..niiss............. 669аа8 (aaaa8f118baa www4e41e2e2222 0003:3//600444440000000000000000
html
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. CORR, abs/2309. 12284, 2023. doi: 10.48500ARXXV2230022222
URL
https://doi: wrw/10..oo........o 48550/ wwt.S//aaxiivvv.. 2309 20...120222221 12284
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao. Automathtex:: Autonomous data selection with language models for mathematical texts. CoRR, abs/2402.00625.. 2024. doi: 10. 48550/AARXI.22400.77555 URLpttps:: http:////oo...//......ooeeoo.. 410p://1..855/aaa0/
2402. 07625
Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, Kuai Li, Chen Chen, TAO YANG, and Maosong Sun. CA-loRA: Adapting existing loRA for compressed LLMs to enable efficient multi-tasking on personal devices. First Conference on Language Modeling, 2024. URL
httpeeewweweeeeeeeeeeeeeeeeeeeee forum? d=kpffuonss...
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. "going vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3361- 3367. Association for Computational Linguistics, 2019. doi: 10.18653/N1/D19-13322 URL https: /ttp:s:///oi..or/1...18.8 wttp:///...111//1/////// / 1332
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettle- moyer, and Omer Levy. LIMA: less is more for alignment. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL httllhhttppp........eee nips wwtp::spppppppppppppppppppppppae fiapdeerfflll///ppppeeeeee wwte///aappe//////////eaaaa 8 htt.://000..aaaaaaaaoooaaaaaaaor html
14

---

Published as a conference paper at ICLR 2025
Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. Visual in in-context learning for large vingon-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11 11-16, 2024, pp. 18890-15502. Association for Computational Linguistics, 2024a. doi: SWW.:///WWI224..IINNNNAAAAAA
Utthhttps::///// wttp//..rrr111) 10 1883386511////
Findings ac1. 940
Yucheng Zhou, Jihai Zhang, Guanjie Chen, Jianbing Shen, and Yu Cheng. Less is more: Vision representation compression for efficient video generation with large language models, 2024b.
Yucheng Zhou, Lingran Song, and Jianbing Shen. Training medical large vision-languuge models with awwremal-wware feedback. CoRR, abs/2501 001377, 2025. doi: 10445500ARRXIV23310017777 URL
arriv...vvvv.m 2501. 01377
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth Interna- tional Conference on Learning Representations. ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:/ pettps://openzevieee net http://eeoooooaa---22228-8--
15

---

Published as a conference paper at ICLR 2025
A DETAILS OF DATASETS ON DIFFERENT CAPABILITIES
In this section, we provide a comprehensive overview of the datasets utilized in our study, which are designed to train and evaluate various capabilities of LLMs. Table summarizes the datasets employed, along with their sources and descriptions of the tasks.
Table 1: Datasets on multi-capabilities tasks
Dataset
Original Source
Description
CREAK
Onoe et al. 2021
Pact-Checking and Commonsense Reasoning
ECQA
Aggarwal et al.
2021
Explainable Commonsense Reasoning
e-SNLI
Camburu et al.
2018
Logical Reasoning
GSM8K
Cobbe ct al.
(20TT)
Mathematical Problem Solving
MC-TACO
Zhou ct al. (019)
Temporal Reasoning
OpenBookQA
Mihaylov al.
(2018
Fact Questions
SCAN
Lake & Baroni
2018
Robot Planning
SciQ
Welbl al. (201)
Science Question Answering
B
RELATED WORK
B.1 LARGE LANGUAGE MODELS
Recent advancements in AI have been driven by the development of large language models (LLMs). These models exhibit remarkable capabilities not only language reasoning but also in visual understanding (Zhu et al. 2024. Zhou al., 2024a) and have even demonstrated impressive results across a variety of applications (Zhou et al. 2025 2024b). These models can be categorized into closed-source and open-source LLMs. Closed Closed-source source models such as GPT-4 OpenAl 2023), Gemini (Anil et al. 2023). and Claude 3 (Anthropic 2024) excel in text and visual processing. reasoning, various benchmarks due to significant resource investments but with restricted access. In contrast, open-source models like Mistral 7B (Jiang al. 2023), LLaMA (Touvron et al.) 2023). LLaMA-3 (AI@ Metal 2024) and Qwen (Bai ct al. 2023) promote broader research and innovation. Mistral 7B, with 7.3 billion parameters, and LLaMA, ranging from 700 million 65 billion parameters, provide competitive performance on public datasets, enhancing accessibility and community contributions. Qwen encompasses a variety of models ranging from 0.5 billion to 110 billion parameters, and its range makes suitable for research on model size and capability, from weak to strong generalization (Burns et al. 2023).
and coding tasks, demonstrating near-human performance
B.2 WEAK STRONG GENERALIZATION
weak strong generalization leverages weaker models to supervise stronger ones, improving their performance significantly (Burns et al.) 2023) Guo et al. 2024). Several studies in weak to strong generalization have focused on optimizing the capabilities of large language models (LLMs) using weak supervision. For instance, Tong et al. (2024) introduces a self-reinforcement approach, starting with supervised fine-tuning (SFT) small set of annotated questions. The model then iteratively improves by learning from the differences in responses between the SFT and the un-fine-tuned model on unlabeled questions. In mathematical reasoning, (Zhang al.) 2024) discuss the "AutoDS" method and "AutoMath Text" dataset, and uses foundational language models as zero-shot validators for autonomous data selection, significantly improving performance on mathematical tasks. addition,
Li et al. (2024a) investigates using smaller, weaker models for data filtering to fine-tune larger, stronger models in instruction tuning. Despite the performance gap, there is high consistency between weak and strong models in perceiving instruction difficulty and data selection outcomes.
Other studies have applied weak to strong generalization to specific tasks. For example, Gambashidze et al. (024) introduces framework for 3D object detection that addresses sparsity and occlusion issues in GiDAR-based detection. In scalable aligmment, [Sun et al. (2024) proposes an easy-to-han to-hard generalization method. This approach trains process supervision reward models on easy problems
16

---

Published as a conference paper at ICLR 2025
and uses them to evaluate policy models harder problems, resulting in significant performance improvements through re-ranking or reinforcement learning. Morcover, Liu & Alahi (2024) presents co-supervised learning method that enhances weak to strong generalization by using group of specialized teachers instead of a single generalist teacher. Furthermore, [Ji et al. (2024) introduces Aligner, efficient alignment paradigm that bypasses the RLHF process by learning the correction residuals between aligned and misaligned answers.
B.3 TRAINING DATA SELECTION
In recent years, various methods have been proposed to enhance the performance of large language models (LLMs) by selecting high-quality training data. These methods can be broadly categorized into algorithms focusing the efficiency of data selection and those optimizing for specific quality attributes.
afficiency-based data selection methods prioritize computational efficiency and scalability. The DSIR framework (Xie et al. (023) estimates importance weights in simplified feature space and uses importance resampling select data efficiently, which can select 100M documents from the entire Pile dataset within 4.5 hours. LIMA (Zhou et al. 2023) and LESS (Xia et al. 2024) improve instruction-follovii capabilities by fine-tuning models using high-quality instruction tutiioonntningggg data. In addition, Muennighoff et al. (2023) explores scaling language models under data constraints, proposing a method that considers diminishing returns from repeated tokens and excess parameters
to achieve computational optimality.
Quality- -based data selection methods focus on selecting data that meets certain quality criteria. QuRating (Wettig et al., 2024) aims select pre-training data that captures abstract qualities of text perceived by humans, such as writing style, required expertise, factuality, and educational value. Moreover, MetaMath YYu et al., 2023) focuses mathematical reasoning by celf-bootstrapping new mathematical questions from multiple perspectives without adding extra knowledge, resulting in the MetaMathQA dataset. To address the complexities and sensitivities of reinforcement learning from human feedback (RLHF), Song et al. (2024) introduces PRO, an efficient supervised fine-tuning algorithm that iteratively compares candidate responses guide L.LMs towards optimal responses, aligning with human values.
C PROOF FOR REWARD MODEL
To prove that training a classifier using defined positive and negative samples will result positive samples with higher accuracy and distribution that is further from the strong dataset, we follow these steps:
C.1 NOTATION AND ASSUMPTIONS
Let:
D Weak dataset.
D Strong dataset.
D:: Consistent dataset where Dj = f x E D n E D.).
Dr: Inconsistent data DD where Dj = (r x E D. D.).
Dr: Inconsistent data D. where D; = (a E Dj D.J.
R(z): Reward assigned to sample x by the reward model.
C.2 OBJECTIVE FUNCTION
The objective function for the reward model is given by:
max
EENDE [R(x)] E B R(x)
(11)
R
17

---

Published as a conference paper at ICLR 2025
C.3 PROOF
We start by defining the expected accuracy of the weak model M and the strong model M on their respective datasets:
Accuracy of M AM B2 y
(12)
Accuracy of M: AM = ( - =yll
(13)
where y the true label æ.
By construction, the consistent dataset D includes samples where both models agree, which implies higher confidence:
( v}1 De (M y}1
(14)
DE [[M(x)= y}1 Dy [[M(z) y
(15)
Next, consider the inconsistent datasets Dj and Dj. These sets include samples where the weak and strong models disagree, indicating lower confidence and potentially higher noise:
EZ-D; [L(M(x) = 3}} < RDND; [ = v}l
(16)
E DE (1(((x) = y}1l = E [1M((x) = yHl
(17)
We aim to show that the reward model R trained using Dj (positive samples) and Dr (negative samples) will lead to higher accuracy the positive samples. The reward model objective is:
max (e-D:([(a))
EE [Rxx)]
(18)
By optimizing this objective, the reward model learns assign higher rewards to samples from Dj and lower rewards to samples from Dj. This training process effectively filters out noise and emphasizes samples where the weak and strong models agree, leading to a classifier with higher accuracy.
Furthermore, because the consistent samples Dj are derived from both weak and strong datasets, they are more representative of high-confidence data, ensuring that the positive samples (i.e., samples with high R(x)) are less noisy and have distribution that is different from the noisy parts of the strong dataset (D:).
Thus, we have shown that the classifier trained in this manner will produce positive samples with higher accuracy and distribution that is further from the noisy strong datasets.
D MODEL PERFORMANCE ON DIFFERENT CAPABILITY
In this section, present the performance of models of various scales across multiple capabilities. The results are shown in the following figures. The Figure (Figure 15) shows the performance of different-scale models on multi-capabilities tasks. Moreover, Figure shows the performance of weak to strong generalization with single-capability and multi-capabilities.
E IMPACT OF DIFFERENT CAPABILITY
Figure 17 shows the performance of a strong model when trained with various weak supervision. In Figure 18 and Figure 19 we observe the results when the model's capabilities are deliberately weakened on the some dataset. Figure 20 Figure 21] Figure and Figure 23 the generalization performance of models when certain capabilities are removed.
18

---

Published as a conference paper at ICLR 2025
0.9
0.98
Performnne
0.90
0.5
Deffrmence
0.96
Deffrrmance
0.45
0.50
0.88
0.7
0.90
0.92
0.6
0.75
0.30
0.70
0.5
6.88
0.25
Model
CREAK
(b) ECQA
(c) e-SNLI
(d) GSMSK
0.8
0,90
0.85
Peffomarce
Perfomaace
0.75
Perforaance
0.6
Perfomarce
0.85
0.80
0.70
0.75
0:55
(c) MC-TACO
(1) OpenBookQA
(g) SCAN
(h) SciQ
Figure 15: Performance of different-scale models on multi multi-capabilities tasks.
Weak to Strong Performance
Weak to Strang Performance ancce
Weak to Strong Perform ance
ance
Weak WTS
Weak Strong Performa
0.90
0.8
0.92
0.50
0.85
0.45
0.7
0.90
0.40
0.35
Weak
0.75
WTS-5
WTS-5
0.30
WTS-5
0.70
0.5
WTS
0.86
WTS
0.25
WTS
0.58
1.8848
0.5B
1.88B
0.58
1.80
0.59
1.88
4B
Weak Model
Weak Model
Weak Model
Weak Model
(a) CREAK
(b) ECQA
(c) e-SNLI
(d) GSM8K
Weak Strong Performance
Weak
Weak to Strong Performance
0.80
Weok Strong Performance
0.4
Weak
Weak to Strong Performance
0.90
WTS-S
WTS
WTS
0.70
0.2
0.75
0.60
Weak
0.1
Weak
0.70
WT5-5
0.75
WT5-5
0.0
0.58 1.B
0.58
1.88
48
0.58
1.88
4B
Weak Model
Weak Model
Weak Model
Weak Model
(e) MC-TACO
(1) Oook
(g) SCAN
(h) SciQ
Figure 16: Performance weak strong generalization with single-capability multi-capabilities.
19

---

Published as a conference paper at ICLR 2025
Model
Defommancc
9.6
0.92
Strong Model (2.05)
Strong (oodl1446)
Strong Mide/ (70)
Derformance
5.45
Performa
9.7
5.40
0.90
Wesk Nood
Woar Nocil
Strang Nosel (3.83)
Starg
0.88
530
Stong Notel
Stong Hotel (3 (.33)
6,70
Storng
0.55
Storg
189
058
1.90
0.50
1.00
1.85 Week Model
Week Mode
Weak Model
Week Model
(a) CREAK
(b) ECQA
(c) C-SNLI
(d) GSM8K
0.22
6.90
Strong (2, (48)
0.73
Performnce
Nuffrrancc
(75)
Performance
6.35
0.79
0.2
0.75
0.45
Storg Mocel (3,83)
West Nood
0.62
Wusk Souia
Strong Model (3 0B)
0.1
Strong Hosel (3.83)
Wasie Nazi
Storg Muld (485
Storg Mold (780
0.3
Strong Modil (08)
St/ong Model (78)
5.75
Stoong Moll 648)
Strong Molid
3.68 Weak Model
Weck Madel
Weak Model
Weak Model
(c) MC-TACO
(f) OpenBookQA
SCAN
(h) SciQ
Figure 17: Performance of strong model trained on different weak supervision.
0.75
0.75
0.4
0.50
0.50
0.50
0.2
0.25
0.00
0.00
0.00
0.0
CREAK
ECQA
(c) e-SNLI
GSM8K
0.6
0.75
0.4
0.02
0.2
0.2
0.01
0.25
Weaken
ROW
Weaken
RGW
Weaken
RDW
Weaken
RaW
(c) MC-TACO
(f) OpenBookQA
(g) SCAN
(h) SciQ
Figure 18: Results under weaken capability on GSM8K.
0.75
0.75
0.4
0.50
0.50
0.50
Seress
0.2
0.25
0.25
0.25
0.00
0.00
0.00
0.0
Weakknn
Weaken
Weaken
Wiaaken
(a) CREAK
(b) ECQA
(c) C-SNLI
(d) GSMBK
0.6
0.6
0.75
0.02
0.50
0.2
0.2
0.01
0.25
0.00
0.00
Weaken
Raw
Weaken
Raw
Weaken
Raw
Weaken
Raw
(e) MC-TACO
(I) OpenBookQA
(g) SCAN
(h) SciQ
Figure 19: Results under weaken capabilities ECQA and c-SNLI.
20

---

Published as a conference paper at ICLR 2025
0.75
0.75
0.50
0.50
0.50
04
0.25
0.25
0.25
02
0.00
0.00
0.00
0.0
(a) CREAK
(b) ECQA
(c) e-SLI
(d) MC-TACO
0.6
0.03
Scores
Scones
Scores
0.01
(c) OpenBookQA
(f) SCAN
g SciQ
Figure 20: Results of 1.8B to 4B generalization under remove capability on GSM8k.
0.75
0.75
0.50
0.50
0.50
0.25
0.25
CREAK
(h) ECQA
e-NLL
(d) MC-TACO
0.75
0.3
0.50
Scores
0.2
Scores
0.50
0.25
0.1
0.25
0.00
0.0
0.00
(c) OpenBookQA
(f) SCAN
(g) SciQ
Figure 21: Results 4B to 7B generalization under remove capability on GSM8K.
0.4
0.6
Scores
Scores
0.50
0.2
(a) CREAK
(b) ECQA
(c) MC-TACO
0.6
0.75
Scores
Scones
scores
0.4
0.02
0.50
0.2
0.01
0.25
0.0
0.00
0.00
(d) OpenBookQA
(c) SCAN
(f) SciQ
Figure 22: Results of 1.8B 4B generalization under remove capabilities ECQA and e-SNLI.
0.75
0.4
0.50
0.50
0.25
0.25
0.00
0.0
0.00
(a) CREAK
(b) e-SNLI
(c) MC-TACO
0.75
Scores
0.2
Scorez
0.50
0.25
Boo
(e) SCAN
(I) SciQ
Figure 23: Results of 4B to 7B generalization under remove capabilities on ECQA and e-SNLI
21

---

Published as a conference paper at ICLR 2025
F CONSISTENCY ANALYSIS ON WEAK TO STRONG GENERALIZATION
This section presents an analysis of the consistency in prediction results between models varying sizes. The analysis focuses on comparing the predictions of weaker models with those of stronger models. The following Figure 24 Figure 25 and Figure 26 illustrate the prediction results for different pairs of models.
0.5
0.4
0.3
Humbeer
Hamber
0.3
0.2
0.2
0.1
WTS Phat
Week Wlad
Fatee Mecon.
0.0
WTF Fase
Wead Ras
Falm Irco
Ture Com
Fatte Incn
0.0
Tiv Core
W75 Frov
Heak Flod
Faler Rico
Fadie Cor
Tree
Pakle
Palae
WTS
Weak
False
(a) CREAK
(b) ECQA
(c) C-SNLI
(d) GSM8K
0.4
0.5
0.6
0.3
Number
Number
0.3
0.3
0.2
Tue Comt
WTB Rae
Week Raw
Falee arcm.
Cors
com.
trco..
Falee con.
Tra Com.
WFS Rue
Wink Riue
Fiitee lecon.
Patie Com.
Cork.
Patle Com.
Wit
Wesk
WiB
Week
(e) MC-TACO
OpenBookQA
SCAN
(h) SciQ
Figure 24: Prediction results of 1.8B strong model (WTS) and 0.5B weak model (Weak),
0.5
0.4
0.6
Number
8.4
0.3
6.3
0.0
The Cark
False Rocon
Palie Con
0.0
Roe Can
WTB mos
Weat Ros
Fate con
Cant.
W75 Tiee
Weak Piee
ralte lecort
Fatie Cart
0.0
Tie Car
WTS Row
Halk Row
False Reom
Falte Con
WTS
Weak
Fabe
CREAK
ECQA
(c) C-SNLI
GSM8K
0.6
0.8
0.5
0.5
0.6
Number
0.4
Number
6.2
0.2
0.1
0.0
6.0
6.5
0.0
Tha Corb.
WIS Rue
Rue
False Rrc..
Paker Con
WBB Rue
Walk Rue
Filte fric
losa Con.
Falke fresn
Fatel Com.
WiFl Rue
Weak Rue
Fales From.
Ruit
Fatte
WIS
Weak
Pise
Fater
(e) MC-TAN -ACOOCCO
OpenBonkQA
SCAN
(h) Sci
Figure 25: Prediction results of 4B strong model (WTS) and 1.8B weak model (Weak).
22

---

Published as a conference paper at ICLR 2025
0.8
0.6
0.5
0.6
0.6
Number
Number
0.4
0.4
0.4
0.4
0.3
6.2
0.1
WTS Tio
Weak Tiay
Fadoe Heoes
cont
Rakie Incoss
cen
Tou con.
WTS mue
Herll Tiue
Felee Incrss
Tue Cont
Fabie Antes
Con
Tie
Fabse
False
WTS
Heal
(a) CREAK
(b) ECQA
(c) C-SNLI
(d) GSM8K
Number
Number
0.4
0.3
0.4
0.2
Cais
WTS Rup
Wask Wap
Fale Roces
Fats Cor
The can
WTs Boy
Weat Boy
Faise fnces
Falt con
the Con
WTS Rot
Whak Rite
rabe fhree
Falie Corl
Cor
WTS Rue
Whak Fue
Rafl Cent
faise
(e) MC-TACO
OpenBookQA
SCAN
(h) SciQ
Figure 26: Prediction results of 7B strong model (WTS) and 4B weak model (Weak).
G CONSISTENCY ANALYSIS ON STRONG MODEL SELF-BOOTSTRAPPING
This section presents detailed consistency analysis of the self- seff-bootstrapping used in training strong models. The bootstrapping involves using predictions from a previously trained strong model (WTS) to supervise the training of new strong model (WTS-2). We evaluate the prediction consistency between these models across different scales in Figure Figure 28
and Figure 29
0.5
0.8
0.6
0.4
0.3
Number
Number
0.3
0.3
0.2
0.0
Tue Cans
WTB-2 Tise
WPS Wot
Pake Cors.
Con
W15-2 mue
Fair Con.
Tur Com
WTS- Pree
WT5 Wie
Folke iecon.
Pake Cor
WYS-3 Tue
WTS Tuue
Falie Incor.
Feise
WTS
Palse
CREAK
(b) ECQA
(c) e-SNLI
(d) GSMSK
0.4
0.6
Number
Number
0.3
0.4
0.4
6.2
0.1
Cont
WTS-2 Rol
WTS Ria
Patoe Con
Rui Com
w1s-2 mar
wis Fro
Falie Inco
Fais Con
Rree
WT5 Tite
tecoe.
Patle Con
Rue Con-
w1S-2 Pue
wis Ri
Fales from
Fatee Con
Faise
(e) MC-TACO
OpenBookQA
SCAN
(h) Sci
Figure 27: Prediction results of 1.8B strong model (WTS-2) and its supervision from strong model (WTS).
23

---

Published as a conference paper at ICLR 2025
0.5
0.6
0.6
0.4
Number
Number
0.3
0.4
0.2
0.0
W7S- Bue
False Boon
0.0
WTS Tue
Falle Frion
0.0
mom Can
WT5-2 liue
WTS Pue
ratie fren
Falle Con
0.0
WIS-2
False Roon
WTS
Falte
Thw
WTS-2
Fabse
Tow
WIF
Fabre
CREAK
(b) ECQA
(c) C-SNLI
(d) GSMSK
0.6
0.6
0.30
0.25
0.4
Number
0.4
humber
0.15
0.10
0.95
0.0
0.0
0.0
Fadse Ayron.
Tat Con
WTS Froe
Con.
Trse con
WTS-2 Tae
WTS Tlu
Fatso tecon
WTS mom
False Incen
Falte Con
Brue
WTS
False
WTB:2
Falle
False
Tise
WTS:3
(c) MC-TACO
OpenBookQA
SCAN
(h) SciQ
Figure 28: Prediction results 4B strong model (WTS-2) its supervision from strong model (WTS).
6.8
0.5
0.6
Number
0.6
0.4
0.4
0.4
0.2
0.1
0.0
6.0
0.
WFS-2 Pae
Pales Rcoo.
WTS True
Palns frco
Trse Com.
WT5-2 Houe
WTS Tree
Parra lecon
Paise Con.
WTS Rue
Feles from.
Tue
Fassie
WTS-2
Fains
Tue
W7S-3
Palie
CREAK
(b) ECQA
(c) e-SNLI
(d) GSMSK
0.8
0.5
0.5
Number
Number
0.6
0.4
0.4
0.2
62
0.2
Cass
Feise tocoos
Palie Con
Con
WT5-2 Yoe
WTS Time
Incos.
Faie ceo.
Bur Con.
Palke IncNe
Rue Con
WTS Rus
Febee Jicess
Pabie cru
WTB-2
W75-2
WTS
Palse
WTS-2
(e) -C-TACO
OpenBookQA
(g) SCAN
(h) SciQ
Figure 29: Prediction results 7B strong model (WTS-2) its supervision from strong model (WTS).
24

---

Published as a conference paper at ICLR 2025
H IMPACT OF WEAK DATA
This section examines the impact of weak data on model performance. The analysis is presented through Figure 30 and Figure 31 which illustrate the effects of training a strong model with clean and noisy weak datasets.
0.925
0.90
0.25
0.55
Peffrmmree
0.900
peformance
0.85
0.00
0.75
1.0
0.5
Performance
0.91
Performance
0.50
0.850
0.75
0.50
0.45
0.825
Clean
Clenn
0.40
Clean
0.25
0.65
0.89
0.775
0.750
0.75
0.5
1.0
9.60
0.88
0.75
1.0
0.5
0.75
1.0
0.30
0.38
1.85
48
0.58
1.8848
0.58
1.66
46
0.58
1.88
48
Weak Mocel
Weak Model
Weak Model
Weak Model
(a) CREAK
(b) ECQA
(c) SNSLL
(d) GSM8K
0.30
Clean
0.25
0.05
0.75
0.25
Peffemance
0.00
Peformance
0.70
performence
6.20
Nerfrmance
0.15
0.84
0.75
Clean
Ckan
0.10
0.52
Clean
0.25
0.60
0.25
0.25
0.70
0.5
0.5
0.05
0.80
0.5
0.75
0.55
0.75
1.0
0.00
0.78
1.0
Weak Model
Weak Model
Weak Modell
Weak Model
(e) MC-TACO
(1) OpenBookQA
(g) SCAN
(h) SciQ
Figure 30: Each capability performance of strong model trained on clean samples and different proportions of noise samples of weak datasets.
0.32
0.94
Strong Data
Miced Data
0.90
Strong Data Mixedd Daria
0.920
Weok
0.55
Neffrrmnce
0.90
Performacce
0.85
Performance
forrmancc
0.50
0.80
0.80
0.45
0.06
0.919
0.75
0.94
0.62
8.70
0:305
0.35
Weak
Stroin Dati
0.m
0.65
0.300
Mised Data
0.58
1.88
48
0.58
1.8848
0.50
1.40
40
0.58
1.88
49
Weak Model
Weak Model
Weak Modil
Weak Model
(a) CREAK
(b) ECQA
(c) c-SNLI
(d) GSM8K
Wheak Date
0.80
Week Dataa
0.85
Strong Data
Micedd Data
0.75
0.30
Strong Data
Mixed Data
0.08
Neformance
Peffrrancc
Performacc
0.25
Irformence
0.00
0.20
0.84
0.15
0.75
0.65
0.10
0.10
-H- Weak Data
Strong Dale
0.05
0.80
-2-- Weak Data
Stronng Delin
0.55
0.00
0.78
0.58
1.58
49
1.89
0.58
48
0.59
1.88 weak Model
Model
Weak Model
Weeak Modell
(e) MC-TACO
(l) OpenBookQA
(g) SCAN
(h) SciQ
Figure 31: Each capability performance of strong model trained on weak datasets, strong datasets and their combination.
25

---

Published as a conference paper at ICLR 2025
1 PERFORMANCE ON EACH CAPABILITY
This section provides a detailed analysis of the weak to strong generalization performance of each capability. The results are illustrated in the Figure 32 and Figure 33
Store Mookt
0.00
OITSO
0.50
0.09
Dwnel
0.50
Danel 330
0.85
0:35
0.80
Stinng Model
WT%
Shoog Made
0.92
0.40
Scring Madel
0.05
6.0
-are-sa
0.75
Gwini 20.50
Owani
Owers 56.50
0.0
-4- Geini, 2-. 53
Raani F41
0.55
0.30
42
Model
(a) CREAK
(b) ECQA
(c) CNSLLI
(d) GSM8K
Stang Model
NIG 6.53
0.83
6.7
WTS-L.6
Op5-48
(Wei co Semrig Perfomarce
Dum-40
Guent 5-0.
Weak to Srman Dertomatce
0.00
Owail 3-33
00:2
0.5
MISS-150
wrs
Ownit
0.5
Dee-t 40
(weell X 1.90
Oweel
1.4k
1.5
44
183
48
7
1.4
Strong Mede
String Hood
Strong Modal
Stoing Model
(e) MC-TACO
(1) OpenBookQA
(g) SCAN
(h) SciQ
Figure 32: Each capability performance of weak to strong generalization.
W75
1.0
VIT--3
WTS-40
WTS 48
Porfomancc Dap TA00001
0.3
Curo-188
0.55
Oun-a.50 50
Gun-LL88
0.4
Our-- 50
Curs
Peffrmmne Dap Tacoroned
3.7
8-1
WT5 0.35
WTS 1.85
0ur-1150
Duri-100 00
Our--13
1.00
70
1.00
40
75
103
4B
70
1.0
Strong Middl
Strong Model
Strong Modsl
Storng Model
(a) CREAK
(b) ECQA
(c) e-SLI
(d) GSM8K
0.0
WIS-2A6
2.5
W75-850
1/4
WTS: 1.93-.93
W.5-6.53
Puffrr aifurrnnc Recoren
025
CRIN-S.5 SR
Dury-LED
Rufommrce Cup Aucoveroi
W15-22
Ours-
0.00
Catsia
2.1
Our--83
Puferarnes.o Sup
3.0
0.35
9.4
0.00
WTS-3.56
W7T--.08
-0.8
WT5-40
0.rr
-0.4
6.7
0urs-4 49
String Model
Strong Model
S0o0g Made
Storg Model
(e) MC-TACO
(1) OpenBookQA
(g) SCAN
(h) SciQ
Figure 33: Performance gap recovered of weak to strong generalization on each capability.
26

---

Published as a conference paper at ICLR 2025
J PERFORMANCE ON REWARD MODEL
In this section, we present the performance analysis of the reward model compared to weak and strong models across various capabilities. The analysis is visualized through Figure 34 Figure 35 Figure 36 and Figure 37 illustrating the accuracy and consistency of the models.
1.0
0.6
0.5
0.4
04
02
Weak
02
-4-
Rawaed
9.2
Weak
RRAMS Relorckan NUGS
Wrak
RAWA'd Iddavinn NAMA
WT5
In Tato
11
Saturion SAN
M13
W15
0.50
1.00 Model
0.50
Model
Model
Model
(a) CREAK
(b) ECQA
(c) C-SNLI
(d) GSM8K
Wunt
Sevard
0.85
0.6
0.4
0.0
0.2
0.30
1.33
(e) MC-TACO
OpenBonkQA
(g) SCAN
(h) Sci
Figure 34: Accuracy of datasets from reward model, weak model and strong model on each capability.
Weak Reward
Weak Reward
Week Reword
0.4
Week Reward
0.4
Number
Number
0.6
Number
0.3
0.3
0.2
0,1
0.3
0.0
Rite com
Felax Rrrm.
Fatie Cor
Thun cor
WTS Fiue
Weat Fiue
Fales Irron.
Fals Con
6.9
Bow con
WTS War
Wenti Piu
Becmo
Fatle Con
0.0
Rue Com
WTS Rua
Heak Rual
Pelse Rrom..
Patir Con
WTS
CREAK
ECQA
(c) e-SNLI
(d) GSMSK
0.5
Wcak Roward
Weak Reward
Weak Reward
0.6
Number
Number
0.4
0.2
0.1
0.2
0.1
0.3
The Cork.
Fale Incon.
c.0
WPS Rue
Weat Ruse
Fales moon..
0,a
Trub Con
Faka Incor
Pabie Cort
0.
Weak Root
Fales Hoom
WTS
Heat
Palse
Tue
fales
WTS
The
lalie
(e) MC-TACO
(1) OpenBookQA
SCAN
(h) SciQ
Figure 35: Performance of reward model (Reward) on (in cinnconsittnt part between 0.5B weak model (Weak)
1.8B strong model.
27

---

Published as a conference paper at ICLR 2025
0.5
0.6
Weak
Weak
0.8
6.5
0.4
Reward
Number
Number
0.3
0.4
0.2
0,2
0.1
Rue Cots
WTS Rup
Weak Rap
Fadoe Meren
Falser Cors
con
Rakie Encont
Fais Cen
Bue Cont.
WPS mue
Herll Rue
Felee inene
Tue Con
whs Pup
Heak Pup
Fabiee Hocet
Cont.
wTs
West
False
(a) CREAK
(b) ECQA
(c) e-SNLI
(d) GSM8K
Weak
Weak
Weak Rewward
Rewarr
0.5
Rewwrd
0.6
0.4
Number
Number
0.3
Number
83
0.4
0.3
0.2
6.1
0.1
0.2
Cats
WTS Rup
Pade Roces
Pare Car
The Can
WTs Boee
Wealt Roee
Patse tnces
Palt Con
the can
WTS moe
Whak Rile
Feino threeg
False Corl
Cost
WTS Rue
Whak Fue
Palse Roced
Pabe Cent.
Wask
(c) MC-TACO
OpenBookQA
(g) SCAN
(h) Sci
Figure 36: Performance of reward model (Reward) on (in (in)consistent part between 1.8B weak model (Weak)
and 4B strong model.
0.8
Weak Reward
0.8
Weak Reward
Weak Roward
0.6
Weak Reward
0.5
0.6
0.6
0.6
Number
0.4
0.4
0.3
0.2
0.2
0.3
0.3
WTS Roo
West PPo
Fate Recons
0.0
WPS tue
Weat tuu
True Cor.
WTS Thue
Pake INcOI
Palie Cork
0.0
WPT Ruo
Week Fuo
Falet tnco
The
Padie
Falre
The
Radis
(a) CREAK
(b) ECQA
(c) SNLL
(d) GSMSK
0.8
Number
Number
0.4
Number
0.6
0.4
0.4
0.3
0.4
0.2
Tue Cons
WTS Rui
Weal Rui
Palse Incos
Pabse Cor
Con
Faste Incon.
Fale Con.
Troe Com.
WTS Rue
Henll Rue
Feine lecmn
Rise Con
Meri Fual
Felsr Irom
Pabse Cen
W75
Weat
Patle
WTS
(e) MC-TACO
OpenBookQA
(g) SCAN
(h) Sci
Figure 37: Performance of reward model (Reward) on (in consistent part between 4B weak model (Weak) and
7B strong model.
28

---

Published as a conference paper at ICLR 2025
K MORE DETAILED FIGURES
We have expanded on the content of Figure Figure Left), and Figure Right) in greater detail. The
detailed breakdown is presented Figure 38
Figure
39
and Figure 40 below for better observation.
Weak Strong Performance
Weak to Strong Performance
0.76
0.675
0.74
0.650
0.72
0.625
0.600
0.70
0.575
0.68
Weak
Weak
WTS-s
0.550
WTS-S
0.66
WTS
WTS
0.525
0.5B
1.8B
1.8B
4B
Weak Model
Weak Model
Figure 38: Performance of strong model with sangle-capability and multi-capabilitis weak strong generalization.
0.78
Clean
0.675
0.76
0.25
Performance
0.5
Performance
0.650
0.74
0.75
1.0
0.625
Clean 0.25
0.72
0.600
0.5
0.70
0.75
0.575
1.0
0.68
0.5B
1.8B
1.8B
4B
Weak Model
Weak Model
Figure 39: Performance of strong model trained clean samples and different
proportions noise samples of weak datasets.
0.78
Weak Data
Weak Data
0.68
Strong Data
0.76
Strong Data
Performance
0.66
Mixed Data
Performance
Mixed Data
0.74
0.64
0.72
0.62
0.70
0.60
0.68
0.5B
1.8B
1.8B
4B
Weak Model
Weak Model
Figure 40: Performance strong model trained on weak, strong, and combination datascts.
29

---

Published as a conference paper at ICLR 2025
L ABLATION STUDY ON DATA SELECTION METHODS
We experiment to compare the effectiveness of our reward model-based data selection with a random selection baseline. Random selection was performed by choosing the same amount of data as selected by the reward model, as it is considered a strong baseline in LLM data selection. The results of these experiments are presented in 2
Weak Model
Strong Model
Random
Ours
0.5B
1.8B
52.49
59.17
1.8B
4B
63.52
69.05
4B
7B
74.31
77.07
Table 2: Performance comparison between random data selection and reward model-based data selection.
The results show that random selection leads to a significant drop in performance compared to reward model-based approach. This performance gap be attributed to the reward model's ability to identify and prioritize higher-quality training samples, which is critical for achieving better generalization. These findings underscore the importance of the reward model in improving the data selection process.
M PERFORMANCE VERIFICATION WITH IN-CONTEX: LEARNING
To better verify the performance of the strong model before training, we assessed its capabilities using in-context learning (ICL). Specifically, we evaluated the strong model's 's performance before training by providing 3 ground truth examples for ICL. The experimental results are presented in
Table 3
Weak Model
Strong Model
Ours
WTS
ICL
0.5B
1.8B
59.19
56.10
45.92
1.8B
4B
69.07
68.00
58.38
4B
7B
77.06
76.11
70.02
Table 3: Performance comparison between our method, standard weak-to-strong generalization (WTS), and in-context learning (ICL).
The results demonstrate that weak-to-strong generalization (including both our method and the standard WTS approach) significantly outperforms ICL across all model sizes. This performance gap highlights the limitations of ICL, which relies solely on few examples and cannot fully learn the task's intent. In contrast, weak-to-strong generalization effectively learns the task's intent from noisy weak data and exhibits remarkable resistance to noise. This robustness underscores the superiority of weak-to-a seak-to-strong generalization in scenarios where data quality is inconsistent or noisy, further validating its effectiveness for large-scale model training.
30


Reviews:
[{"review_id": "cGXLYEWwdz", "reviewer": "Reviewer_1qFU", "summary": "This paper first conducts a series of analyses to explore the factors influencing the spectrum from weak to strong generalization in large language models (LLMs) with multi-capabilities. It then proposes a two-stage training framework that uses reward models to select valuable data based on the insights gained from the analyses. Experimental results demonstrate improvements in the multi-capabilities of the LLMs.", "questions": "- In line 132, there is a missing comma. Additionally, it would be better to explain the difference between your work and related work in the related work section.\n- In Figure 1 (right), it would be helpful to annotate that the performance pertains to the strong model.\n- In line 186, \"Figure 4\" should be \"Figure 3\". In line 191, \"Figure 3\" should be \"Figure 4\". Furthermore, in Figures 3 and 4, it would be better to keep the colors of the \"weaken\" and \"remove\" bars consistent.\n- In Section 3.5, the experiments show that the same strong model cannot improve itself. However, I am interested in the outcomes when training another weak model or a smaller but strong model.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is well-written and easy to understand. Moreover, the structure is complete and coherent.\n- Section 3 provides valuable insights, and the proposed method makes sense.", "weaknesses": "- The setups of \"WTS-S\" and \"WTS\" are not clearly explained upon their first mention. It appears that \"WTS-S\" involves training different models for different datasets and computing their average score, while \"WTS\" involves training all datasets together in a single model.\n- In Sections 3.3 and 3.4, it would be better to reference the setups under \"WTS\" rather than \"WTS-S\".\n- What is your setup for the \"mix data\" in Section 3.6? Are the datasets directly merged? There is concern about potential conflicting cases in the \"mix data\" that could confuse the model. The total number of correct cases in \"mix data\" should be larger than \"weak data\". So I am doubtful about the conclusion regarding Figure 7 (right) because it seems a little contradictory to Figure 7 (left). Additionally, \"mix data\" should be better termed as \"mixed data\".\n- The performance advantages of \"WTS\" in Figure 8 are not obvious when the weak model is scaled up. However, the gap to the strong model remains significantly large.\n- I doubt the results regarding \"the dependence of data quality across various capabilities\" as claimed in lines 191-192. In Figure 4, there are only two cases: \"w/o GSM8k\" and \"w/o ECQA & e-SNLI\". While these can provide some insight, they are not comprehensive. If, as the conclusion suggests, the abilities in each capability are independent of each other, then why are the mixed training results of multi-capability data better than the single-capability averages?\n- **[Important]** Despite the focus on multi-capabilities in the title, I would prefer to see the performance in single capabilities and comparison results with existing methods. It seems that most of the analysis and the proposed method could also apply to single capabilities, which are more commonly used. This paper does not convincingly demonstrate the necessity of working on multiple capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "25pRzPZZbp", "reviewer": "Reviewer_r2a7", "summary": "This paper analyzes a multi-capabilities setting under the weak to strong generalization paradigm. Based on this analysis, a training framework for this paradigm is proposed, which includes a reward model for selecting valuable weak supervision data, followed by a two-stage training of a strong model on both the weak dataset and the selected dataset. Further discussion on the reward model is also provided.", "questions": "1. What is the connection between the conclusion regarding the independence of different capabilities mentioned in Weakness 1 and the subsequent methodology and experiments?\n\n2. In the scaling experiments, it is observed that as model size increases, the difference between the proposed method and the baseline becomes smaller. Considering the reward model's size is consistent with that of the strong model and the data used for the reward model is from the weak and strong model training, does the proposed method have certain limitations? (In simple terms, is the reward model scalable?)", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper analyzes the current gaps in research on the weak to strong generalization paradigm from a \"super-alignment\" perspective, proposing a new setup for experimental analysis and providing a novel training approach under this setup.", "weaknesses": "1. The significance of the multi-capabilities setting is unclear. In Section 3.3, the paper concludes that since the quality of weak data is relatively independent across different capabilities and does not interfere with each other during generalization, the setup seems only quantitatively different from a single capability setup. Therefore, is it necessary to study this setup separately? It is recommended to clarify the differences between the single capability setup and the one proposed in this paper.\n\n2. The paper does not clearly define the diversity of the weak dataset. It is mentioned that the diversity of a weak dataset can lead to better results as previous works. The experiment the third chapter focuses on the consistency of results generated by weak and strong models, which is typically not an explicit measure of diversity. \n\n3. The paper lacks ablation experiments. It only presents the performance without using the reward model for data selection. It is suggested to conduct experiments by randomly selecting the same amount of data as picked by the reward model (as random selection is a strong baseline in LLM data selection).\n\n4. The legends in the figures are unclear, and some contain labeling errors. In the first three sections of Chapter 3, \"weak model\" is used as the axis label, yet most of the time, the data shown is based on the performance of the strong model, causing confusion. Additionally, there are errors in the numbering of Figures 3 and 4 in Section 3.3. Besides, there is inconsistent usage of \"weak to strong generalization\" and \"weak-to-strong generalization.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fiiu8CLuIk", "reviewer": "Reviewer_5nVp", "summary": "This paper first investigates multi-capability weak-to-strong generalization in large language models (LLMs). It then confirms that diverse, high-quality data from weak models improves generalization for strong models. To enhance this process, the authors introduce a two-stage training framework that uses reward models to select optimal weak supervision data.", "questions": "1. I am curious whether the second version of the strong model in Section 3.5 is further trained on the existing trained strong model, or if it is untrained. If an untrained strong model is used to relabel data and then continues to train itself, would that improve performance? Additionally, while greedy search can exacerbate the issue of overconfidence, would sampling help mitigate this? Specifically, by using multiple sampling and voting (self-consistency) to select more diverse and accurate samples, could it lead to greater self-bootstrapping gains?\n2. Is a separate reward model trained for each weak-to-strong training instance? If so, wouldn’t this make the overall training pipeline somewhat lengthy? Additionally, I understand that the reward model’s training data and inference data are identical, so how many weak data points with the same labels are actually filtered out? If not, how does the reward model generalize?\n3. To what extent might rule-based data selection methods serve as alternatives to the reward model? For instance, could calculating semantic similarity or n-gram redundancy be effective approaches?\n4. What is the performance of the strong model before training? How does it compare with the supervised trained weak model and the strong model after weak-to-strong training? I am curious about the significance of studying weak-to-strong generalization.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. As the first study to investigate multi-capability weak-to-strong generalization, it can provide some relevant conclusions for this field.\n2. This study explores data selection directions for weak-to-strong generalization and proposes corresponding methods to improve performance.", "weaknesses": "1. The research on multi-capability weak-to-strong generalization seems rather isolated from the later stages of weak data selection and also lacks a connection to the proposed method. It is a relatively shallow and separate exploration.\n2. The validation of self-bootstrapping's ineffectiveness is insufficient. Concerns are detailed in the 'Questions' section.\n3. The necessity and effectiveness of the reward model lack sufficient validation and investigation. For example, an ablation study on Equation 7 was not conducted. Additional concerns are detailed in the 'Questions' section.\n4. The necessity of the warm-up stage lacks further validation. For example, running only 2 epochs might be insufficient, as the model's optimal performance is achieved after a total of 4 epochs, suggesting that it may be the total training time, rather than the distinct nature of the two stages, that drives performance improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M6jcHoZA2v", "reviewer": "Reviewer_fdJr", "summary": "This paper conducts a thorough exploratory study on multi-capability alignment within the realm of weak supervision for strong model training. The authors make several key observations, such as the relative independence of different tasks when weak supervision is applied and the tendency of models to become overly reliant on limited supervision through bootstrapping. These insights highlight the challenges and limitations of current approaches. Based on these findings, the authors propose a theoretically and experimentally grounded two-stage training framework, which includes training reward models for data selection and training strong models. This framework effectively enables learning from weak datasets and the selected data, addressing the inconsistencies between weak and strong datasets.", "questions": "As mentioned in the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research topic is novel and meaningful, providing valuable insights into multi-capability alignment under the concept of super-alignment.\n2. The writing is rigorous and standardized, with a deep and thorough analysis that enhances the overall clarity of the paper.\n3. The proposed two-stage approach effectively addresses key challenges within the research paradigm, supported by solid theoretical analysis and experimental validation.", "weaknesses": "1. While the paper exhibits excellent logical flow and robust experimentation, the presentation of key figures, such as Figure 1, lacks clarity and attention to detail. The meanings of the axes, calculation formulas, and the implications of the legends are somewhat ambiguous, leading to confusion regarding the data sources. The overlapping results of different methods detract from the visual appeal, and the depiction of performance proportional to model size raises questions, especially when the same-sized methods are represented in different colors. It is advisable to optimize this section for better clarity and presentation.\n   \n2. The width of the model size representation could be expanded to better align with the theme of super-alignment, thereby more effectively demonstrating the robustness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0WLA3OlTyL", "reviewer": "Reviewer_orGJ", "summary": "This paper investigates the weak-to-strong generalization capabilities of LLMs across multiple capabilities. Through extensive experiments, key findings include the relative independence of capabilities, the significant impact of data quality on weak supervision, and the overconfidence of strong models in specific knowledge areas, leading to performance degradation. To enhance weak-to-strong generalization, the authors propose a novel training framework. This framework uses reward models to select valuable weak supervision data, generated by weaker models, for training stronger models. A reward model is trained to identify datasets that differ in distribution from the strong datasets, ensuring diversity. A two-stage training method is then applied, allowing the strong model to learn from both weak and selected datasets, resulting in improved multi-capabilities weak-to-strong generalization.", "questions": "The paper explores the weak to strong generalization for LLMs across multiple capabilities. However, there are several aspects that require further clarification and improvement to strengthen the validity and robustness of the proposed method. Below are my detailed comments:\n1. The paper frequently uses the term \"performance\" to describe the experimental results of the comparison models without clearly specifying the particular evaluation metrics. Could the authors clarify what is meant by \"performance\"? Is it accuracy, or another metric? Please provide a clear definition and the specific metrics used.\n2. In Equation 6, why is Dci considered low-confidence data? Is it because this data is generated by a strong model through relabeling? Could the rationale behind treating it as low-confidence data be to enhance data diversity and thereby prevent model collapse?\n3. Could you provide more details about the reward model R(x)? Specifically, what are the training data and model architecture used for R(x)? Are the reward model, weak model, and strong model all instances of Qwen 1.5, or do they differ in their configurations?\n4. In the experiments conducted with different model sizes, were the sizes of the datasets (e.g., the labeled dataset and the unlabeled dataset) varied? For instance, was a larger amount of data used for the 4B model compared to the 1.8B model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The authors present how weak supervision affects strong models, revealing that strong models do not always follow the guidance of weak models. This insight is crucial for understanding the limitations and potential pitfalls of using weak supervision. And the proposed training framework, which uses reward models to select valuable data for weak supervision, is innovative. It addresses the issue of overconfidence in strong models and leverages the diversity of weak datasets to improve generalization. This paper clearly outlines the steps involved in the proposed framework, making it easy to understand.", "weaknesses": "Although the paper points out that overconfidence is an important problem in strong models, it does not provide a detailed description of the reward model used to select valuable weak supervision data, such as the reward model's framework and training data. In addition, this paper uses 'performance' to refer to the experimental results of the comparison models in several places, without clearly describing the specific evaluation metrics of the performance. Clearer metrics would help readers better understand the improvements achieved by the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)