Weak to Strong Generalization for Large Language Models with Multi-capabilities

Authors: Yucheng Zhou, Jianbing Shen, Yu Cheng
Primary area: foundation or frontier models, including LLMs
Keywords: Weak to Strong Generalization, Large Language Model, Superalignment

Abstract:
As large language models (LLMs) grow in sophistication, some of their capabilities surpass human abilities, making it essential to ensure their alignment with human values and intentions, i.e., Superalignment. This superalignment challenge is particularly critical for complex tasks, as annotations provided by humans, as weak supervisors, may be overly simplistic, incomplete, or incorrect. Previous work has demonstrated the potential of training a strong model using the weak dataset generated by a weak model as weak supervision. However, these studies have been limited to a single capability. In this work, we conduct extensive experiments to investigate weak to strong generalization for LLMs with multi-capabilities. The experiments reveal that different capabilities tend to remain relatively independent in this generalization, and the effectiveness of weak supervision is significantly impacted by the quality and diversity of the weak datasets. Moreover, the self-bootstrapping of the strong model leads to performance degradation due to its overconfidence and the limited diversity of its generated dataset. To address these issues, we proposed a novel training framework using reward models to select valuable data, thereby providing weak supervision for strong model training. In addition, we propose a two-stage training method on both weak and selected datasets to train the strong model. Experimental results demonstrate our method significantly improves the weak to strong generalization with multi-capabilities.

Decision: accept (poster)
Reviewer scores: 6, 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730686576130

thread_type:
official_review

<thread_object>
creator: reviewer_orgj

<content>
summary:
This paper investigates the weak-to-strong generalization capabilities of LLMs across multiple capabilities. Through extensive experiments, key findings include the relative independence of capabilities, the significant impact of data quality on weak supervision, and the overconfidence of strong models in specific knowledge areas, leading to performance degradation. To enhance weak-to-strong generalization, the authors propose a novel training framework. This framework uses reward models to select valuable weak supervision data, generated by weaker models, for training stronger models. A reward model is trained to identify datasets that differ in distribution from the strong datasets, ensuring diversity. A two-stage training method is then applied, allowing the strong model to learn from both weak and selected datasets, resulting in improved multi-capabilities weak-to-strong generalization.

strengths:
The authors present how weak supervision affects strong models, revealing that strong models do not always follow the guidance of weak models. This insight is crucial for understanding the limitations and potential pitfalls of using weak supervision. And the proposed training framework, which uses reward models to select valuable data for weak supervision, is innovative. It addresses the issue of overconfidence in strong models and leverages the diversity of weak datasets to improve generalization. This paper clearly outlines the steps involved in the proposed framework, making it easy to understand.

weaknesses:
Although the paper points out that overconfidence is an important problem in strong models, it does not provide a detailed description of the reward model used to select valuable weak supervision data, such as the reward model's framework and training data. In addition, this paper uses 'performance' to refer to the experimental results of the comparison models in several places, without clearly describing the specific evaluation metrics of the performance. Clearer metrics would help readers better understand the improvements achieved by the proposed framework.

questions:
The paper explores the weak to strong generalization for LLMs across multiple capabilities. However, there are several aspects that require further clarification and improvement to strengthen the validity and robustness of the proposed method. Below are my detailed comments:
1. The paper frequently uses the term "performance" to describe the experimental results of the comparison models without clearly specifying the particular evaluation metrics. Could the authors clarify what is meant by "performance"? Is it accuracy, or another metric? Please provide a clear definition and the specific metrics used.
2. In Equation 6, why is Dci considered low-confidence data? Is it because this data is generated by a strong model through relabeling? Could the rationale behind treating it as low-confidence data be to enhance data diversity and thereby prevent model collapse?
3. Could you provide more details about the reward model R(x)? Specifically, what are the training data and model architecture used for R(x)? Are the reward model, weak model, and strong model all instances of Qwen 1.5, or do they differ in their configurations?
4. In the experiments conducted with different model sizes, were the sizes of the datasets (e.g., the labeled dataset and the unlabeled dataset) varied? For instance, was a larger amount of data used for the 4B model compared to the 1.8B model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702056465

thread_type:
official_review

<thread_object>
creator: reviewer_fdjr

<content>
summary:
This paper conducts a thorough exploratory study on multi-capability alignment within the realm of weak supervision for strong model training. The authors make several key observations, such as the relative independence of different tasks when weak supervision is applied and the tendency of models to become overly reliant on limited supervision through bootstrapping. These insights highlight the challenges and limitations of current approaches. Based on these findings, the authors propose a theoretically and experimentally grounded two-stage training framework, which includes training reward models for data selection and training strong models. This framework effectively enables learning from weak datasets and the selected data, addressing the inconsistencies between weak and strong datasets.

strengths:
1. The research topic is novel and meaningful, providing valuable insights into multi-capability alignment under the concept of super-alignment.
2. The writing is rigorous and standardized, with a deep and thorough analysis that enhances the overall clarity of the paper.
3. The proposed two-stage approach effectively addresses key challenges within the research paradigm, supported by solid theoretical analysis and experimental validation.

weaknesses:
1. While the paper exhibits excellent logical flow and robust experimentation, the presentation of key figures, such as Figure 1, lacks clarity and attention to detail. The meanings of the axes, calculation formulas, and the implications of the legends are somewhat ambiguous, leading to confusion regarding the data sources. The overlapping results of different methods detract from the visual appeal, and the depiction of performance proportional to model size raises questions, especially when the same-sized methods are represented in different colors. It is advisable to optimize this section for better clarity and presentation.
   
2. The width of the model size representation could be expanded to better align with the theme of super-alignment, thereby more effectively demonstrating the robustness of the proposed method.

questions:
As mentioned in the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715376113

thread_type:
official_review

<thread_object>
creator: reviewer_5nvp

<content>
summary:
This paper first investigates multi-capability weak-to-strong generalization in large language models (LLMs). It then confirms that diverse, high-quality data from weak models improves generalization for strong models. To enhance this process, the authors introduce a two-stage training framework that uses reward models to select optimal weak supervision data.

strengths:
1. As the first study to investigate multi-capability weak-to-strong generalization, it can provide some relevant conclusions for this field.
2. This study explores data selection directions for weak-to-strong generalization and proposes corresponding methods to improve performance.

weaknesses:
1. The research on multi-capability weak-to-strong generalization seems rather isolated from the later stages of weak data selection and also lacks a connection to the proposed method. It is a relatively shallow and separate exploration.
2. The validation of self-bootstrapping's ineffectiveness is insufficient. Concerns are detailed in the 'Questions' section.
3. The necessity and effectiveness of the reward model lack sufficient validation and investigation. For example, an ablation study on Equation 7 was not conducted. Additional concerns are detailed in the 'Questions' section.
4. The necessity of the warm-up stage lacks further validation. For example, running only 2 epochs might be insufficient, as the model's optimal performance is achieved after a total of 4 epochs, suggesting that it may be the total training time, rather than the distinct nature of the two stages, that drives performance improvements.

questions:
1. I am curious whether the second version of the strong model in Section 3.5 is further trained on the existing trained strong model, or if it is untrained. If an untrained strong model is used to relabel data and then continues to train itself, would that improve performance? Additionally, while greedy search can exacerbate the issue of overconfidence, would sampling help mitigate this? Specifically, by using multiple sampling and voting (self-consistency) to select more diverse and accurate samples, could it lead to greater self-bootstrapping gains?
2. Is a separate reward model trained for each weak-to-strong training instance? If so, wouldn’t this make the overall training pipeline somewhat lengthy? Additionally, I understand that the reward model’s training data and inference data are identical, so how many weak data points with the same labels are actually filtered out? If not, how does the reward model generalize?
3. To what extent might rule-based data selection methods serve as alternatives to the reward model? For instance, could calculating semantic similarity or n-gram redundancy be effective approaches?
4. What is the performance of the strong model before training? How does it compare with the supervised trained weak model and the strong model after weak-to-strong training? I am curious about the significance of studying weak-to-strong generalization.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720478420

thread_type:
official_review

<thread_object>
creator: reviewer_r2a7

<content>
summary:
This paper analyzes a multi-capabilities setting under the weak to strong generalization paradigm. Based on this analysis, a training framework for this paradigm is proposed, which includes a reward model for selecting valuable weak supervision data, followed by a two-stage training of a strong model on both the weak dataset and the selected dataset. Further discussion on the reward model is also provided.

strengths:
The paper analyzes the current gaps in research on the weak to strong generalization paradigm from a "super-alignment" perspective, proposing a new setup for experimental analysis and providing a novel training approach under this setup.

weaknesses:
1. The significance of the multi-capabilities setting is unclear. In Section 3.3, the paper concludes that since the quality of weak data is relatively independent across different capabilities and does not interfere with each other during generalization, the setup seems only quantitatively different from a single capability setup. Therefore, is it necessary to study this setup separately? It is recommended to clarify the differences between the single capability setup and the one proposed in this paper.

2. The paper does not clearly define the diversity of the weak dataset. It is mentioned that the diversity of a weak dataset can lead to better results as previous works. The experiment the third chapter focuses on the consistency of results generated by weak and strong models, which is typically not an explicit measure of diversity. 

3. The paper lacks ablation experiments. It only presents the performance without using the reward model for data selection. It is suggested to conduct experiments by randomly selecting the same amount of data as picked by the reward model (as random selection is a strong baseline in LLM data selection).

4. The legends in the figures are unclear, and some contain labeling errors. In the first three sections of Chapter 3, "weak model" is used as the axis label, yet most of the time, the data shown is based on the performance of the strong model, causing confusion. Additionally, there are errors in the numbering of Figures 3 and 4 in Section 3.3. Besides, there is inconsistent usage of "weak to strong generalization" and "weak-to-strong generalization."

questions:
1. What is the connection between the conclusion regarding the independence of different capabilities mentioned in Weakness 1 and the subsequent methodology and experiments?

2. In the scaling experiments, it is observed that as model size increases, the difference between the proposed method and the baseline becomes smaller. Considering the reward model's size is consistent with that of the strong model and the data used for the reward model is from the weak and strong model training, does the proposed method have certain limitations? (In simple terms, is the reward model scalable?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1731058718869

thread_type:
official_review

<thread_object>
creator: reviewer_1qfu

<content>
summary:
This paper first conducts a series of analyses to explore the factors influencing the spectrum from weak to strong generalization in large language models (LLMs) with multi-capabilities. It then proposes a two-stage training framework that uses reward models to select valuable data based on the insights gained from the analyses. Experimental results demonstrate improvements in the multi-capabilities of the LLMs.

strengths:
- The paper is well-written and easy to understand. Moreover, the structure is complete and coherent.
- Section 3 provides valuable insights, and the proposed method makes sense.

weaknesses:
- The setups of "WTS-S" and "WTS" are not clearly explained upon their first mention. It appears that "WTS-S" involves training different models for different datasets and computing their average score, while "WTS" involves training all datasets together in a single model.
- In Sections 3.3 and 3.4, it would be better to reference the setups under "WTS" rather than "WTS-S".
- What is your setup for the "mix data" in Section 3.6? Are the datasets directly merged? There is concern about potential conflicting cases in the "mix data" that could confuse the model. The total number of correct cases in "mix data" should be larger than "weak data". So I am doubtful about the conclusion regarding Figure 7 (right) because it seems a little contradictory to Figure 7 (left). Additionally, "mix data" should be better termed as "mixed data".
- The performance advantages of "WTS" in Figure 8 are not obvious when the weak model is scaled up. However, the gap to the strong model remains significantly large.
- I doubt the results regarding "the dependence of data quality across various capabilities" as claimed in lines 191-192. In Figure 4, there are only two cases: "w/o GSM8k" and "w/o ECQA & e-SNLI". While these can provide some insight, they are not comprehensive. If, as the conclusion suggests, the abilities in each capability are independent of each other, then why are the mixed training results of multi-capability data better than the single-capability averages?
- **[Important]** Despite the focus on multi-capabilities in the title, I would prefer to see the performance in single capabilities and comparison results with existing methods. It seems that most of the analysis and the proposed method could also apply to single capabilities, which are more commonly used. This paper does not convincingly demonstrate the necessity of working on multiple capabilities.

questions:
- In line 132, there is a missing comma. Additionally, it would be better to explain the difference between your work and related work in the related work section.
- In Figure 1 (right), it would be helpful to annotate that the performance pertains to the strong model.
- In line 186, "Figure 4" should be "Figure 3". In line 191, "Figure 3" should be "Figure 4". Furthermore, in Figures 3 and 4, it would be better to keep the colors of the "weaken" and "remove" bars consistent.
- In Section 3.5, the experiments show that the same strong model cannot improve itself. However, I am interested in the outcomes when training another weak model or a smaller but strong model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732692833207

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Feedback Before Discussion Deadline

comment:
Dear Reviewers and Area Chair,

As the discussion deadline approaches, we kindly request all reviewers to review our responses to your comments and share any remaining concerns or questions. We highly value your feedback and want to ensure that all issues are fully addressed before the final decision.

We have carefully responded to all of the concerns raised during the review and discussion phases by adding additional experiments and explanations. We hope that these clarifications might lead to a reconsideration of the current score.

We sincerely appreciate your valuable time and effort in reviewing our submission.

Best regards,

The authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733130845310

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Feedback Before Extended Discussion Deadline

comment:
Dear Reviewers and Area Chair,

As the extended discussion deadline is fast approaching, we kindly request all reviewers to provide feedback regarding our responses to your comments. We have carefully addressed all concerns raised in the review, including conducting additional experiments and providing more detailed explanations. We are hopeful that these updates will prompt a reconsideration of the current rating. Your feedback is immensely valuable to us.

Thank you again for your time and effort in reviewing our work.

Best regards,  

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733198329076

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers and Revision Submitted

comment:
Dear all reviewers and ACs,

Thank you very much for your valuable time in providing these constructive comments and suggestions. We have addressed each of the comments and misunderstanding by adding more experiments or explanations.

### We also thank the reviewers for identifying the strengths of our work below:
1. Reviewer 1qFU highlights Section 3 (analysis of multi-capabilities weak to strong generalization) in our study provides valuable insights, and the proposed method makes sense.
2. Reviewer r2a7 appreciates our work analyzing the current gaps in research on the weak to strong generalization paradigm from a "super-alignment" perspective and providing a novel training approach under this setup.
3. Reviewer 5nVp highlights our work provides relevant conclusions for multi-capabilities weak-to-strong generalization and explores data selection directions for weak-to-strong generalization.
4. Reviewer fdJr praises our research for its novelty and significance, highlighting valuable insights into multi-capability alignment under super-alignment. The writing is commended for its rigor and depth, and the proposed two-stage approach is recognized for effectively addressing key challenges, supported by solid theory and experimental validation.
5. Reviewer orGJ appreciates our insight into how weak supervision affects strong models, revealing their limitations. The proposed framework, using reward models to select valuable data, is innovative and addresses overconfidence while improving generalization. The paper is clearly presented and easy to understand.


### We have revised the paper to address the reviewers’ concerns. Below we summarize the major revisions (the main revisions are marked with blue text in the pdf, we also made some minor layout changes to fit the page limit), while we reply to the comments of each reviewer separately. The major revisions are:


1. We have further clarified the content in the figures, the details of the multi-capability setting, and the experimental setup (Reviewers 1qFU, r2a7, 5nVp, fdJr, orGJ).
2. We have added ablation experiments (randomly selecting datasets) to verify the effectiveness of our reward model (Reviewer r2a7).
3. Our method can also be applied to single-capability weak-to-strong generalization, achieving results comparable to those in the multi-capability setting. Moreover, it outperforms the state-of-the-art weak-to-strong generalization method (CSL) [1]. We hope these results address the concerns raised by Reviewer 1qFU.
4. We compared our method with an untrained strong model with in-context learning, demonstrating the significance of studying weak-to-strong generalization (Reviewer 5nVp).

We report a simplified version of some of the added results in the table below, and the detailed complete results are in Appendix of the paper.

1. Our Method for Single-Capability and Multi-Capabilities Weak to Strong Generalization
| Weak Model | Strong Model | Ours (single-capability averages) | Ours |
|------------|--------------|-----------------------------------|------|
| 0.5B       | 1.8B         | 59.18                             | 59.17 |
| 1.8B       | 4B           | 69.04                             | 69.05 |
| 4B         | 7B           | 77.06                             | 77.07 |


2. Comparison with SOTA (CSL) [1]
| Weak Model | Strong Model | CSL  | Ours |
|------------|--------------|------|------|
| 0.5B       | 1.8B         | 57.84| 59.17 |
| 1.8B       | 4B           | 68.62| 69.05 |
| 4B         | 7B           | 76.91| 77.07 |

3. Ablation Study
| Weak Model | Strong Model | Random | Ours |
|------------|--------------|--------|------|
| 0.5B       | 1.8B         | 52.49  | 59.17 |
| 1.8B       | 4B           | 63.52  | 69.05 |
| 4B         | 7B           | 74.31  | 77.07 |


4. Comparison with Untrained Strong Model (w/ In-Context Learning)
| Weak Model | Strong Model | Ours  | ICL   |
|------------|--------------|-------|-------|
| 0.5B       | 1.8B         | 59.19 | 45.92 |
| 1.8B       | 4B           | 69.07 | 58.38 |
| 4B         | 7B           | 77.06 | 70.02 |

[1] Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the superalignment challenge in LLMs by exploring the weak to strong generalization with multi-capabilities. The authors propose a novel training framework using reward models to select valuable data for weak supervision and introduce a two-stage training method that enhances model performance. The paper is well-written, providing valuable insights into the multi-capability generalization and presenting experimental results that demonstrate improvements over baseline methods.

The reviewers raised some questions about the clarity of certain setups, such as the "WTS-S" and "WTS" configurations, and the handling of mixed data which could potentially introduce conflicting cases. There were also concerns about the necessity of focusing on multi-capabilities when the results could potentially apply to single capabilities as well. During the rebuttal, the authors addressed these issues by clarifying their experimental setups and providing additional justifications for the focus on multi-capabilities. The reviewers were generally satisfied with the authors' responses.