Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

Authors: Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, Rui Wang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: weak-to-strong, model alignment

Abstract:
Aligning language models (LMs) with human preferences has become a key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by a weaker model can consistently outperform its weak supervisor, we extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models can be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak-to-Strong Preference Optimization (WSPO), which achieves strong model alignment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding performance, improving the win rate of Qwen2-7B-Instruct on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlled win rate on AlpacaEval 2, and scoring 7.33 on MT-bench. Our results suggest that using the weak model to elicit a strong model with a high alignment ability is feasible. The code is available at https://github.com/zwhong714/weak-to-strong-preference-optimization.

Decision: accept (spotlight)
Reviewer scores: 5, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730058758885

thread_type:
official_review

<thread_object>
creator: reviewer_ssa6

<content>
summary:
This paper introduces Weak-to-Strong Preference Optimization (WSPO), a novel method for aligning large language models (LLMs) with human preferences. Inspired by the concept of weak-to-strong generalization, WSPO leverages distribution shifts observed during the alignment of weaker models to guide the fine-tuning of stronger models.

strengths:
WSPO is broadly applicable to Reinforcement Learning with Human Feedback (RLHF) tasks, and the benchmark results validate its effectiveness. Remarkably, even though a weaker model like Qwen2 performs poorly on its own (as shown in Table 1), the distribution changes induced by reference and DPO-tuned models provide meaningful guidance. This enables the stronger model to achieve significant performance improvements, demonstrating the method's potential to enhance model alignment despite the initial limitations of weaker models.

weaknesses:
1. All experiments are conducted using the Qwen2 model family, and it would strengthen the study to include results from other open-source LLMs, (e.g., Llama3). Additionally, incorporating a Proximal Policy Optimization (PPO) baseline in Table 1 would provide a more comprehensive comparison.

2. While the proposed method and experiments are innovative and comprehensive from my perspective, the manuscript requires significant improvements in clarity and organization. Specifically, in Section 3.2, the process for obtaining $\pi_{r}^{weak}$ and $\pi_{ref}^{weak}$ should be explicitly detailed, as currently, **readers have to refer to the appendix** to understand their derivation through Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), and for the summarization task, via PPO.

3. Furthermore, the methods section needs clearer explanations of each notation upon the first introduction. The experimental section also lacks essential details and attempts to cover three sets of experiments simultaneously. To enhance readability and focus, it is recommended that Section 4.3 be emphasized with more comprehensive details, while at least one of the experiments in Sections 4.1 and 4.2 could be relocated to the appendix to streamline the presentation.

questions:
1. The MMLU score (shown in Table 2) seems very low for QWen2 model, can you explain why it’s in the range of 40s? 

2. Do you have experiment results for other model families?

3. You showed that alignment from an even weaker model like Qwen2-0.5B can be helpful. I am curious about what happens if the weaker model is not weak anymore (e.g. use 7B model’s SFT and DPO checkpoint to compute the ratio as if they are pi_{r}^{weak} and \pi_{ref}^{weak}).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484976674

thread_type:
official_review

<thread_object>
creator: reviewer_lzug

<content>
summary:
The paper introduces a novel method called Weak-to-Strong Preference Optimization (WSPO), which aims to align large language models (LLMs) with human preferences by leveraging the alignment signals of weaker models. The proposed method builds on the concept of weak-to-strong generalization, where a strong model can amplify the alignment capability learned from a weaker model. WSPO works by learning the distributional differences before and after the alignment of a weaker model, which enables a stronger model to effectively inherit and improve upon the alignment behavior of the weaker one. The authors demonstrate that WSPO improves the win rate and alignment quality of the Qwen2-7B model in several benchmarks, including Arena-Hard, AlpacaEval 2, and MT-bench.

strengths:
Novelty: WSPO is a unique approach that effectively transfers alignment signals from weak to strong models, providing an alternative to reward model training. This is a valuable addition to existing alignment techniques such as RLHF and DPO.

Quality: The experimental results demonstrate that WSPO performs competitively with strong models trained using PPO, and in some cases even surpasses it in alignment capabilities.

Clarity: The paper provides a well-organized discussion of prior work, situating WSPO within the broader context of model alignment. The use of diagrams to illustrate alignment processes helps in understanding the proposed method.

Significance: WSPO has practical significance for reducing the computational cost of aligning strong models, as it eliminates the need for repeated reward model training.

weaknesses:
Complexity of Theoretical Derivations: The theoretical sections, particularly those related to deriving the WSPO objective and gradient, are quite dense and might be challenging for readers without a strong background in reinforcement learning. Consider providing more intuitive explanations or additional examples.

questions:
1. The method relies on aligning the weak model first and then transferring this alignment to a stronger model. Are there scenarios where aligning the strong model directly would be more beneficial, and how does WSPO compare in terms of resource efficiency in such cases?
2. Is WSPO applicable to non-language model tasks, such as reinforcement learning agents or vision-language tasks? It would be interesting to understand the generalizability of the approach beyond language models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495116659

thread_type:
official_review

<thread_object>
creator: reviewer_opbs

<content>
summary:
The paper presents a novel approach called Weak-to-Strong Preference Optimization (WSPO), designed to enhance model alignment by leveraging alignment signals from weaker models to improve stronger ones. Inspired by weak-to-strong generalization, WSPO transfers the alignment abilities of a weaker model to a stronger one by optimizing the differences in distributions before and after weak model alignment. WSPO demonstrated competitive results across various benchmarks, showing a significant performance improvement over existing approaches, such as Direct Preference Optimization (DPO).

strengths:
1. **Innovation in Model Alignment**: WSPO introduces a unique method of transferring alignment signals from weak to strong models, showcasing effective model alignment without the need for a dedicated reward model.
2. **Experimental Validation**: The paper provides thorough experimental evidence across several tasks, including summarization, single-turn dialogue, and complex benchmarks like MT-Bench and AlpacaEval 2. The results demonstrate WSPO's strong performance relative to baselines like DPO.
3. **Efficiency**: WSPO presents a more streamlined alignment process, avoiding the costly RLHF approach. This approach may offer practical advantages for resource-constrained settings, as the alignment signal is derived directly from weaker models.
4. **Comprehensive Analysis**: The authors conduct various analyses, including the impact of model size and the role of key hyperparameters, providing insights into the approach's scalability and stability.

weaknesses:
1. **Limited Cross-Model Analysis**: While the paper demonstrates the impact of weak-to-strong transfer within a specific model family, it lacks analysis of WSPO’s effectiveness across diverse architectures.
2. **Lack of Theoretical Explanation**: Although the approach works empirically, the authors do not provide a detailed theoretical rationale for why transferring alignment from weak to strong models amplifies alignment effectiveness.
3. **Reliance on Specific Datasets**: The experimental results heavily rely on specific datasets (UltraChat, Anthropic HH), potentially limiting the generalizability of WSPO to other alignment contexts. Broader dataset testing could validate the approach further.
4. **Parameter Sensitivity**: The method’s effectiveness appears sensitive to hyperparameter tuning (e.g., γ in WSPO), which might complicate deployment across different model types and alignment scenarios.

questions:
1. Have you tested WSPO across different model architectures or families beyond Qwen? If so, could you provide insights into its effectiveness across various architectures?

2. Could you explain why the alignment capability amplifies when transferred from a weak model to a strong model? Are there theoretical or empirical insights that could clarify this phenomenon?

3. Given WSPO's reliance on certain hyperparameters (e.g., γ), do you have any practical guidelines for setting these across different model sizes or tasks? Could automatic tuning methods be integrated to simplify deployment?

4. Could you provide further details on WSPO’s computational efficiency, particularly in comparison to RLHF in real-world applications? How significant are the resource savings, and are there trade-offs in alignment quality for complex, large-scale tasks?

5. How critical is the quality of the initial alignment in the weak model to WSPO's success? Would a weak model with suboptimal alignment still provide useful signals for the strong model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798167621

thread_type:
official_review

<thread_object>
creator: reviewer_4ugn

<content>
summary:
The paper presents Weak-to-Strong Preference Optimization (WSPO), a pioneering approach for aligning language models (LMs) with human preferences by leveraging the alignment capabilities of weaker models to enhance stronger ones. Through learning distributional differences before and after the alignment of the weaker model, WSPO effectively transfers and amplifies alignment behaviors, demonstrating improved performance on key benchmarks such as Arena-Hard, AlpacaEval 2, and MT-bench.

strengths:
1. Innovative Transfer of Alignment Capability: WSPO innovatively transfers alignment capabilities from a weaker model to a stronger one, which is a novel approach in the field of LM alignment.
2. Performance Improvement: The paper shows significant improvements in win rates on challenging benchmarks, indicating the effectiveness of WSPO in aligning stronger models.
3. Empirical Evidence: The paper provides extensive experimental results, demonstrating WSPO's effectiveness across various tasks and datasets, including summarization, dialogue, and reasoning tasks.
4. Generalization Beyond Weak Labels: WSPO not only imitates weak model behavior but also generalizes beyond it, as shown by the strong model's improved performance on benchmarks.

weaknesses:
1. Dependence on Quality of Weak Model Alignment: The success of WSPO is heavily dependent on the quality of alignment in the weak model, which might not always be optimal.
2. Hyperparameter Sensitivity: The performance of WSPO seems to be sensitive to the choice of hyperparameters like γ, which might require careful tuning for different models and tasks.
3. Efficiency Considerations: During WSPO training, the algorithm requires the loading of parameters from four models, two of which are weak models with limited parameters. The paper omits a comparison with previously established efficient methods like SimPO, which could potentially deliver better performance and higher training efficiency.

questions:
see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732760700528

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Revision Summary

comment:
**We greatly appreciate the reviewers' thoughtful, high-quality review of our work.**  

**All reviewers** found our work **innovative**  and **novel**. Reviewer `4UGn` described it as "**pioneering,**" while Reviewers `oPbS` and `LZuG` described it as "**unique.**"

As the revision deadline approaches, we present the following summary of the key updates. 

------

**Revisions to Address Q&W:**

1. **[Content Highlighted in Red]** Improved clarity of notation and simplified the method description (`addressing Reviewers LZuG and ssa6`).
2. **[Experiment 4.3.2]** Added empirical insights into alignment enhancement (`addressing Reviewer oPbS`).
3. **[Analysis 5.1]** Demonstrated that suboptimal weak model can still provide alignment signal (`addressing Reviewers 4UGn, oPbS`).
4. **[Analysis 5.2, Appendix D]** Provided guidelines for hyperparameter tuning (`addressing Reviewers 4UGn, oPbS`).
5. **[Appendix B]** Included experiments with Llama families, further showing the effectiveness of WSPO (`addressing Reviewers oPbS, ssa6`).
6. **[Appendix C]** Analyzed the impact of the dataset, demonstrating that WSPO focuses on learning distributional differences rather than relying on a specific dataset (`addressing Reviewer oPbS`).
7. **[Appendix D]** Evaluated equivalent models as weak models, showing WSPO's role in adjusting alignment strength (`addressing Reviewer ssa6`).
8. **[Appendix E]** Extended WSPO to vision-language tasks, showing its applicability beyond language models (`addressing Reviewer LZuG`).
9. **[Appendix F]** Added an efficiency comparison with SimPO and RLHF (`addressing Reviewers 4UGn, oPbS`).

We sincerely hope these **comprehensive updates** meet the reviewers' expectations and look forward to receiving positive feedback in the remaining discussion period.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new approach to alignment where the larger model's reward is regressed against the smaller model's reward, where both the reward models are parameterized as a ratio of the policy and the sft model (i.e., as in DPO). The authors find that this approach can result in stronger aligned models than in regular DPO.

On the positive side, this paper presents an interesting approach to alignment that is coupled with some surprising empirical results. (I was genuinely surprised to see this approach outperform a DPO-aligned stronger model.) On the negative side, this approach introduces additional overhead (since the smaller models need to be loaded), and is missing obvious baselines (e.g., PPO with the smaller model as the reward). I also think there is a substantial ensembling effect happening with this model, which is not discussed by the authors (e.g., I bet an ensemble of the strong-DPO and weak-DPO model will do quite well).

But given the strong(ish) empirical results, I think this paper should be accepted.