Weighted-Reward Preference Optimization for Implicit Model Fusion

Authors: Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, Xiaojun Quan
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Model Fusion, Preference Optimization, Large Language Models

Abstract:
While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9\% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2\% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730108522284

thread_type:
official_review

<thread_object>
creator: reviewer_duir

<content>
summary:
The paper investigates the progress of fusing heterogeneous open-source LLMs with varying architecture and size. They then summarise the challenges in this scenario, including vocabulary alignment and merging distribution matrices. To solve these challenges, they propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which eliminates the need for vocabulary alignment and matrix distribution and improves the performance on multiple benchmarks.

strengths:
1. The paper explores the direction of improving the capabilities of individual models by combining the strength of multiple LLMs, which has potential benefits to improve the ability of individual models.
2.  They propose a novel implicit fusion method that eliminates the need for vocabulary alignment and matrix fusion.
3. Extensive experiments demonstrate the effectiveness of proposed methods in multiple aspects.

weaknesses:
1. Although the proposed methods demonstrate a certain degree of improvement on AlpacaEval-2 and Arena-Hard, they only have weak influences in MT-Bench. This weakens the generalization of the proposed methods.
2. The object of the proposed WRPO is to increase the likelihood of a preferred response while decreasing the occurrence of the dispreferred response. Preferred responses come from source and target models and dispreferred responses only come from the source model, which means dispreferred responses from the source model will not influence training. Whether some potential small experiments that support this phenomenon as it is intuitively we not only need to learn recognition capability in correct data but also learn exclusion ability.
3. Although the experiment section provides some interpretation, it lacks an in-depth analysis of the rationale behind the proposed phenomenon, such as, how WRPO achieves a balance between effectiveness and efficiency.
4. They lack the discussion of limitations and future work.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730526171857

thread_type:
official_review

<thread_object>
creator: reviewer_42p3

<content>
summary:
This paper introduces Weighted-Reward Preference Optimization (WRPO), an implicit model fusion approach for large language models (LLMs) that bypasses the need for complex vocabulary alignment and distribution merging found in traditional fusion methods. WRPO uses a weighted reward mechanism and a progressive adaptation strategy to smoothly transfer preference weights from the target model to source models. Benchmark results on AlpacaEval-2 and MT-Bench show that WRPO consistently outperforms traditional fusion and preference optimization methods, marking it as a promising advancement in LLM model fusion.

strengths:
1. WRPO effectively introduces implicit model fusion, bypassing the need for complex vocabulary alignment and distribution merging, a significant advancement over traditional methods.
2. The novel reward mechanism balances contributions from source and target models and helps mitigate distributional discrepancies, leading to a smoother optimization process.

weaknesses:
1. The success of WRPO appears to depend heavily on the choice and quality of source models, yet the paper does not fully address criteria or strategies for selecting these source models, which could be a limiting factor in practice.
2. The need for dynamic tuning of the fusion coefficient introduces complexity, and the paper does not sufficiently detail how this parameter was optimized across different datasets and tasks.
3. While WRPO is more efficient than traditional methods, it still requires significant computational resources for training and fine-tuning, especially when handling multiple large LLMs, which may limit its accessibility
4. Although WRPO is presented as a robust method, more ablation studies could have provided a clearer understanding of the specific impact of each component, particularly the fusion coefficient and its progressive adjustment.

questions:
1. How sensitive is WRPO to the selection of source LLMs? Could varying the quality and diversity of source models affect the final performance, and if so, what are the selection criteria?
2. What are the specific criteria or methods used for tuning the fusion coefficient? How was it determined for different tasks, and does it require extensive hyperparameter optimization?
3. Could WRPO be applied to multilingual or cross-domain tasks where distributional shifts might be more pronounced, and if so, are any modifications required?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730575741905

thread_type:
official_review

<thread_object>
creator: reviewer_mcsp

<content>
summary:
This paper aims to align the LLM prediction from multiple source models to a target model. Specifically, it focuses on fuse multiple heterogeneous LLMs and combine their preference into the target one. Technically, it proposes a weighted-reward strategy to implicitly fuse the source models. Experiments on several evaluation benchmarks show the method effectiveness.

strengths:
1. LLM alignment is popular and valuable research topic, especially aligning heterogenous models into a target one.
2. The proposed weighted-reward is a straightforward to to achieve an implicit model fusion process.
3. Comprehensive evaluation supports the paper statement with rich analysis.

weaknesses:
1. Adding an analysis of alignment efficiency and ablation will be helpful, what is the difference of using different source models? if multiple source models are used, will it increase the alignment cost?
2. Some analysis parts can be more informative such as figure 3 and 4. Including more analysis and make them dense will further enhance the draft.

questions:
Please refer to the weaknesses section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732349031568

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking Forward to the Opportunity for Further Discussion

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you have devoted to providing thoughtful reviews and valuable feedback. We have carefully addressed your concerns in the following ways:

- Clarified misunderstandings about WRPO's computational efficiency and implementation details.
- Conducted new experiments to analyze the impact and combinations of source models.
- Expanded our analyses with additional ablation studies and detailed explanations.

We hope these revisions and discussions have adequately addressed your concerns. As the Author-Reviewer discussion phase is ending soon, we would be grateful for any additional comments or questions that could further enhance our work.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732720865415

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Major Revisions

comment:
Dear Reviewers,

We sincerely appreciate your thorough review and valuable feedback, which has significantly enhanced the quality of our manuscript. We have carefully addressed all the comments and suggestions through comprehensive revisions. Below, we outline the major changes made to the manuscript (with key revisions highlighted in ${\color{blue} blue}$ text in the PDF, alongside other refinements to fit the page limit).

> **Key Revisions**

1. We have revised our experimental analysis in **Section 4** (Line 414-417, Line 463-519, Figure 3). (Reviewer **McSp, dUiR**)

2. We have added a training cost analysis of WRPO in **Appendix C**, demonstrating that increasing the number of source LLMs does not raise the time complexity of our method during training. (Reviewer **McSp, 42p3**)

3. We have elaborated on the selection criteria of source LLMs in **Appendix G** and conducted additional experiments to discover the impact of different source LLM combinations in **Appendix D**. (Reviewer **McSp, 42p3**)

4. We have detailed the dynamic tuning strategy for the fusion coefficient and presented ablation experiments comparing it to the static strategy in **Appendix E**. (Reviewer **42p3**)

5. We have investigated the impact of incorporating extra dispreferred responses from the source models in **Appendix F**. (Reviewer **dUiR**)

6. We have added a discussion on limitations and future work in **Appendix H**. (Reviewer **dUiR**)

7. We relocated Training Details from Section 4.1 to Appendix A.1 to fit the page limit. 

We are deeply grateful for your insightful feedback, which has been instrumental in strengthening our work. We hope these revisions and additional analyses thoroughly address all raised concerns. As we approach the conclusion of the Author-Reviewer discussion phase, we welcome any further suggestions that could help us further improve the manuscript.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an implicit model fusion that leverages preference optimization between two LLMs for knowledge transfer. The reviewers are generally satisfied by the current work in the final stage. The AC has checked all files and agreed to reviewers. The authors shall take review suggestions for revision in the camera-ready version.