Weighted-Reward Preference Optimization for Implicit Model Fusion
Authors: Ziyi Yang
Fanqi Wan
Longguang Zhong
Tianyuan Shi
Xiaojun Quan
Venue: ICLR 2025 Poster (2025)
Keywords: Model Fusion
Preference Optimization
Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: In this work, we propose a new framework for implicit model fusion through preference optimization.

Abstract: While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9\% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2\% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.

Published as a conference paper at ICLR 2025
WEIGHTED-REWARD PREFERENCE OPTIMIZATION FOR IMPLICIT MODEL FUSION
Ziyi Yang" Fanqi Wan" Longguang Zhong Tianyuan Shi Xiaojun Quan School Computer Science and Engineering, Sun Yat-sen University, China yangzy3 94 mai 12. sysu. edu. en, quanxj3ema 1. sysu. edu.cn
ABSTRACT
While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. this paper, we propose an implicit fusion method, Weighted-
RewghttedRReward Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capa- bilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces progressive adaptation strategy that gradually shifts reliance on preferred examples from the target L.LM to the source L.LMs.. Extensive experiments on the MT-Beech, AlpacaEval-22 2,, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning basclines. When applied to Llama-3-8B 101-8--IInseee as the target model, WRPO achieves a length- controlled win rate of 55. 999% against GPT-4-Previeww 106 on AlpacaEval-22 and a win rate of 46.2% against GPT-4-0314 on Arena- Hard. Our code is available at
/ttps::/gitte com Bt/P:/SSSTTAAAAPAAA AI/WRPO
INTRODUCTION
Combining the strengths of multiple large language models (LLMs) can potentially enhance the capabilities of individual models. Model ensemble techniques (Jiang et al., 2023b Wang et al. 2024b) aggregate predictions from several models to improve overall performance and robustness over a single model. However, this approach requires substantial computational resources, as all models must remain active during inference. The Mixture of Experts (MoE) (Komatsuzaki al.) 2023 Feng et al. 20244, Sukkhaataar et al. 2024) leverages sparse expert networks to boost capacity by activating only subset of parameters. Despite reduced activation, MoEs still incur significant memory overhead, as all parameters must be maintained. Model merging (Wortsman et al.) 2022 Matena & Raffel 2022 Yadav et al. 2023), which combines independently trained instances of the same model through arithmetic operations, allows a single model to be maintained during inference. While more efficient, this method is restricted to models with identical architectures and sizes.
Another approach is to fuse these LLMs into a target model through multi-teacher knowledge distillation (Wan et al. 2024a by Shi al. 2024). Unlike traditional knowledge distillation (Gou el al. 202I), which usually leverages diverse sources (e.g., logits, features, and relations) of knowledge from teacher models, this method relies exclusively on the probabilistic distribution matrices generated by source L.MS transfer knowledge the target model. We refer to this method explicit model fusioon (EMF) because it involves a well-defined knowledge transfer process. While applicable to heterogeneous models with varying architectures and sizes, and without increasing memory overhead during inference, this approach presents notable challenges such as vocabulary alignment and the merging of distribution matrices from different LLMs. These issues complicate model fusion, reduce its efficiency, and may introduce noise and errors and affect the fusion results.
Contributed equally. Corresponding author.

---

Published as a conference paper at ICLR 2025
0.8
Rate (7%) avee GP GPT.4
50-
Langlh-Cattelled
48.2 47,5
Raw
0.6
1.0
40-
Density
0.4
30
25.3
25,9 26.1
0.5
20-
0.0
0.0
10
-3 Avg. log-prob on
Av
Avg. log-prob on Aveaual
Modela
Source L.Mss
Trrget
(a) Original
(b) After DPO
(c) Results on AlpacaEval-2
Figure 1: Distribution deviations between responses from heterogeneous source LLMs and the Llama-3-8D Instruct target LLM before (a) and after (b) DPO fine-tuning, with the prompts from Ultraeedback (Cui et al.) 2024) as input. Subfigure (c) shows the results (TDr of preference optimization with this deviated preference dataset, compared to the results (Te) from dircetly applying the target model and those (TD from DPO fine-tuning on on-deviated preference data sampled from the target model.
This work aims to enhance the capabilities of a single LLM by implicitly learning from robust open- source LLMs, a process we term implicit model fusion (IMF). The concept of IMF has been widely utilized to improve the performance of weaker models. For instance, a weak model can be boosted
through fine-tuning with outputs from stronger LLMs (Ranaldi & Freitas 2024 Tian al. 2024
Kang et al. 2023). Morcover, a reward model can be trained using outputs from various LLMs (Cui
al., 2024 Zhu et 2004a), enabling it learn and capture the differences in capabilities between
the LLMs. Zephyr (Tunstall et al. 2023) further collects responses from multiple LLMs and ranks them with GPT-4 to obtain preference data for training the policy using DPO. One advantage of
IMF over EMF (Wan et al. 2024a b) Shi et al. 2024) is that it eliminates the need for challenging alignment of vocabularies and fusion of distributions among different L.LMs. Inspired by recent alignment techniques such as Direct Preference Optimization (DPO) (Rafailov et al.) 2023) and Simple Preference Optimization (SimPO) (Meng et al. 2024). we propose a novel IMF method to transfer the capabilities of source LLMs to target LLM through preference optimization. However, directly applying preference learning to outputs from heterogeneous LLMs presents challenges. Previous works have shown that DPO is highly sensitive distribution shifts between the policy model and the preference data (Xu et al. [024bb] Tajwar et al. 2024: Zhou et al., 2024), and training policy model on this preference data can lead to sub-optimal performance.
To demonstrate this, we conduct a preliminary experiment on the UltraFeedhack dataset (Cui et al. 2024), using Jlama-3-8B-Instruue (Dubey al. 2024) the target model and 10 strong open-source LLMs as source models| ! For each prompt, we first ask each source model to generate several responses and use the ArmoRM reward model (Wang et al., 2024a) to select the highest-reward response among all source LLMs as the preferred response, with the dispreferred response coming from the target LLM's completions. Figure aa] visualizes the average pogobabbbiyltyy distribution of the target LLM wo for both response types, which reveals a significant deviation between the distributions of the source and target models. Although applying DPO directly on this deviated dataset marginally enhances the aog-robabbllitiss of source LLMs" responses relative to those of the target LLM, as shown in Figure [bb), this results in sub-optimal performance compared to sampling both response types exclusively from the target LLM, illustrated in Figure I[c..
To address the distributional deviations during implicit model fusion, we introduce a novel approach called Weighted- -eeghteedRReardd Reward Preference Optimization (WRPO). Instead of directly relying on the source LLMs to provide preferred responses, we propose a progressive adaptation strategy that begins with the target L.M providing preferred responses and gradually shifts this responsibility to source LLMs. Specifically, this progressive adaptation is implemented in two stages. First, for each prompt .r, we construct preference quadruple (z, yee. Miee x 31). where Yu. a preferred response generated by the source LLMs, and yw, and Yt are preferred and dispreferred responses, respectively. from the target LLM. Second, we gradually decrease the weight of internal reward for yue, and increase the weight
1 Refer to Section 4.1 for more details.
"W use "internal reward" to refer to the reward generated during preference optimization for preferred or dispreferred responses, in contrast to the reward provided by external reward model.
2

---

Published as a conference paper at ICLR 2025
for Vie during preference optimization. This smoothing process facilitates the integration of strengths from the source models into the target model while mitigating the distributional discrepancies.
To assess the effectiivnnes of WRPO in implicit model fusion, select 10 prominent open-source LLMs as the source models, with parameter sizes ranging from 9B to 236B. We chose Llama-3-88 Instruct (Dubey al., 2024) the target model due to its strong performance relative to its size. Our experiments are conducted on three widely-used instruction folllooinnn benchmarks, namely, MT-Bench (Zhnng et al. 2023), AlpacaEval-2 (Li et 2023), and Arena-Hard (Li et al. 2024). The results show that WRPO consistently outperforms existing fusion methods and various baselines. This highlights its ability to allow a model to implicitly learn from the diverse capabilities of heterogeneous LLMs while addressing distributional shifts. Notably, the fused model, Llama-3-8P Saama-3-8B-Instiee Instruct-t WRRO. surpasses all source models on AlpacaEval-2 with length-controlled win rate of 55.9%.
2 RELATED WORK
Collective LLMS Given that LLMs are trained with various architectures and sizes on different datasets, it is reasonable to assume they possess unique capabilities and strengths. Therefore, leveraging the distinct advantages of different LLMs becomes a natural approach to developing more robust and capabe models. Recent studies have increasingly emphasized the development of collective LLMs through the integration of diverse heterogeneous models.
LLM-Blender (Jiang et al., 2023b) presents an ensemble framework that first employs pairwise ranking mechanism to identity the top- outputs generated by different L.Mss. These selected outputs are then refined by a seq2seq model to produce enhanced results. Mixture-of-Agents (MoA) (Wang) it al. 2024b) utilizes a hierarchical structure where each layer consists of multiple LLM agents. The outputs from a previous layer are concatenated and refined by each agent in the subsequent layer. However, this approach significantly increases the number of LLMs needed during inference. In addition to the sequence-level ensemble, Xu et al. (2024c) explored a token-level ensemble method that aggregates the distributions of LLMs at each decoding step through global alignment matrix. Similarly, P LMs (Mavromatis et al., 2024) conducts distribution ensembling during inference utilizing sequence-level weights derived from the perplexity of each LLM on the input.
FuseLLM (Wan et al. 2024a) and FuseChat (Wan et al. 2024b) aim to fuse LLMs of various architectures and sizes into more robust model through multi-teacher knowledge distillation. They start by aligning the vocabularies and probabilistic distributions of the source LLMs, followed by merging their distributions and continuously fine-tuning the target LLM. ProFuser (Shi et al.) 2024) goes further by integrating both training mode (through cross-entropy loss) and inference mode (via model outputs), which provides more comprehensive understanding of the capabilities of source LLMs. Although applicable to models with varying architectures and sizes, these methods face challenges such as vocabulary alignment and merging distribution matrices from different LLMs, which are complex and may also introduce noise and errors that affect the fusion results.
Direct Preference Optimization Aligning LLMs with human preferences is crucial for their success. Reinforcement learning from human feedback (RLHF) (Christiano al., 2017 Schulman et al. 2017 Ziegler et al., (2019) is a widely used approach to achieve this alignment. However, RLHF depends on complex reinforcement learning techniques such as Proximal Policy Optimization (PPO), which are challenging to implement and often unstable during training. To address these challenges, approaches such as SLiC-HF (Zhao al. 2023) and RRHF (Yuan et al.) (202)) replace reinforcement learning with a ranking loss on preference pairs to better align LLMs with human preferences, while also incorporating a regularization term based on reference responses. Similarly, DPO (Rafailov et al., 2023) directly optimizes the policy model by training the reward model on human preference data. In addition to providing more stable training, lower computational costs, and easier implementation, this approach ensures high-quality alignment with human preferences.
Subsequent research aims to address the potential limitations of DPO. For example, IPO (Azar et al. 2024) tackles the risk of overfitting by optimizing nonlinear preference function, thus avoiding the transformation of pairwise preferences into pointwise rewards. KTO (Ethayarajh et al. 2024) is based new alignment objective of human-awan loss (HALO), which maximizes the utility of generated outputs directly from a binary signal indicating whether the output is desirable, rather than maximizing the likelihood of preferences. CPO (Xu al. 2024a) and ORPO (Hong et al.) 2024)
3

---

Published as a conference paper at ICLR 2025
aim to eliminate the need for reference model by streamlining the optimization process, combining supervised fine-tuning (SFT) and preference alignment into a single step. R-DPO (Park et al. 2 introduces lengah--egulariizationn term into the DPO objective to mitigate length biases that may be exploited by DPO. Similarly, SimPO (Meng et al., 2024) revises the reward component in DPO to use the average log probability of positive negative responses from the policy model. Another motivation for this method is that the training process aligns more closely with inference.
However, none of the above works consider the hybrid scenario where one response is generated by the policy itself while the other comes from a different LLM. This situation may introduce serious distribution shifts relative to the policy, which in turn affects the policy's 's optimization. The work closely related to our setup is WPO (Zhou et al. (024), which assigns weights to off off-policy preference pairs based on their likelihood under the policy model. These weights indicate the degree of deviation from the policy's 's distribution and mitigate the influence of preference pairs with notable deviations.
3 METHOD
In this section, we begin with a problem statement for implicit model fusion, followed by the preliminaries of direct preference optimization (DPO) (Rafailov et al. 2023). Finally, we provide a detailed explanation of our proposed method,
Preference Optimization (WRPO).
3.1 PROBLEM STATEMENT
Previous works on model fusion primarily focus on transferring knowledge from various heteroge- neous LLMS into a unified model via multi-teacher knowledge distillation (Wan et al. 2024a] Shi et al. 2024). We refer to this method explicit fusion (EMF) because it involves dell-defined knowledge transfer process. As mentioned earlier, this approach requires complex alignment of vocabularies and merging of distribution matrices across heterogeneous L.MMs. In contrast, this work proposes implicit model fusion (IMF) to enhance the capabilities of a target L.MM by implicitly learning from the outputs of robust source LLMs, thereby bypassing the difficulties of vocabulary alignment and distribution fusion. Another advantage of IMF is that the source LLMs can be either open-source or proprictary; however, for comparison with previous fusion approaches, we focus on open- open-source LLMs. Inspired by recent alignment techniques like DPO (Rafailov et al. 2023) and SimPO (Meng et al. 2024), we propose implementing IMF through preference optimization.
For each prompt 2; in the training dataset D, we first sample N (e.g., 5= responses from each of the source L.LMs. Then, an external reward model is employed to identify the response with the highest reward score among all source models as preferred, denoted as Yw., Next, dispreferred response can be sampled from the target LLM. However, as illustrated in Figure [ significant deviations may exist between the distributions of the preferred and dispreferred responses, and directly applying preference optimization under these conditions could yield problematic results.
To address this issue, we propose progressive adaptation strategy. Specifically, we sample N responses from the target model and evaluate them using the reward model. The response with the highest score is labeled another preferred response Yur., while the lowest-scoring response is regarded as the dispreferred response yt- To tackle the challenges of distributional discrepancies and effectively utilize data from the source models, we introduce novel optimization objective called Brighted-Reward Preference Optimization (WRPO). As shown in Figure 2 this objective introduces a fusion coefficient that dynamically balances the internal reward of the preferred response yo, from source models and that of Yw, from the target during training. This approach enables the target LLM to transition smoothly from its distribution to align with that of the source LLMs.
3.2 PRELIMINARIES: DIRECT PREFERENCE OPTIMIZATION
Conventional alignment methods such as reinforcement learning from human feedback (RLHF) (Christiano et al.) 2017 Schulman et al. 2017 Ziegler al. 2019) often involve complex training pipelines that are unstable and resource-intensve (DPO) (Rafallov et al. 2023) provides a more efficient alternative by fine-tuning LLMs to align with human preferences through straightforward supervised learning objective using human-labeled preference data. DPO optimizes the policy to generate outputs that match human preferences without requiring explicit reward functions tndd--and-error updates. Specifically, DPO reformulates the
contrast, Direct Preference Optimization
4

---

Published as a conference paper at ICLR 2025
Source Models
Off poliyy
Responses:
Sample
Preferred
WRPO objective
LLM
Y
Rank
Sample
LIM
WRRPO Toi Tred)
Sample
LLM
log
r(a, Yar.)
()) r(r,, (hn.)
r(1,
Toly
where r(z, y)
Prompts
Reward Model
Preferred
y1
reduced
Target Model
Orypolicy
Rank
Target model
Sample
Dispreferred
Responses
training
After training
Data Construction
Preference Optimization
Figure 2: Overview of our proposed WRPO for implicit model fusion.
reward function to yield a closed-form solution for the optimal policy. Given the optimal policy x", the reparameterized form of the optimal reward function r" (xx y) is expressed as follows:
(t x)) Tirl x)
r" 3) = B log
+ log Z(x).
(1)
where (xx) is the partition function, कe denotes the reference policy. typically supervised fine-tuned (SFT) model, which also serves the starting point for the policy. Given human preference dataset D {xx, Ve: 21)" 21)"] N i=1' where Yhe and yI represent the preferred and dispreferred completions for prompt
x, the reparamcterized reward function r r'(x, y) is incorporated into the Bradley-Terry model (Bradley & Terry
1952 which yields the probability of preference between yoe and yı
(y xx) (ye x))
log (yc 2r)
(v > v xx)
log
Tee x))
The maximum likelihood objective for parameterized policy wo then:
Te (Ver a:)
mo (31
x)) xx)
Luvo Coeo(TTe vieet
E ao
log
B log
3 log
Teer y/
The preference dataset for DPO training can be sampled from the reference model sourced from publicly available data. In the latter casc, supervised fine-tuning process typically required for the reference model
mitigate distribution shift between the true reference distribution and the dataset used for DPO.
3.3 WRPO: WEIGHTED-REWARD PREFERENCE OPTIMIZATION
While fine- nee-tuning the target LLM with high-reward responses from source LLMs can alleviate the distribution issue in implicit model fusion, empirical results suggest that the distribution deviation remains, particularly when compared to preference data fully sampled from the target model. Therefore, propose progressive adaptation strategy with new optimization objective called Weighted-Rewt Reward Preference Optimization (WRPO), which enables target LLM smoothly transition align its distribution with that the source LLMs.
Derivation of the WRPO objective The preference dataset for WRPO consists of a set of quadruples
(a, Yu., Mwe. vt), where you, is the highest-reward response from the source LLMs for prompt and ye, and y are the responses with the highest and lowest reward from the target LLM,, respectively. Based on this setup, define new pair preferred completions yu (uw., Dor and updated preference triple (a, yws v.). We then extend the DPO framework by introducing weighted-rewan mechanism. In particular, the Bradlcy-Terry (BT) model is reformulated as:
ply Vi z)) ar((,, yw) r(a, v1)).
(4)

---

Published as a conference paper at ICLR 2025
where yw) is compound reward calculated as weighted average of r(a, r(a,yw,) and r(a, yws):
r(z,
r(z, Ve,) (1 a) ra Bwe).
(5)
where a represents the fusion coefficient that dynamically balances the internal reward of the preferred response Yo, from source models and that of Yun, from the target model during training. Next, by substituting r" from Eq. into Eq. (4) and Eq. ( we derive the WRPO training objective:
Lwaro (o: Fraet)
EI, Veex Muss 31) DD
(o( x)) Teet (x) Teet(// x)) ]
log a a Gog To(Uw, x) (ye + (1 ai) log
To (Ver [x) - 3 log
(6)
which can be reformulated Lwxro Aeter)
-E(r, Yorr So v1
log o
c- [Boog
TOo (B (x) Teer (Shur
log
To T
x))
+1--a)
a).
(log
To Ter (Yae
xr) xr)
log To (32 x)) Tier yy
Tiet yy
hybri--plkky interall reward murgin
pollccy internal reward
(7)
The above process seeks maximize the margin of internal rewards between preference responses, utilizing both on-policy sampling from the target model and hybrid-policy sampling from the source and target models. Initially, it emphasizes on-policy sampling and gradually transitions hybrid-policy sampling. This process helps mitigate distributional deviations and ensures a smoother optimization process.
Gradient analysis We examine the gradient of WRPO to understand the impact of the weighted-reward
mechanism on the training process. The gradient of loss function LwRFO in Eq. (6) with respect to the policy model me can be expressed as:
Vo WoLwwro nee)) = B.. Dws, Bws DLiD a
Po(or, 31) - fo fo vh..) - (1 a) no (xr Mere)
hightrr weight roward estimation is worong
a (a- Ve log (o +(1 a) log (Y (z) Ve log Ne (u1
(8)
increase likelhood on
increase likelibood
decrease likelibood
mailua)) represents the internal reward function. Intuitively, the gradient flow of LWrPo tends to increase the likelihood of preferred respouses you and yor while decreasing the likeli- hood of dispreferred lr. The function a(.) represents the reward estimation error that controls the rate of increasing or decrcasing the likelihood of preferred or dispreferred complctions in WRPO. When the reward estimation is incorreet, WRPO will accelerate the gradient flow for updates. To further on gradient update, (ax (fo(x,, 31) - fo (I. V),)) + (1 - at) (fo(z, yr) fa(2, D)). We can see that a serves as a constraint term on the gradient update for the policy model learning from yw.- A larger a means the policy will absorb more gradient information from y At the beginning of training process, since there exists distributional gap between yau from source models and y, from target model, assign relatively low the estimation term (Fe 31) - fe yw,)) )) and progressively increase a during the training process. In this way, we smoothly shift the target model Ty from the distribution yw, to that of yj..
where Fz(x,,) 2) = log =o(v)z)
alyze the impact of ye, and hyperparameter
reformulate the (-) term
Therefore, WRPO balances the contributions of diverse responses from heterogencous L.Ms and provides richer preference signals for preference optimization. Moreover. this weighted approach reduces distribution mismatches and enhances the fusion process by leveraging the strengths of both target and source models.
EXPERIMENTS
our experiments, we use Llama-3-58 3-8B--nstruce (Dubey et al. 2024) as the target LLM. we include ten advanced open-source models of varying architectures and sizes, as detailed in Table
source LLMs.
4.1 EXPERIMENTAL SETUP
Following prior work (Meng et al. 2024 Zhou et al. 2024). we chose UltraFeedback (Cui et al. (2024) to construct our training dataset. UltraFeeddaal includes approximately 64K prompts gathered from six established datasets that emphasize instruction- followwngg,..... truthfulness. honesty, and helpfulness. However, the original dataset comprises preference data derived from old versions of LLMss, which are often less capable than our target model. Therefore, we discarded the original responses and instead used their
Training Dataset

---

Published as a conference paper at ICLR 2025
prompts construct a new preference dataset D for our implicit model fusion, as described in Section 3.] Specifically, for each prompt in the dataset, we sam- pled N = 5 responses from cach source model using top-p sampling (p = 0.95) with a temperature of 0.8.
Table I: Details the source LLMSs used in our experi- ments along with the percentage of the highest-scoring responses from each source LLM.
Source LLMSs
Percentage
This approach aims to ensure that the sampled outputs capture the capabilities est extent possible.
Mistral-J Ciaral-Taaree warge-Instract-2221 [iang et al. 20233a
28.24%
the source LLMs to the great- -aaaaMM--llaa--BBBB..
Gemmu 2 2-27B-ii (Team el al (UA4)
15.45%
Qwen-2-72B- 720---2--tts Yang ml 2024
12.38%
(Wang) et al. 2024a) is then employed as the reward model score and rank these responses. We selected the highest-scoring response across all source models Iw,, with the percentage contribution from each source LLM detailed in Table
Llama-3-700 Instruct (IKibey el
9,92%%
Gemma- Team et 2324
9,91%
NecrmL.M-2.5-205--00
2024
7.54%
DeepSock-V2 V2-Chat LLu || 2024
6,20%
CeopSook-Cazz-V--1ige
al.
2024b
4.01%
V1-1.5-34B-Chat Youngg et al 20/4
3.86%
Phi-3-medium--e 1istrnt
et
2024
2.49%
Training Details Wc conducted experiments with a batch size of 128 and a maximum length of 2048 tokens on 8x80GB NVIDIA A800 GPUs. The training performed on a single epoch for method. A cosine learning rate schedule with warmup ratio of 0.1 is employed. The training process is divided into two stages. In the first stage, we applied supervised fine-tuning (SFT) on the set of yw, with one-third the dataset, with the learning rate empirically set to 7e-6. The resulting fine-tuned model, Target-SFT,, is the foundation for subsequent preference optimization. In the next stage, the remaining dataset is used for preference optimization, during which Yie, and yI are generated from the SFT model, i.e., Target-SFT. For WRPO, we used learning rate of 3 and set / = 0.01, with the weight a assigned to yoe, linearly increasing from to 0.1. Further details on hyperparameter tuning can found in Appendix [A]
Evaluation Benchmarks We assess the performance of our models on three representative instruction- following benchmarks: MTT-eench (Zheng et al.] 2023), AlpacaFval-2 (Li et al. (2023). and Arena-Hard (Li et al. 2024.. These benchmarks are well-regarded for their comprehensive coverage of diverse tasks and their effectiveness in providing robust evaluations of the instruction-follen foolowing capabilities of L.MMs.
MT-Bench contains 80 multi-turn dialogues with 160 questions across eight categories, including writing,
roleplay, reasoning, math, coding, extraction, STEM, and humanities. Each response is evaluated by GPT-4 on a scale from to 10, with the average score reported for each dialogue turn across the 80 dialogues.
Different from the official setting, we follow the latest works (Wang et al. 2024c Wan et al. 2024b) to
adopt 4PT
as the evaluator and baseline.
AlpacaEval-2 comprises 805 instructions from five different datasets and assesses models using two metrics: length- controllee (LC) win rate and raw win rate (WR) (Dubois al.. 2024). In this benchmark, GPT-4-Previcw- I 1106 serves as both the baseline model and the evaluator for the other models.
Arena-Hard is a more challenging benchmark that closely aligns with the human preference ranking from Chatbot Arema (Chiang et al. 2024), crowd-sourced platform for evaluating LLMs. It spans 250 high-quality topic clusters including 500 well-defined technical problem-solving queries. We report the win rate against GPT-4-0314 14 using 11-4-Preeiew-1000 as the judge model.
Baselines We compare WRPO with three categories of haselines, including source&target L.Mss, collective LLMs, and preference optimization methods. For source&targe LLMs, the results obtained from official leaderboards or our local tests unavailable. For collective LLMs. we include PackLLM-Top Too--Top-PPP- (PPp-PPL (Mavromatis
et al. 2024). LLM- Blender- Top1 (Jiang et al. 2023b), MoA (Wang et al. 2024b). FuseL.M (Wan el al. (202a)), and FuscChat (Wan et al. (024bb) For Pack LMoToo-P-Pppp we select the response from the source or target rank LLM outputs via pairwise comparisons and select the top response P For MoA (Wang al 2024b). we select Mistral- Large- Instruct-2407 as the aggregator LLM to combine input responses into single response. For FuseL_LM (Wan)
LLMs with the lowest perplexity on the test instruction. For L. BleMe--eeee--Tol..
ct al.
2024a) and FuseChat (Wan et al.) 2024b). limited by the complex vocabulary alignment and distribution merging process, we only include femmaa-2-2BB-i.. Gemma-2-913 source LLMs, with Llama-3-83 3---nssttt
Qwen-2-72B- InIstuutt L-ama-3-700 Instruct serving as the target/pivot LLM to reimplement their methods. For a fair comparison, we select the same 5 source LLMs to implement WRPO and obtain Target-SFT- SOTTT-RRPO-Meduur
Instruct, and Yi- Vi-1.5-34B--hr.
For preference optimization methods, we include DPO (Rafailov et al. 2023), SimPO (Meng et al. 2024), and IPO (Azar ct al. 2024). The results on A ApaccFFva--2 and Arena-Hard are referenced from [Meng et al. 20024) ; while the results MT-Bench are obtained by running the checkpoints released by Meng et al. (2024). In the following experimental results, these baselines are denoted Target-DPO., Target-SimPO, and Iar, respectively.
4.2 OVERALL RESULTS
In Table 2 we present the overall results of our WRPO method compared to various baseline methods of different categories, architectures. and scales on AlpacaEval-22, Arena- Hard, and MT-Bench benchmarks. These results offer valuable insights into WRPO's performance and efficiency detailed below.
Due the fuser model's limited input length, we only use the ranker model select the st-ranked output.
7

---

Published as a conference paper at ICLR 2025
Table 2: Overall results of proposed WRPO method with Llama-3-8B Instruct the target model, compared against various baseline categories on AlpacaEval-22 Arena-Hard, and MT-Bench. "T1" and "T2" represent average scores for the first and second turns, respectively. Bold indicates the best performance 8B models.
AlpacaEval-2
Arena-Hard (GPT-4-100--Preveen)
MT-Bench
Model
Size
(GPT-4-1106-Previen)
(GPT-4--125-Previen)
LC(%)
WR(%)
WR(%)
TI
T2 Overall
Source& Target LLMs
Target
8B
26.0
25.3
20.6
7.41
7.04
7.23
Haarrl--arge--nstuutt---
123B
54.3
46.8
70.4
8.83
8.31
8.57
Remma-2-27B--1
27B
55.5
41.0
57.5
8.34
8.03
8.19
Owen-2-72B-Innsee
72B
38.1
29.9
46.9
8.44
7.84
8.15
-lama-3-70B-Instree
70B
34.4
33.2
46.6
8.61
7.77
8.19
Gemma-2-99B-e
9B
51.1
38.1
40.8
8.27
7.44
7.86
20B
37.4
45.3
31.2
8.03
7.23
7.64
DeepSeek- -V2-chht
236B
51.4
51.3
7.96
8.31
DaepSeek-Coder-V22
Booddrr-V-Instruue
236B
50.7
54.0
66.3
7.42
8.13
Yi-1.5-34B-Chrr Chat
34B
37.5
44.5
42.6
7.99
7.64
7.81
1.n-3-Medium-4K-nssse
14B
29.8
24.2
33.4
8.63
7.46
8.04
Collective LLMs
PackLLM-Toop--P Top---PP
849B
49.1
48.0
64.8
8.29
8.20
LL.M-Blender-Top1
849B
46.2
44.3
58.2
8.06
8.38
MoA
849B
61.3
77.2
83.1
9.04
8.03
8.54
Fanget-Fusel..M.
8B 36.0
33.8
32.1
7.53
7.13
7.33
Chattt
8B
38.1
35.2
32.7
7.68
7.07
7.38
Preference Optimization Methods
Target-DPO
8B
48.2
47.5
7.23
Target-SimPO
8B
53.7
47.5
36.5
7,00
7.38
Target-IPO
8B
46.8
42.4
36.6
7.89
7.19
7.54
Our Methods
Target-SFT
8B
27.2
26.0
24.7
7.69
7.03
7.36
Target-SFT-DDR
8B
50.7
53.1
40.2
7.98
7.23
7.61
Bangt--FT-WRPP-Meddue
8B
53.5
53.8
41.6
7.80
7.03
7.42
Turget-FFT-WRPO
8B
55.9
57.6
46.2
7.95
7.31
7.63
WRPO strikes a balance between effectiveness and efficiency compared to collective LLMs. target LLM and involving the same source LLMs, as FuseLLM and FuseChat by notable margins. achieves
Starting with the performs existing model fusion techniques improvements of and 15.4 points the length-controlled (I.C) win rate on AlpacaEval-2, and 9.5 and 8.9 points in the win rate (WR) on Arena-Hard, respectively. This highlights the superior effectiveness of WRPO for implicit model fusion (IMF) compared to previous explicit model fusion (EMF) methods. Particularly. fused model, Target-SFFs SFT- WRPO, surpasses all larger source LLMs AlpacaEval-2, showcasing WRPO's potential enable a target model outperform much larger models. Furthermore, compared collective LLM fusion architectures that are 106 times larger in scale, WRPO outperforms most of these models, only falling short of MoA on AlpacaEval-2, while incurring substantially lower computational costs. While WRPO may not exceed the absolute performance of larger ensemble systems across all evaluation metries, its ability to achieve comparable results with far lower computational demands presents an elegant solution the ongoing efficiency-effccctvveee trade-off in language model deployment.
out-
WRPO consistently outperforms preference optimization baselines. In terms preference opti- mization, WRPO delivers notable improvements over prior methods. After fine-tuning you, using one-third of the data, Target-SFT performs slightly hetter than the target model, Following further optimization on the remaining two-thirds of the dataset, WRPO consistently outperforms all preference optimization baselines. Specifically. WRPO outperforms the best-performing preference optimization baseline three benchmarks by 2.2, 9.6, and 0.09 points, respectively. Besides, starting from Target-SFT, WRPO achieves 5.2 points improvement over DPO in the dength-controlle win rate on AlpacaEval-22, 2, and a 6.0 points increase in win rate on Arena-Hard. Compared to these approaches which utilize responses exclusively from the target LLM, the proposed WRPO method effectively incorporates responses sampled from various source LLMs for preference optimization, thus facilitating the integration diverse knowledge and capabilities through implicit model fusion.
out
4.3 ADAPTABILITY OF WRPO TO VARIED OBJECTIVES AND SOURCE LLM SCALING
this section, we examine how WRPO adapts to diverse preference optimization objectives scales with varying numbers of source LLMss, demonstrating its flexibility both dimensions.
8

---

Published as a conference paper at ICLR 2025
Adaptation to different preference optimization objec- tives Beyond DPO, we also investigate integrating our WRPO mechanism with alternative preference optimization objectives, utilizing the same SFT target model as the above experiments for DPO. Specifically, we experiment with IPO, which employs similar internal reward formulation DPO but optimizes nonlinear objective, as well as SimPO, which defines its re- ward function based on the average bgglikelihood of a response, thereby eliminating the need for a reference model. Detailed descriptions of the training objectives and the hyperparameter search ranges for these methods are provided in Appendix [A]
Table 3: Results of WRPO combined with different preference optimization objectives.
AlpacaEval-2
MT-Bench
Method
LC(%)
WR( %6)
Overall
SimPO
53.9
49.9
7.39
7.67
WRPOsinro
55.8
51.8
7.42
WRPO
53.3
57.7
7.72
We refer to the methods combining WRPO with SimPO and TPO as WRPO and WRPO respectively, The performance of these methods on Al AlpacaEval-2 and MT-Bench is summarized in Table 3 We note that combining WRPO with IPO and SimPO consistently improves their performance, highlighting our WRPO's s generalizability and efficacy in integrating preference signals from heterogeneous LLMs into the target LLM across various preference optimization objectives.
Scaling with different numbers of source LLMs We conduct experiments with varying numbers of source L.Mss to implement the WRPO framework. For five source LLMs con- figuration, we select Gemma-2-27B-it, Gemma-2299-it, Qwen-2- 72B-Instruct, Llama-3-70B-Inste ing setup with the comparisons made in FuseLLM and FuseChat. Moreover, we utilize two subsets of these five source LLMs for experiments involving fewer source LLMs. One subset includes a single LLM, Gemmma-222B--- while the other con- sists of two L.LMs: Gemma-2-27B-ir and -eeen-2-7BBInntruuee The results in Tahle/4show that the performance of WRPO ex- hibits an overall upward trend as the number of source L.LMs increases on AlpacaEval-2 and MT-Bench. This trend demonstrates the potential effectiveness of scaling up the number of source LLMs to enhance our method.
Table 4: Results of our WRPO implemented with varying numbers of source LLMs on AlpacaEval-2 and MT-Bench.
and Yi-1.5-34B-Chrn align-
AlpacaEval-2
MT-Bench
Num
I.C())
WR(%)
Overall
48.9
50.3
7.29
52.3
50.4
7.54
53.5
53.8
7.42
10
55.9
58.0
7.63
4.4 ANALYSIS OF THE WEIGHTED-REWAN MECHANISM IN WRPO
this section, we conduct an in-depth analysis the weighted-reward mechanism our implicit model fusion framework, focusing on three distinctive views.
Balancing internal reward dynamies Figure demonstrates the evolution of internal reward dynamics during preference optimization the Target-SFT model across various preference pairs, with consistent learning rate and 3 parameters. The internal reward margin, as defined in Eq. 77. comprises an on on-policy reward margin r(z, Mw. r2, 31) weighted by 1 a, and hybrid-polics reward margin r r(x, yw,) -r(z, 3t) weighted by Figure presents the analysis of solely utilizing the on-policy reward margin (a = 0). The observed reward margin approximates 0.2, indicating a relatively conservative optimization approach. This modest margin growth can attributed to the model's limited exploration capability due to its exclusive reliance on on-policy samples. In contrast, Figure b)) illustrates the effect of employing only the hybrid-policg reward margin (a = 1). This configuration exhibits more aggressive optimization behavior, yielding reward margins exceeding 1.0. While this suggests enhanced discriminative capability between positive and negative samples, the substantial distribution shift inherent in the hybrid setting may compromise training stability and ultimately yicld suboptimal results. Figure Be) showcases our proposed weighted-rewan mechanism, which synthesizes both on-policy and hybrid-policy reward margins through dynamic weighting. This approach achieves an optimal balance between the aforementioned extremes, generating moderate reward margins of approximately 0.5 and facilitating smooth margin transitions throughout the training process. The harmonious integration of on-policy and hybrid-policy components, as evidenced by the balanced optimization process, appears to be instrumental in the superior performance of our weighted-reward mechanism.
Effectiveness of weighted-rewant reward mechanism Figure 4 illustrates the ablation studies on the effec- tiveness of incorporating preferred responses from both source and target LLMs. We conduct these studics on two configurations: the baseline target model (Target) and its fine-tuned version (Target-SFT) to ensure a comprehensive evaluation. The analysis involves systematically removing either the source model's 's chosen response yo, or the target model's chosen response Yee from the optimization objective in Eq. ( by setting 0 or a I, respectively. In the Target setting. the removal yo, leads to a substantial decline of 25.8 points in the length-come controlled win rate, indicating that the distribution shift between yo, and y! creates challenges in directly utilizing source model responses for preference optimization. Moreover, this finding emphasizes the crucial role of ye, in bridging this distribution gap. In the Target-SFT setting, although SFT helps mitigate the performance deterioration caused by removing ywuy its performance still lags behind our

---

Published as a conference paper at ICLR 2025
0.5
0.0
Reward
0.0
0.2
-1
-0.5
-0.4
-
-1.0
-0.6
-1.5
50
100
150
200
250
300
50
100
150
200
250
300
0
50
100
150
200
250
300
Training
Training
Training stop
Closan revord
Rijoced rowudd
Rowal morrin
Byteal rowad margie
Oupolky reward eeargin
(a) DPO-on
(b) DPO-hybrid
(c) WRPO a= 0.5
Figure 3: Internal reward dynamics on Target-SFT model under different preference optimization setups. (a)
DPO-on: DPO training on on-policy preference pairs (ar Mer. M1). (b) DPO-hybrid: DPO training on hybrid- policy preference pairs (z, Vw (1). (c) WRPO a = 0.5: WRPO training with a increasing from 0 to 0.5.
60
Ongiial
WRROO S+
Length- Bingh-Controlled Win Rate (%)
Hybrid-Poo Policy Reward Accurcc
Length Cuntrolled Win Rate (9)
WEPO
WIPOO
55.9
$1.7
50.7
50-
48.2
55
84
40
53
78
30
26.0
25.9
27.2
20
Target
Target-SFT
51
0.3
0.7
72
Fusion coefficient
Figure Results of ablation studies for WRPO method on AlpacaEval-2, utilizing the length-controlled win rate metric.
Figure 5: AlpacaEval-2 contr-lont win rate and hybrid ppolic reward accuracy under different fusion coefficient a settings.
WRPO hy 6.3 points, which combines both yeu, and Yeu On the other hand, removing Yeu, reduces WRPO DPO based solely on sall-sd on-policy data. Notably, the exclusion source model responses leads to performance declines of 3.5 points and 5.2 points in the Target and Target-SFT settings, respectively, highlighting the important role of yo, in providing valuable preference signals through the weighted-reward mechanism.
Influence of fusion coefficient We evaluate the impact of varying the fusion coefficient in the weighted- reward mechanism, with a [0.1, 0.7, 0.9), by recording the ength-controlle (LC) win rate on AlpacaEval-2 and the sybrid-policy reward accuracy on a held-out set of the UltraFeedback dataset. Hybrid- policy reward accuracy is defined as the percentage of instances where the internal reward r(r., Ve,) from source LLMs surpasses r(z, 3t) from the target I.I.M. As shown in Figure 5 hybrid-policy reward accuracy improves increases, indicating that progressively increasing accuracy. However, the win rate on AlpacaEval-2 shows an initial decline followed by improvement. This suggests that although increasing & may provide richer preference signals and achieve higher hybrid-police reward accuracy on the UltraFeedback held-out set, docs not always correlate with real-world benchmark performance. Nonetheless, WRPO consistently outperforms the DPO bascline (50.7) across all ar settings.
wide range leads to higher hybrid-polie reward
5 CONCLUSION
In this work, we introduce Weighted- of heterogeneous open-source I.LMs with diverse architectures and sizes, aiming create more capable and robust target I. address distributional deviations hetween source and target L.MMs, WRPO utilizes a progressive adaptation strategy that gradually shifts reliance on preferred responses from the target LLM to the source LLMs. Extensive experiments on three public benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines.
Weighled-Reward Reward Preference Optimization (WRPO) for the implicit model fusion
This study concludes with three notable findings. First, implicit model fusion presents promising approach to enhancing the capabilities of LLMs by eliminating the need for vocabulary alignment and distribution merging. Second, the fusion of LLMs can be redefined as a preference optimization task, distinguishing it from conventional methods such as knowledge distillation and fine-tuning. Finally, our WRPO effectively addresses challenges related to hybrid-policy sampling, enabling efficient scaling accommodate various L.Mss.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (No. 62176270) and the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515018832.
REFERENCES
Marah Abdin, Jyoū Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. 14219, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 2024.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard, 2023.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39:324, 1952.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li. Hao Zhang. Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating 11ms by human preference. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Newral Information Processing Systems, 2017.
Kevin Clark, Minh Thang Luong. Urvashi Khandelwal, Christopher D Manning, and Quoe Le. Bam! bom-again multi-task networks for natural language understanding, In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5931-5937 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint DrXivv/803.000 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:2110.1416. 2021.
Ganqu Cui, Lifan Yuan, Ning Ding. Guamming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting language models with high-quality feedback. In Proceedings of the 41 International Conference on Machine Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2188. 2024.
Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment as prospect theoretic optimization. In Proceedings of the 41 International Conference Machine Learning. 2024.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixtureoof-LoRAs: efficient multitask tuning method for large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 11371-11380, 2024.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 12966): 2021.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Mcasuring massive multitask language understanding. International Conference on Learning Representations. 2021.
Jiwon Hong. Noah Lee, and James Thome. ORPO: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'clio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock. Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothéc Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arX1v:2310.06625,, 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 141178 2023b.
Minki Kang, Seanie Lee, Jinheon Back, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge-augmentee reasoning distillation for small language models in knowledge-intensive tasks. Advances in Neural Information Processing Systems, 2023.
Aran Komatsuzaki, Joan Puigcerver, James Loe-Thorp. Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslic, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upeycling: Training mixture-of-expee -xpeeessts from dense checkpoints. The Eleventh International Conference on Learning Representations, 2023.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. /939, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-followt models, 2023.
Stephanie Lin. Jacob Hilton, and Owain Evans. TruthfulQA: Mcasuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the A Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, 2022.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenggi Dengr, Chong Ruan, Damai Dai. Daya Guo, et al. DeepSeck-V V2: A strong, economical, and efficient mixture-of-cxpone language model. arXiv preprint ar arXiv:2405.000 044344 2024.
Michael Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances Neural Information Processing Systems, 2022.
Costas Mavromatis, Petros Karypis, and George Karypis. Pack of LLMs: Model fusion at test-time via perplexity optimization. In First Conference on Language Modeling, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: reward. In Advances in Neural Information Processing Systems, 2024.
Simple preference optimization with a reference-free
Ryan Park, Rafacl Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pp. 4998 5017, 2024.2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023.
Leonardo Ranaldi and Andre Freitas. Aligning large and small language models via chain-of of th-uthtuht reasoning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1812-1827, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Tianyuan Shi, Fanqi Wan, Canbin Huang, Xianjun Quan, Chenliang Li, Ming Yan, and Ji Zhang. ProFuser: Progressive fusion of large language models. arXiv preprint arXiv:2408.0400 2024.
12

---

Published as a conference paper at ICLR 2025
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Wop Wen Li, Wen tau Yih, Jason E Weston, and Xian Branch-Train Train-MiX: Mixing expert LLMs into mixture-of-expern LLM. In First Conference on Language Modeling, 2024.
Fahim Tajwar. Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size, arXiv preprint 2rXiv:2408.00/ 18, 2024.
Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and N. Chawla. TinyLLM: Learning small student from multiple large language models. arXiv preprint arXiv:2402.04616 2024.
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexan- der M. Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. arXiv preprint 2xXiv:2310.19444 2023.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations, 2024a.
Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi, FuseChat: Knowledge fusion of chat models. arXiv preprint arXiv:2402. 16107, 2024b.
Haoxiang Wang, Wei Xiong. Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi- objective reward modeling and mixture-of-expert In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582- 10592, 2024a.
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-Agents enhances large language model capabilities. arXiv preprint Nal:24006046992 2024b.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang. Makesh Nar- simhan Sreedhar, and Oleksii Kuchaiev. HelpSteer 2: Open-source dataset for training tor-perfonning reward models. In The Thirty-eight Conference on Neural Information Processing Systems Datosets and Benchmarks Track, 2024c.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-Lopes Ari S Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Haoran Xu, Amr Sharaf, Yunmo Chen. Weiting Tan, Lingleng Shen. Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Proceedings of the 41st International Conference on Machine Learning, 2024a.
Shusheng Xu, Wei Fu. Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Proceedings of the 41st International Conference on Machine Learning, 2024b.
Yangyifan Xu, Jinliang Lu, and Jiajun Zhang, Bridging the gap between different vocabularies for Ilm ensemble. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Pupers), pp. 7133-7145, 2024c.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1066 10671, 2024.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by Ol.ii. arXiv preprint 043642 2024.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedhack. In Advances in Neural Information Processing Systems, 2023.
13

---

Published as a conference paper at ICLR 2025
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp.
2019.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a- as-auue-- with MT--Bench and Chatbot Arena. In NeurlPS Datasets and Benchmarks Track, 2023.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. WPO: Enhancing RLHF with weighted preference optimization. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 8328 8328-8340, 2024.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. Starling- 7B: Improving helpfulness and harmlessness with RLAIF. In First Conference Language Modeling, 2024a.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang. Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. DeepSeek-Coders V2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 1931, 2024b.
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodci, Paul Christiano, and Geoffrey Irving. Fine- Sine-tuning language models from human preferences. arXiv preprint arXiv: 1909.08593 2019.
14

---

Published as a conference paper at ICLR 2025
A HYPERPARAMETER TUNING
Table 5: Various preference optimization objectives and hyperparameter search range.
Method
Objective
Hyperparameter
DPO
Rafailev et al.
2023
llog o log sutuola)
Telaule)
Flog
Fa(ly i2) mudlill))
e [0.01,0.05, 0.1]
SimPO Mong al.
2024
log
[px] low/l((y//a))
bai log mo(s)(>)) 7)
E 2.00)
c [5.0, 10.0)
IPO
Azar et
2024
log xelawia)) sarlsule)
log mnfaria) swalante)
e [0.00,0.1,10 1.00
WRPC
logg o (8 loy, Selbe, surlDe, ())
Telp.,it) (n) (1 ay- log munlte, 18]
1 loy malania) Fwilanle
0.01
[0.1, 0.3, 0.5, 0.7.0.9
WRPO
log or (o d log Ry Edl. ix) (1 ()) Tauy
log To(oorla)
A log me(srle)
[0.1, 0.3,0.5
10.0,
WRPO
log
neltt [n)
oi log lel
e
0.300.
[0.01,
Prior works such as SimPO (Meng et al. 2024) suggest that hyper-
Table 6: Hyperparameter settings for preference optimization methods using Target-SFT as the policy model. "LR" denotes the learning rate.
parameter tuning is crucial for achieving optimal performance of preference optimization methods. To avoid getting suboptimal base-
line results, followed the recommendation by/eng et al. [2024] to
apply hyperparameter tuning for preference optimization methods, including DPO (Rafailov et al. 2023). SimPO (Meng et al.) 2024). and IPO (Azar et al.) 2024). Specifically, we individually search the learning rates in the range of [3e-7, 5e-7, 6e-7, Ic-6] for cach preference optimization method. The specific training objectives and hyperparameter search ranges for these preference optimization base- lines, along with our method, are outlined in Table We used a batch size of 128 and trained these methods for a single epoch. The best hyperparameter values under the Tar setting are summarized Table 6 Besides, a learning rate of 7c-6 was used with single epoch for supervised fine-tuning (SFT), For the model fusion methods, including FuseLLM (Wan et al.] (2024a) and FuseChat (Wan et al.] (2024b), we used a learning of 7e-6 and conducted training over three epochs, with the parameter À empirically set to 0.9.
Method
9
LR
DPO SimPO
0.01
3e-7
10
1.0
6c-7
IPO
0.01
1c-6
WRPO
0.01
0.1
3e:7
WRPOs
10
0.5
6e-7
WRPO
0.01
0.1
1c-6
B EVALUATION ON ADDITIONAL BENCHMARKS
To further investigate the impact of WRPO on downstream tasks, we evaluate the models we trained using six tasks from the Huggingface Open LLM Leaderboard (Beeching et al. 2023). These tasks include:
AI2 Reasoning Challenge (ARC) (Clark et al. 2018): A collection of grade-school science questions in a 25-shot setting.
HellaSwag (Zellers et al.
2019 A commonsense inference task in a 10-shot setting.
2021): ; A set of diverse tasks spanning high-school and college subjects, social sciences, STEM, and others, in a 5-shot setting.
MMLU [Hendrycks et al.
TruthfulQA (Lin et al. 2022 : A set of measuring how language models mimic human falsehoods with 6-shot setting
Winogrande (Lvesquee et al.
2012):
A set of adversarial and difficult Winograd benchmarks for commonsense
reasoning 5-shot setting.
GSMSK (Cobbe et al. 2021): A set of grade-school math word questions evaluates mathematical reasoning capabilities in a shot setting.
We followed the established evaluation pipelines by using the in-evaluation-harress too for our evaluation. The results are presented in Table from which we draw several key observations. Firstly, after undergoing SFT with one-third of the data entries, Target-SFT shows a significant performance decline compared to Target, particularly on ARC and GSMSK, likely due to catastrophic forgetting during training. Next, all preference optimization methods display varying performance drops on MMLU and GSMSK, which may stem from the UltraFeedback
A Ithough TruthfulQA is traditionally regarded as 0-shot, it technically a 6-shot task hecause each example associated with Q&& pairs. 'We used updated version of v0.4.3 at https: htt:::/ggttttttt/// com/ / e.lehhhrrrl mm eeaaluatti www.iiionnnneeeeeeeeee.
4.3 for more accurate evaluation.
15

---

Published as conference paper at ICLR 2025
Table 7: Results of evaluations Huggingface Open LLM Leaderboard. "Target" denotes Llama-3-8B Instruct.
Model
ARC
HellaSwag
MMLU
TruthfulQA
Winogrande
GSMSK
Avg.
Target
61.43
78.48
65.71
51.64
75.61
75.21
68.01
Target-SFT
51.19
79.83
45.93
76.87
62.77
Furget-SFT-DPO
60.67
64.98
76.95
68.76
67.23
Target-SFT-SimmP
61.77
82.23
65.13
54,76
78.45
68.66
Target-SFT-IPO
60.58
81.68
53.93
77.9
68.21
Target-SFT WRPO
62.63
82.38
64.91
54.72
78.53
71.57
69.12
Target-SFT- WRPOs
61.69
81.95
65.08
57.11
78.69
68.69
68.87
Target-SFT WRPO
59.98
81.53
65.35
53.48
78.14
69.83
68.05
dataset's focus on alignment over general knowledge and mathematics. In contrast, these preference optimization methods consistently improve performance on HellaSwag and Winogrande, suggesting the presence relevant prompts for commonsense inference UltraFeedback. Similarly, all preference optimization methods show consistent gains TruthfulQA, except for Sagett-FFT-DO.. Lastly the performance of ARC demonstrates only minor improvements declines across all methods. In summary, while not explicitly designed for these tasks, our fused model. Frrget-FFT-WRFF surpasses the initial Target model while preserving general knowledge and mathematical abilities with minimal decline. This illustrates the generalization potential of our WRPO method.
C TRAINING COST ANALYSIS
Increasing the number of source LLMs does not affect the time complexity of our method during training. First, the interaction with source LLMs occurs exclusively during the data collection phase before training, where conduct offline sampling from the source LLMs and utilize ArmoRM responses select one response with highest reward score for each prompt. This step constitutes a fixed, one-time computational cost that independent participate in the actual training phase. Therefore, the inclusion of additional source L.Mss does not introduce additional computational costs during WRPO training. Furthermore, comparative analysis in Table & shows that WRPO maintains consistent computational efficiency across different numbers of source LLMs. Notably, WRPO incurs only modest overhead of approximately 16% training time compared to DPO (which does not involve source LLMs) on 8xA800 GPUs, regardless of the number of source LLMs involved.
reward model evaluate the
training process. Importantly, the source .Ms not
Table Runtime comparisons DPO WRPO across different numbers of source LLMs.
Num.
Runtime of DPO (min)
Runtime of WRPO (min)
Increase (%)
183
212
15.88%
185
215
16.22%
16.13%
10
185
215
16.22%
D DIFFERENT COMBINATIONS OF SOURCE LLMS
To explore the impact of different source model combinations, we conducted additional experiments using the AlpacaFval-2 examined the influence of the sponse quality from source LLMs by comparing responses different reward rankings. The experimental results in Table dicate that responses from top-ranked source models consistently outperform those from sccond-ranke rankeed models. This reinforces the importance selecting high-qualit, responses to achieve optimal performance, addition, we investigated the impact of model composition by dividing our ten source models into two balanced groups, each comprising five models with strong perfor- mance characteristics. The first group includes Gemma-2-27B-it, Gemma- 2-9B-it it Qwen-2-72B Insttrcct, Llama-3-70B Instruct
Table Results our WRPO implemented with varying combinations of source LLMs on AlpacaEval-2.
benchmark. Specifically,
AlpacaEval-2
Method
LC(%)
WR(%)
Length
Rank1 Rank2
55.9
$7.6
2159
53.7
55.4
2143
Groupl Group2
53.5
53.8
2098
53.7
60.7
2440
and Yi-1.5- 1--34--Chal The second group comprises Mistral-Large- Large-Instruch dastruct-24077 IntemLM -.0..---55-020 2.5-20t. DeepSeek- 22-Chat, DeepSeek DeepSeek-Coder- Coder-V. Instructt and Phi-3-Mediume Medium-4K, Instruct The experimental results in Table show that various combinations of source models achieve comparable length-controlled (I.C) win rate.
These hindinss demonstrate the robust performance of WRPO across range of source model configurations.
16

---

Published as a conference paper at ICLR 2025
E TUNING STRATEGIES FOR FUSION COEFFICIENT
WPRO, we implement a dynamic adjustment mechanism for the fusion coefficient a to facilitate a gradual transition of the target model's distribution toward that of the source models. In practice, the fusion coefficient initialized at 0.0 and increases linearly throughout the training process until it reaches a pre- determined target value (Clark ct al. 2019). To determine the optimal target value, we employ simple greedy search over the range 10.1, 0.3, 0.5, 0.7, 0.9]. This dynamic adjustment strategy effectively balances the contributions from both source and tar- get models while addressing potential distribution discrepancies, making it suitable for various tasks and eliminating the need for complex parameter configurations or exhaustive optimiza- tion procedures. Moreover, conducted ablation experiments comparing static and dynamic tuning strategies. In the static strategy, a remains fixed at target value throughout training, while in the dynamic strategy, a linearly increases from O to the target value. The experimental results in Figure 6 show that the dynamic tuning strategy generally outperforms the static strategy, except setting a 0.7, further demonstrating the effectiveness the dynamic tuning approach.
Longthh Cenrrelled Win Rate (%)
Dynamic
36
54
0.1
0.3
0.5
0.7
Fusion coefficient az
Figure 6: Comparisons of dynamic and static tuning strategies for the fusion coefficient on AlpacaEval-2 utilizing the length-controlles win rate metric.
F INCLUDING DISPREFERRED RESPONSES FROM SOURCE MODELS
We conducted additional experiments investigate the impact of incorporating extra dispreferred responses from the source models. Specifically, we use an extension of the WRPO loss Eq. (9). where 30, denotes the dispreferred response from the target model, and 3r, denotes the dispreferred response from the same source model corresponding to the preferred response Yw..
LWxPOw//1, (Ta: Riet) B(x, Ywus Mure Mi )1 logo a Blog To FoVu |x) keet 1
To (Bews [x) + +11 a) log
a Blog To (31, | x) Tea (31. x))
To xr) Ted bt. x)) ]] (9)
(1 a) log
(Be | x)
First, perform comparative analysis of the reward scores across four categories of responses. The average reward scores for Yeu,, Bwe. Vt., and Yl, are 0. 0.180, 0.152, 0.158, and 0.132, respectively. We observe that the source model's 's dispreferred responses y have higher average scores than the target model's preferred responses Vwr This finding indicates that incorporat- ing dispreferred responses from source models into training objective could potentially lead to an undesirable reduction in the probability of niihee-sccringg responses. Such results would contradict our optimization objectives and potentially compro- mise the overall training effectiveness. Furthermore, the results in Table 10 show that the inclusion of rejected responses from the source model leads to a decrease performance on AlpacaEval-2 and MTTBeench. Morcover, this approach increases computational costs due to the need for extra forward passes during training.
Table 10: Results of WRPO combined with additional dispreferred responses from source models.
AlpacaEval-2
MT-Bench
Method
LC(()
WR(S)
Overall
WRPO WRPO
55.9
57.6
7.63
54.0
56.0
7.52
G DETAILS OF OPEN-SOURCE MODELS AND THE DATASET
The selection of source models and the dataset is primarily determined by specific objectives. When a target model exhibits limitations in particular domains, domain-specific source models and datasets can be strategically used to enhanee its capabilities. In study, we focus preference optimization research. Therefore, selected ten prominent open-source LLMs with parameter sizes ranging from 9B to 236B, all which exhibit strong performance on relevant benchmarks. Moreover, we chose one of the most popular instruction-followw our training dataset. In Table II we provide the Huggingface repository names of the target LLM, source LLMs, reward model, and preference optimization baseline checkpoints used in our experiments. For the UltraFeedback (Cui et al. (2024) dataset, we select the same prompts as provided by MMng al. 2024 in -aaaaannnnnllllama-----iie armmorm for fair comparison baselines.
estructionnfollowee tasks align with prior
datasets, the UltraFeedback (Cui et al. 2024), as
17

---

Published as a conference paper at ICLR 2025
Table 11: Details of source models in experiments. "Target" denotes -ama-3-8B--. I-Ittrucc.
Model
Huggingface ID
Target
Mistral-L
Gemma-2-27B-ii
300gl-/-emma---271--11
Bwen-2-72B-Instmun
1aima-3-70B-nnstrucc
meta- meta-llama Meainna/MMMa-eee Multtuu/Mett-Liama----0---------
Gemma-2-99Bii
guogle/
MM2.5 BhB-h
interni/m/mmemmm223 20b-chhh
DeepSeck--2-Chat
deepseekaa//Deepseeeeee-
DeepSeck-Codes
Deepseckai/DDeepeeeecooorr
Yi-1.5-34B-CUI
Ul-a//Y-133334B
Phi-3-mcdium-44--nnttmm
AmmoRM-Llama--8BB-000
Target-DPO
Target-SimPO
Target-IPO
H LIMITAIONS AND FUTURE WORK
First, the WRPO training ohjective currently incorporates only the highest-scoring response from source models as the preferred output for each prompt. This selective approach may overlook other valuable responses, potentially underutilizing the full range of capabilities offered by the source models. Future work could explore inclusive methods that incorporate multiple responses from source models into the training objective. Second, while WRPO demonstrates strong empirical performance, relies heavily on existing preference optimization frameworks. A more rigorous thcoretical analysis is needed to provide deeper insights into the internal fusion dynamics of WRPO and to further strengthen its thcorctical foundation. Finally, while WRPO significantly improves performance such MMLU. This limitation can largely be attributed the narrow domain coverage of the training dataset. Future studies could address this by incorporating more diverse datasets from wider range domains.
cntrructionfoollowiii tasks, it may not perform as well on other tasks,
I
CASE STUDY
Finally, we evaluate the performance of our WRPO method using sample instruction from Arena-Hard (Li et al. 2024). which features math puzzle centered on Situation Puzzle Challenges. Using Llama-3-88 3-8B-Instruct the target model, compare the performance of Target-SFT, Target-SFT-DPO, and SiggetSFFTWRRPG against the baseline model GPT-4-0314.
shown table L2 Tar misinterprets puzzle, providing an incorrect answer of dots as maximum. contrast, Target-SFT-DPO approaches the problem through a graph theory, arriving at solution of 5 dots. Although this answer is logically derived from graph theory, it remains incorrect. Notably, Target-SFT- SFT-WRPO demonstrates significant improvement over the other models, surpassing the GPT-4-0314 baseline correctly solving the puzzle and offering comprehensive explanation. This response not only accurately addresses the puzzle also discusses practical limitations related to page size and provides effective strategies for solving similar puzzles in real- world contexts. The combination of factually sound reasoning and thorough, user-friendly explanation underscores the superiority of the WRPO approach.
18

---

Published as a conference paper at ICLR 2025
Table 12: Case study of math puzzle Situation Puzzle Challenges from Arena- Hard. Our Target-SFT- SFT-WRPO model delivers mathematically sound and user-oriented response, highlighting the efficacy of WRPO in enhancing model performance challenging problem-sol solving scenarios.
Topic: Situation Puzzle Challenges
Instruction: Help me find out how to solve math puzzle. It's: "How many dots you can joint down page while drawing lines between them, without three of them ever forming straight line?"
This math puzzle is related to the concept of "no three points being collincar." In this case, you want to find the maximum number of points you can place on a page such that no three of them are in straight line. The answer to this puzzle is that you can place an infinite number of points without having three of them form straight line. One way to achieve this is by placing the points in a non-linear pattern, such as arranging them in spiral or circle. However, if you are looking for a more specific answer, you can consider placing the points in a triangular grid pattern. In this arrangement, you can place as many points as you want without having three of them form straight line. Each new point added will be connected to the existing points without creating straight line with any other two points.
GPT-4-0314 Reference Answer:
Target-SFT Answer: This a classic puzzle known as the "Three Dots Puzzle" or "No Straight Lines Puzzle." The answer 3. Here's why: If you draw lines between 3 dots, you can't avoid creating at least one straight line. you draw lines between dots, you can create a triangle with two straight sides, which means three dots are forming straight line. If you draw lines between 5 dots, you can create a pentagon with three straight sides, which means three dots are forming straight line. Therefore, the maximum number of dots you can connect without three of them forming straight line 3. This puzzle is a classic example of graph theory problem, specifically a planar graph challenge. Here's a breakdown and solution: Understanding the Problem:
Target-SFT-DPO Answer:
Graph: Think of each dot as a node, and the lines connecting them as edges.
Planarity: The key constraint is that no three nodes (dots) can be connected by straight lines, forming a triangle. This ensures no three edges intersect at a single point.
Goal: Find the maximum number of dots you can connect without violating this planarity condition.
Solution: The maximum number of dots you can connect while maintaining planarity is 5. Why 5?
Proof: Imagine arranging the dots in pentagon (a five-sided shape). Each edge of the pentagon connects two dots, and no three dots form a straight line within the pentagon.
Generalization:
For any number of dots greater than 5, it's impossible to draw planar graph without three dots forming a straight line. This is because as you add more dots, the likelihood of forming triangles increases, eventually leading to intersections.
Key Takeaway: For any planar graph, the maximum number of nodes (dots) without three forming straight line is 5. This fundamental limit arises from the inherent geometry of planar graphs
hope this explanation helps you solve your math puzzle!
Srree--SFT-WRO Answer: Unraveling Puzzle: This problem is a classic example "Non- Collinear Point Arrangement" with a twist. Let's break it down step by step:
Key Insights:
The core constraint is that no three dots (points) should ever form a straight line. 2. Geometric Considerations:
1. No Straight Triungles:
A straight line requires three points a specific arrangement (collinear).
We need to avoid this configuration.
Strategies for Solving:
1. The Fundamental Approach:
Inductive Reasoning: Start with two dots. Adding third dot that doesn't 't create a straight line always possible (e.g., forming a triangle with slightly obtuse or acute angles).
Generalization: If you can place three dots without a straight line, you can always add more dots, ensuring each new addition maintains this property.
2. Mathematical Proof (for the curious):
Proof by Contradiction: Assume, for the sake of contradiction, that you've 've found set of dots where three form a straight line.
Analysis: This implies a triangle with all angles equal (180"), which is impossible in a non-degenerate triangle (no sides are zero-length).
Therefore, it's impossible to construct a set of dots where three always form a straight line. Practical Implications: Infinite Solutions: There are infinitely many ways to arrange dots without creating straight lines between any three.
Conclusion:
Page Limitation: While theoretically infinite, the practical limitation is your page size. You can draw as many dots as your paper allows, ensuring no three forms a straight line.
Solving on Puper:
Start with two dots: This sets the foundation.
Add dots gradually: Choose each new dot carefully to avoid creating straight line with the existing ones.
Visual Inspection: Continuously check that no three dots form a straight line as you add more.
Remember: The beauty of this puzzle lies in its simplicity and the creative process of finding arrangements that satisfy the constraint. Happy dot-connecting!
19


Reviews:
[{"review_id": "agKLBUN5rn", "reviewer": "Reviewer_McSp", "summary": "This paper aims to align the LLM prediction from multiple source models to a target model. Specifically, it focuses on fuse multiple heterogeneous LLMs and combine their preference into the target one. Technically, it proposes a weighted-reward strategy to implicitly fuse the source models. Experiments on several evaluation benchmarks show the method effectiveness.", "questions": "Please refer to the weaknesses section above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. LLM alignment is popular and valuable research topic, especially aligning heterogenous models into a target one.\n2. The proposed weighted-reward is a straightforward to to achieve an implicit model fusion process.\n3. Comprehensive evaluation supports the paper statement with rich analysis.", "weaknesses": "1. Adding an analysis of alignment efficiency and ablation will be helpful, what is the difference of using different source models? if multiple source models are used, will it increase the alignment cost?\n2. Some analysis parts can be more informative such as figure 3 and 4. Including more analysis and make them dense will further enhance the draft.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FrCxftzG2v", "reviewer": "Reviewer_42p3", "summary": "This paper introduces Weighted-Reward Preference Optimization (WRPO), an implicit model fusion approach for large language models (LLMs) that bypasses the need for complex vocabulary alignment and distribution merging found in traditional fusion methods. WRPO uses a weighted reward mechanism and a progressive adaptation strategy to smoothly transfer preference weights from the target model to source models. Benchmark results on AlpacaEval-2 and MT-Bench show that WRPO consistently outperforms traditional fusion and preference optimization methods, marking it as a promising advancement in LLM model fusion.", "questions": "1. How sensitive is WRPO to the selection of source LLMs? Could varying the quality and diversity of source models affect the final performance, and if so, what are the selection criteria?\n2. What are the specific criteria or methods used for tuning the fusion coefficient? How was it determined for different tasks, and does it require extensive hyperparameter optimization?\n3. Could WRPO be applied to multilingual or cross-domain tasks where distributional shifts might be more pronounced, and if so, are any modifications required?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. WRPO effectively introduces implicit model fusion, bypassing the need for complex vocabulary alignment and distribution merging, a significant advancement over traditional methods.\n2. The novel reward mechanism balances contributions from source and target models and helps mitigate distributional discrepancies, leading to a smoother optimization process.", "weaknesses": "1. The success of WRPO appears to depend heavily on the choice and quality of source models, yet the paper does not fully address criteria or strategies for selecting these source models, which could be a limiting factor in practice.\n2. The need for dynamic tuning of the fusion coefficient introduces complexity, and the paper does not sufficiently detail how this parameter was optimized across different datasets and tasks.\n3. While WRPO is more efficient than traditional methods, it still requires significant computational resources for training and fine-tuning, especially when handling multiple large LLMs, which may limit its accessibility\n4. Although WRPO is presented as a robust method, more ablation studies could have provided a clearer understanding of the specific impact of each component, particularly the fusion coefficient and its progressive adjustment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cftozpU1JK", "reviewer": "Reviewer_dUiR", "summary": "The paper investigates the progress of fusing heterogeneous open-source LLMs with varying architecture and size. They then summarise the challenges in this scenario, including vocabulary alignment and merging distribution matrices. To solve these challenges, they propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which eliminates the need for vocabulary alignment and matrix distribution and improves the performance on multiple benchmarks.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper explores the direction of improving the capabilities of individual models by combining the strength of multiple LLMs, which has potential benefits to improve the ability of individual models.\n2.  They propose a novel implicit fusion method that eliminates the need for vocabulary alignment and matrix fusion.\n3. Extensive experiments demonstrate the effectiveness of proposed methods in multiple aspects.", "weaknesses": "1. Although the proposed methods demonstrate a certain degree of improvement on AlpacaEval-2 and Arena-Hard, they only have weak influences in MT-Bench. This weakens the generalization of the proposed methods.\n2. The object of the proposed WRPO is to increase the likelihood of a preferred response while decreasing the occurrence of the dispreferred response. Preferred responses come from source and target models and dispreferred responses only come from the source model, which means dispreferred responses from the source model will not influence training. Whether some potential small experiments that support this phenomenon as it is intuitively we not only need to learn recognition capability in correct data but also learn exclusion ability.\n3. Although the experiment section provides some interpretation, it lacks an in-depth analysis of the rationale behind the proposed phenomenon, such as, how WRPO achieves a balance between effectiveness and efficiency.\n4. They lack the discussion of limitations and future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)