InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
Authors: Yanggan Gu
Yuanyi Wang
Zhaoyi Yan
Yiming Zhang
Qi Zhou
Fei Wu
Hongxia Yang
Venue: NeurIPS 2025 spotlight ()
Keywords: Model Fusion
Preference Optimization
LLMs
Primary area: deep_learning
TL;DR: We propose InfiFPO, a novel model fusion method for preference alignment that integrates multi-source probability information to enhance LLM performance, outperforming existing approaches across 11 benchmarks.

Abstract: Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods. Existing works on model fusion focus primarily on supervised fine-tuning (SFT), leaving preference alignment (PA) —a critical phase for enhancing LLM performance—largely unexplored.
The current few fusion methods on PA phase, like WRPO, simplify the process by utilizing only response outputs from source models while discarding their probability information. To address this limitation, we propose InfiFPO, a preference optimization method for implicit model fusion.
InfiFPO replaces the reference model in Direct Preference Optimization (DPO) with a fused source model that synthesizes multi-source probabilities at the sequence level, circumventing complex vocabulary alignment challenges in previous works and meanwhile maintaining the probability information.
By introducing probability clipping and max-margin fusion strategies, InfiFPO enables the pivot model to align with human preferences while effectively distilling knowledge from source models.
Comprehensive experiments on 11 widely-used benchmarks demonstrate that InfiFPO consistently outperforms existing model fusion and preference optimization methods. When using Phi-4 as the pivot model, InfiFPO improves its average performance from 79.95 to 83.33 on 11 benchmarks, significantly improving its capabilities in mathematics, coding, and reasoning tasks.

InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
Yanggan Gu 2
Yuanyi Wang's
Zhaoyi Yan?
Yiming Zhang'
Qi Zhou' Fei Wu
Hongxia Yang!
The Hong Kong Polytechnic University
ainfi..a *Zhejiang University
yangganguuouttlook.
y hk
Project Page: https///tthbb.cmm/iiXAAA//iii
Abstract
Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods. Existing works on model fusion focus primarily on supervised fine- tuning (SFT), leaving preference alignment (PA) a critical phase for enhancing LLM performance- largely phase, like WRPO, simplify the process by utilizing only response outputs from source models while discarding their probability information. To address this limitation, we propose InfiFPO, a preference optimization method for implicit model fusion. InAFPO replaces the reference model Direct Preference Optimiza- tion (DPO) with fused source model that synthesizes multi-source probabilities at the sequence level, circumventing complex vocabulary alignment challenges in previous works and meanwhile maintaining the probability information. By introducing probability clipping and max-margin fusion strategies, InfiFPO en- ables the pivot model to align with human preferences while effectively distilling knowledge from source models. Comprehensive experiments on widely- benchmarks demonstrate that InfiFPO consistently outperforms existing model fusion and preference optimization methods. When using Phi-4 as the pivot model, InfiFPO improve its average performance from 79.95 to 83.33 11 benchmarks, significantly improving its capabilities in mathematics, coding, and reasoning tasks.
largely unexplored. The current few fusion methods on PA
1
Introduction
Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of natural language tasks. Yet, no single model is universally diffneren LLMs often possess distinct strengths due to variations in architecture, pretraining data, and objectives. This has motivated surge of interest in model fusion, which aims to integrate knowledge from multiple source models into a single pivot model enhance its overall performance [15).
While existing work on model fusion has primarily focused on the supervised fine-tuning (SFT) phase, little attention has been paid to integrating fusion techniques into the preference alignment phase, a critical step in reinforcement learning from human feedback (RLHF) pipelines that substantially boosts performance and usability. Applying model fusion to this phase is non-trivial due to the discrete nature of preference data and the challenge of aligning both preferred and dispreferred outputs across heterogeneous models.
The closest prior work, WRPO 6 attempts to bridge this gap by incorporating high-quality source model responses additional preference signals. However, WRPO suffers from major limitations:
"Coresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1) it discards the probabilistic outputs of source models and only uses response-level supervision; and 2) it focuses solely on preferred responses, missing valuable contrastive signals for dispreferred ones. As result, WRPO only partially leverages the capabilities of the source models. The broader question-how to systematically fuse model knowledge during preference alignment- remains largely unanswered.
To overeome these limitations, we propose InfiFPO, principled and efficient framework for performing model fusion during the preference alignment phase. Our key insight is that the reference model in preference optimization (e.g., in DPO) can be replaced with a fused source model, thereby enabling the pivot model to learn not only from preference data but also from the probabilistic behaviors of multiple source models. Unlike WRPO, InñiFPO utilizes sequence probability from all source models including for both preferred and dispreferred responses making the fusion process more principled and information-rich. We call this fusion that utilizes sequence-level probabilities as implicit model fusion, distinguishing it from previous work on token-level model fusion.
To instantiate InfiFPO efficiently, we derive from an RLHF-style constrained optimization frame- work called FuseRLHF, which encourages the pivot model to maximize preference rewards while remaining close-in sequence-level KL divergence--to each source model. We further transform this constrained RL problem into a fully offline optimization objective that avoids expensive online sampling and reward model training.
To improve the robustness and effectiveness of InfiFPO, we introduce three key enhancements: (I) Length Normalization, which reduces bias arising from varying token sequence lengths across models; (2) Probability Clipping, which limits the influence of underperforming source models by suppressing noisy gradients; and (3) Max-Margin Fusion, which adaptively prioritizes source models that offer the most distinctive and informative deviations from the pivot model.
We conducted comprehensive evaluation of InfiFPO using Phi-4 7 the pivot model and selecting five mainstream open-source LLMs with parameters ranging from 9B to 24B as source models. Our experiments spanned 11 widely-used benchmarks covering diverse tasks, including mathematics, coding, instruction following, and so on. Results demonstrate that InfiFPO consistently outperforms existing model fusion and preference optimization methods, improving Phi-4's average performance from 79.95 to 83.33 across 11 benchmarks. Furthermore, InfiFPO exhibits great versatility, effectively combining with various preference optimization objectives to further enhance performance.
In summary, our contributions are threefold:
We propose InfiFPO, novel preference optimization framework that performs implicit model fusion by replacing the reference model DPO with a fused source model, derived via an offline relaxation of a sequence- KL constrained RLHF objective.
We introduce three stabiliyy-enhaniing strategies length normalization, probability clipping. and max-margin fusion to avoid degradation from tokenization mismatch and probability incon- sistencies source models.
We conduct extensive experiments on 11 preference benchmarks, demonstrating that InfiFPO consistently and significantly outperforms existing model fusion and preference optimization baselines, verifying its effectiveness for implict model fusior P7
2 Preliminaries
In this section we briefly revisit the two foundations of our work: Model Fusion 1 Preference Optimization (DPO)
2
4
and Direct
The goal of model fusion is to integrate knowledge from source models with different architectures, parameter sizes, and training data into one pivot model to enhance the pivot model's capabilities. Given N source models {M;) Ji and pivot model (also called target model) MP, the model fusion can be cast as KL-constrained optimization problem:
Model Fusion:
arg max Ex DDD MP
log (og(((y
(1)
S.t. Em, BDDD
DTKL.
(M((y)=)))MMM ( ( se, Vi
"Our project is available at http::/gjiilbbccmm///aalllllaalt

---

where each sample in the dataset D contains a prompt sequence æ and its corresponding response sequence y. TTK, indicates token-level KL divergence. Each constraint here keeps the pivot model within an e-ball of every source model, thereby fusing their behaviors. While effective, Eq. ( suffers from vocabulary conflict: the source models often employ incompatible tokenisers, forcing previous work to rely on heuristic vocabulary matching.
Direct Preference Optimization: DPO an offline replacement for Reinforcement Learning from Human Feedback (RLHF). The objective of RLHF can be formalized as
arg max P. (r(r, az, y)l M
fiDKL [M' [| (y|=z)]
where r is a reward model that evaluates how good the response y generated by policy model M is. DsKL indicates sequence-level KL divergence Usually, the base reference model M is the initial M and / is a parameter controlling the deviation from Mrref.
After deriving Eq. (2). the relationship between r and the optimal policy M - is as follows
MM (yllaz)
r(x, y) Bllog (
+ +Blog Z(x).
(3)
where 7(x)) = Ey (g)=z)) exp (r((x,y)))
Since the reference model M remains unchanged during training, the optimal policy model is theoretically determined only by the reward model r. A natural idea is to indirectly train the optimal policy model through training the reward model, which the optimization objective of DPO:
M (v)l ( -E. log a ( [Blog Mre (x)
(y Mrel (
log
(4)
where each sample the preference dataset 'DP contains a prompt æ, a preferred response ywu and dispreferred response y/-
3 InfiFPO: Preference Optimization for Model Fusion
This section presents InfiFPO,, our novel approach to preference optimization for Model Fusion. We begin by introducing the FuseRLHF objective ($ 3.1 which integrates model fusion with RLHF. Given the inherent complexity of reinforcement learning frameworks, direct implementation proves challenging. Consequently, propose efficient offline InfiFPO methodology and develop three performance-enhannin strategies ($ 3.22) Finally, we provide gradient analysis to further understand
the optimization mechanism of InfilPO 3.3)
3.1 FuseRLHF: RLHF for Implicit Model Fusion
Constrained objective. We consider model fusion during the preference alignment phase to be constrained optimization problem. Based on Eq. (1) and (2), we obtain the objective of FuF
r(x, 2)
arg max E DDD MP
(5)
s.t. E BYDD
DEKL [M((Ul))]|MM(((())))) se, Vi (1.
where the initial pivot model is included in the source models. Fach constraint keeps the pivot policy within e-ball (in KL divergence) of every teacher, thereby fusing their behaviour while still allowing preference alignment through the reward r.
Please note that in Eq. (I). previous works on model fusion used token-level KL and faced the vocabulary conflict
KL constraints the sequence-level instead of the token-level. As shown
DDue to the discrete nature of language generation, this function is not differentiable and usually be simplified DSku [M' (y [M (y) (y)))) log(M (M (y]x)) "o (ylæ)))

---

Pivot Model
Optimization Objective
veglt
Fusion
MP
Normalized Probabilitt Pairs
Cussoeuu (M, (M
Sequence Probability Pairs
Length Norm
Probability Clipping
fro mede
wright7s
(
0.5
Inde
0.1
ML(vi)) (M)())=))
0
Source Models [M
difiered begghn
Max-margin fasion
Figure 1: Overview of InfiFPO for implicit model fusion. We compute probabilities for preferred (zww) and dispreferred (yr) responses using both pivot and source models. Following length normalization and probability clipping, identify the source model with the maximum normalized probability difference from the pivot model for fusion and preference alignment.
issue, where heterogeneous models have different vocabularies, making their token-level output distributions incompatible for divergence calculation. To solve this issue, those works introduce complex vocabulary matching processes and only calculate KL divergence on the matching portions. In contrast, our sequence-level KI. constraints avoid this issue while preserving the source models' probability information, learning source models' preferences for the whole sentence. We call this fusion that utilizes sequence-level probabilities implicit model fusion (IMF), distinguishing from previous work on token-level model fusion.
Unconstrained objective. Introducing non-negative multipliers {y and relaxing the constraints Eq. (5) yields the following unconstrained objective:
arg max Ee v)1 MP
Σ DsKL [M"(]]))
(6)
where 2 0 and Σ 7i = By setting to different values, various fusion strategies can be adopted. Specifically, when N and taking the source model the initial pivot model, Eq. (14 reduces to classical RLHF.
3.2 InfiFPO: Efficient Preference Optimization for IMF
Deriving the FPO objective. Directly training the pivot model with FuseRLHF requires significant time and resources. On one hand, it requires training additional reward model; the other hand, during RL, the pivot model needs to sample responses online, while all source models also need to
calculate sequence probabilities simultaneously. To reduce these costs, we follow Rafailov al. [8] by converting online FuseRLHF to offline infiFPO, improving training efficiency. Following prior works, it clear to show that the optimal pivot model M to the FuseRLHF objective in Eq. (14 takes the form:
1 Mi (y exp( 1 ( v)).
MP*(y/ = ((x)
where a fused source model My I ())=)))) and partition function ((x) Ey Mi, (g) exp (fr(a, v)). See Appendix A.1 for complete derivation. Then we can rearrange
Eq. 7 show the relationship between the reward model and the optimal pivot model as:
M (yl (x))
r(x, y) = log
+ log Z(x).
(8)
Mi, (y(z)
We can see that theoretically the optimal pivot model M only depends on the reward model r, since the source models remain unchanged during training. Therefore, can derive the optimization

---

objective of FPO as follows:
My x)) Mi, tu yy x:)
M M (r) Ma (2 m
LFFO (M;)E))
log
log
B log
Especially, when .N- = 1 and using the initial pivot model as the source model, this loss function reduces to the original DPO, i.e., Eq. 4 While the FPO objective is simple and theoretically supported, faces length normalization and source model degradation issues in model fusion. Besides, the fusion multipliers {%/} ) 1 for source models requires a strategy for determination. To address these issues, we propose the following three strategies.
Length Normalization. Models with larger vocabularies tend to produce longer segmentations whose log-likelihood sums are lower, even when the semantic adequacy of the response is unchanged. Similar effects have been presented by Wu et al. [9] To address the length bias issue, we introduce the length normalization, which divides the sequence log probability from model by the length of the sequence after tokenization with that model.
log M (
y log (
(10)
where M can be either source model or pivot model.
Probability Clipping. Source models may occasionally assign lower probability the preferred response V (or higher probability to the dispreferred response yt) than the pivot model, injecting misleading gradients and causing instability. To avoid the source model degradation issue, clip the sequence probabilities of the source model. Specifically, for y we define the minimum probability of the source model as the sequence probability of the initial pivot model M for yı. we define the maximum sequence probability of the source model as the sequence probability of the initial pivot model.
Clip(M((yll)) { [max(M55 x)), M (u/)z)), if y is Yw:
(11)
(y)la, M M (ylaz),, else.
This piecewise definition is weakly monotone: for all t S to we have Clip(t)) S Clip(tz). so the logarithm that follows preserves (or ties) the order of probabilities and cannot invert preferences.
Max-margin Fusion. To determine the multipliers {7 for source models, we propose max- margin fusion strategy that maximizes the diversity of information incorporated into the pivot model. Specifically, select the source model that differs most from the current pivot model, as this model likely contains the most unique and complementary information.
1, if = arg max, MMM) 0, otherwise.
%
(12)
where
(y)()))) measures the probability difference between source model and the pivot model. This wanner-takes-all approach simplifies training while effectively capturing diverse capabilities across source models. $4.3) we also evaluate alternative fusion strategies, including averaging and dynamic weighting methods.
abs(M)
In Summary. Combining the strategies listed above, we have the final InfiFPO objective:
MP - Blog M)(y)(0)
LIMEPO((MP (MM = -EG logo Glog
Maclip (y (x)
Maclip (y((z))
(13)
where Man
Maclipyyla I [Clip(M((gl=))
3.3 Gradient Analysis.
To gain deeper insights into InfiFPO's optimization dynamics, we analyze the gradient of the loss function with respect to the pivot model parameters 6. This analysis elucidates how information from source models influences the pivot model's 's learning trajectory. The gradient can be expressed as:
V
- WiDDW
(BlogR, Rl
preforence disparity coefficicut
imcresse liketihord of yuu decreuse likelihood of
5

---

where Rg Mas ( (M..l))
and Rp M (2.l)
MM(y)l) Analogous to DPO, the InfiFPO gradient increases
Mr SMpalpp (M,la))
the likelihood of preferred responses yı while decreasing that of dispreferred responses Yt- However, the critical distinction lies in the preference disparity coefficient, which weights training samples based on the divergence between the source and pivot models' preference assessments. This coefficient becomes larger when source models exhibit stronger preference between Y and y: than the pivot model does. Consequently. samples where the source models strongly differentiate between responses but the pivot model does not yet reflect this distinction receive greater optimization emphasis. This adaptive weighting mechanism efficiently transfers preference knowledge from source models to the pivot model. The full derivation of the gradient is in Appendix A.2
Experiments
We evaluated InfiFPO using Phi-4 as the pivot model and five mainstream open-source LLMs (9B-24B parameters) source models across 11 diverse benchmarks. Results show InfiFPO outperforms existing fusion and preference optimization methods.
4.1 Setup
Model. We use Phi-4 7 as the pivot model. The source models consist of three general-purpont mod- els (Qwen2.5- 14B---111B-Iin Instruct [I0]. Seadl--alll-----nmene two dopain-specific models (Qwen2.5-Coder 14B-Instruct [12] and Qwen2.5-Math-77 -n-streettttt [33..
4l and Gemma-3-12B-Inssteee (II) and
By including these domain-specific models as additional source models, we can also investigate whether specialized expertise can enhance the overall performance of general model
Dataset. We constructed new training dataset comprising 150k examples across mathematics, coding, and general tasks. Data sources include Infinity- NuminaMath-11. 1.5 [15], and KodCode-VI V1-
Types
General Data
Math Data
Code Data
Dataset
Infinity-Instrue
NuminaMath-1155 1.4M
KodCode-VI-SFT 268k
Original Size
14M
[1),
Sample Size
60K
45K
SET [6. with detailed statistics provided in Table capable than our pivot model, we retained only the prompts to build a new preference dataset. For each prompt, we generated responses using multiple models: 4 from each source model and 8 from the pivot model, all with a sampling temperature of 0.8. We then employed reward mode 6 [77 to evaluate these responses, selecting the highest-scored response y and the lowest-scored y!-
Table I: Dataset Statistics.
I Since the original dataset may contain responses from outdated LLMs, potentially less
Training Detail. Our training process involved two stages with batch size of 128 and a maximum sequence length of 4,096 tokens, using 16 NVIDIA A800-80GB GPUs. We implemented cosine learning rate schedule with 10% warmup ratio. In the first stage, we performed SFT on half of our dataset with Vu for 3 epochs, using a learning rate of 1e-6 to build the SFT model. This model then served as the foundation for the second stage, where we conducted Preference Optimization on the remaining half of the data for a single epoch, with learning rate of 1e-7 and 3 2.5.
Evaluation. We conduct a comprehensive evaluation across 11 diverse benchmarks assess the model's 's capabilities. Our evaluation spans five critical dimensions: (1) General reasoning (BBH (18), ARC-C 19I. MMLU 220). (2) Math (GSM8K (21) MATH [22]. TheoremQA (23). D. (3) Code
(MBPP 24) Eval (25)) (4) Text reasoning (DROP 26. HellaSwag (27). and (5) Instruction
following (IFEval (.)). This multifaceted evaluation strategy enables to systematically analyze
the model's strengths and limitations across spectrum of tasks requiring different cognitive abilities.
More evaluation details are listed in Appendix [B
Baseline. We compare InfiFPO with three categories of baseness, including pivot &source model, model fusion, and preference optimization methods. For model fusion methods, include FuseL.MM M. FuseChat (2) and InfiFusion Limited by the complex vocabulary alignment
htts://uuginggface http:://huggiigfaaee.ccmmmssiiii ww/.iisttaaaiiiisseettooooooooeo 24a
"s shown in Table domain- domain-specific models Codee-Coerr and Qwen-Matt perform poorly outside their specialized domains. To prevent these limited capabilities from being fused into the pivot model, we selectively fused Qwen-Cnder only on code data and Qwen- Math only on mathematical data.
htttss//huggingiace.coo http://gg...nn.eooooooooo////ooo Reward Reward-Gemma- Gemma- 2-27B

---

Math
Code
General Reasoning
InstFol
Text Reasoning
Model
GPU
Models
GSM8K MATH
ThmQA
MBPP
HEval
BBH
ARC
MMLU
IFEval
DROP
A
Size
Hours
Pivot Model
Phi-4
87.41
80.04
51.12
75:40
83.54
68.84
93.90
85.62
77.34
88.67
87.62
79.95
14B
~-100
Models
Qwen2.5. Instruct
91.13
78.16
47.25
81,70
83.54
77.59
92.20
80.22
85.01
85.56
88,28
80.97
14B
1.88M
Mistral-Small
92.42
69.84
48.50
68.80
84.15
81.59
91.86
81.69
82.25
86.52
91.84
79.95
24B
1.66
Qwen2.5 Coder
89.16
74.18
38.88
85.40
90.90
75.4
89.49
75.08
74.70
84,34
79.83
77.94
14B
~ ~-8M
Gomma-3-Insten
93.71
82.90
49.62
72,60
82.32
85.70
71.19
77.61
90.77
86,43
83.34
79.65
12B
Qwen2.5 MaMahh
92.27
81.70
20.25
1.40
46.34
33.12
65.76
40.20
35.49
81.96
25.57
47.64
7B
~~0.5M
Model Fusion Methods
L
90.24
80.25
53.52
79.28
84.00
77.62
92.08
83.92
78.56
88.74
87.81
81.46
14B
225
FuseChat*
91.21
77.52
51.88
81.80
84. 15
83.37
93.56
84,23
78.90
89.23
87.42
82.12
14B
650
InliFusion
90.07
80.94
55.62
81,80
83.54
80.94
94.24
85.81
76.02
89.27
87.91
82.38
14B
160
Preference Optimization Methods
SFT
88.70
79.58
55.12
78.20
86.59
74.66
93.56
84.36
80.06
88,72
87.75
81.57
14B
15
SFT-DPO
89.76
80.02
57.88
82,50
84.76
77.86
94.58
84,27
81.89
88.56
87.31
82.67
14B
50
SFT-IPO
90.45
80.18
55.25
82.50
85.37
77.13
94.24
84,08
80.94
88.67
87.36
82.38
14B
50
SPPO
86.59
83.98
InAFPO
InAFPO
89.92
79.88
57.00
82.00
85 98
81 81.26
94.24
83.33
80.46
88.68
87.36
82.74
14B
55
InáFPO
90.07
80.10
57.25
82,50
87.80
82.02
94.24
84.27
82.25
88.83
87.29
83.33
14B
Table 2: Main Results. indicates that this method only uses Qwen2.5-Instrrn Instruct, Qwen2.5-Coder and Mistral-Small as source models. Abbreviations: InstFol (Instruction Following), ThmQA (TheoremQA), HEval (HumanEval), HS (HellaSwag). We use the packing technique for SFT, where multiple short examples are packed into single input sequence increase training efficiency.
and distribution merging process, we only include Qwen2.5-Instie Inntrruct, Qwen2.5-Coo Small as source models. For fair comparison, we select the same source models to implement InfiFPO (marked with asterisks). For preference optimization methods, we include DPO [8] IPO [29], and WRPO (6). All these preference optimization methods adopt the same two-stage training approach as InfiFPO, i.e., first using half of the data for SFT, then using the remaining half for preference optimization.
and Mistral-
4.2 Main Results
Table 2 presents comprehensive evaluation results across 11 benchmarks for different model types. From these results, we can draw several findings about InfiFPO.
InfiFPO effectively integrates the capabilities of the source models. InfiFPO significantly im- proved the pivot model's average performance across 11 benchmarks from 79.95 83.33. This integration is particularly remarkable InfiFPO manages inherit specialized strengths from di- verse source models, such as mathematical reasoning from Qwen2.5-Math and code generation from Qwen2.5- while avoiding their respective weaknesses. For instance, while Qwen2.5-Man excels on GSM8K and MATH (92.27 and 81.70) but performs poorly on other tasks, and Qwen2.5- Coder achieves top scores MBPP and HumanEval (85.40 and 90.90) but underperforms theorem questions, InOiFPO maintains balanced high performance across these diverse task categories.
InfIPPO outperforms model fusion baselines in both efficiency and effectiveness. Compared to InfiFusion the best-performing baseline on average, InfiFPO** shows an average improvement of 0.36 across 11 benchmarks. The improvements are particularly significant for instruction-followinn and code tasks, with a 4.4 improvement on IFEval and 2.4 improvement on HumanEval. More importantly, InfiFPO* requires only 34% of the GPU hours compared to InfiFusion (55 vs 160). This is thanks to our implicit model fusion objective, which replaces token-level KL with sequence-level KL. This strategy preserves probability information while avoiding complex vocabulary alignment processes, making InfiFPO more efficient and scalable.
InfiFPO consistently outperforms preference optimization baselines. After training on half of the data's preferred responses y SFT improved by 1.62 on average compared to the original model. Then, using the remaining half of the data for preference optimization, InfIPP outperformed all preference optimization baselines. Compared to SFT-WRPO (the best-performing baseline), InfiFPO showed an average improvement of 0.53 across 11 benchmarks while using about the same
7

---

amount of GPU hours (58 vs 57). This demonstrates that our InfiFPO can better incorporate source model knowledge to enhance performance without significantly increasing training time compared to existing preference optimization baselines.
4.3 Analysis
We conduct two analyses: 1) examining the effectiveness of InfiFPO when combined with other preference optimization objectives; and evaluating how different fusion strategies influence InfiFPO's performance. Due to space limitations, we only report three metrics: Math represents the average results from GSM8K, MATH, and ThmQA; Code represents the average of MBPP and HEval; and All represents the average results across all 11 benchmarks.
Adaptation to Other Preference Optimization Objectives. InfiFPO is fundamentally an improve- ment on the reference model, making it applicable not only to DPO but also to other DPO variants. To verify the versatility of InfiPPO, we integrated it with IPO and WRPO, creating InfiFPO and InfiFPO wrFo respectively. Table T demonstrates InfiPPO's versatility as general enhancement framework that can be effectively integrated with various preference optimization objectives beyond standard DPO. When integrated with WRPO, InfiFPO delivers modest but consistent gains in mathematical reasoning (+0.4), code generation (+0.2), and overall performance (+0.5). The improvements are more pronounced when combining InfiFPO with IPO, yielding notable enhancements Math (+1.1) and Code (+0.5) benchmarks, with an overall improvement of +0.8 across all tasks. These results empirically validate that In- fiFPO's fusion-based approach provides complementary benefits to existing preference optimization techniques by leveraging diverse source models while maintaining the pivot model's optimization objective. The loss functions for InfiFPO and InfiFOORRo can be found in Appendix C
Method
Math
Code
All
WRPO
75.94 10.0
84.84
10.0
82.80 10.0
InfiFPOwaP
76.31 10.4
85.20
10.2
83.32
10.5
IPO InfiPPOPo
75.29 10.00
83.93
[0.0
82.38 L0.00
76.37 111
84.54
10.5
83.15 10.8
Table 3: Results of InfiFPO combined with diffcrent preference optimization objectives.
Different Fusion Strategies. In $3.2 we proposed the max- marginal fusion strategy extract unique information from source models. Additionally, we explored two other fusion strategies: an average-base based strategy and confidence- strategy. For the baaede strategy, we treat all source models equally, assigning each weight of 1/N. For the confidence-bart strategy, weight source models accord- ing to their confidence the response, measured by se- quence probability, meaning models with higher confidence 2 illustrates the performance
86
Average
85.1
84
$3.5
53.1
margin
B2
scone
80
78
76
75.0
24.0
receive greater weights. Figure
Math
Code
All
of InfiFPO under different fusion strategies. We observe Figure 2: Different fusion strategies. that the Confidence-bassee strategy slightly outperforms the Average-bass based strategy, likely because it emphasizes tighcconfidence models during fusion, thus acquiring better information. The Max- marginal strategy consistently outperforms both the Average-based and Confidence- based strategies, as it focuses on the most distinctive models, thereby learning more unique and complementary information. Detailed fusion strategies can be found in Appendix D
4.4 Albation Study
To investigate the effects of different components in InfiFPO,, we conducted ablation experiments in three parts: Length Normalization & Probability clipping, Number of Source Models.
Length Normalization and Probability Clip- ping. demosttdessssssttsss the critical importance of both techniques. Without these optimizations, InfiFPO actually underperforms the baseline Phi4 model. Length Normalization alone provides sub- stantial improvements across all metrics (+2.4 in Math, +4.5 in Code, +2.9 in All), addressing the length bias issue. Similarly, Probability Clipping
Method
LN
PC
Math
Code
All
Phs4
72.85
00.0
79.47
10.0
79.95
10.0
72.72 J0.1
78.42
11.0
_0.4
InfiPPO
75.06
12.4
83.99 14.5
82.86
12.9
74.39
81.32
11.9
11.
75.80 13.1
85.15 15.6
83.33 13.3
Table 4: Albation Study on Length Normalization
(LN) and Probability Clipping (PC).
8

---

yields notable gains (+1.3 in Math, +1,9 in Code,
+1.5 in All) by preventing source model degradation. The combination of both techniques delivers the strongest performance boost, with significant improvements across all benchmarks (+3.1 in Math, +5.6 in Code, +3.3 in All), confirming their complementary nature enhancing model capabilities.
Number of Source Models. We conducted experiments on InfiPPO with an increasing number of source models, adding different models one by one for fusion. The experiment started with a single source model, Qwen2.5-Instrus. gradually added Mistral-Small, Swwe2..5-ooden Gemma- 3-Instruct, and Qwen2.5-Mathe Table shows clear per- formance improvements as the number of source models increases. With each additional model, we observe consis- tent gains across all metrics, particularly in Code tasks (+3.2 points with 5 models). The diminishing returns after 4-5 models suggest balance between performance benefits and computational resources when selecting source models for fusion.
Num
Math
Code
All
then
74.91 00.0
81.91 10.0
81.54 10.0
75.53 10.6
82.49
10.5
82.20
10.6
3
75.60 10.6
83.99
12.0
82.74
f1.2
4
75.25 70.3
85.21
13.3
83.14
11.6
75.80 10.8
85.15
13.2
83.33 11.7
Table 5: Albation Study on Number of Source Models.
5 Related Work
Preference Optimization. Aligning LLMs with human preferences often relies RLHF B0 31], where a reward model is trained from human comparisons and used in PPO to optimize the policy. While effective, RLHF is issturceintensivve and complex converge. To simplify preference alignment, DPO 8 reformulates the objective as offline learning from preference pairs, removing the need for reward model and RL. Azar al. (29] unify RLHF and DPO under a general preference optimization framework, and propose IPO, a variant that avoids overfitting by using the identity transformation, offering more stable learning in low-data or biased settings. Gu et al. (32 further propose the PAD framework, which models preference knowledge as a probability distribution over responses, providing more nuanced supervisory signals for preferences distilling.
Model Fusion. Integrating models aims to combine multiple LLMs into a single model that inherits their respective strengths. While earlier methods like model merging [3888 require architectural compatibility, fusion techniques relax such constraints, making them more suitable for heterogeneous models. Another line of work, knowledge distillation (KD) [1 offers alternative approach for transferring capabilities across models without requiring structural homogeneity. KD transfers the generalization ability of larger models to smaller ones by leveraging "soft targets" the output probability distributions (logits) of the teacher model enabling efficient deployment while preserving performance. However, traditional KD typically assumes that the teacher model is larger than the student and fully covers the target capability space. Moreover, most prior work has focused on the single teacher setting [42 43,, limiting its flexibility multi-teacher scenarios.
FuseLLM (2 introduced LLM fusion, showing gains in reasoning and code tasks. Later works extended this idea across domains [44), but struggled with structural mismatches. To address this, pairwise fusion (2) sequentially integrates models into the pivot model, and InfiFusion [4) improved it via adaptive merging and unified output aggregation. The most explicit fusion approaches operate at the token level and face vocabulary alignment issues. WRPO [6] mitigates this by shifting fusion to the sequence level using reinforcement learning, enabling implicit fusion without vocabulary conflicts. However, WRPO has two major limitations: it discards probabilistic outputs using only response-level supervision and focuses solely on preferred responses while ignoring contrastive signals from dispreferred ones, resulting in only partial leverage of source model capabilities.
Conclusion
We introduce InfiFPO. a novel framework that enables model fusion during the preference alignment phase by replacing the reference model in DPO with a fused sequence-level distribution over multiple source models. Unlike prior work, InfiFPO leverages full-sequence probabilities rather than token- level outputs, thereby avoiding vocabulary alignment issues across heterogeneous models while preserving rich preference signals. To make this optimization practical and efficient, we derive an offline training objective from sequence- KL constrained RLHF formulation, termed Fus.RLHF.

---

We further enhance stability through three key strategies: length normalization to mitigate sequence bias, probability clipping suppress noisy gradients, and max margin fusion to prioritize diverse and informative sources. Experiments across 11 benchmarks demonstrate that InfiFPO consistently outperforms strong baselines in both model capability and alignment quality. Our results highlight that preference optimization not only accommodates but can significantly benefit from principled model fusion, offering a robust and scalable path to integrating diverse LLMs.
Limitation
Despite InfiPPO demonstrating significant empirical performance, it still relies on existing preference optimization methods such DPO. More rigorous theoretical analysis is needed better understand InfiFPO's ' fusion mechanisms and further strengthen its theoretical foundation. Additionally, due to computational resource constraints, we only selected five mainstream open-source LLMs as source models for our experiments, which cannot represent the SOTA performance of current advanced LLMs. Experiments with larger-scale models and datasets remain unexplored.
Funding Transparency Statement
Funding in direct support of this work. This research was supported by the team of Prof, Hongxia Yang at The Hong Kong Polytechnic University. GPU resources were donated by InfiX.ai.
Competing Interests. The authors declare no competing interests.
References
[1] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In ICLR, 2024.
[2] Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. Fusechat: Knowl- edge fusion of chat models, 2025.
[3] Yuanyi Wang. Zhaoyi Yan, Yiming Zhang, Qi Zhou, Yanggan Gu, Fei Wu, and Hongxia Yang. Infigfusion: Graph-on-logits distillation via efficient gromov-wassersten for model fusion, ab//2/0/...5././/////////// 13893.
2025. http:::t////rr//t
[4] Zhaoyi Yan, Yiming Zhang, Baoyi He, Yuhao Fu, Qi Zhou, Zhijie Sang, Chunlin Ji, Shengyu Zhang, Fei Wu, and Hongxia Yang. Infifusion: A unified framework for enhanced cross-model reasoning via Ilm fusion, 2025.
[5] Qi Zhou, Yiming Zhang, Yanggan Gu, Yuanyi Wang. Zhijie Sang, Zhaoyi Yan, Zhen Li, Shengyu Zhang, Fei Wu, and Hongxia Yang. Democratizing AI through model fusion: A comprehensive review and future directions. Nexus, October 2025, ISSN 2950-1601. URL
https://doi. 1tt.:oo.///........11 1016/j.ynexs ynexs. 20121100102
[6] Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, and Xiaojun Quan. Weighted-reward preference optimization for implicit model fusion. In The Thirteenth International Conference on Learning Representations, 2025.
[7] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffimann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024.
[8] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.
[9] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arviv preprint arXiv:1609.0 00 144, 2016.
10

---

[10] Qwen, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[11] Gemma Team. Gemma 3 technical report, 2025.
[12] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024.
[13] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li. Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Owen2.55math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 12122, 2024.
[14] Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfilternng: Weak-to strong data filtering for fast instruction-tuning In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 44255-14273 2024.
[15] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: 'he largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
[16] Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. 2025.
[17] Chris Yuhao Liu, Liang Zeng. Jiacai Liu, Rui Yan, Jujie He, Chaojic Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 18451, 2024.
[18] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, al. Challenging big-bench tasks and whether of-in-of-tthuuht can solve them. In ACL (Findings), 2023.
[19] Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLP), pages 2578 7789, 2019.
[20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.
[21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arxiv 10. 14168, 2021.
[22] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
[23] Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing. Association for Computational Linguistics, 2023,
11

---

[24] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arxiv preprint arXiv:2108.0775. 077332, 2021.
[25] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
[26] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages
2019.
[27] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019.
[28] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cerruction-oolowinng evaluation for large language models, arXiv preprint arXiv:231 1.07911, 2023.
[29] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences, 2023.
[30] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv1707.0633 2017.
[32] Yanggan Gu, Junzhuo Li, Sirui Huang, Xin Zou, Zhenghua Li, and Xuming Hu. Capturing nuanced preferences: Preference-aligneee distillation for small language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages -1973, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 8 891 89176-256-56..1 doi: 18.18653/ URL http::/s httpp:://aclanthholoyy org/ 2025. findings-acl.82222
11/2025.findingss.
[33] Guillermo Ortiz-Jimencz, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 66727-66754. Curran Associates, Inc., 2023.
[34] Prateck Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing 3677933-15. 2023.
[35] Mitchell Wortsman, Gabriel Ilharco. Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965 229998. PMLR, 2022.
[36] Yuanyi Wang, Yanggan Gu, Yiming Zhang, Qi Zhou, Zhaoyi Yan, Congkai Xie, Xinyao Wang, Jianbo Yuan, and Hongxia Yang. Model merging scaling laws in large language models, 2025. https//s://axxi org/
24244
[37] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch, 2024.
[38] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, pages 1-10, 2025.
12

---

[39] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL httttt://aaxxii>> www./rgg////////e/e/..oo.ee a503 02531
[40] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint atXiv:1908.000 00355,955, 2019.
[41] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.01 108, 2019.
[42] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthicu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024.
[43] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint ar Xiiv:3306.08544 2023.
[44] Yiming Zhang, Baoyi He, Shengyu Zhang, Yuhao Fu, Qi Zhou, Zhijie Sang, Zijin Hong, Kejing Yang. Wenjun Wang, Jianbo Yuan, Guanghan Ning, Linyi Li, Chunlin Ji. Fei Wu, and Hongxia Yang. Unconstrained model merging for enhanced Ilm reasoning, 2024. URL fttps://arxxv.org/abb/////
13699
[45]
OpenCompass Contributos, Opencompass: A universal evaluation platform for foundation 2023.
models. htttps://github..comm// http:///enccoopppppppeceoooo///p
[46] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https
//openreview //operreriee..ne/fforrrr// wttp://eeee..ooomrrod-ioo//.00.0
[47] Balaji lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[48] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, pages 5574 5584, 2017.
[49] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting lan- guage models with scaled feedback, 2024. URL //arxivv org/ 2bs/2310.01377
A Mathematical Derivations
A.1 Deriving the FPO Objective
In this appendix, we will derive Eq. Based on Eq. (14, we optimize the following objective:
N
arg max EE Mew. - Σ MP
3 % (ogl - log Mi((l=z) (14)
then we have:
N
arg max Ear Be-D b)] Mr
D.KL. (MP(g))))LMM(())))))
M x))
= arg max MP
r r(x, v) - :logg
Mi, (v)2z)
log ()(y) 1
arg min EewDEyy
r(z, r(
MP
M ())0))
3
arg min En-DEy-yMe MP
log M (3) exp r 4))
log 2(x)
(15)
7(x)
13

---

where the fused source model Mi (z)(=) = II (M)3)))))) and the partition function Z(x) EMMM (zylz) exp (3r(2.))) v)).
Note that the partition function a function of only æ and the fused source model Min but does depend on the pivot model MP. Hence we have the optimal solution MP:
1
M (ylæ) =
Ma (y)z) exp( 3 rzx y)).
(16)
Z(x)
This completes the derivation.
A.2 Deriving the Gradient of the InfiFPO Objective
In this section, derive the gradient of the InfiFPO objective, where the parameters of the pivot model are denoted as A:
Blog (M M (z)) Maclip (V,,la)) (x) Maclip (1 (2)1z²)
VOE (o. WODDD
log
-llgg
(17)
where (tyla) = II:m [Clip(M](y/lz))]
We can rewrite Eq. 17 as
ViLinarro (MP: (M;)
VOR (o. Wi)
loga ['(u) (o(u)
VB(u))
(18)
where u Flog
(x) Glog
M lar) M Machip (sr)
M pelip (y x)
With the properties of sigmoid function - (1-x))1-o[p)) =(())_(=)(o and (xx) = 1 - o(-2), we can get the gradient:
VOLInFFP
- -a Von PL)-DP
plog Mr (m)l=))
sellip (yp imn)
Blog
MP (y mn) so1l (Ww
Volog
MP
Mr
After rewriting the first term, we obtain the final form of the gradient in
3.3
B Evaluation Setup
We adopt OpenCompass 45] and EvalPlus 46 to conduct evaluation 11 benchmark datasets. We revise the prompts of certain datasets within OpenCompass to ensure more reliable answer extraction via its regex-based matching mechanism. The detailed prompts listed in Table
For TA and HumanEval, we follow the default evaluation settings without modifying the prompts. For MBPP, we employ EvalPlus 46) for rigourous evaluation of LLM-synthesizzed code.
C Adaptation to Other Preference Optimization Objectives
The original objective of WRPO is
LwRrolMo: Meet) E(a,, Mup Wi)nD
Mo(y (r)
Mo B log Mo(Uilz)) ].
logo 2..Blo log Mierly (x) + a) Blog Merryy,,1a
(19)
M
where yar is the preferred response generated by source models, and yıe, is the dispreferred response generated by the pivot model. represents the fusion coefficient that dynamically balances the reward of Yw, and w After adapting InfiFPOO to this objective, we can obtain
CnrIrO (M))))) - -EE (2 .WD
logo a Flogoog Moly (x)
+ (1 - a) Blogg Mo(yw, Glog Mo(y)(z) ]
(20)
M ( |x)
M scllip ( (B x)
M (s)ll(()
14

---

Table 6: Prompt format used for different datasets.
Dataset
Prompt Format
[FEval ARC-C
directly give the correct answer: Question: {question)\nA (textAMnB (textB)AnC {textCHnn give the correct {ctx) Inquestion Which ending makes the most bense?\nDirectis me the correct choice, you can further explain it or not. \nA (AHAnB {B}nC {CHnD {D> may choose from 'A', 'B', CC' "D" Andnswer
Sprompt/\nPPeasee
option, and then explain: give
Hellaswag
BBH
Follow the given examples and answer the question. An {_hintHan LaA Let's think step by step.
DROP
You will be asked to read a passage and answer a question. Some examples of passages and Qu& are provided below. \n (drop_examples)ln An #* Your Taskin --An (prompt/\nThink step by step, then write line of the form "Answer: \$ANSWER" the end your response.
MMLU
{(input))\a ((A)) {(B))nC (()))nn (nD))n AnFor simple problems: AnDirectly provide the answer with minimal explanation.
(_hint) (oint)nnQuestion
AnFor complex problems \nUse this step-by- step format: #**# Step 1: [Concise description] \n [Brief explanation] 'n #### Step 2: [Concise description] \n [Brief explanation] (n
inRegardless of the approach, always conclude with: \nThe ansver is [the]_ saeesssrwweeeeterr Anwhere the "che\_answerl_leterr is of A, or AnLet think step by step.
GSM8K
@question\lnPeaase reason step by step, and put final answer within W\boxed((V).
MATH
pproblem\\nPPeaae reason step by step, and put your final answer within (lboxed((V.
The original objective of the IPO is
( (Boog Mrer ( ax)
Mo(yul - Blog Mo(y)
1
(M; Mier) = (x. (z. wi
(21)
M,r((y,
2,3
Since IPO already implements length normalization in the reference model part.
7
the main changes when adapting to InfiFPO
log (y log Maclip (y (ymll))
MP (x)
- (MM (MMP)) -E -E... WiDD
M (x)
2,3
(22)
D Different Fusion Strategies
When focus the fused source model Mi which the weighted geometric mean of all source model sequence probabilities, we can find that it originates from the model fusion objective. By changing weights, we can adopt different fusion strategies. In addition to the Max-margin fusion strategy mentioned in $ 3.2 we design two following strategies. Their results can be viewed in 4.3
Average-based Fusion. A common fusion strategy is to treat N source models equally, adopting the same weight of N. thus the fusion model becomes
M II (M) 1=1
(23)
Confidence- Fusion. To dynamically weight the source models, we introduce a confidence- based strategy and define the confidence of each source model Mi as the inverse of its negative log-likelihood:
(y)f)
(24)
log M5 te
15

---

where is a small constant ensuring numerical stability [7]. This design follows the intuition that llog M (z)az) quantifies the information content of prediction, where smaller values indicate higher confidence. Taking its inverse reflects the common practice of inverse-uncertainty weighting 48.. assigning larger weights more confident models. This fusion mechanism emphasizes source models with higher confidence, allowing Mi to adaptively inherit more reliable information from the sources. The corresponding normalized weight for M is given by SoftMax:
Confidence; (y x))
Wi (y ())
(25)
Coufidence, (3/[z)
===
Based on these confidence-derived weights, the fused model Mi, aggregates source models via:
(y) II (
(26)
E
Supplementary Experiments and Analysis
E.1
Additional Experimental Results
E.1.1
Data Diversity Validation
To validate InfiFPO's robustness to data diversity, conducted experiments UltraFeedbacc 49). which contains approximately 60k preference pairs and has been widely adopted in preference optimization research. We treated the preferred (chosen) responses labels for model fusion baselines.
We selected three general-purpose purpose models for fusion: 1) Qwen2.5-12 1eeenB.--14B-. 10B.55-11---nnuuue Instruct; 2) Mistral-Small; 3) Gemma-3-Instruce We compared against InfiFusion and SFT-WRPO, which demonstrated strong per- formance in our main experiments. As shown in Table 7 InfiFPO remains effective UltraFeedback and outperforms relevant baselines.
Model/Method
Math
Code
All
Phi-4
72.86
79.47
79.95
InfiFusion
73.23
82.43
81.44
SFT-WRPO
73.15
82.21
81.58
InfiFPO
73.59
83.15
82.19
Table 7: Experimental results on UltrFeedback
However, the optimal performance UltraFeedback (82.19) is notably lower than constructed dataset (83.33), validating the effectiveness of our data construction process.
E.1.2 Strong Backbone Validation
We conducted experiments using Qwen2.5-144 1eon22.5-14B-I-see 1-11-4--Bnstcctt as the target (pivot) model, selecting three source models: 1) Phi-4; 2) Mistral-Small; 3) Gemma-3-Instruue We used FuseChat, InfiFusion, and WRPO as baselines with identical experimental settings to our main experiments.
Model Method
Math
Code
All
Owen-2.5-Instreee
72.18
82.62
80.97
FuseChat
75.96
83.50
83.22
InfiFusion
76.09
83.38
83.16
SFT-WRPO
76.42
84.29
83.46
InfiFPO
76.46
85.41
84.01
Table Experimental results using 10-n2.55-1B-Instrruae as target model
16

---

Table 8 ddemonstrates that when using Qwen2.5 Instructt as the target model, our method still outper- forms other baselines, proving its generalizability. In our main experiments, chose Phi-4 over Qwen2.5- Instrrctt as the target model because Phi-4 is heterogeneous with all source models (different architectures, vocabularies, etc.), representing a more general and typical scenario for model fusion.
E.1.3 Reward Model Sensitivity Analysis
To examine data sensitivity, we tested the impact of different reward models on method performance. We tested two reward models: Beaork--ewwad--Laaa 1aamma-..--B---- 1-82-v0.2 and ArmoRM-Llama3-88 both widely used in preference optimization.
Model Method
Math
Code
All
Boywork-Reward-Lllma----
Phi-4
72.86
79.47
79.95
InfiFusion
74.32
82.47
81.96
SFT-WRPO
74.28
83.44
81.93
InfiFPO
74.53
84.88
82.67
AmmoM--Llama3-BB--
Phi-4
72.86
79.47
79.95
InfiFusion
73.38
82.54
81.59
SFT-WRPO
73.07
83.04
81.62
InfiFPO
73.45
84.94
82.46
Table 9: Results using different reward models
We observe two phenomena from Table 9 (1) InfiFPO consistently outperforms baselines when using different reward models; (2) Different reward models yield different performance improvements. According to Reward Bench v2. Skywork-Reward other two reward models, indicating a positive correlation between reward model performance and final model performance.
-antmma---2BB--a- 2-------.BB significantly outperforms the
E.1.4 Sequence Alignment Ablation
conducted ablation experiments to investigate the impact of sequence alignment on model fusion methods. We removed sequence alignment from all model fusion methods for fair comparison.
Model / Method
Math
Code
All
Phi-4
72.86
79.47
79.95
FuseChat
73.54
82.97
82.12
without sequence alignment
72.30
79.16
79.43
InfiFusion
75.54
82.67
82.38
without sequence alignment
73.47
79.98
80.11
InfiFPO
75.60
83.99
82.74
without sequence alignment
73.72
81.28
80.82
Table 10: Impact of sequence alignment on model fusion methods
Results in Table 10 show that even without sequence alignment, InfiFPO outperforms other model fusion baselines. Token-level fusion methods like FuseChat without sequence alignment even perform worse than the base Phi-4 model, indicating greater dependence on sequence alignment. Therefore, consider addressing vocabulary conflicts essential for model fusion methods.
E.1.5 Data Scaling Analysis
We tested different data volumes (25k/50k/75k/100k)) during preference optimization to analyze performance scaling.
17

---

Model / Method
25k
50k
75k
100k
Phi-4
79.95
79.95
79.95
79.95
SFT-WRPO
81.58
82.14
82.80
82.89
InfiFPO
82.55
82.99
83.33
83.56
Table 11: Performance scaling with data volume
Table shows consistent performance improvements across different data volumes, with gains increasing with larger datasets, demonstrating the scalability of our approach.
E.1.6 Small-scale Model Validation
We experimented with Sween25--nstrrutt-15BB//B1 effectiveness across different model scales.
pivot models validate method's
Model Method
Math
Code
All
Qwen2.5-1.5B Instruct
Base Model
46.02
47.16
50.31
InfiFusion
49.59
50.49
53.11
SFT-WRPO
50.16
51.21
53.70
InfiFPO
50.70
52.75
54.61
Qwen2.5-3B-Instrucc
Base Model
55.88
62.26
64.07
InfiFusion
58.82
65.33
66.72
SFT-WRPO
59.31
65.63
67.01
InfiFPO
60.05
66.64
67.96
Owen2.5-7B-Inntruue
Base Model
62.48
70.13
73.04
InfiFusion
65.09
72.24
75.35
SFT-WRPO
65.45
72.51
75.65
InfiFPO 66.14
73.31
76.24
Table 12: Results with different small-scale pivot models
Table 12 demonstrates that method outperforms model fusion and preference optimization baselines across different pivot model sizes, confirming scalability and effectiveness.
E.2
Technical Details and Method Analysis
E.2.1 Technical Implementation Details
Pre-computation Strategy: In practice, pre-compute log probabilities from target (pivot) and source models before training, which provides two key advantages: (1) Reduced training memory footprint by avoiding the need load multiple source models during training; (2) Accelcrated computation through efficient third-party inference libraries.
We employ vLLM for acceleration, requiring only approximately 8 GPU hours to compute log probabilities for 5 source models. This explains why our GPU hours increase by merely 10% compared to vanilla DPO when fusing multiple source models.
Memory Efficiency: InnIPPO offers significant memory efficiency advantages. It performs model fusion at the sequence level, requiring only sequence-level log probabilities (a few floating point numbers per sequence). When the number of source models increases, training efficiency remains largely unaffected. In contrast, token-level fusion methods must load probability distributions over
18

---

the entire vocabulary ( 100k tokens per token postion), requiring 100k-scale floating-pone numbers per token that scale linearly with source model count.
E.2.2 In-depth Analysis of Method Principles
We provide detailed discussion of how source models are selected during Probability Clipping (PC) and Max-Margin fusion strategies.
Probability Clipping and Max-Margin Process: PC aims to avoid interference from weak source models by clipping the corresponding log probabilities, ensuring that source models do not assign lower probabilities to preferred responses or higher probabilities to dispreferred responses than the pivot model. Max-Margin selects the source model with the most different preference knowledge from the pivot model as the reference model.
When using PC, Max-Margin selects the "smartest" source model, meaning the source model that assigns probabilities to preferred responses no lower than the pivot model, probabilities to dispreferred responses higher than the pivot model, and has the largest margin between preferred and dispreferred responses.
Example Illustration: Given two responses, the probability values assigned by models are:
Model type
Logps (preferred)
Logps (dispreferred)
Pivot model
-0.4
-0.7
Source model
-0.6
-0.5
Source model 2
-0.3
-0.8
Table 13: Example of model probabilities before PC
After PC, the source models' log probabilities become:
Model type
Logps (preferred)
Logps (dispreferred)
Selected
Pivot model
-0.4
-0.7
Source model
-0.4
-0.7
Source model
-0.3
-0.8
Table 14: Example of model probabilities after PC
Based on the Max-Margin strategy, source model serves as the reference model. It is the "smartest" source model, having the highest log probabilities for preferred responses and the lowest log proba- bilities for dispreferred responses.
This strategy aligns with findings in FuseLLM, which adopts the MinCE strategy to select source models with the lowest cross-entropy for ground truth. Both approaches indicate that model fusion, fusing knowledge from multiple source models may not be as effective as fusing knowledge from the "best" source model.
Case Study
F.1
Advantages in Code Generation
To evaluate the effectiveness of our proposed InfiFPO strategy, we conduct a detailed comparison with the baseline Supervised Fine-Tuning (SFT) model on the HumanEval benchmark.
Through case studies, we observe that InfiFPO consistently exhibits superior capabilities in two critical aspects: (1) precise task understanding and (2) concise and robust code generation. For instance, in the aanove_duplicater task, SFT misconstrues the requirement by retaining the first occurrence of duplicates, leading to incorrect outputs. In contrast, InfiFPO accurately identifies the need to eliminate all repeated elements, employing frequency-counting approach that aligns with the task's semantics. Similarly, in the vowels count task, InfiFPO adopts minimal yet effective implementation
19

---

OpenAi HumanEval-
Tass Type: Code Generation
InfiFPO Behavior
Baseline Behavior
"from typing import Listlnbnlndeff remove remove_duplicates(tieee duplicatesnuumbers: Listfint)) ->List VY"Y" From list of integers, remove all elements that occur more than once. in Keep order of elements left the same as in the input.In =esxremooe_dupiicctess(// 2, 3, 2, 4]An [1, 3, 4Jun V"\"\"nn
"prompt_ II
"completion": "def
"completion": "from typing import List remove_ remove_dupl caroove_ddllccteseenn (numbers: List [int]) ->List [int]: \"\"(" From list of integers, remove all elements that occur once. Keep order elements left the same as the input. >>>remove >xpremove_duplieen dupllicates(11, 2, 3, 2, An)
remove remove_duplicateeeeeeeee duplicates (numbera: (numberr: [iitss[[nt]:i :An An \"\"\" From list of integers, remove all elements that occur elements left duplicatee input. \n >>>remove >>rremoe_dupplcateee 2, 3, 2, in [1, 3, 4n \"\"\"\n counts = OAn for number in numbers: An counts [number] = counts.ged (number, 0) + i'n In result = [1 for number numbers: if counts [number] == 1: result. result.append (number) (aubbbbr \n return resultin", "result": "passed", "passed";
List [int])
than once. An Keep order
than
same as the
[1, 3, 4n \"\"\"ln seen = sn duplicates set() result - Wn for number in numbera: \n if number in duplicates: An continue in if number in seen:\n duplicates. addmber)) \n else: seen add (number) \n reppld (muutt)) (number) \n'n return resultin", "result"; "failed: "p"aased": false
true
"prompt_2": "InFIX A"dd" more cases, "I"nnn"" vowels_count(s) co W"Vte"Write a function vowels_cooe countt which takes a string representing'n a word as input returns the number of vowels the string.An Vowels in this case are 'a', 'e', 'i', 'o', 0, 'u'. Here, 'y" is also aln vowel, but only when is at the "aabdel"") in 2 Sosvowels cocreess__otto"" ACEDY\")nn end of the given wnld.Innn Example:in bsovowels_count/ "aaaeel
3 \"V"'"'n"
vowels_count(3 "completion"; "def vowels_ count (s) : An ""\" function vowels_coume count which takes string representingle i word input and returns the number of vowels in the string. \n Vowels this case are 'a', 'e', 'i', 'o', 'u'. Here, 'y' is also a'n vowel, but only when it is at the end of the given word. An\n Example: An >>>vomelsccount (aabcdee""" 2\n >>>vovels_comm (ttpeeel_scoottteecaaae (V"ACEDY\"" 3\n A"("(""n vowels A \"aeiouEEIOU\ "'n
"completion"; "def vowels_ (s) An ""\"X"Write a function vowels, count which takes string zepresenting in word input and returns the number of vouels the string. \n Vowels in this case are 'a', 'e', 'i', 'o', Pu'. Here, 'y' also a'n vowel, but only when is at the end of the given word. An/n Example:\n >>>vowels_come (\ "aabdde"")\n 2)n >>>vowels_counte (1 (ACEYY ") An 3'n \"\"\"An vouels \"aeiouΛEIQUV" "An count O'n for i, char in enumerate (a) An if char vovels: count 1\n elif char len(s) 1: count
"abcde\")
count O'n for char in 8: An char in vowels:An count l'n s[-1] .lower() 'y":'n count 1lm return count\n", "result"; "passed", "passed"; true
'y' and 1'n return count \n Additional test cases (wprintvvoelleeeeneeeeet (vovele_count (\ (\"hell\"")
# (nporitttvooll___ses "rhythm\*))
()) #
Dlaprint(vovels_comm (1"fly\"))
Nxprrint (("oooottnt"""""))))))
2'mprint (varelsssvoo (("xyy()))
A'pprint (vwwels_ccont (\mm))"))
1 \upp:ant(roella___ottt1111111 #
1 imprint (voxels_ccoutt((aa\\
#
(vowels_couutt
Amprint Inprint(vowelee
inprinttvooels__oomm (\"u\"))
1\n",
"result": "failed:
Test 5", "passed":
false,
Table 15: Case study on code generation.
20

---

that avoids boundary errors, whereas SFT introduces unnecessary loop-index conditions prone to edge-case failures.
These results suggest that InfiFPO not merely replicating seen patterns but actively refining the reasoning chain to bridge the gap between instruction semanties and executable code. By reducing spurious complexity and enhancing robustness against implicit corner cases, InfiFPO demonstrates stronger alignment with real-world coding practices.
F.2
Mathematical Reasoning Perspective
Case
Problem Prompt
SFT Output
FPO Output
Gold
Comment
Problem: How many ways there to divide set of 8 elements into 5 non-empty ordered subsets? ? Please ana- lyze and write your linal un- swer after the phrase "The is":
predicts 126000, mistakenly recall- ing S(8, 5) and apply- ing 5! permutations, leuding to significant overestimate.
InfiFPO
cor-
11760
InfiPO demonstrates
rectly
computes
recursive computation of Stirling numbers, while SFT simply re- calls a wrong value, lacking procedural rea- soning.
5(8.
105.
applies 5! = 120,
yielding 105 120 12600.
Problem: What is the value I_ sin(3r)
SFT predicts 0, over- simplifying based on oscillation intuition, neglecting the struc- ture of the integrul.
InfiPPO expands prod- uct of sines, applies Fourier integral prop- erties, identifies zero singular contribu- tion, yielding the cor- rect result.
1.0
InfiFPO's success stems from correct application orthog- onality and singulurity analysis, while SFT lacks symbolic manip- ulation depth.
sin(t/x)/Ft dt? Please ana- lyze and write your final an- swer after the phrase "The answer is":
Problem: Given 0.157, what is the value of æ x [T-1 (11-517))
Both SFT and InfiFPO predict 0.157, reoog- nizing Euler's reflec- tion formula, correctly simplifying numerator and denominator.
Both predict 0.157. FPO's derivation ex- plicitly links product form to sine function identity.
0.157
Both models give cor- rect answer, but In- ÍiFO provides more explicit and pedagogi- cal derivation path.
sin(er) 2irl2 Please alyze and write your final answer after the phruse "The
is":
Table 16: Comparison between SFT and InfiFPO on TheoremQA benchmark. InfiFPO shows stronger symbolic reasoning, procedural consistency, and structural understanding, leading to better robustness and interpretability.
We conduct detailed comparison of InfiFPO and SFT the Theorem benchmark assess their mathematical reasoning capabilities. Our evaluation reveals that InfiFPO demonstrates superior performance in symbolic manipulation, stepwise reasoning, and the application of mathematical identities. While SFT often relies on pattern matching and heuristic recall, InfiFPO constructs solutions via complete, logically consistent derivation chains.
For instance, concerning the evaluation of the definite integral f sin(3)) dt, InfiFPO suc- cessfully decomposes the product of sine functions using trigonometric identities and correctly identifies the contribution of singularities and symmetry in determining the integral's value. In contrast, SFT prematurely coneludes that the integral vanishes due oscillation, neglecting deeper analysis of the integrand's behavior.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction align with the paper's contributions and scope, providing a clear and accurate overview.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
The complete mathematical derivation can be found in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided implementation details in section and included repro- ducible anonymous code repository URL in the abstract.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: In section 4 we provide the data sources, and the abstract, we include an anonymous code repository URL for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/giee guwde//ddee/// Cot//ooodeeeieseeeeeeee wwwpibmmsssiooooiioooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided implementation details in section and appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Given that calculating statistical significance requires additional resources, we did not perform these calculations. Most of our experimental results show improvements of more than percentage point, indicating certain level of significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In section 4 we provide information about the computational resources used.
Moreover, Table 2
we show the GPU hours required for model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research fully adheres to the NeurlPS Code of Ethics, ensuring integrity, fairness, and transparency throughout the study.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our method focuses improving the performance of general models, which does not directly cause social impact. Additionally, we will provide licenses for our code and model weights.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We will provide licenses for code and model weights in the future.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [No]
Justification: Based the double-blind review process, all code provide now has been anonymized and is intended only for reviewers to check and reproduce results.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Based the Mouble-bline review process, all code provide has been anonymized and is intended only for reviewers to check and reproduce results.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
28


Reviews:
[{"review_id": "aiSLv8o0FV", "reviewer": "Reviewer_Pa3G", "summary": "This paper introduces InfiFPO, a novel framework designed to fuse the capabilities of multiple Large Language Models (LLMs) during the preference alignment phase. The authors identify a clear gap in existing model fusion research, which has predominantly concentrated on the Supervised Fine-Tuning (SFT) stage while overlooking the critical preference alignment step. The core innovation of InfiFPO is replacing the static reference model used for KL-divergence computation in Direct Preference Optimization (DPO) with a dynamic, fused source model, enabling simultaneous preference optimization and model fusion. Empirical results demonstrate strong performance of the proposed method.", "questions": "[Q1] Are the log probabilities of reference models pre-computed and cached, or computed on-the-fly when training? The training time of InfiFPO seems to be only slightly larger than DPO+SFT, despite that numerous reference models are adopted.", "limitations": "See Weaknesses. I suggest a brief discussion of the practical limitations of this approach and the trade-offs between the number of source models and memory and computational costs.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n[S1] Integrating preference optimization and model fusion through KL-divergence is technically sound and elegant. The derivation of the loss is clearly presented.\n\n[S2] Compared with multi-stage methods, simultaneous fusion and preference optimization can simplify the implementation and thus enhance practical applicability. Convincing empirical results show that InfiFPO can effectively achieve the two goals.\n\nWeaknesses:\n\n[W1] The data for preference optimization is generated by either the pivot model or reference models, leading to low KL-divergence. Is the InfiFPO method still applicable when data is from diverse sources, such as human-written or generated by models other than reference models?\n\n[W2] Memory usage scales linearly with the number of reference models. Parallelization and distributed strategies need to be carefully tuned when introducing larger reference models.", "was_revised": true, "final_justification": "The authors addressed weaknesses 1 and 2 through additional experiments and clarification.\n\nFor W1, UltraFeedback's answers are generated by earlier language models (MPT, LLaMA 1/2, Bard), which are weaker than the reference model but still achieve performance improvements with InfiFPO.\n\nFor W2, InfiFPO uses sequence-level log-probabilities, which can be precomputed and consume reasonable space. These probabilities can be loaded during training without loading the reference model into VRAM.", "rebuttal": ""}, {"review_id": "NRUH4fkUef", "reviewer": "Reviewer_v33j", "summary": "This paper proposes InfiFPO that fuses multi-source LLM probabilities at the sequence level to better align with human preferences, avoiding complex vocabulary alignment in traditional token-level fusion.", "questions": "- As infiFPO uses half the data for SFT and the remaining half for preference optimization, is the total data volume equivalent for SFT and model fusion methods?\n\n- In line 242, the authors state “… (55 vs. 160) … while avoiding complex vocabulary alignment.” Does this imply that the vocabulary alignment process is included in the 160 GPU hours for infiFusion and 225 GPU hours for FuseLLM? From my understanding, FuseLLM’s vocabulary alignment uses dynamic programming, which is CPU-intensive rather than GPU-intensive, would it be fair to count it in GPU hours?\n\n- Since Qwen2.5-Coder excels in coding tasks (achieving the highest code score in Table 2), would fusing Phi-4 with Qwen2.5-Coder alone lead to notable gains in coding or other related capabilities?\n\n- What is the shot setting used for the MMLU and other evaluations, zero-shot or few-shot? This should be clearly disclosed.\n\n- Could model performance improve further by increasing the data size (e.g., from 150k to 200k samples)?", "limitations": "- Besides only testing Phi-4 (14B) as the pivot, it would be interesting to see results for smaller models, such as 1B, 3B, or 7B backbones.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The implementation details are disclosed comprehensively, which makes potential reproduction feasible and straightforward for readers.\n- The derivation of the InfiFPO objective is clear and technically sound.\n- The preliminaries on model fusion and DPO are well-explained and effectively set the stage for the proposed method, making the paper easy to follow.\n\nWeaknesses:\n- The diversity of experiments appears quite limited. The main results are conducted solely using Phi-4 as the pivot model. Could the authors clarify why a stronger backbone, such as Qwen2.5-Instruct (which achieves an average performance of 80.97 vs. 79.95 in Table 2), was not used instead? I am concerned that the results may fluctuate significantly when changing the pivot model, especially given that infiFPT only slightly outperforms SFT-WRPO (by 0.53 on average) and no error bars or statistical significance tests are provided.\n- Regarding the dataset in the setup: since the authors constructed a new preference dataset, did they observe any sensitivity to sample quality? If so, I strongly encourage the authors to release their sampled dataset for reproducibility.\n- In the ablation study (Table 4), infiFPO without LN and PC performs significantly worse than all other model fusion and preference optimization methods. Does this imply that sequence-level modeling is less effective than token-level methods, despite this being a major claim and advantage of the paper? A more detailed and rigorous analysis would be valuable here.\n- In Table 2, why do all model fusion methods fuse exactly three source models? Under the same setting, would FuseChat outperform infiFPO if all five models were fused instead?\n\nI would be happy to discuss further with the authors in case I have missed some details.", "was_revised": true, "final_justification": "The authors’ responses have addressed most of my technical concerns, and I acknowledge that the additional experiments and analyses have significantly improved the work. If these discussions are incorporated into the revision, the paper would be substantially strengthened.\n\nHowever, I have two fairness concerns:\n1. The detailed response to my comments was posted after the July 30 rebuttal deadline. According to NeurIPS guidelines, such rebuttals should be ignored to ensure fairness to other submissions that complied with the timeline. If I strictly follow this guideline, I would need to retain my original rating of 3.\n2. The combined length of the authors’ responses appears to exceed the 10,000-character rebuttal limit. While the additional materials are informative, posting them during the discussion phase rather than the rebuttal phase risks circumventing the official limit and could set a precedent that disadvantages other submissions.\n\nGiven these considerations, I will raise my rating to 4 in recognition of the technical improvements demonstrated, but I would be happy to have a further discussion with the ACs, SACs, and PCs.", "rebuttal": ""}, {"review_id": "09pPvgHWLl", "reviewer": "Reviewer_xt73", "summary": "This paper presents InfiFPO, which performs model fusion during the preference alignment (PA) stage rather than the more common supervised fine-tuning. The method improves a single pivot model by learning from multiple source models. It modifies the Direct Preference Optimization (DPO) objective by replacing the fixed reference model with a fused source model that combines sequence-level probabilities from all source models. This approach avoids vocabulary alignment issues and preserves full probabilistic information for both preferred and dispreferred responses. The design includes techniques such as length normalization, probability clipping, and max-margin fusion to enhance training stability and effectiveness. Experiments using Phi-4 as the pivot model show improved performance across 11 benchmarks, increasing the average score from 79.95 to 83.33, with strong results in tasks involving math, coding, and reasoning.", "questions": "1. InfiFPO's Max-margin Fusion strategy assigns a weight of 1 to the source model with the largest deviation, and 0 to all others. This reduces the fusion term M_fu(y|x) to a single, dynamically chosen reference model. Consequently, the loss function seems to become a form of DPO with a changing external reference. How does this align with the goal of model fusion, which is typically understood as combining knowledge from *multiple* sources at once?\n\n2. On line 152, the paper notes that source models can sometimes assign anomalous probabilities, such as a low probability to a preferred response y_w. The Max-margin Fusion strategy favors models with the greatest deviation from the pivot. Does this not create a risk of preferentially selecting these anomalous models, which could in turn harm the final performance?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. A novel approach to model fusion with a practical design. The paper proposes integrating model fusion into the preference alignment phase, which is a departure from common practice. The method operates at the sequence-probability level, which effectively addresses the vocabulary alignment issue often encountered in token-level fusion. This design makes the framework applicable to a range of models with different tokenizers without requiring complex alignment heuristics.\n2. The proposed method is well-motivated with a clear derivation. The InfiFPO objective is derived from a constrained reinforcement learning formulation, providing a transparent theoretical basis for the approach. This derivation helps to interpret the final loss function as a trade-off between maximizing preference rewards and maintaining proximity to the knowledge of multiple source models.\n3. The framework demonstrates notable versatility. The analysis in the paper shows that InfiFPO is not restricted to the DPO algorithm. It can be integrated with other preference optimization methods, such as IPO and WRPO, leading to performance improvements in those contexts as well. This suggests that the core idea of replacing the reference model with a fused model could function as a general enhancement for a class of preference optimization algorithms.\n\n\n\n## Weaknesses\n\n1. The reported performance gains are consistent but marginal. While InfiFPO consistently outperforms the baselines across 11 benchmarks, the margin of improvement is often modest, in many cases less than a single percentage point over the next best method (e.g., SFT-WRPO).\n2. The effectiveness of the method is not demonstrated in a more challenging fusion scenario. In the experiments, the selected source models exhibit performance that is on par with, or in some cases inferior to, the pivot model (as shown in Table 2). A common and practical use case for model fusion is to enhance a pivot model using knowledge from source models that are significantly stronger. The current evaluation does not sufficiently show whether InfiFPO remains effective in such a scenario.\n3. The analysis of the proposed optimization strategies could be deeper. While the ablation studies confirm the utility of the three strategies (length normalization, probability clipping, max-margin fusion), the paper offers limited insight into *why* they work as they do. For example, a more detailed analysis of the max-margin strategy—perhaps by examining which source models are selected under different contexts—would provide a richer understanding of the fusion dynamics and strengthen the paper's contribution.", "was_revised": true, "final_justification": "The rebuttal has addressed some of my concerns, so I decide to increase my rating accordingly.", "rebuttal": ""}, {"review_id": "mciDYyY0TT", "reviewer": "Reviewer_kiK4", "summary": "The paper introduces InfiFPO, a novel preference optimization framework that enables implicit model fusion during the alignment stage of large language model (LLM) training. Instead of relying solely on a reference model as in DPO, InfiFPO replaces it with a fused distribution of multiple source models’ sequence-level probabilities, preserving richer preference signals. The authors propose a theoretically grounded offline training objective and introduce three enhancements—length normalization, probability clipping, and max-margin fusion—to improve robustness. Extensive experiments across 11 benchmarks demonstrate that InfiFPO consistently improves performance over existing model fusion and preference optimization baselines, especially in math, coding, and instruction-following tasks.", "questions": "1. While InfiFPO demonstrates strong overall performance, it remains unclear how the method behaves when different source models exhibit strongly conflicting preferences for the same prompt. In such cases, does the max-margin fusion strategy reliably identify the most reasonable source signal, or could it amplify noise from an outlier model? It would be helpful if the authors could include a robustness analysis or case studies illustrating how InfiFPO handles such disagreements in practice.\n\n2. Although InfiFPO improves benchmark performance, the paper lacks analysis on how the fused model enhances its internal representations or decision behavior. Could the authors provide deeper insight into this? Visualization techniques could strengthen the understanding of how information from different sources is integrated.\n\n3. The current training pipeline relies heavily on the reward model to select preferred and dispreferred responses. However, it is unclear how sensitive InfiFPO is to the quality and bias of the reward model. If the reward model introduces systematic bias, does InfiFPO risk overfitting to such signals? It would be valuable for the authors to discuss or empirically test the robustness of InfiFPO under different reward models.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper proposes InfiFPO, a novel framework that introduces implicit model fusion into the preference optimization stage of large language model alignment. By replacing the reference model in Direct Preference Optimization (DPO) with a sequence-level probabilistic fusion of multiple source models, InfiFPO effectively circumvents the vocabulary mismatch issues that have limited prior fusion methods. The approach is both conceptually elegant and practically efficient, avoiding the need for token-level alignment while preserving rich preference signals from heterogeneous sources. The framework is further enhanced by three principled strategies—length normalization, probability clipping, and max-margin fusion—which collectively improve stability and robustness. Extensive experiments across 11 benchmarks show that InfiFPO achieves consistent performance gains over strong model fusion and preference optimization baselines, particularly in math, code, and instruction-following tasks.\n\nDespite its clear contributions, some aspects of the work warrant further clarification. In particular, it remains unclear how the method handles conflicting preferences from source models—a scenario that may challenge the reliability of the max-margin fusion strategy. Additionally, while the empirical gains are evident, the paper lacks analysis on how the fused model's internal representations or calibration behavior are improved. The framework’s reliance on a fixed reward model to construct preference data also raises questions about sensitivity to reward quality or bias. These concerns do not undermine the core contributions, but addressing them—through robustness diagnostics, representation analyses, or ablations on the reward model—would significantly strengthen the work. Overall, the paper presents a meaningful and well-executed advancement in aligning LLMs via model fusion, with room for further insight into its underlying dynamics.", "was_revised": true, "final_justification": "The authors have addressed all my concerns, but I think the current score is valid for this work.", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper introduces a neat and technically solid idea: fusing multiple LLMs during preference alignment rather than just at the supervised fine-tuning stage. I like the clean formulation and the way the authors address vocabulary alignment issues by working at the sequence-probability level. It feels both elegant and practical. While some reviewers noted the improvements are modest and raised valid fairness concerns about rebuttal timing, the additional experiments strengthen the case that the method generalizes beyond the initial setup. Overall, I think this is a solid contribution with interesting potential, and I lean toward acceptance.
Decision: Accept (spotlight)