Title: InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model Fusion

Preference Optimization

LLMs
Abstract: Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods. Existing works on model fusion focus primarily on supervised fine-tuning (SFT), leaving preference alignment (PA) —a critical phase for enhancing LLM performance—largely unexplored.
The current few fusion methods on PA phase, like WRPO, simplify the process by utilizing only response outputs from source models while discarding their probability information. To address this limitation, we propose InfiFPO, a preference optimization method for implicit model fusion.
InfiFPO replaces the reference model in Direct Preference Optimization (DPO) with a fused source model that synthesizes multi-source probabilities at the sequence level, circumventing complex vocabulary alignment challenges in previous works and meanwhile maintaining the probability information.
By introducing probability clipping and max-margin fusion strategies, InfiFPO enables the pivot model to align with human preferences while effectively distilling knowledge from source models.
Comprehensive experiments on 11 widely-used benchmarks demonstrate that InfiFPO consistently outperforms existing model fusion and preference optimization methods. When using Phi-4 as the pivot model, InfiFPO improves its average performance from 79.95 to 83.33 on 11 benchmarks, significantly improving its capabilities in mathematics, coding, and reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces InfiFPO, a novel preference optimization framework that enables implicit model fusion during the alignment stage of large language model (LLM) training. Instead of relying solely on a reference model as in DPO, InfiFPO replaces it with a fused distribution of multiple source models’ sequence-level probabilities, preserving richer preference signals. The authors propose a theoretically grounded offline training objective and introduce three enhancements—length normalization, probability clipping, and max-margin fusion—to improve robustness. Extensive experiments across 11 benchmarks demonstrate that InfiFPO consistently improves performance over existing model fusion and preference optimization baselines, especially in math, coding, and instruction-following tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes InfiFPO, a novel framework that introduces implicit model fusion into the preference optimization stage of large language model alignment. By replacing the reference model in Direct Preference Optimization (DPO) with a sequence-level probabilistic fusion of multiple source models, InfiFPO effectively circumvents the vocabulary mismatch issues that have limited prior fusion methods. The approach is both conceptually elegant and practically efficient, avoiding the need for token-level alignment while preserving rich preference signals from heterogeneous sources. The framework is further enhanced by three principled strategies—length normalization, probability clipping, and max-margin fusion—which collectively improve stability and robustness. Extensive experiments across 11 benchmarks show that InfiFPO achieves consistent performance gains over strong model fusion and preference optimization baselines, particularly in math, code, and instruction-following tasks.\n\nDespite its clear contributions, some aspects of the work warrant further clarification. In particular, it remains unclear how the method handles conflicting preferences from source models—a scenario that may challenge the reliability of the max-margin fusion strategy. Additionally, while the empirical gains are evident, the paper lacks analysis on how the fused model's internal representations or calibration behavior are improved. The framework’s reliance on a fixed reward model to construct preference data also raises questions about sensitivity to reward quality or bias. These concerns do not undermine the core contributions, but addressing them—through robustness diagnostics, representation analyses, or ablations on the reward model—would significantly strengthen the work. Overall, the paper presents a meaningful and well-executed advancement in aligning LLMs via model fusion, with room for further insight into its underlying dynamics.\n\nQUESTIONS:\n1. While InfiFPO demonstrates strong overall performance, it remains unclear how the method behaves when different source models exhibit strongly conflicting preferences for the same prompt. In such cases, does the max-margin fusion strategy reliably identify the most reasonable source signal, or could it amplify noise from an outlier model? It would be helpful if the authors could include a robustness analysis or case studies illustrating how InfiFPO handles such disagreements in practice.\n\n2. Although InfiFPO improves benchmark performance, the paper lacks analysis on how the fused model enhances its internal representations or decision behavior. Could the authors provide deeper insight into this? Visualization techniques could strengthen the understanding of how information from different sources is integrated.\n\n3. The current training pipeline relies heavily on the reward model to select preferred and dispreferred responses. However, it is unclear how sensitive InfiFPO is to the quality and bias of the reward model. If the reward model introduces systematic bias, does InfiFPO risk overfitting to such signals? It would be valuable for the authors to discuss or empirically test the robustness of InfiFPO under different reward models.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents InfiFPO, which performs model fusion during the preference alignment (PA) stage rather than the more common supervised fine-tuning. The method improves a single pivot model by learning from multiple source models. It modifies the Direct Preference Optimization (DPO) objective by replacing the fixed reference model with a fused source model that combines sequence-level probabilities from all source models. This approach avoids vocabulary alignment issues and preserves full probabilistic information for both preferred and dispreferred responses. The design includes techniques such as length normalization, probability clipping, and max-margin fusion to enhance training stability and effectiveness. Experiments using Phi-4 as the pivot model show improved performance across 11 benchmarks, increasing the average score from 79.95 to 83.33, with strong results in tasks involving math, coding, and reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. A novel approach to model fusion with a practical design. The paper proposes integrating model fusion into the preference alignment phase, which is a departure from common practice. The method operates at the sequence-probability level, which effectively addresses the vocabulary alignment issue often encountered in token-level fusion. This design makes the framework applicable to a range of models with different tokenizers without requiring complex alignment heuristics.\n2. The proposed method is well-motivated with a clear derivation. The InfiFPO objective is derived from a constrained reinforcement learning formulation, providing a transparent theoretical basis for the approach. This derivation helps to interpret the final loss function as a trade-off between maximizing preference rewards and maintaining proximity to the knowledge of multiple source models.\n3. The framework demonstrates notable versatility. The analysis in the paper shows that InfiFPO is not restricted to the DPO algorithm. It can be integrated with other preference optimization methods, such as IPO and WRPO, leading to performance improvements in those contexts as well. This suggests that the core idea of replacing the reference model with a fused model could function as a general enhancement for a class of preference optimization algorithms.\n\n\n\n## Weaknesses\n\n1. The reported performance gains are consistent but marginal. While InfiFPO consistently outperforms the baselines across 11 benchmarks, the margin of improvement is often modest, in many cases less than a single percentage point over the next best method (e.g., SFT-WRPO).\n2. The effectiveness of the method is not demonstrated in a more challenging fusion scenario. In the experiments, the selected source models exhibit performance that is on par with, or in some cases inferior to, the pivot model (as shown in Table 2). A common and practical use case for model fusion is to enhance a pivot model using knowledge from source models that are significantly stronger. The current evaluation does not sufficiently show whether InfiFPO remains effective in such a scenario.\n3. The analysis of the proposed optimization strategies could be deeper. While the ablation studies confirm the utility of the three strategies (length normalization, probability clipping, max-margin fusion), the paper offers limited insight into *why* they work as they do. For example, a more detailed analysis of the max-margin strategy—perhaps by examining which source models are selected under different contexts—would provide a richer understanding of the fusion dynamics and strengthen the paper's contribution.\n\nQUESTIONS:\n1. InfiFPO's Max-margin Fusion strategy assigns a weight of 1 to the source model with the largest deviation, and 0 to all others. This reduces the fusion term M_fu(y|x) to a single, dynamically chosen reference model. Consequently, the loss function seems to become a form of DPO with a changing external reference. How does this align with the goal of model fusion, which is typically understood as combining knowledge from *multiple* sources at once?\n\n2. On line 152, the paper notes that source models can sometimes assign anomalous probabilities, such as a low probability to a preferred response y_w. The Max-margin Fusion strategy favors models with the greatest deviation from the pivot. Does this not create a risk of preferentially selecting these anomalous models, which could in turn harm the final performance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes InfiFPO that fuses multi-source LLM probabilities at the sequence level to better align with human preferences, avoiding complex vocabulary alignment in traditional token-level fusion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The implementation details are disclosed comprehensively, which makes potential reproduction feasible and straightforward for readers.\n- The derivation of the InfiFPO objective is clear and technically sound.\n- The preliminaries on model fusion and DPO are well-explained and effectively set the stage for the proposed method, making the paper easy to follow.\n\nWeaknesses:\n- The diversity of experiments appears quite limited. The main results are conducted solely using Phi-4 as the pivot model. Could the authors clarify why a stronger backbone, such as Qwen2.5-Instruct (which achieves an average performance of 80.97 vs. 79.95 in Table 2), was not used instead? I am concerned that the results may fluctuate significantly when changing the pivot model, especially given that infiFPT only slightly outperforms SFT-WRPO (by 0.53 on average) and no error bars or statistical significance tests are provided.\n- Regarding the dataset in the setup: since the authors constructed a new preference dataset, did they observe any sensitivity to sample quality? If so, I strongly encourage the authors to release their sampled dataset for reproducibility.\n- In the ablation study (Table 4), infiFPO without LN and PC performs significantly worse than all other model fusion and preference optimization methods. Does this imply that sequence-level modeling is less effective than token-level methods, despite this being a major claim and advantage of the paper? A more detailed and rigorous analysis would be valuable here.\n- In Table 2, why do all model fusion methods fuse exactly three source models? Under the same setting, would FuseChat outperform infiFPO if all five models were fused instead?\n\nI would be happy to discuss further with the authors in case I have missed some details.\n\nQUESTIONS:\n- As infiFPO uses half the data for SFT and the remaining half for preference optimization, is the total data volume equivalent for SFT and model fusion methods?\n\n- In line 242, the authors state “… (55 vs. 160) … while avoiding complex vocabulary alignment.” Does this imply that the vocabulary alignment process is included in the 160 GPU hours for infiFusion and 225 GPU hours for FuseLLM? From my understanding, FuseLLM’s vocabulary alignment uses dynamic programming, which is CPU-intensive rather than GPU-intensive, would it be fair to count it in GPU hours?\n\n- Since Qwen2.5-Coder excels in coding tasks (achieving the highest code score in Table 2), would fusing Phi-4 with Qwen2.5-Coder alone lead to notable gains in coding or other related capabilities?\n\n- What is the shot setting used for the MMLU and other evaluations, zero-shot or few-shot? This should be clearly disclosed.\n\n- Could model performance improve further by increasing the data size (e.g., from 150k to 200k samples)?\n\nLIMITATIONS:\n- Besides only testing Phi-4 (14B) as the pivot, it would be interesting to see results for smaller models, such as 1B, 3B, or 7B backbones.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces InfiFPO, a novel framework designed to fuse the capabilities of multiple Large Language Models (LLMs) during the preference alignment phase. The authors identify a clear gap in existing model fusion research, which has predominantly concentrated on the Supervised Fine-Tuning (SFT) stage while overlooking the critical preference alignment step. The core innovation of InfiFPO is replacing the static reference model used for KL-divergence computation in Direct Preference Optimization (DPO) with a dynamic, fused source model, enabling simultaneous preference optimization and model fusion. Empirical results demonstrate strong performance of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n[S1] Integrating preference optimization and model fusion through KL-divergence is technically sound and elegant. The derivation of the loss is clearly presented.\n\n[S2] Compared with multi-stage methods, simultaneous fusion and preference optimization can simplify the implementation and thus enhance practical applicability. Convincing empirical results show that InfiFPO can effectively achieve the two goals.\n\nWeaknesses:\n\n[W1] The data for preference optimization is generated by either the pivot model or reference models, leading to low KL-divergence. Is the InfiFPO method still applicable when data is from diverse sources, such as human-written or generated by models other than reference models?\n\n[W2] Memory usage scales linearly with the number of reference models. Parallelization and distributed strategies need to be carefully tuned when introducing larger reference models.\n\nQUESTIONS:\n[Q1] Are the log probabilities of reference models pre-computed and cached, or computed on-the-fly when training? The training time of InfiFPO seems to be only slightly larger than DPO+SFT, despite that numerous reference models are adopted.\n\nLIMITATIONS:\nSee Weaknesses. I suggest a brief discussion of the practical limitations of this approach and the trade-offs between the number of source models and memory and computational costs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}