Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement

Authors: Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li
Primary area: foundation or frontier models, including LLMs
Keywords: Model Merging, Large Language Models

Abstract:
Merging Large Language Models (LLMs) aims to amalgamate multiple homologous LLMs into one with all the capabilities. Ideally, any LLMs sharing the same backbone should be mergeable, irrespective of whether they are Fine-Tuned (FT) with minor parameter changes or Pre-Trained (PT) with substantial parameter shifts. However, existing methods often manually assign the model importance, rendering them feasible only for LLMs with similar parameter alterations, such as multiple FT LLMs. The diverse parameter changed ranges between FT and PT LLMs pose challenges for current solutions in empirically determining the optimal combination. In this paper, we make a pioneering effort to broaden the applicability of merging techniques from FT to PT LLMs. We initially examine the efficacy of current methods in merging FT and PT LLMs, discovering that they struggle to deal with PT LLMs. Subsequently, we introduce an approach based on **W**e**I**ght **D**is**EN**tanglement (WIDEN) to effectively extend the merging scope, which first disentangles model weights into magnitude and direction components, and then performs adaptive fusion by considering their respective contributions. In the experiments, we merge Qwen1.5-Chat (an FT LLM with instruction-following skills) with Sailor (a PT LLM with multilingual abilities) across 1.8B, 4B, 7B, and 14B model sizes. Results reveal that: (1) existing solutions usually fail when merging Sailor, either losing both abilities or only retaining instruction-following skills; (2) WIDEN successfully injects the multilingual abilities of Sailor into Qwen1.5-Chat and make it proficient in Southeast Asian languages, achieving enhancements in the fundamental capabilities. In light of previous research, we also merge multiple 13B FT LLMs and observe that WIDEN achieves a balance of instruction following, mathematical reasoning, and code generation skills.

Decision: reject
Reviewer scores: 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730391972570

thread_type:
official_review

<thread_object>
creator: reviewer_3upt

<content>
summary:
The paper presents WIDEN, a novel merging technique for Large Language Models (LLMs), which extends the applicability of merging from finetuned (FT) to pretrained (PT) models by disentangling weights into magnitude and direction components. This weight disentanglement enables adaptive merging by quantifying each LLM's alteration relative to a shared backbone. The disentangled weights are ranked using normalized divergence scores compared to the pretrained baseline, and this ranking is used to compute an automated importance factor for each LLM. This results in a generalized form of several existing arithmetic methods for LLM merging. Experimental results suggest that WIDEN effectively balances multiple capabilities, such as multilingual and instruction-following skills, across FT and PT LLMs.

strengths:
- The paper’s effort to expand merging capabilities from FT to PT models is well-motivated and addresses a crucial gap in existing merging techniques.
- The methodology has a sound technical foundation, with a detailed four-step framework integrating weight disentanglement, ranking, and adaptive score calibration.
- The experimental setup is thorough, covering both conventional FT merging tasks and the new FT-PT merging setting. WIDEN’s performance across SEA and Open LLM Leaderboard benchmarks and comparison with multiple baselines highlights its applicability to diverse LLMs.
- The impact of each component within WIDEN is evaluated with an ablation experiment in Figure 2, demonstrating the importance of weight disentanglement and score calibration.

weaknesses:
- Although WIDEN generalizes across FT and PT models, it does not consistently outperform Task Arithmetic on all benchmarks. For instance, Task Arithmetic often shows competitive results on Open LLM Leaderboard tasks, raising concerns about WIDEN’s scalability and stability. For example, on the SEA benchmark, the performance improvement on 14B models is smaller than the 7B model, with the gap between Task Arithmetic and its claimed generalized form WIDEN narrowing as the LLMs become larger.
- The improvement WIDEN demonstrates is noticeably higher on SEA benchmarks than on the OpenLLM Leaderboard, yet the paper does not clarify why performance fluctuates between benchmarks. This omission raises questions about its adaptability to different domains or task settings.
- While grid search is used for tuning, the choice of hyperparameters (particularly t and s) lacks justification beyond empirical results. A clearer rationale or theoretical insight into their selection would enhance the robustness of WIDEN’s methodology.
- Although score calibration is a novel addition to ensure adaptive ranking and merging, values other than 1.0 should be evaluated in score calibration. The "ease of implementation" rationale is not good enough.

questions:
- How does WIDEN handle cases where the backbone (reference pretrained model) diverges substantially in structure or task specificity from both FT and PT models? Would WIDEN work with heterogeneous LLMs beyond those sharing the same backbone?
- Did the authors attempt to merge more than two or three models to evaluate WIDEN’s scalability and robustness? If so, what were the results, and how does performance change as the number of LLMs increases?
- Given WIDEN’s better performance on SEA benchmarks than on the OpenLLM Leaderboard, could the authors elaborate on why this discrepancy exists? Is WIDEN more suited to particular types of tasks or linguistic benchmarks?
- On tasks where Task Arithmetic performs better, why might WIDEN’s performance lag?
- Since WIDEN modifies weights adaptively, would it be feasible to incorporate it into a continual learning setup where multiple LLMs are progressively merged over time? Could this method be used for models other than LLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589870254

thread_type:
official_review

<thread_object>
creator: reviewer_kkew

<content>
summary:
This paper presents a pioneering effort in extending model merging to Pretrained LLMs utilizing weight disentanglement. Extensive studies on previous methods demonstrate their inability to perform when applied to pretrained LLM while the method proposed in the paper is able to solve the task with minimal performance drop compared to the models to be merged.

strengths:
1. The paper is the first successful attempt to incorporate the ability of PT LLM into model merging techniques.
2. Extensive experiments and analyses have demonstrated the effectiveness of the proposed method.
3. The paper is well-written and easy to follow.

weaknesses:
1. The experiments are only limited to Sailor, more results on different models could validate the effectiveness of the proposed method.
2. Despite being indicated by the method, the experiments didn't show evidence that the proposed method could work for multiple LLM cases. Some experiments from this perspective would be appreciated.

questions:
See weaknesses.

I'm not familiar with LLM merging and am open to discussion if misunderstood any part of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692285870

thread_type:
official_review

<thread_object>
creator: reviewer_w7xb

<content>
summary:
Merging multiple LLMs, particularly those with substantial parameter shifts from pre-training (PT), presents challenges for traditional merging methods. To address this issue, the paper introduces WIDEN (Weight Disentanglement), a novel approach for merging large language models (LLMs) that have undergone either fine-tuning (FT) or pre-training (PT). This method expands the applicability of model merging beyond conventional fine-tuned models.

strengths:
1. The paper makes a valuable contribution by identifying a critical limitation in existing model merging methods: their ineffectiveness when applied to continually pre-trained (PT) models. This insight is essential, as it highlights a gap in current merging techniques, which are generally only effective for fine-tuned (FT) models with minimal parameter shifts.
2. The paper introduces WIDEN (Weight Disentanglement), an innovative method that automatically computes the importance of weights during the merging process. WIDEN disentangles each model’s weights into magnitude and direction components, and then adapts the merging decisions based on the divergence of these components from a shared backbone. This approach removes the need for manually assigning scaling factors and effectively addresses the challenges posed by the varied parameter changes in both fine-tuned (FT) and pre-trained (PT) models.
3. The experimental results demonstrate that WIDEN outperforms existing merging methods by effectively combining both instruction-following and multilingual capabilities. The paper also evaluates WIDEN in traditional FT-only merging scenarios, where it achieves competitive performance compared to established methods.

weaknesses:
1. The paper assumes that continually pre-trained (PT) models inherently experience larger weight shifts than fine-tuned (FT) models, which serves as the justification for a new merging approach. However, this assumption may not hold universally, as the degree of weight change in PT models depends on factors such as the data domain and dataset size. This raises questions about the paper’s motivation and the general applicability of its problem formulation. A more thorough exploration or empirical verification of weight changes across PT and FT models would help substantiate this claim. The authors are expected to provide empirical evidence comparing the distribution of weight changes between PT and FT models across different domains, model sizes, and dataset sizes.
2. The proposed ranking mechanism in WIDEN calculates divergences in magnitude and direction separately for each weight relative to the backbone model. However, the reliability of comparing magnitudes across models with different directional vectors is questionable. When calculating magnitude differences, direction is not considered, meaning that the importance of weights in different models could be misinterpreted if their directions diverge. Similarly, comparing directional differences might be misleading if the corresponding magnitudes differ significantly between models. So, have the authors considered alternative approaches that jointly consider both magnitude and direction? Additionally, have the authors empirically analyzed how often misinterpretation occur in practice due to treating these components separately?
3. Although WIDEN is intended to be a general merging technique applicable to both FT and PT models, its performance in merging FT models is comparatively weak (as shown in Table 5). Given that the method is designed to be adaptable across model types, this underperformance raises concerns about its overall efficacy. Are there certain characteristics of FT models that WIDEN struggles with?
4. The experiments primarily focus on merging a specific PT model (Sailor) with a FT model, which limits the generalization ability of the results. Evaluating WIDEN on other PT models, particularly in diverse domains such as finance or healthcare, would provide stronger evidence of its effectiveness.

questions:
1. Equation 1: The shape of mD is equal to that of W. This equation should be corrected.
2. Equation 7: Could you provide a reason for not averaging all the differences by multiplying by 1/N?
3. Have the authors considered an alternative approach that compares each weight matrix on a column-by-column basis between the tuned model and the original backbone? Specifically, this approach would involve calculating and ranking differences column by column, rather than disentangling weights into separate magnitude and direction components.
4. How to grid search the hyperparameters for baselines methods? What validation dataset is used?
5. The paper should provide a figure to visually illustrate the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732255312264

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank all the reviewers for their valuable feedback and insightful comments on our work. We are encouraged by the recognition that our research is **well-motivated** and **addresses the limitations of existing merging methods when applied to Pre-Trained (PT) Large Language Models (LLMs)** (Reviewers W7XB, KKeW, and 3uPt), the proposed WIDEN is **innovative** and **technically sound** (Reviewers W7XB and 3uPt), our **experimental results are effective, extensive, and thorough** (Reviewers W7XB, KKeW, and 3uPt), and the paper is **well-written and easy to follow** (Reviewer KKeW).

To the best of our efforts, we have diligently addressed each concern raised by the reviewers through the following enhancements:

- We explored parameter changed ranges across a variety of existing LLMs spanning different domains, architectures, and model sizes;
- We conducted further merging experiments on 1.8B and 4B PT Sailor and Fine-Tuned (FT) Qwen1.5-Chat;
- We discussed the characteristics of LLMs that are more suitable for WIDEN;
- We provided explanations for the varied performance gaps between WIDEN and baselines across different benchmarks;
- We attempted to merge three FT LLMs using WIDEN and compared its performance with baselines;
- We investigated the performance of WIDEN under different settings of hyperparameters $t$ and $s$;
- We elaborated on the importance of the weight disentanglement operation in WIDEN and provided details of the grid search process.

All these improvements are highlighted in blue in the revised manuscript. We think that all the comments are very constructive and believe that the revisions can enhance the quality of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the model merging problem between PT and FT LLMs. The approach focuses on merging homogeneous models that are of a significant parameter divergence. The basic idea is to separate the weight into the magnitude and the direction components to automatically determine the merging weights. The authors took the Sailor as the PT LLM and showcased positive results. However, this paper received some common and critical concerns, especially that the method is limited to significant parameter change scenarios and performs less effectively with FT model merging and other PT models beyond the Sailor series. After discussion, Reviewer W7XB improved the rating score from 5 to 6 but decreased the confidence score from 4 to 3. Other reviewers kept their scores unchanged. After reviewing all materials, the AC found that the critical concerns were not all addressed. Considering the overall rating, the final recommendation is reject.