Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement

## Abstract

Abstract Merging Large Language Models (LLMs) aims to amalgamate multiple homologous LLMs into one with all the capabilities. Ideally, any LLMs sharing the same backbone should be mergeable, irrespective of whether they are Fine-Tuned (FT) with minor parameter changes or Pre-Trained (PT) with substantial parameter shifts. However, existing methods often manually assign the model importance, rendering them feasible only for LLMs with similar parameter alterations, such as multiple FT LLMs. The diverse parameter changed ranges between FT and PT LLMs pose challenges for current solutions in empirically determining the optimal combination. In this paper, we make a pioneering effort to broaden the applicability of merging techniques from FT to PT LLMs. We initially examine the efficacy of current methods in merging FT and PT LLMs, discovering that they struggle to deal with PT LLMs. Subsequently, we introduce an approach based on W e I ght D is EN tanglement (WIDEN) to effectively extend the merging scope, which first disentangles model weights into magnitude and direction components, and then performs adaptive fusion by considering their respective contributions. In the experiments, we merge Qwen1.5-Chat (an FT LLM with instruction-following skills) with Sailor (a PT LLM with multilingual abilities) across 7B and 14B model scales. Results reveal that: (1) existing solutions usually fail when merging Sailor, either losing both abilities or only retaining instruction-following skills; (2) WIDEN successfully injects the multilingual abilities of Sailor into Qwen1.5-Chat and make it proficient in Southeast Asian languages, achieving enhancements in the fundamental capabilities. In light of previous research, we also merge multiple 13B FT LLMs and observe that WIDEN achieves a balanced amalgamation of instruction following, mathematical reasoning, and code generation skills.

## 1 Introduction

In recent years, model merging has sparked significant interest as a prominent topic, which intends to integrate multiple homologous models (sharing the same backbone) into a singular one that encapsulates all the abilities (Wortsman et al., ; Matena & Raffel, ; Ilharco et al., ; Jin et al., ; Jang et al., ; Yadav et al., ; Davari & Belilovsky, ; Yu et al., ). Distinct from other approaches that can also amalgamate various skills (e.g., ensemble learning (Mohammed & Kora, ), multi-task learning (Crawshaw, ; Zhang & Yang, )), model merging is lauded for its computational frugality, especially when applied to Large Language Models (LLMs). Notably, it achieves integration without using additional training data or even GPUs, establishing a new paradigm for efficiently combining LLMs’ capabilities (Yu et al., ).

Technically, there are predominantly two strategies to equip LLMs with desired capabilities (Zhao et al., ): fine-tuning to elicit existing skills (Wang et al., ; Zhang et al., ) and pre-training to inject new abilities (Wu et al., ). Existing merging methods mainly focus on integrating the skills of Fine-Tuned (FT) LLMs with minor parameter changes relative to the backbone, typically within 0.002 (Yu et al., ). However, it is crucial to acknowledge that pre-training is the cornerstone for fundamentally enhancing the capabilities of LLMs. The practicality of merging techniques in scenarios where Pre-Trained (PT) LLMs undergo substantial parameter shifts remains unexplored, as depicted in Figure 1. Consequently, if the application of merging is restricted to FT LLMs, its potential for broader improvement would be significantly constrained.

Figure: Figure 1: Issues of existing merging techniques.
Refer to caption: extracted/5776892/figures/motivation.jpg

To fill in the aforementioned blank, this work makes two key technical contributions.

We examine the feasibility of existing approaches in absorbing the abilities from PT LLMs. We investigate the performance of widely used arithmetic-based (Wortsman et al., ; Ilharco et al., ), geometric-based (Shoemake, ; Jang et al., ), and pruning-based (Yadav et al., ; Davari & Belilovsky, ; Yu et al., ) methods when merging FT and PT LLMs. As illustrated in Table 1, we find current methods either lose efficacy in retaining the abilities of PT LLMs (leading to a decrease of approximately 10 to 20 points on average) or fail to preserve both capabilities (resulting in an average degradation of about 15 and 30 points, respectively). One possible reason is that existing methods depend on manually assigned scaling terms to gauge the model contribution, which is only applicable when multiple LLMs depict comparable parameter alterations. Nonetheless, when confronted with diverse parameter changed ranges between FT and PT LLMs, deriving the optimal scaling factors according to human expertise becomes exceedingly arduous.

We propose a new solution grounded in WeIght DisENtanglement (WIDEN) to expand the scope of merging techniques from FT to PT LLMs. WIDEN tackles the drawbacks of existing works by automatically computing the model importance in the merging process without requiring manual specification, mitigating the influence induced by diverse parameter changed ranges between FT and PT LLMs. To be specific, WIDEN first disentangles each weight of a given LLM into two components: magnitude and direction. Then, the divergence of each component relative to the backbone is quantified to provide a numerical measure of how much each LLM has been altered. Next, WIDEN employs a ranking mechanism within each LLM to obtain the weight importance, tackling the diversity in parameter changed ranges between FT and PT LLMs. Finally, WIDEN performs adaptive merging on multiple LLMs by Softmax with the score calibration design.

We experiment with Qwen1.5-Chat (Bai et al., ) (an FT LLM with instruction-following skills) and Sailor (Dou et al., ) (a PT LLM with multilingual abilities for South-East Asia) across 7B and 14B model scales to verify the effectiveness of WIDEN for model merging(^1^11To the best of our knowledge, Sailor is one of the few publicly accessible LLM that has undergone extensive continued pre-training upon the open-source Qwen1.5 model, ideally suitable to our experimental scenarios. Therefore, Sailor and its homologous counterpart, Qwen1.5-Chat, are selected for our study.). Experimental results indicate that WIDEN outperforms existing methods by not only absorbing the multilingual abilities of Sailor but also preserving the instruction-following skills of Qwen1.5-Chat. For example, in Table 1, WIDEN slightly causes an average reduction of 0.23 and 1.33 points for Sailor-14B and Qwen1.5-14B-Chat, respectively. These observations demonstrate that WIDEN effectively extends the applicability of merging techniques from FT to PT LLMs. Considering previous works, we further merge three FT LLMs including WizardLM-13B (Xu et al., ) for instruction following, WizardMath-13B (Luo et al., ) for mathematical reasoning, and llama-2-13b-code-alpaca (Chaudhary, ) for code generation. Results show that WIDEN is also feasible under the conventional setting and can strike a favorable balance among these capabilities.

Resources are available at [https://github.com/yule-BUAA/MergeLLM](https://github.com/yule-BUAA/MergeLLM).

## 2 Related Work

Fine-Tuning and Pre-Training of LLMs.
Generally, LLMs can be adapted to various tasks via two strategies: fine-tuning and pre-training (Zhao et al., ). Fine-tuning is designed to elicit backbones with specific skills by optimizing them on a limited set of task-specific data, obtaining FT LLMs with skills such as instruction following (Rafailov et al., ; Song et al., ) and mathematical reasoning (Yuan et al., ; Luo et al., ). The fine-tuning process typically brings minor modifications to the model parameters (Yu et al., ), holding true for both full fine-tuning approaches (Radford et al., ; Devlin et al., ) and parameter-efficient fine-tuning techniques (Houlsby et al., ; Li & Liang, ; Lester et al., ; Hu et al., ).
In contrast to fine-tuning, pre-training trains LLMs on large-scale raw corpora to enhance models with domain knowledge (Ke et al., ; ; Cheng et al., ), deriving PT LLMs with fundamental abilities like finance analysis (Xie et al., ) and law assistance (Colombo et al., ). Pre-training often leads to more obvious parameter shifts than fine-tuning due to extensive data used during the phase. Different from current merging methods that are only applicable to FT LLMs, this paper proposes a new solution to innovatively harness the capabilities of PT LLMs.

Merging of LLMs.
Model merging aims to amalgamate multiple homologous models (derived from the same backbone) into a single one that possesses all the abilities (Wortsman et al., ; Matena & Raffel, ; Ilharco et al., ; Jin et al., ; Jang et al., ; Yadav et al., ; Davari & Belilovsky, ; Yu et al., ). The allure of the model merging technique stems from its minimal computational expense, particularly favorable for LLMs, which can be realized without retraining or GPUs (Yu et al., ).
Existing merging techniques that are feasible for LLMs can be broadly categorized into three groups, which are based on arithmetic, geometric, and pruning. Average Merging (Wortsman et al., ) and Task Arithmetic (Ilharco et al., ) belong to arithmetic-based approaches. The former utilizes averaged parameters to create the merged model, whereas the latter introduces the concept of task vector (i.e., parameter difference between an FT model and its backbone) and uses a scaling term to regulate the importance of various models. As geometric-based methods, both SLERP (Shoemake, ) and Model Stock (Jang et al., ) consider the geometric properties in weight space. In particular, SLERP is specifically designed for the integration of two models, which performs spherical interpolation of model weights. Model Stock approximates a center-close weight based on several FT models, utilizing their backbone as an anchor point. TIES-Merging (Yadav et al., ), Breadcrumbs (Davari & Belilovsky, ), and DARE (Yu et al., ) are methods based on pruning. TIES-Merging eliminates parameter interference among multiple models by first removing delta parameters with low magnitudes and then merging parameters with consistent signs after resolving disagreements. Breadcrumbs masks out the extreme tails (also known as outliners) of the absolute magnitude distribution of task vectors to obtain the final model. DARE is a versatile plug-in for existing merging approaches, which first randomly drops delta parameters and then rescales the remaining ones to maintain model performance. However, most of the current methods manually determine the importance of each model, suitable only for LLMs with similar parameter changes. When the parameter changed ranges are diverse between FT and PT LLMs, determining the optimal combination becomes overwhelmingly challenging. This paper initially verifies the limitations of existing methods in combining the abilities of PT LLMs. Subsequently, an approach based on weight disentanglement is introduced to effectively expand the scope of merging techniques from FT to PT LLMs.

## 3 Methodology

### 3.1 Preliminaries

Merging Beyond FT LLMs. Given a collection of $N$ homologous LLMs characterized by parameters $\left\{\Theta^{1},\Theta^{2},\cdots,\Theta^{N}\right\}$, all of which share the same backbone with parameters $\Theta_{\text{PRE}}$, model merging aims to amalgamate the parameters of $N$ LLMs into a singular model with all the capabilities, denoted as $\Theta_{\text{M}}$. Previous studies only focus on combining the skills of FT LLMs parameterized by $\left\{\Theta_{\text{FT}}^{1},\Theta_{\text{FT}}^{2},\cdots,\Theta_{\text{FT}}
^{N}\right\}$, where each model exhibits slight parameter changes, usually within 0.002 (Yu et al., ). In this paper, we extend the scope of merging techniques from FT to PT LLMs, intending to absorb the abilities of PT LLMs. Therefore, the parameters targeted for merging become $\left\{\Theta_{\text{TYPE}_{1}}^{1},\Theta_{\text{TYPE}_{2}}^{2},\cdots,\Theta
_{\text{TYPE}_{N}}^{N}\right\}$, where $\text{TYPE}_{n}$ ($1\leq n\leq N$) can be either FT or PT.

Weight Disentanglement. As outlined in Salimans & Kingma (); Liu et al. (), a weight $\bm{W}\in\mathbb{R}^{d\times k}$ can be disentangled into two components: a row vector $\bm{m}\in\mathbb{R}^{1\times k}$ that captures the magnitudes and a matrix $\bm{D}\in\mathbb{R}^{d\times k}$ that stores the direction vectors. Here, $d$ and $k$ represent the output and input dimensions. Mathematically, the disentanglement of weight $\bm{W}$ is achieved by

$$ $\bm{W}=\bm{m}\bm{D}=\|\bm{W}\|_{c}\frac{\bm{W}}{\|\bm{W}\|_{c}},$ (1) $$

where $\|\cdot\|_{c}$ denotes the vector-wise $l_{c}$-norm of a matrix across each column. Such a decoupling operation guarantees that each column $\bm{D}_{:,j}$ ($1\leq j\leq k$) is a unit vector, and scalar $m_{j}\in\bm{m}$ signifies the magnitude of direction vector $\bm{D}_{:,j}$. Since the primary challenge of extending merging scope to PT LLMs lies in the manual assignment of model importance, we employ weight disentanglement to initially decouple weights into magnitudes and directions, and then automatically compute the weight importance without human expertise based on these two components.

### 3.2 Exploring Efficacy of Current Methods When Merging PT LLMs

We investigate the efficacy of seven commonly used merging techniques when integrating the abilities of PT LLMs. To be specific, Average Merging (Wortsman et al., ) and Task Arithmetic (Ilharco et al., ) are arithmetic-based methods. SLERP (Shoemake, ) and Model Stock (Jang et al., ) belong to geometric-based approaches. TIES-Merging (Yadav et al., ), Breadcrumbs (Davari & Belilovsky, ) and DARE (Yu et al., ) are pruning-based solutions. Please see Section A.3 for detailed descriptions of these methods. To evaluate the performance, we attempt to combine the instruction-following skills of an FT LLM, Qwen1.5-Chat (Bai et al., ), and the multilingual abilities of a PT LLM, Sailor (Dou et al., ). Experimental setup, results, and analysis can be found in Section 4.

Since this part mainly concentrates on the feasibility of merging techniques when applied to PT LLMs, we highlight the key conclusion pertinent to PT LLMs: existing merging approaches face difficulties in preserving the abilities of PT LLMs. As evidenced in Table 2, the performance of all merging methods on the multilingual abilities significantly declines. This phenomenon is largely attributed to the reliance of most methods on manually assigned scaling factors to determine the contribution of each model at various levels throughout the merging process, encompassing model level (Ilharco et al., ; Yadav et al., ; Davari & Belilovsky, ), layer/module level (Goddard et al., ), and parameter level (Shoemake, ). The diverse parameter changed ranges between FT and PT LLMs complicate the manual assignment of model importance, making it intractable to define optimal scaling factors case by case.

### 3.3 Extending Merging Scope to PT LLMs via Weight Disentanglement

We present a new approach based on WeIght DisENtanglement (WIDEN) to innovatively broaden the applicability of model merging techniques from FT to PT LLMs, whose key concept is to adaptively assess the importance of weights during the merging process for neutralizing the effects of diverse parameter changed ranges between FT and PT LLMs. The framework of WIDEN comprises four main steps. Given the weights of LLMs (including the backbone as well as models to be merged), WIDEN 1) disentangles each weight into a row vector of magnitudes and a matrix of direction vectors; 2) estimates weight divergence relative to the backbone founded on absolute values of magnitude alterations and cosine similarities between direction vectors; 3) ranks the weights inside each LLM grounded in their divergence to derive the weight importance, thereby mitigating the impact of diverse parameter changed ranges; 4) merges multiple LLMs into a single one according to the obtained weight importance via Softmax with score calibration.

Disentangling Weights of LLMs.
Given multiple homologous LLMs (each LLM can be obtained by either FT or PT) with parameters $\left\{\Theta^{1},\Theta^{2},\cdots,\Theta^{N}\right\}$ as well as the backbone with parameters $\Theta_{\text{PRE}}$, we first perform weight disentanglement for the parameters. Take $\bm{W}^{n}\in\Theta^{n}$ with shape $\mathbb{R}^{d\times k}$ as an example(^2^22Note that $\Theta^{n}$ represents the collection of parameters of the $n$-th LLM, consisting of a multitude of weights.). $\bm{W}^{n}$ can be decoupled into $\bm{m}^{n}=\|\bm{W}^{n}\|_{c}\in\mathbb{R}^{1\times k}$ and $\bm{D}^{n}=\frac{\bm{W}^{n}}{\|\bm{W}^{n}\|_{c}}\in\mathbb{R}^{d\times k}$. After applying this disentanglement across all the LLMs, we can obtain the sets of row vectors of magnitudes $\{\bm{m}^{n}\}_{n=1}^{N}\cup\{\bm{m}_{\text{PRE}}\}$ and matrices of direction vectors $\{\bm{D}^{n}\}_{n=1}^{N}\cup\{\bm{D}_{\text{PRE}}\}$.

Estimating Weight Divergence Relative to Backbone.
We estimate the weight divergence of each LLM relative to the backbone from the perspective of magnitudes and directions with two measurements. To be specific, we compute the absolute values of magnitude alterations and determine the changes between direction vectors based on cosine similarities as follows,

$$ $\begin{split}&\Delta\bm{m}^{n}=|\bm{m}^{n}\ -\bm{m}_{\text{PRE}}|\in\mathbb{R} ^{1\times k},\ \text{for}\ 1\leq n\leq N,\\ \Delta D_{j}^{n}=1-&\text{CosineSimilarity}(\bm{D}_{:,j}^{n},{\bm{D}_{\text{ PRE}}}_{:,j})\in\mathbb{R},\ \text{for}\ 1\leq j\leq k,\ 1\leq n\leq N,\end{split}$ (2) $$

where $\text{CosineSimilarity}(\bm{x},\bm{y})=\frac{\bm{x}\cdot\bm{y}}{\|\bm{x}\|_{2}
\cdot\|\bm{y}\|_{2}}$. Thus, we obtain the divergences of the LLMs relative to the backbone in both magnitudes $\{\Delta\bm{m}^{n}\in\mathbb{R}^{1\times k}\}_{n=1}^{N}$ and directions $\{\Delta\bm{D}^{n}\in\mathbb{R}^{1\times k}\}_{n=1}^{N}$.

Ranking Weights Inside Each LLM.
We design a ranking mechanism to alleviate the potential impact of diverse parameter changed ranges among various LLMs, which assigns importance to the weights within each LLM according to their divergence relative to the backbone (greater divergence indicates higher essentiality). The ranking mechanism is applied to both the magnitudes and the directions of weights. To illustrate, consider the magnitudes as an instance. Given $\Delta\bm{m}^{n}\in\mathbb{R}^{1\times k}$ of the $n$-th LLM, we initially sort $\Delta\bm{m}^{n}$ in ascending order, yielding an index row vector $\bm{m}_{\text{IND}}^{n}\in\mathbb{R}^{1\times k}$ that contains values ranging from 1 to $k$. Subsequently, we derive a row vector $\widetilde{\bm{m}}^{n}\in\mathbb{R}^{1\times k}$ that encapsulates normalized ranking scores based on $\bm{m}_{\text{IND}}^{n}$, which is computed by

$$ ${\widetilde{m}^{n}}_{{m_{\text{IND}}^{n}}_{j}}=j/k,\ \text{for}\ 1\leq j\leq k.$ (3) $$

$\widetilde{\bm{m}}^{n}\in\mathbb{R}^{1\times k}$ represents the normalized importance of each position within the range $[1,\cdots,k]$ for the $n$-th LLM. Following the same procedure, the directions of weights can also be assigned with normalized importance, which can be denoted by $\widetilde{\bm{D}}^{n}\in\mathbb{R}^{1\times k}$. Such a ranking mechanism ensures that, within each LLM, the importance of magnitudes and directions is uniformly distributed between 0 and 1, thereby eliminating the potential influences arising from diverse parameter changed ranges between FT and PT LLMs. After applying the ranking operation for all the LLMs, we can ultimately obtain $\{\widetilde{\bm{m}}^{n}\in\mathbb{R}^{1\times k}\}_{n=1}^{N}$ and $\{\widetilde{\bm{D}}^{n}\in\mathbb{R}^{1\times k}\}_{n=1}^{N}$.

Merging LLMs via Softmax with Score Calibration.
We employ an adaptive merging strategy for multiple LLMs through a Softmax function, complemented by score calibration. Initially, we calculate the importance scores for magnitudes and directions by applying the Softmax function to $\{\widetilde{\bm{m}}^{n}\in\mathbb{R}^{1\times k}\}_{n=1}^{N}$ and $\{\widetilde{\bm{D}}^{n}\in\mathbb{R}^{1\times k}\}_{n=1}^{N}$, yielding $\widetilde{\bm{\mathcal{M}}},\widetilde{\bm{\mathcal{D}}}\in\mathbb{R}^{N
\times k}$ by

$$ $\begin{split}\widetilde{\mathcal{M}}_{n,j}=\frac{\exp{(\widetilde{m}^{n}_{j})} }{\sum_{n^{\prime}=1}^{N}\exp{(\widetilde{m}^{n^{\prime}}_{j})}}\in\mathbb{R}, \ \widetilde{\mathcal{D}}_{n,j}=\frac{\exp{(\widetilde{D}^{n}_{j})}}{\sum_{n^{ \prime}=1}^{N}\exp{(\widetilde{D}^{n^{\prime}}_{j})}}\in\mathbb{R},\ \text{for }\ 1\leq j\leq k,\ 1\leq n\leq N,\end{split}$ (4) $$

However, Softmax restricts the sum of parameter importance across multiple LLMs to 1, potentially diminishing the significance of crucial parameters in certain cases. Thus, we incorporate a score calibration operation to relax the constraint of Softmax for essential parameters. We identify crucial parameters as those whose importance exceeds the average level by a factor of $t$ as follows,

$$ $\mathbb{P}^{n}_{m}=\{j|\widetilde{m}^{n}_{j}>\frac{t}{k}\cdot\sum\nolimits_{j^ {\prime}=1}^{k}\widetilde{m}^{n}_{j^{\prime}}\},\ \mathbb{P}^{n}_{D}=\{j| \widetilde{D}^{n}_{j}>\frac{t}{k}\cdot\sum\nolimits_{j^{\prime}=1}^{k} \widetilde{D}^{n}_{j^{\prime}}\}.$ (5) $$

Subsequently, we calibrate the scores using $\mathbb{P}^{n}_{m}$ and $\mathbb{P}^{n}_{D}$ by

$$ $\displaystyle\begin{aligned} \mathcal{M}_{n,j}&=\begin{cases}s,&\text{if }j\in \mathbb{P}^{n}_{m}\\ \widetilde{\mathcal{M}}_{n,j},&\text{if }j\notin\mathbb{P}^{n}_{m}\end{cases} \end{aligned},\ \ \begin{aligned} \mathcal{D}_{n,j}&=\begin{cases}s,&\text{if }j\in\mathbb{P}^{n}_{D}\\ \widetilde{\mathcal{D}}_{n,j},&\text{if }j\notin\mathbb{P}^{n}_{D}\end{cases} \end{aligned},\ \$ (6) $$

where $s$ regulates the numerical value of score calibration. Finally, we integrate the weights of multiple LLMs into $\bm{W}_{\text{M}}$ by considering the adjusted contributions of both magnitudes and directions,

$$ $\bm{W}_{\text{M}}=\bm{W}_{\text{PRE}}+\sum\limits_{n=1}^{N}\frac{\bm{\mathcal{ M}}_{n,:}+\bm{\mathcal{D}}_{n,:}}{2}\odot\left(\bm{W}^{n}-\bm{W}_{\text{PRE}} \right)\in\mathbb{R}^{d\times k}.$ (7) $$

Remark 1. The aforementioned procedure is designed to deal with two-dimensional weights within LLMs, accounting for both magnitudes and directions. For one-dimensional parameters, such as weights in normalization layers and biases in linear transformations, we handle them as vectors of magnitudes and estimate their changes relative to the backbone by absolute values of the differences.

Remark 2. Existing arithmetic-based merging methods including Average Merging (Wortsman et al., ) and Task Arithmetic (Ilharco et al., ), can be viewed as special instances of WIDEN. Specifically, when $t<0.0$ and $s=1/N$, WIDEN aligns with the principles of Average Merging. When $t<0.0$ and $s=\lambda$ ($\lambda$ is the scaling term in Task Arithmetic), WIDEN becomes Task Arithmetic. Please refer to Section A.4 for the detailed analysis.

## 4 Experiments

We conduct experiments on model merging in two scenarios: 1) integrating both FT and PT LLMs, a new setting not explored before; 2) combining FT LLMs as in previous research.

### 4.1 Experimental Setup

Merging FT and PT LLMs.
We choose Qwen1.5-Chat (Bai et al., ) with instruction-following skills as the FT LLM and select Sailor (Dou et al., ) with multilingual abilities for South-East Asia as the PT LLM. Both models adopt Qwen1.5 (Bai et al., ) as the backbone. Open LLM Leaderboard (Beeching et al., ) and benchmark for South-East Asian languages (Dou et al., ) are used for evaluating the performance of models across 7B and 14B sizes.

Merging FT LLMs.
In accordance with Yu et al. (), we merge three FT LLMs that are based on Llama-2-13b (Touvron et al., ): WizardLM-13B (Xu et al., ) for instruction following, WizardMath-13B (Luo et al., ) for mathematical reasoning, and llama-2-13b-code-alpaca (Chaudhary, ) for code generation. AlpacaEval 2.0 (Dubois et al., ), GSM8K (Cobbe et al., ), MATH (Hendrycks et al., ), HumanEval (Chen et al., ), and MBPP (Austin et al., ) are utilized for evaluation.

Please see Section A.2 for the overview and evaluation metrics of the benchmarks. Also, refer to Table 7 in Section A.1 for the details of FT and PT LLMs. We compare WIDEN with seven popular baselines for model merging, including Average Merging (Wortsman et al., ), Task Arithmetic (Ilharco et al., ), SLERP (Shoemake, ), Model Stock (Jang et al., ), TIES-Merging (Yadav et al., ), Breadcrumbs (Davari & Belilovsky, ), and DARE (Yu et al., ). See Section 3.2 and Section A.3 for more descriptions.

Configurations of Merging Methods.
We apply grid search to identify the optimal settings for various merging techniques. The proposed WIDEN utilizes $l_{2}$ normalization and involves two hyperparameters: $s$ and $t$. For ease of implementation, the score calibration factor $s$ is consistently fixed to 1.0 across all the cases. The factor $t$ is determined by grid search. Please refer to Table 8 in Section A.5 for detailed information about the searched ranges.

Hardware Requirements. The process of merging LLMs requires only CPU resources. To evaluate the merged LLMs, we employ A100 GPUs equipped with 80 GB of memory. Notably, all the experiments can be successfully reproduced using a single A100 GPU.

### 4.2 Performance of Merging FT and PT LLMs

Table 2 shows the results of merging Qwen1.5-Chat and Sailor on South-East Asian language benchmark. Since Average Merging is a special case of Task Arithmetic when the scaling term is 0.5, we thereby only report the results of Task Arithmetic, which inherently include the performance of Average Merging. Note that th, id, vi, and jv are abbreviations of Thai, Indonesian, Vietnamese, and Javanese. The best and second-best results are marked in bold and underlined fonts. From Table 2, two conclusions can be summarized.

**Table 2: Results of merging Qwen1.5-Chat and Sailor on South-East Asian language benchmark.**
| Size | Models | Merging<br>Methods | XQuAD | TydiQA | XQuAD | XCOPA | Belebele | M3Exam | Average | Average<br>Rank |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| th | id | vi | th | id | vi | th | id | vi | jv |  |  |  |  |  |
| 7B | Qwen1.5 | / | 53.79/69.30 | 57.17/77.28 | 56.63/76.99 | 54.20 | 62.20 | 66.20 | 38.33 | 42.00 | 42.89 | 26.15 | 55.63 | / |
| Qwen1.5-Chat | / | 24.28/46.77 | 42.30/67.57 | 45.51/69.91 | 56.20 | 66.80 | 70.40 | 38.67 | 43.11 | 47.11 | 28.30 | 49.76 | / |  |
| Sailor | / | 57.88/71.06 | 60.53/75.42 | 53.81/74.62 | 59.00 | 72.20 | 72.20 | 41.56 | 44.33 | 45.33 | 32.88 | 58.52 | / |  |
| Qwen1.5-Chat<br>& Sailor | Task Arithmetic | 28.20/49.62 | 45.84/65.78 | 37.38/61.53 | 63.20 | 77.60 | 73.40 | 38.89 | 46.89 | 45.11 | 30.46 | 51.07 | 2.15 |  |
| SLERP | 16.62/43.62 | 20.53/54.02 | 33.70/61.49 | 55.80 | 73.40 | 73.00 | 38.44 | 47.89 | 47.56 | 28.30 | 45.72 | 3.23 |  |  |
| Model Stock | 26.72/52.69 | 24.78/58.88 | 43.80/69.50 | 54.60 | 66.00 | 69.40 | 37.33 | 42.78 | 43.67 | 27.76 | 47.53 | 3.31 |  |  |
| TIES-Merging | 0.61/8.84 | 5.66/17.23 | 7.70/20.78 | 50.20 | 62.20 | 59.80 | 30.22 | 35.33 | 35.11 | 25.07 | 27.60 | 5.54 |  |  |
| Breadcrumbs | 6.79/11.38 | 7.61/15.23 | 12.32/27.90 | 51.40 | 66.40 | 57.20 | 31.33 | 34.00 | 32.56 | 24.53 | 29.13 | 5.23 |  |  |
| WIDEN | 42.65/64.21 | 45.84/73.37 | 48.42/73.17 | 60.20 | 77.40 | 73.60 | 40.11 | 51.11 | 48.56 | 32.88 | 56.27 | 1.15 |  |  |
| 14B | Qwen1.5 | / | 55.53/74.39 | 60.35/81.07 | 57.66/77.62 | 58.40 | 70.40 | 72.60 | 41.22 | 48.67 | 44.44 | 26.15 | 59.12 | / |
| Qwen1.5-Chat | / | 33.59/59.98 | 37.17/65.46 | 44.14/71.91 | 61.80 | 75.20 | 71.80 | 44.00 | 51.00 | 52.67 | 29.92 | 53.74 | / |  |
| Sailor | / | 49.43/70.01 | 58.94/77.85 | 57.74/77.34 | 62.60 | 77.60 | 78.60 | 40.89 | 47.67 | 47.11 | 32.88 | 59.90 | / |  |
| Qwen1.5-Chat<br>& Sailor | Task Arithmetic | 8.53/24.39 | 13.45/33.54 | 13.52/25.75 | 59.80 | 82.40 | 78.20 | 46.00 | 56.33 | 53.78 | 33.69 | 40.72 | 2.54 |  |
| SLERP | 14.53/44.70 | 22.48/61.67 | 42.69/69.48 | 61.80 | 75.60 | 74.60 | 43.22 | 52.56 | 50.56 | 29.92 | 49.52 | 2.46 |  |  |
| Model Stock | 25.59/53.10 | 14.87/51.19 | 44.74/70.20 | 58.60 | 70.40 | 71.80 | 42.67 | 49.89 | 45.11 | 27.22 | 48.11 | 3.08 |  |  |
| TIES-Merging | 0.44/8.78 | 1.42/12.87 | 0.00/6.95 | 55.20 | 69.20 | 67.20 | 32.78 | 39.00 | 37.11 | 27.22 | 27.55 | 5.46 |  |  |
| Breadcrumbs | 1.22/6.48 | 2.30/20.88 | 3.17/14.46 | 52.20 | 64.60 | 63.40 | 34.78 | 42.11 | 40.67 | 26.68 | 28.69 | 5.23 |  |  |
| WIDEN | 49.61/73.16 | 50.62/75.09 | 54.75/78.23 | 60.80 | 77.40 | 74.60 | 42.22 | 56.22 | 50.44 | 32.61 | 59.67 | 1.77 |  |  |

Firstly, existing model merging approaches encounter significant challenges when incorporating the multilingual abilities of Sailor, leading to a marked decline in performance. The downturn is probably attributed to the difficulty in determining the optimal combination due to diverse parameter changed ranges between Qwen1.5-Chat and Sailor. We also notice that the reduction is particularly pronounced in pruning-based methods, prompting us to conduct additional verifications. As demonstrated in Table 3, we find that the feasibility of pruning strategies such as DARE and Magnitude-based Pruning (MP) in TIES-Merging and Breadcrumbs is severely compromised with minor parameter drop rates on Sailor-7B, far below the levels reported results in the original studies (i.e., 0.9 in DARE, 0.8 in TIES-Merging, and 0.85 in Breadcrumbs), diminishing the effectiveness of pruning in alleviating parameter interference. As a result, DARE fails to serve as a plug-in for existing merging techniques when considering PT LLMs, and its inferior results are excluded.

**Table 3: Performance of pruning strategies on Sailor-7B for Vietnamese-related tasks.**
|  | Drop Rate | XQuAD | XCOPA | Belebele |
| --- | --- | --- | --- | --- |
| Sailor-7B | / | 53.81/74.62 | 72.20 | 45.33 |
| DARE | 0.1 | 47.56/66.95 | 64.20 | 41.00 |
| 0.3 | 5.90/16.05 | 55.60 | 30.56 |  |
| MP | 0.1 | 54.23/75.16 | 72.80 | 45.44 |
| 0.3 | 52.44/73.53 | 72.20 | 44.78 |  |
| 0.5 | 49.19/70.11 | 70.00 | 43.67 |  |
| 0.8 | 13.77/30.13 | 59.00 | 34.56 |  |

Secondly, WIDEN effectively assimilates the multilingual capabilities of Sailor, emerging as the top performer among all the merging techniques. The key advantage of WIDEN lies in the adaptive computation of weight importance by considering both magnitudes and directions during the merging process, mitigating the influence of diverse parameter changed ranges between FT and PT LLMs.

**Table 4: Performance of merging Qwen1.5-Chat and Sailor on Open LLM Leaderboard.**
| Size | Models | Merging<br>Methods | ARC | Hella-<br>Swag | MMLU | Truthful-<br>QA | Wino-<br>grande | GSM8K | Average | Average<br>Rank |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 7B | Qwen1.5 | / | 54.86 | 78.45 | 60.60 | 51.09 | 71.03 | 56.79 | 62.14 | / |
| Qwen1.5-Chat | / | 56.14 | 78.71 | 60.18 | 53.61 | 67.48 | 54.21 | 61.72 | / |  |
| Sailor | / | 49.57 | 76.13 | 52.91 | 40.07 | 71.35 | 34.65 | 54.11 | / |  |
| Qwen1.5-Chat<br>& Sailor | Task Arithmetic | 52.05 | 75.15 | 59.38 | 50.84 | 69.77 | 25.55 | 55.46 | 3.50 |  |
| SLERP | 54.78 | 76.20 | 60.76 | 50.78 | 71.51 | 55.50 | 61.59 | 2.33 |  |  |
| Model Stock | 55.12 | 76.29 | 61.18 | 49.33 | 71.43 | 55.80 | 61.53 | 2.00 |  |  |
| TIES-Merging | 43.86 | 56.88 | 52.39 | 46.59 | 67.56 | 0.00 | 44.55 | 5.67 |  |  |
| Breadcrumbs | 47.18 | 49.99 | 52.66 | 52.05 | 64.88 | 0.45 | 44.53 | 4.67 |  |  |
| WIDEN | 53.84 | 76.25 | 57.65 | 49.34 | 71.90 | 44.81 | 58.97 | 2.83 |  |  |
| 14B | Qwen1.5 | / | 56.40 | 81.22 | 67.79 | 52.04 | 74.43 | 68.01 | 66.65 | / |
| Qwen1.5-Chat | / | 57.25 | 82.56 | 67.48 | 60.42 | 72.69 | 68.08 | 68.08 | / |  |
| Sailor | / | 55.46 | 80.31 | 62.95 | 46.64 | 76.80 | 61.94 | 64.02 | / |  |
| Qwen1.5-Chat<br>& Sailor | Task Arithmetic | 56.57 | 81.59 | 67.52 | 62.93 | 75.22 | 53.98 | 66.30 | 2.50 |  |
| SLERP | 55.72 | 79.94 | 67.94 | 57.51 | 75.14 | 69.29 | 67.59 | 3.00 |  |  |
| Model Stock | 57.00 | 80.50 | 68.44 | 51.98 | 76.01 | 66.72 | 66.77 | 2.33 |  |  |
| TIES-Merging | 49.74 | 67.23 | 60.54 | 47.43 | 72.14 | 0.30 | 49.56 | 5.67 |  |  |
| Breadcrumbs | 51.88 | 62.22 | 63.47 | 57.90 | 70.32 | 4.55 | 51.72 | 4.83 |  |  |
| WIDEN | 57.17 | 80.05 | 66.00 | 54.85 | 76.09 | 66.34 | 66.75 | 2.67 |  |  |

Table 4 depicts the merging performance on Open LLM Leaderboard. We find that geometric-based approaches (SLERP and Model Stock) excel in retraining the instruction-following skills of Qwen1.5-Chat, indicating that parameters of FT LLMs may potentially exhibit more evident properties in the geometric space. WIDEN shows competitive results alongside SLERP and Model Stock, underscoring its applicability in merging FT LLMs. Moreover, WIDEN outperforms arithmetic-based methods since it is a generalized format of these methods and offers greater flexibility through the adaptive computation of weight importance.

### 4.3 Performance of Merging FT LLMs

Under the setting of merging multiple FT LLMs, we strictly follow the identical protocol in Yu et al. () and report the official results in Table 5 for fair comparisons. One exception is that we use AlpacaEval 2.0 instead of AlpacaEval in Yu et al. () for evaluation, aiming to provide more convincing and reliable verifications. We evaluate the performance of merging pairwise LLMs since some baselines (e.g., SLERP) are only applicable for dealing with two models.

**Table 5: Performance of merging WizardLM-13B, WizardMath-13B, and llama-2-13b-code-alpaca.**
| Models | Merging Methods | Instruction-<br>following | Mathematical<br>Reasoning | Code Generation |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| AlpacaEval 2.0 | GSM8K | MATH | HumanEval | MBPP |  |  |
| WizardLM-13B | / | 12.73 | 2.20 | 0.04 | 36.59 | 34.00 |
| WizardMath-13B | / | / | 64.22 | 14.02 | / | / |
| llama-2-13b-code-alpaca | / | / | / | / | 23.78 | 27.60 |
| WizardLM-13B<br>& WizardMath-13B | Task Arithmetic | 11.85 | 66.34 | 13.40 | 28.66 | 30.60 |
| SLERP | 7.90 | 66.19 | 13.44 | 28.05 | 30.80 |  |
| Model Stock | 0.25 | 0.00 | 0.00 | 3.05 | 25.80 |  |
| TIES-Merging | 10.07 | 15.77 | 2.04 | 37.80 | 35.60 |  |
| Breadcrumbs | 9.85 | 64.75 | 11.80 | 26.22 | 33.20 |  |
| WIDEN | 9.45 | 66.34 | 13.58 | 28.66 | 30.40 |  |
| WizardLM-13B<br>& llama-2-13b-code-alpaca | Task Arithmetic | 10.09 | / | / | 31.70 | 32.40 |
| SLERP | 6.04 | / | / | 32.32 | 35.80 |  |
| Model Stock | 0.25 | / | / | 3.66 | 24.80 |  |
| TIES-Merging | 7.27 | / | / | 0.00 | 0.00 |  |
| Breadcrumbs | 7.23 | / | / | 33.54 | 32.00 |  |
| WIDEN | 6.53 | / | / | 31.70 | 35.60 |  |
| WizardMath-13B<br>& llama-2-13b-code-alpaca | Task Arithmetic | / | 64.67 | 13.98 | 8.54 | 8.60 |
| SLERP | / | 61.41 | 12.50 | 9.15 | 22.40 |  |
| Model Stock | / | 0.00 | 0.00 | 4.27 | 25.60 |  |
| TIES-Merging | / | 63.23 | 13.56 | 9.76 | 22.40 |  |
| Breadcrumbs | / | 62.55 | 12.48 | 9.15 | 16.20 |  |
| WIDEN | / | 64.22 | 13.58 | 9.76 | 9.80 |  |

From Table 5, we observe that the efficacy of certain baselines drastically fluctuates when integrating FT LLMs. For example, Model Stock appears to lose potency, whereas pruning-based methods including TIES-Merging and Breadcrumbs show competitive performance. WIDEN consistently depicts results on par with established merging techniques in most situations, affirming its suitability in the standard setting of merging multiple FT LLMs.

### 4.4 Investigations of Designs in WIDEN

Figure: Figure 2: Effects of various designs in WIDEN.
Refer to caption: extracted/5776892/figures/ablation_study.jpg

The foundational designs in WIDEN consist of three components: weight disentanglement, ranking weights inside each model, and score calibration for Softmax. To assess the contribution of each module, we respectively remove the above components and measure the performance of the remaining parts. Specifically, we eliminate the disentanglement of weights by calculating the discrepancy between the weights of LLM and the corresponding backbone using cosine similarities, denoted as WIDEN w/o WD. We substitute the ranking mechanism with min-max normalization within each model, represented by WIDEN w/o RANK. We discard the score calibration and directly employ Softmax to compute importance scores, identified as WIDEN w/o SC. Figure 2 shows the impact of these three modifications, where OLL and SEA are the abbreviations for Open LLM Leaderboard and South-East Asian language benchmark, respectively. Note that the reported results are the average of metrics across all the datasets within each benchmark.

From Figure 2, we find that each design in WIDEN contributes to enhancing the merging performance, particularly in absorbing the multilingual abilities on the South-East Asian language benchmark. Precisely, the weight disentanglement refines the estimation of weight importance at a granular level, considering both magnitude and direction. The ranking mechanism offers a smoother distribution of weight importance based on continuous indices, effectively mitigating the influence of diverse parameter changed ranges. The calibration of scores computed by Softmax reallocates importance to critical parameters, which maintains the characteristics of essential parameters across multiple models. In summary, the components of WIDEN are indispensable and improve performance with varied benefits; the removal of any module leads to diminished outcomes.

### 4.5 Analysis of Computed Weight Importance

We further delve into the properties of weight importance calculated by WIDEN from both qualitative and quantitative perspectives.
Since Figure 2 demonstrates that the improvements in weight disentanglement and score calibration are notably more pronounced, we qualitatively depict the distribution of weight importance computed by WIDEN, WIDEN w/o WD, and WIDEN w/o SC on 7B model size in Figure 3. Our observations reveal that: 1) WIDEN exhibits a more balanced and reasonable weight importance distribution than WIDEN w/o WD, attributed to the disentanglement of weights. The distribution of WIDEN ranges approximately from 0.3 to 0.8 and 0.9 to 1.0, versus 0.3 to 0.6 and 0.9 to 1.0 for WIDEN w/o WD. WIDEN considers the collective contributions of magnitude and direction, rather than the individual impacts of weights, leading to a more holistic assessment of weight importance with increased numbers of weights falling within the importance range from 0.6 to 0.8. As a result, compared with WIDEN w/o WD, WIDEN achieves 4.98% and 20.08% improvements on average on the Open LLM Leaderboard and the South-East Asian language benchmark, respectively; 2) In contrast to WIDEN w/o SC, WIDEN distinguishes essential weights and assigns high importance within the range of 0.6 to 0.8 as well as 0.9 to 1.0 for certain weights, thanks to the design of score calibration. Therefore, WIDEN ensures the retention of essential weights in both Qwen1.5-7B-Chat and Sailor-7B, resulting in 12.25% and 72.87% average enhancements on the two benchmarks.

Figure: Figure 3: Distribution of weight importance computed by WIDEN and its variations.
Refer to caption: extracted/5776892/figures/computed_weight_importance.jpg

Furthermore, we categorize weight importance into three levels: Low (L), Medium (M), and High (H). The Low tier comprises the first third of weights when sorted by ascending importance, indicating those with the least significance. The Medium tier includes weights from the 1/3 mark to the 2/3 mark, and the High tier contains weights from the 2/3 mark to the end. Table 6 quantitatively illustrates the adjustments of weight importance made by WIDEN when compared to WIDEN w/o WD and WIDEN w/o SC across three levels. We find that WIDEN effectively reallocates the weight importance via three aspects: 1) elevating weights of lower importance from Low to Medium; 2) either demoting or promoting weights of medium importance from Medium to Low or from Medium to High, respectively; 3) decreasing weights of high importance from High to Medium. These adjustments in weight importance explain how WIDEN brings improvements through the designs of weight disentanglement and score calibration.

**Table 6: Adjustments of weight importance made by WIDEN.**
| Adjustments | Models | L$\rightarrow$L | L$\rightarrow$M | L$\rightarrow$H | M$\rightarrow$L | M$\rightarrow$M | M$\rightarrow$H | H$\rightarrow$L | H$\rightarrow$M | H$\rightarrow$H |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WIDEN w/o WD<br>to WIDEN | Qwen1.5-7B-Chat | 18.82% | 11.09% | 3.42% | 13.97% | 10.18% | 9.18% | 0.54% | 12.06% | 20.75% |
| Sailor-7B | 15.34% | 10.50% | 7.48% | 17.80% | 7.72% | 7.80% | 0.18% | 15.10% | 18.07% |  |
| WIDEN w/o SC<br>to WIDEN | Qwen1.5-7B-Chat | 24.78% | 7.69% | 0.85% | 7.93% | 17.51% | 7.88% | 0.62% | 8.12% | 24.61% |
| Sailor-7B | 22.01% | 9.52% | 1.80% | 9.63% | 15.14% | 8.56% | 1.69% | 8.67% | 22.99% |  |

## 5 Conclusion

In this study, we paved the way for extending the merging scope from FT to PT LLMs. Specifically, we first observed that existing methods struggled to integrate the abilities of PT LLMs and then introduced WIDEN, an innovative approach based on weight disentanglement, to effectively deploy merging strategies to PT LLMs. Experimental findings demonstrated that WIDEN not only exhibited an advantage in absorbing the abilities of PT LLMs but also preserved the skills of FT LLMs. Additionally, WIDEN achieved competitive performance with established merging methods in the conventional setting of merging FT LLMs. We further offered a detailed analysis of the designs underlying WIDEN. This work made the first attempt to broaden the sources of combinable abilities, fostering the broader application of model merging techniques.

## Appendix A Experimental Details

### A.1 Details of FT and PT LLMs

Table 7 depicts the versions and correspondences with backbones of FT and PT LLMs.

**Table 7: Versions and correspondences with backbones of FT and PT LLMs.**
| Types | Models | Backbones |
| --- | --- | --- |
| FT LLM | Qwen1.5-7B-Chat(^3^33[https://huggingface.co/Qwen/Qwen1.5-7B-Chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat)) | Qwen1.5-7B(^4^44[https://huggingface.co/Qwen/Qwen1.5-7B](https://huggingface.co/Qwen/Qwen1.5-7B))\saveFN\QwenSevenBfn |
| PT LLM | Sailor-7B(^5^55[https://huggingface.co/sail/Sailor-7B](https://huggingface.co/sail/Sailor-7B)) | Qwen1.5-7B\useFN\QwenSevenBfn |
| FT LLM | Qwen1.5-14B-Chat(^6^66[https://huggingface.co/Qwen/Qwen1.5-14B-Chat](https://huggingface.co/Qwen/Qwen1.5-14B-Chat)) | Qwen1.5-14B(^7^77[https://huggingface.co/Qwen/Qwen1.5-14B](https://huggingface.co/Qwen/Qwen1.5-14B))\saveFN\QwenFourteenBfn |
| PT LLM | Sailor-14B(^8^88[https://huggingface.co/sail/Sailor-14B](https://huggingface.co/sail/Sailor-14B)) | Qwen1.5-14B\useFN\QwenFourteenBfn |
| FT LLM | WizardLM-13B(^9^99[https://huggingface.co/WizardLM/WizardLM-13B-V1.2](https://huggingface.co/WizardLM/WizardLM-13B-V1.2)) | Llama-2-13b(^10^1010[https://huggingface.co/meta-llama/Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf))\saveFN\llamaTwoThirteenBfn |
| WizardMath-13B(^11^1111[https://huggingface.co/WizardLM/WizardMath-13B-V1.0](https://huggingface.co/WizardLM/WizardMath-13B-V1.0)) | Llama-2-13b\useFN\llamaTwoThirteenBfn |  |
| llama-2-13b-code-alpaca(^12^1212[https://huggingface.co/layoric/llama-2-13b-code-alpaca](https://huggingface.co/layoric/llama-2-13b-code-alpaca)) | Llama-2-13b\useFN\llamaTwoThirteenBfn |  |

### A.2 Overview and Evaluation Metrics of Benchmarks

The Open LLM Leaderboard is established to assess open-source LLMs using the Eleuther AI Language Model Evaluation Harness (Gao et al., ), which encompasses six datasets: AI2 Reasoning Challenge (ARC) (Clark et al., ), HellaSwag (Zellers et al., ), MMLU (Hendrycks et al., ), TruthfulQA (Lin et al., ), Winogrande (Sakaguchi et al., ), and GSM8K (Cobbe et al., ). These datasets adopt accuracy as the evaluation metric under various shot settings (25-, 10-, 0-, 5-, 5-, and 5-shot, respectively). The leaderboard ranks models based on the average scores across these six datasets.

The benchmark for South-East Asian languages is designed with four tasks: XQuAD (Artetxe et al., ) (Thai, Vietnamese) and TydiQA (Clark et al., ) (Indonesian) for question answering; XCOPA (Ponti et al., ) (Indonesian, Thai, Vietnamese) for commonsense reasoning; BELEBELE (Bandarkar et al., ) (Indonesian, Thai, and Vietnamese) for reading comprehension; and M3Exam (Zhang et al., ) (Javanese) for examination. All the datasets utilize 3-shot Exact Match (EM) and F1 as evaluation metrics. It is worth noticing that the official code(^13^1313[https://github.com/sail-sg/sailor-llm](https://github.com/sail-sg/sailor-llm)) of Sailor computes multiple metrics for M3Exam on Thai and Vietnamese, which are inconsistent with the originally reported results. Thus, we only present the results of M3Exam (Javanese) in this work.

AlpacaEval 2.0 employs the win rate for assessment, calculated as the proportion of cases where a powerful LLM (GPT-4 Turbo is used in this work) prefers the outputs from the target model over those from GPT-4 Turbo. GSM8K and MATH are evaluated by zero-shot accuracy in addressing mathematical problems. HumanEval and MBPP adopt pass@1 as the evaluation metric, representing the fraction of individually generated code samples that successfully pass the unit tests.

### A.3 Descriptions of Model Merging Baselines

We compare with seven commonly-used model merging methods in the experiments:

- •
Average Merging simply averages the parameters of multiple models for building the merged model (Wortsman et al., ).
- •
Task Arithmetic employs a scaling term to modulate the importance of the backbone and various models to be merged (Ilharco et al., ).
- •
SLERP is tailored for the combination of two models, utilizing spherical interpolation to merge the model weights (Shoemake, ).
- •
Model Stock seeks to approximate a center-close weight by considering several FT models, where the backbone is leveraged as an anchor point (Jang et al., ).
- •
TIES-Merging aims to mitigate task conflicts in model merging by initially pruning delta parameters with lower magnitudes and subsequently fusing parameters that exhibit consistent signs (Yadav et al., ).
- •
Breadcrumbs refines model parameters by filtering out the extreme tails (i.e., outliers) in the absolute magnitude distribution of task vectors to derive the final merged model (Davari & Belilovsky, ).
- •
DARE serves as a versatile module for current merging techniques, which first randomly discards delta parameters and then rescales the remaining parameters to preserve the model performance (Yu et al., ).

### A.4 Analysis of WIDEN and Arithmetic-Based Merging Methods

The computation procedure of Average Merging (Wortsman et al., ) for $N$ LLMs is denoted by

$$ $\bm{W}_{\text{M}}=\frac{1}{N}\sum\limits_{n=1}^{N}\bm{W}^{n}=\bm{W}_{\text{PRE }}+\frac{1}{N}\sum\limits_{n=1}^{N}\left(\bm{W}^{n}-\bm{W}_{\text{PRE}}\right) \in\mathbb{R}^{d\times k}.$ (8) $$

Task Arithmetic (Ilharco et al., ) is implemented as follows,

$$ $\bm{W}_{\text{M}}=\bm{W}_{\text{PRE}}+\lambda\sum\limits_{n=1}^{N}\left(\bm{W} ^{n}-\bm{W}_{\text{PRE}}\right)\in\mathbb{R}^{d\times k},$ (9) $$

where $\lambda$ denotes the scaling term. It is straightforward that in Equation (5), if $t$ is set to be minus, all the parameters can be considered crucial, with their importance scores calibrated to $s$. Thus, Equation (7) can be rewritten as

$$ $\bm{W}_{\text{M}}=\bm{W}_{\text{PRE}}+\sum\limits_{n=1}^{N}\frac{s+s}{2}\left( \bm{W}^{n}-\bm{W}_{\text{PRE}}\right)=\bm{W}_{\text{PRE}}+s\sum\limits_{n=1}^{ N}\left(\bm{W}^{n}-\bm{W}_{\text{PRE}}\right)\in\mathbb{R}^{d\times k}.$ (10) $$

To this end, when $t<0.0$ and $s=1/N$, WIDEN transforms into Average Merging; when $t<0.0$ and $s=\lambda$, WIDEN represents Task Arithmetic.

### A.5 Details of Grid Search on Hyperparameters of Merging Methods

Table 8 presents the searched ranges of hyperparameters of model merging approaches.

**Table 8: Hyperparameter searched ranges of model merging approaches.**
| Model Merging Methods | Search Ranges of Hyperparameters |
| --- | --- |
| Task Arithmetic | scaling term to merge parameters: [0.5, 1.0] |
| SLERP | spherical interpolation factor: [0.3, 0.5, 0.7] |
| Model Stock | / |
| TIES-Merging | scaling term to merge parameters: [0.5, 1.0],<br>ratio to retain parameters with largest-magnitude values: [0.5, 0.7, 0.9] |
| Breadcrumbs | scaling term to merge parameters: [0.5, 1.0],<br>ratio to mask parameters with largest-magnitude values: [0.01, 0.05],<br>ratio to retain parameters [0.9] |
| WIDEN | factor to indicate the multiple above the average: [1.0, 2.0],<br>factor to calibrate scores: [1.0] |

### A.6 Parameter Changed Ranges of FT and PT LLMs

We depict the statistics about the deciles of parameter changed ranges of both FT and PT LLMs in Table 9, which are derived by first sorting the entire ranges and then indexing at positions corresponding to 0%, 10%, 20%, …, 100%.

**Table 9: Statistics about the deciles of parameter changed ranges of FT and PT LLMs.**
| Models | 0% (min) | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% (max) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen1.5-7B-Chat<br>vs. Qwen1.5-7B | -2.43e-02 | -4.27e-04 | -2.44e-04 | -1.22e-04 | -3.05e-05 | 0.00 | 3.05e-05 | 1.22e-04 | 2.44e-04 | 4.27e-04 | 2.29e-02 |
| Sailor-7B<br>vs. Qwen1.5-7B | -0.27 | -0.57e-02 | -0.37e-02 | -0.23e-02 | -0.11e-02 | 0.00 | 0.11e-02 | 0.23e-02 | 0.37e-02 | 0.57e-02 | 0.25 |
| Qwen1.5-14B-Chat<br>vs. Qwen1.5-14B | -2.34e-02 | -4.27e-04 | -2.44e-04 | -1.22e-04 | -3.05e-05 | 0.00 | 3.05e-05 | 1.22e-04 | 2.44e-04 | 4.27e-04 | 2.06e-02 |
| Sailor-14B<br>vs. Qwen1.5-14B | -0.36 | -0.78e-02 | -0.51e-02 | -0.31e-02 | -0.15e-02 | 0.00 | 0.15e-02 | 0.31e-02 | 0.51e-02 | 0.78e-02 | 0.42 |
| WizardLM-13B<br>vs. Llama-2-13b | -3.93e-02 | -0.16e-02 | -0.10e-02 | -0.06e-02 | -0.03e-02 | 0.00 | 0.03e-02 | 0.06e-02 | 0.10e-02 | 0.16e-02 | 4.81e-02 |
| WizardMath-13B<br>vs. Llama-2-13b | -0.69e-02 | -0.06e-02 | -0.04e-02 | -0.02e-02 | -0.01e-02 | 0.00 | 0.01e-02 | 0.02e-02 | 0.04e-02 | 0.06e-02 | 0.74e-02 |
| llama-2-13b-code-alpaca<br>vs. Llama-2-13b | -8.42e-02 | -3.05e-05 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.05e-05 | 7.98e-02 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Equation 1: The shape of mD is equal to that of W. This equation should be corrected.
2. Equation 7: Could you provide a reason for not averaging all the differences by multiplying by 1/N?
3. Have the authors considered an alternative approach that compares each weight matrix on a column-by-column basis between the tuned model and the original backbone? Specifically, this approach would involve calculating and ranking differences column by column, rather than disentangling weights into separate magnitude and direction components.
4. How to grid search the hyperparameters for baselines methods? What validation dataset is used?
5. The paper should provide a figure to visually illustrate the proposed method.
soundness: 2
strengths: 1. The paper makes a valuable contribution by identifying a critical limitation in existing model merging methods: their ineffectiveness when applied to continually pre-trained (PT) models. This insight is essential, as it highlights a gap in current merging techniques, which are generally only effective for fine-tuned (FT) models with minimal parameter shifts.
2. The paper introduces WIDEN (Weight Disentanglement), an innovative method that automatically computes the importance of weights during the merging process. WIDEN disentangles each model’s weights into magnitude and direction components, and then adapts the merging decisions based on the divergence of these components from a shared backbone. This approach removes the need for manually assigning scaling factors and effectively addresses the challenges posed by the varied parameter changes in both fine-tuned (FT) and pre-trained (PT) models.
3. The experimental results demonstrate that WIDEN outperforms existing merging methods by effectively combining both instruction-following and multilingual capabilities. The paper also evaluates WIDEN in traditional FT-only merging scenarios, where it achieves competitive performance compared to established methods.
summary: Merging multiple LLMs, particularly those with substantial parameter shifts from pre-training (PT), presents challenges for traditional merging methods. To address this issue, the paper introduces WIDEN (Weight Disentanglement), a novel approach for merging large language models (LLMs) that have undergone either fine-tuning (FT) or pre-training (PT). This method expands the applicability of model merging beyond conventional fine-tuned models.
weaknesses: 1. The paper assumes that continually pre-trained (PT) models inherently experience larger weight shifts than fine-tuned (FT) models, which serves as the justification for a new merging approach. However, this assumption may not hold universally, as the degree of weight change in PT models depends on factors such as the data domain and dataset size. This raises questions about the paper’s motivation and the general applicability of its problem formulation. A more thorough exploration or empirical verification of weight changes across PT and FT models would help substantiate this claim. The authors are expected to provide empirical evidence comparing the distribution of weight changes between PT and FT models across different domains, model sizes, and dataset sizes.
2. The proposed ranking mechanism in WIDEN calculates divergences in magnitude and direction separately for each weight relative to the backbone model. However, the reliability of comparing magnitudes across models with different directional vectors is questionable. When calculating magnitude differences, direction is not considered, meaning that the importance of weights in different models could be misinterpreted if their directions diverge. Similarly, comparing directional differences might be misleading if the corresponding magnitudes differ significantly between models. So, have the authors considered alternative approaches that jointly consider both magnitude and direction? Additionally, have the authors empirically analyzed how often misinterpretation occur in practice due to treating these components separately?
3. Although WIDEN is intended to be a general merging technique applicable to both FT and PT models, its performance in merging FT models is comparatively weak (as shown in Table 5). Given that the method is designed to be adaptable across model types, this underperformance raises concerns about its overall efficacy. Are there certain characteristics of FT models that WIDEN struggles with?
4. The experiments primarily focus on merging a specific PT model (Sailor) with a FT model, which limits the generalization ability of the results. Evaluating WIDEN on other PT models, particularly in diverse domains such as finance or healthcare, would provide stronger evidence of its effectiveness.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: See weaknesses.

I'm not familiar with LLM merging and am open to discussion if misunderstood any part of the paper.
soundness: 3
strengths: 1. The paper is the first successful attempt to incorporate the ability of PT LLM into model merging techniques.
2. Extensive experiments and analyses have demonstrated the effectiveness of the proposed method.
3. The paper is well-written and easy to follow.
summary: This paper presents a pioneering effort in extending model merging to Pretrained LLMs utilizing weight disentanglement. Extensive studies on previous methods demonstrate their inability to perform when applied to pretrained LLM while the method proposed in the paper is able to solve the task with minimal performance drop compared to the models to be merged.
weaknesses: 1. The experiments are only limited to Sailor, more results on different models could validate the effectiveness of the proposed method.
2. Despite being indicated by the method, the experiments didn't show evidence that the proposed method could work for multiple LLM cases. Some experiments from this perspective would be appreciated.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - How does WIDEN handle cases where the backbone (reference pretrained model) diverges substantially in structure or task specificity from both FT and PT models? Would WIDEN work with heterogeneous LLMs beyond those sharing the same backbone?
- Did the authors attempt to merge more than two or three models to evaluate WIDEN’s scalability and robustness? If so, what were the results, and how does performance change as the number of LLMs increases?
- Given WIDEN’s better performance on SEA benchmarks than on the OpenLLM Leaderboard, could the authors elaborate on why this discrepancy exists? Is WIDEN more suited to particular types of tasks or linguistic benchmarks?
- On tasks where Task Arithmetic performs better, why might WIDEN’s performance lag?
- Since WIDEN modifies weights adaptively, would it be feasible to incorporate it into a continual learning setup where multiple LLMs are progressively merged over time? Could this method be used for models other than LLMs?
soundness: 3
strengths: - The paper’s effort to expand merging capabilities from FT to PT models is well-motivated and addresses a crucial gap in existing merging techniques.
- The methodology has a sound technical foundation, with a detailed four-step framework integrating weight disentanglement, ranking, and adaptive score calibration.
- The experimental setup is thorough, covering both conventional FT merging tasks and the new FT-PT merging setting. WIDEN’s performance across SEA and Open LLM Leaderboard benchmarks and comparison with multiple baselines highlights its applicability to diverse LLMs.
- The impact of each component within WIDEN is evaluated with an ablation experiment in Figure 2, demonstrating the importance of weight disentanglement and score calibration.
summary: The paper presents WIDEN, a novel merging technique for Large Language Models (LLMs), which extends the applicability of merging from finetuned (FT) to pretrained (PT) models by disentangling weights into magnitude and direction components. This weight disentanglement enables adaptive merging by quantifying each LLM's alteration relative to a shared backbone. The disentangled weights are ranked using normalized divergence scores compared to the pretrained baseline, and this ranking is used to compute an automated importance factor for each LLM. This results in a generalized form of several existing arithmetic methods for LLM merging. Experimental results suggest that WIDEN effectively balances multiple capabilities, such as multilingual and instruction-following skills, across FT and PT LLMs.
weaknesses: - Although WIDEN generalizes across FT and PT models, it does not consistently outperform Task Arithmetic on all benchmarks. For instance, Task Arithmetic often shows competitive results on Open LLM Leaderboard tasks, raising concerns about WIDEN’s scalability and stability. For example, on the SEA benchmark, the performance improvement on 14B models is smaller than the 7B model, with the gap between Task Arithmetic and its claimed generalized form WIDEN narrowing as the LLMs become larger.
- The improvement WIDEN demonstrates is noticeably higher on SEA benchmarks than on the OpenLLM Leaderboard, yet the paper does not clarify why performance fluctuates between benchmarks. This omission raises questions about its adaptability to different domains or task settings.
- While grid search is used for tuning, the choice of hyperparameters (particularly t and s) lacks justification beyond empirical results. A clearer rationale or theoretical insight into their selection would enhance the robustness of WIDEN’s methodology.
- Although score calibration is a novel addition to ensure adaptive ranking and merging, values other than 1.0 should be evaluated in score calibration. The "ease of implementation" rationale is not good enough.

Meta-review: This paper tackles the model merging problem between PT and FT LLMs. The approach focuses on merging homogeneous models that are of a significant parameter divergence. The basic idea is to separate the weight into the magnitude and the direction components to automatically determine the merging weights. The authors took the Sailor as the PT LLM and showcased positive results. However, this paper received some common and critical concerns, especially that the method is limited to significant parameter change scenarios and performs less effectively with FT model merging and other PT models beyond the Sailor series. After discussion, Reviewer W7XB improved the rating score from 5 to 6 but decreased the confidence score from 4 to 3. Other reviewers kept their scores unchanged. After reviewing all materials, the AC found that the critical concerns were not all addressed. Considering the overall rating, the final recommendation is reject.

Decision: Reject