Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs
Authors: Rui Dai
Sile Hu
Xu Shen
Yonggang Zhang
Xinmei Tian
Jieping Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language model
Task Arithmetic
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Task arithmetic is a straightforward yet highly effective strategy for model merging, enabling the resultant model to exhibit multi-task capabilities. Recent research indicates that models demonstrating linearity enhance the performance of task arithmetic. In contrast to existing methods that rely on the global linearization of the model, we argue that this linearity already exists within the model's submodules. In particular, we present a statistical analysis and show that submodules (e.g., layers, self-attentions, and MLPs) exhibit significantly higher linearity than the overall model. Based on these findings, we propose an innovative model merging strategy that independently merges these submodules. Especially, we derive a closed-form solution for optimal merging weights grounded in the linear properties of these submodules. Experimental results demonstrate that our method consistently outperforms the standard task arithmetic approach and other established baselines across different model scales and various tasks. This result highlights the benefits of leveraging the linearity of submodules and provides a new perspective for exploring solutions for effective and practical multi-task model merging.

Published as a conference paper at ICLR 2025
LEVERAGING SUBMODULE LINEARITY ENHANCES TASK ARITHMETIC PERFORMANCE IN LLMs
Rui Dai Sile Hu Xu Shen Yonggang Zhang Xinmei Tian Jieping Ye"
National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China
PIndependent Researcher
'Hong Kong Baptist University
"sheexuusttc@@gaa i 1. com, xi nime @uste.uedeee
ABSTRACT
Task arithmetic is straightforward yet highly effective strategy for model merg- ing, enabling the resultant model to exhibit multi-task capabilities. Recent re- search indicates that models demonstrating linearity enhance the performance of task arithmetic. In contrast to existing methods that rely on the global linearization of the model, we argue that this linearity already exists within the model's sub- modules. In particular, we present a statistical analysis and show that submodules (e.g., layers, leff-attentions, and MLPs) exhibit significantly higher linearity than the overall model. Based on these findings, we propose an innovative model merg- ing strategy that independently merges these submodules. Especially, we derive a closed-form solution for optimal merging weights grounded in the linear proper- ties of these submodules. Experimental results demonstrate that our method con- sistently outperforms the standard task arithmetic approach and other established baselines across different model scales and various tasks. This result highlights the benefits of leveraging the linearity of submodules and provides new perspective for exploring solutions for effective and practical multi-task model merging.
INTRODUCTION
In recent years, the growing scale of large language models has significantly increased the demand for data and training costs fine-tuning multi-task models. To integrate the capabilities of various existing single-task models into unified model, various model merging techniques have been de- veloped (Yu et al. 2024 Jin et al. [023 Matena & Raffel 2022 Yadav et al. (002).. Task arithmetic [Tharco et al. 2023) is one of the simplest and most efficient model merging strategies. It involves a straightforward weighted combination of the weights from single-task models, allowing the final merged model to exhibit multi-task capabilities without the need for additional training data.
Recent studies (Ortiz-imménzz et al. 2023 Zhou et al. 2024) have revealed a connection between the linearity of fine-tuned models and the effectiveness of the weighting operations in task arith- metic. Here, linearity specifically refers to the linear relationship between the differences in model weights and the differences in output features caused by fine-tuning" (Ortiz-Jiménez et al. 2023 Zhou et al. 2024). which is very different from the Traditional Linear Propertie This linearity intuitively aligns with the linear weighting operations in task arithmetic (Ilharco et al. 2023). Re-
search (Ortiz-Jiménc) et al. [023] Tang et al. 2024 Jin et al. 22024 Liu et al. 2024) has shown that models exhibiting linearity retain their individual task performance better when merged using task arithmetic, leading to superior multi-task models. However, models produced through conventional finettuning often lack this ideal linear property (Ortiz-Jimént et al., 2023). To address this issue,
existing methods (Ortiz-liménez et al. 2023 Tang et al. 2024 Jin et al. 2024 Liu et al. 2024 have
"Corresponding authors. Code: htt.r:/hc.p/gggggttoooooe/eeeeea 'a;; 0 + aT) x f(z;; 0b) + aAf(;;; 6. + T) where 7 = - 0o is the weight difference caused hy fine-tuning and (x 6o + r) = f(x; 0n + TT f(x; 6o) is the corresponding output feature difference.
'These properties easily confused. More discussions Appendix A.5.1

---

Published as a conference paper at ICLR 2025
0.4
Io Llama2-7B
Llama-2-78
Bnt Jamma-1138
1.lama---13R
0.3
Non-Iinaaitty
Non-linaarity
0.2
0.1
Layer
Head
Model
Layer
Artn
MLP
Head
(a) Non- Non-lineurity on fine- tuned models
(b) Non-linearity
of model merging
Figure 1: (a) Comparison of non-linearity in full models and submodules for fine-tuned models, measured using Non-linearity Score defined in Definition 2 A lower value indicates better lincarity, based on data from three fine-tuned models detailed in Section 4.1 (b) Comparison of non-linearity in full models and submodules within the context of model merging, assessed via Eq (Zhou et al,) 2024). A lower value indicates better linearity, based data from three merging combinations outlined in the caption of Figure Error bars represent standard deviation.
explored adjustments to the fine-tuning paradigm, retraining the model using original training data to enhance the linearity and the final merging outcomes.
In contrast to existing methods that rely on global linearization of the model, which impose require- ments for retraining with the original data, we argue that this linearity has already existed within the model's s submodules. Intuitively, the most straightforward approach seek linearity from a complex nonlinear function is to decompose it into simpler ones. Following this idea, we divide the model into multiple shallower submodules (e.g., layers, self-attentions, MLPs) and discover that these sub- modules exhibit a level of linearity that significantly surpasses that of the overall model. Figure [ compares the lincarity characteristics full models and their constituent modules. In panel (a), the non-linearity of full models is significantly higher than that of the decomposed submodules, which exhibit scores close to zero, indicating their strong linearity. In panel (b), we further evaluate the linear properties in the context of model merging using the metric introduced by Zhou al. (2024).
The results reveal a distinct gap in non-linearity between full models and submodules, suggesting that the latter can demonstrate good linearity even when the full models do not.
Inspired by these observations, we propose an innovative training-free model merging strategy: 1) Decompose the models into submodules that exhibit linear characteristics. 2) Compute the optimal merging weights for each submodule by leveraging their linear properties. 3) Apply these derived weights to facilitate the merging of the models. To obtain the optimal merging weights under the linearity assumption of the modules, we derived a closed-form solution by minimizing the distance between the output features of the merged module and those of the corresponding module across fine-tuned tasks (Jin et al. (023). Notably, this computational process necessitates only a minimal amount of data (30 samples per task) and a limited number model inference iterations.
To validate the effectiveness of our proposed method, we conducted merging experiments across models fine-tuned from Llama-2-7B and Llama-2-133 (Touvron et al. 2023) for three distinct tasks: Math, Coding, and Translate. The experimental results demonstrate that method significantly outperforms various baseline techniques, including standard arithmetic approaches, across different model scales and diverse tasks, particularly with respect to the decomposition level of layers and attn/MLPs. These findings provide strong evidence that leveraging the linearity inherent in submod- ules holds substantial potential for enhancing model merging performance with task arithmetic.
ANALYZING THE LINEARITY OF MODEL AND MODULES
In this section, we first introduce some definitions regarding lincarity (Ortiz- Jiménzi et al., 2023) and task arithmetic[Ilhan et al. 2023). We then present score that can quantitatively compare the
$ f[x: d + ET acTs) ~ fax 0o) + DT ar As, do + Tt), where 7, 0c - Go is the weight difference caused by fine-tuning in the 4th model and A Og + T.) = f(;; 9g + T.) f(z; 0o) is the corresponding output feature difference in the tth model.
2

---

Published as a conference paper at ICLR 2025
linearity of models and modules. Subsequently, following the insight outlined in the introduction, we divide the model into submodules to examine linearity, discovering that the linearity of the module surpasses that of the model. Finally, we further validate the linearity when merging multiple modules, laying a foundation for the methodologies proposed in the next section.
2.1 PRELIMINARY
Notation. Let f X x - Y be a neural network taking inputs I E X and parameterized by set of weights O. Consider T tasks, where D, S X is a dataset used fine-tune the models starting from the pre-trained weights Oo and obtain the fine-tuned weights O,.
Task Arithmetic. (Ihharco al. 2023) Task arithmetic is a simple, efficient, and widely used model merging strategy. It assigns multi-task performance to the merged model through simple weighted combination of the weights of models, without the need for additional training or data support.
Task vector for task t is defined the difference between the fine-tuned and the pre-trained weights, namely, T. = 0c Og. The approach task arithmetic involves a weighted combination of the task vectors, which is then added to the pre-trained model 0g. This can be expressed as
T
merrg = 0o + Σ Te
where an represents the weight corresponding to Tt-
Here, linearity specifically refers the linear relationship between the differences in model weights and the differences in output features caused by fine-tuning
Definition (Linearity). Let be a fine-tumed model and 0g be its corresponding pre-trained model. If assert that O exhibits linearity, the differences in model weights caused by fine-tuning are linearly related to the differences in output features caused by fine-tuning for any E X, i.e.
[ax; Oo + +T)) R [ax; 0o) + o1I(1; +T)
(2)
0o is the differences in model weights before and after fine-tuning, and A Oo + 7) = f);; +T)- [xx; 0g) the differences in model outpul features before and after fine-tuning.
where T = 0
It is very different from the Traditional Linear Properties 11
2024) has shown that models demonstrating linearity retain higher single-task performance when merged via task arithmetic, leading to enhanced multi- task models. However, models developed through conventional fine-tuning methods typically lack this desired linearity (Ortiz-Jiménez et al. 2023).
Research (Ortiz-Jiménez et al. 2023 Jin al.
Proposed Linearity Metric. The interpolation model 6mner between the model @ and the pre- praitrained model 6g lies on the line connecting and 6g in the parameter space. the model satisfies linearity in Definition [ then its output features [00merr 2r) should also lie the line formed by the output features f(0, σ.) and [fo. x). Moreover, the distance between [00meer; and (00. should be proportional to the interpolation weight. Based on this idea, to measure and compare the linearity of models quantitively, we propose the Non-linearity Score.
Definition 2 ANon-lineate Score). For fine-tumed model O and its corresponding pre-trained model 0.. if 9 satisfies linearity in Definition I under any distance metric D(a, b) = Ia bll in the feature space, where denotes the norm operation, then for any 0, E [0, 1), the following approximation holds:
+ ((,, ( (U(
where 7 = 0 Gg-
"The preliminary investigation into the relationship between task arithmetic and lincarity commenced with the linearity conclusion derived from the research on the Neural Tangent Kernel (Jacot et al. 2018 Chizat et al. (019). It was established that the output of neural network can be approximated using first-order Taylor expansion: (x; + ar) f (a: 0o) + ar fof 0o), which presents more rigorous conclusion.
ChizaCizzal
3

---

Published as a conference paper at ICLR 2025
When replace the approximation in E with equality, it indicates state of perfect linearity. Therefore, we define the Non- con-line Score of model as the measure of the deviation of model from this ideal linear condition:
Non-linearity Score = S. S. (Pufle, 8o + (T), 8o + Br)) - -lar - 81) dad
(4)
Dif(x; 0), (x:
We calculated the Non-linearity Score for models fine-tuned on various tasks and presented the results in Figure 2 Our observations indicate that nearly all models exhibit a lack of satisfactory linearity the model level. This finding aligns with conclusions drawn in other research Ortiz- Jiménez et al. (2023) employing different methodologies.
2.2 ANALYZING THE LINEARITY OF SINGLE FINE-TUNED MODELS
Following the idea of seeking linearity by decomposition, we divide the entire model according its architecture into multiple levels, including layers, self-attention, MLP, and head levels. Analogous to Definition the linearity at the module level can be expressed as follows:
Property (Linearity of Module). Let of be a module weight from the fine-tumed model and 0h be its corresponding module's 's weight from the pre-trained model. If we assert that O' exhibits linearity, the differences in module weights caused by fine-tuning are linearly related to the differences in this module's output features caused by fine-tuning for any x H1², i.e.
[xx 0g aTT) [x;;0) (1;
where r' = 0g is the differences module weight before and after fine-tuning, and A (x; Og + T') = f(:;; 8 + +') fz; 0)) is the differences in module output features before and after fine-
tuning with same input x H', H' corresponds to the imput feature space of module
Math
Math
3
Non- lincaritt Score
Coding
Coding
Translate
TTrnnsate
2
Model
Layer
Attn
ML.P
Head
Model
Layer
Attn
MLP
Head
(a) Llama-2-7B
Llama
Figure 2: Comparison of Non- -ineaiinyy Score in full models and submodules for models fine-tuned from tasks of Math, Coding, and Translate in Llama-2-7B and Llama-2-13BB A lower value indicates better linearity. Error bars represent standard deviation across different modules.
Linearity in Models and Modules. We can employ the same Non-Linearity Score defined in 2 to assess the degree of satisfaction concerning this property for multiple fine-tuned models and different sets of corresponding submodules. Figures ((a) and (b), we compare the average linearity at both the module level and model level across models fine-tuned on three distinct tasks, on two different scales Llama-2 backbones. Several observations can be derived from these figures: 1) All submodules exhibit degree of linearity that significantly exceeds that of the overall model. 2) The Non-Lincarity scores for all submodules are near zero, suggesting an adequate approximation indicated in 5 3) The pronounced linearity disparity between the model level and the module level remains consistent across all three fine-tuned tasks and both scales of the model.
$ Forr practical reasons, this paper, we employ discrete approximation of the metric, which we continue to refer to as Non-lincarity Score for simplicity. Further details can found in Appendix A.2 'We make slight abuse of notation ff by omitting the subscripts that distinguish between different archi- tectures for the sake of simplicity.
4

---

Published as a conference paper at ICLR 2025
In conclusion, these observations provide compelling evidence that the modules can exhibit linear- ity even when the models do not. This suggests a considerable opportunity to leverage the linear properties the submodule level when integrating fine-tuned models with task arithmetic, even if the overall model does not exhibit the same degree of linearity.
2.3 ANALYZING THE LINEARITY OF MERGING MULTIPLE FINE-TUNED MODELS
Property I pertains to the linearity of modules within a single fine-tuned model. However, in the con- text of model merging, multiple fine-tuned models are involved. Consequently, we extend Property to encompass multiple models:
Property 2 (Linearity of Merged Modules). Let 0 be a module weight from the fine-tunel model and 0g be its corresponding module's 's weight from the pre-trained model. If we assert that O ex- hibits linearity of merging, we imply that the differences in ouiput features originating from different models can be linearly separated, i.e. for any x E H
features
T f(x; 0b + Σ Cait)) (z) + E E=1
(6)
1=1
where TÝ 0f 0g is the differences in module weight from model t and (f 0g + +/) fæ; 06 + T - f(x; 0)) is the differences in module output features before and after fine- fine-tuning.
Assesing Linearity in the Context of Model Merging. Property 2 is only feasible in the presence of Property [ for all corresponding models; however, its occurrence is not guaranteed. The com- bination of multiple models fine-tuned for distinct tasks may introduce additional non-linearity due to the approximations inherent in these equations. To ascertain whether the previously mentioned observations regarding linearity remain applicable within the context of model merging, adopted the methodology delineated by Zhou et al. (2024), which evaluates linearity using a feature space metric. For clarity and convenience, we refer this metric as Projection Distance in this paper.
Specifically, we firstly calculate the cosine similarity between two sets of feature differences: the differences between the output features of the merged module (derived from multiple fine- tuned models) and those of the pre-trained module, denoted as Af(x; 0f. + Σi af7!)
f(; 66 + ET= afT')- - (f; 6)). as well as the corresponding weighted combination of the dif- ferences in output features from each fine-tuned model, represented as D af (a; 0g + +/)) =
ΣT a4 (f (x; 0g + r/) (x; ())) This similarity is denoted by the following equation
T
T
cosine (I; at = cos Af(1;
+
a af
t=1
where cos(a, b) a b|(||||bbb is the Cosine Similarity. As follow, the Projection Distance projfnerge (af...T) .) between these two sets of feature deltas is represented
Ax of + ET afT!) mo of
T
proimerge (af...))
Exen
(8)
Σ ag (z; 03 + +/))
where E represents expectation and H represents the input feature set corresponding to module i, with further details provided in Appendix A.3.3
A.3.3
Results of Projection Distance In Figure 3 (a) and (b), we present the projection distances result- ing from the merging of two and three models. Both configurations reveal trend that submodules exhibit superior linearity compared to the model level, consistent with the observations made for single models presented in Figure 2
noteworthy that the projection distance for merging three models is significantly lower than that observed for the merging of two models across layers, atten- tions, and MLPs. This suggests that there exists an even more favorable condition for linear merging
Follow Zhou al. 22024) we also analyze the metric of cosine similarity, please refer to Appendix A.I.1
for results and analysis.
5

---

Published as a conference paper at ICLR 2025
0.4
0.4
ImMerging
M
Exprof
033
luMerging Models
Exprofmars
0.3
i iMerging Models
0.2
0.22
0.1
0.1
0
0
Model
Layer
Atta
MLP
Head
Model
Layer
MLP
Head
(40 Projection Distance
I.lama--77
Projection
Distance
1.ama-2-155
Figure 3: Comparison of Projection Distances in full models and submodules within the context of model merging. lower value indicates better linearity. For each backhone, we computed the results across all possible combinations of three fine-tuned models, alongside the corresponding merging weights a and 02 (and Ora). In the case of merging two models, there are three combinations fine- tuned models, yielding 25 possible configurations for merging weights ar [0.2, 0.4, 0.6, 0.8, 1] where i E [1, 2]. For merging three models, there's only one combination of fine-tuned models, resulting in 27 configurations for merging weights ae [0.33, 0.5, 0.7] with t [1 3]. Error bars represent standard deviations.
when three models are considered compared to when only two models are merged, particularly when analyzed at these granular levels.
In summary, these findings validate the applicability of Property within submodules, demonstrat- ing its relevance to the merging of both two and three models. As illustrated in Section 3 this property facilitates a closed-form solution for determining the optimal merging weights, thereby enhancing the task arithmetic performance while minimizing associated costs.
3 MERGING MODULES WITH LINEARITY
The selection of merging weights is a pivotal factor influencing model merging performance in task arithmetic. In (Ilharco et al. 2023), the authors propose determining this parameter through abla- tion while maintaining equal weights across different models to constrain the search space within a manageable range. Although this method demonstrates commendable balance between cost and efficiency at the model level, it becomes impractical when merging models at the module level. In such cases, the search space expands exponentially relative to the number of modules; for instance, ablating n candidate weights results in a search space of n for m modules. Fortunately, by lever- aging Property [ we derive closed-form solution for the optimal merging weights, thereby circumventing the cumbersome ablation process and expanding the solution space by relaxing the constraint of equal weights. Here we first introduce commonly used objective in model merging Jin et al. 2023 Ortiz-Jiménez et al. 2023), which is also regarded as the objective of task arithmetic Ortiz- Jimenez al. 2003).
The Objective of Model Merging, Let D (D. C hhe be collection of T datasets, with ......, Or representing the parameters of models fine-tuned from O using the corresponding datasets. The parameters resulting from the merging process are denoted by Omerge. The objective of model merging is defined as:
f(x; (l)
if D. if U._, D.
f(x; mmerge)
Next, we will independently apply the objective to each module that satisfies linearity, using standard weighted operations of task arithmetic,, i.e., 6 6nnngg 0g "
Objectives Function for Each Module. For a module located at position (which could be layer/attentionn/ILPPPeee
let Of represent the weights of this module from model and f(z; 0f)
denote the output of this module given input s. The merging parameters of .f... or for 0; 0....., f. can be obtained by
.
T
Σ ..... of = arg min
E
f
0: +
Σ t'
a Th
-f(x;
(10)

---

Published as a conference paper at ICLR 2025
where r/ 0i 8.. Hi represent the set of imput features of module corresponding Dj. In this paper, H; is obtained by inputting all 2 E Dr into the model 8g and collecting the input features corresponding to module more details please refer to Appendix A.3.3
module satisfies linearity, by applying Property and Property
we have
T
ai 022 .,,, arp arg min
Σ
B
BEEH
Σ
ai (4As(1;0)) (1;0g) + + 1)21 1)As(1;0))) (1:
(1I)
where A (I; 0i) (z; 0.)
(z; 66)
Solving this equation allows us to directly obtain the closed-form solutions for ai a...........
[ai.
..... "i."a"".,,f1"" A bb
(12)
where A C RTXT and b C R..
T Σ
T bj Σ
AjA
Bijk:
Bije
(13)
tm1
tm1
and
Babo Ey Σ
As (x; 08) AS (x;
(14)
k=1
Af (x; (z; - f(z; 6%).
where d denote the dimension of vector f (x; The detailed derivation process can be found in the Appendix A
We present our merging process at the layer-level decomposition detailed Algorithm T Initially, the input feature sets H; compiled for all layers by feeding randomly sampled subset from the task-related datasets into the pre-trained model 0n- Subsequently, for each layer, we apply Eq 12
to derive the optimal merging weights, which then utilized to perform linear merging. 1 Te adaptation of this algorithm to other granularities straightforward, as described in Appendix A.3.3
Algorithm Merging Modules with Linearity at Layer Level
Input: T Fine-tuned Models with L layers
{0] f.............. .... 0f.) ..., OT {0). 0 04 0%)
Pre-trained Model 0, = {0), og .... 0%) Task-Related Datasets D1, D2, .... DT Output: Merged Model 0merge = {0 (0merger 02 merger.... .... 0L 0merge) for t T do
d, < {x E D; x is sampled randomly, Idel N}
> Sample small set of task-related data
H < de
D Set the input of first layer
for
1 to L --11 do
Hi+1 + f(x; 0)), V x E Hi
> Prepare input feature sets for Eq
12
for i = 1 to L do
aj. C app f- Eq 12
Calculate optimal merging weights with Eq I2
Oi merge + 0j + Σt
ai 06)
Merge modules from different models linearly
EXPERIMENTS
4.1 EXPERIMENTS SETUP
Models, Datasets and Evaluation Metries. We utilize Llama-2-7B and Llama-2-13B as our back- bone models. Fine-tuning is conducted across three distinct tasks: mathematics, coding, and trans- lation. For the mathematics task, we employ the GSMSK (Cobbe al.) (021) training set. For the coding task, we utilize the Code Alpaca (Chaudhary 2023) dataset. Lastly, for the translation task, apply the zh dataset from Xu et al. (2024a). During training, adopt the prompt of the

---

Published as a conference paper at ICLR 2025
Table 1: Results of merging fine-tuned models by different methods with Llama-2-13h the back- bone. For each setting, we fully replicate our method three times with different data sample seeds and compute the mean and standard deviation of the three results. The best and second-best results are highlighted in bold and underlined, respectively. For the hyperparameters of other baselines, we conducted a grid search in each merging setting based on the recommendations in the original paper and reported the best results. We only report the average results of different task evaluations here; for more detailed results, please refer to Appendix A.112
Math & Coding
Math & Translate
Coding Translate
Math & Coding & Translate
Methods
Fine-tuned Model
34.02
65.16
51.26
50.14
Weight Avg
33.57
64.57
52.28
46.95
DARE
32.97
62,47
54.33
47.51
Task Arithmetic
34.60
64.72
53.22
48.24
Ours Layer Level
34.47 20.17
66.08
52.42.
50.80
Ours Attn/MLP Level
35.11 #0.29
66.11
52.51 +026
51.05
FastChat (Zheng al. 2023) template and fine-tune the models for 3 epochs with a batch size of 128 and a learning rate of x 10
For evaluation, test mathematical capability using the GSMSK (Cobbe et al. 2021 test set, coding capability with the HumanEval (Chen et al. 2021), and translation capability with the tools and datasets from (Xu et al., 2024a).
Algorithm Implementation Details. In practice, we observed varying levels of bias in the fea- ture magnitudes across different datasets. To mitigate potential adverse impact, implemented mechanism to balance the influence of each sample, thereby resulting in minor modification of the original formula. For further details, please refer the Appendix [A.3.2] of our compact computational and storage budget, we use limited dataset of only 30 samples per task for calculating the merging weights. Surprisingly, this minimal data size demonstrates a com- mendable performance, underscoring the efficiency of our method in terms of data requirements and computational costs.
(A.3.2 Due the constraints
4.2 MAIN RESULTS
We compare our proposed method with several baseline approaches: Weight Average (Wortsman) et al., 2022a b): Weight Average is the simplest and most direct model merging method, which involves taking the average of the parameters of fine-tuned models. Task Arithmetic(Tharco et al.) 2023 Task arithmetic entails straightforward weighted combination of the task vectors and weight of the pre-trained model. DARE Yu et al., 2024]: DARE builds upon task arithmetic by employing random dropout of parameters within the task vectors to reduce conflicts between different task vectors during the merging process.
We present the results of merging fine-tuned models employing various methodologies, utilizing Llama-2-133 and Llama-2-7b (Touvron et al. 2023) as the fine-tuning backbones. The findings are detailed in Table [ and Table 2 respectively. Each value indicates the average evaluation results across multiple related tasks. It is evident that our approach surpasses the baseline in most merging settings, particularly in the case of merging three models, where performance improvement of 3% is observed, albeit slightly lower results were achieved in the merging of code and translation models. This validates the effectiveness of our method. Furthermore, the similar outcomes on both Llama-2-7b and Llama-2-134 demonstrate that our method has consistent performance.
4.3 ABLATION AND DISCUSSION
Since the optimal merging weights are derived through a closed-form solution, our proposed merg- ing method lacks adjustable hyperparameters. Consequently, our discussion centers the influence
8

---

Published as a conference paper at ICLR 2025
Table 2: Results of merging fine-tuned models by different methods with Llama-2-7b as the back- For more results, please refer to the Appendix
bone. The details of this table are the same as Table I
Math & Coding
Math & Translate
Coding & Translate
Math & Coding & Translate
Methods
Fine-tuned Model
32.35
62.62
50.99
48.65
Weight Avg
25.14
53.70
49.25
38.30
DARE
19.11
50.77
50.15
39.00
Task Arithmetic
27.45
54.65
51.11
39.59
Ours Layer Level
27.18+0o4
49.79
50.64
42.68 +0.26
Ours Attn/MLP Level
27.33+0.10
55.86
49.80
42.88+0z.
Table 3: Ablation results of merging fine-tuned models by different levels in method. For each setting, we fully replicate our method three times with different seeds and compute the mean and standard deviation of the three results.
Math
Math
Coding & Translate
Math & Coding & Translate
Level
Coding
& Translate
Llama-2-7B
Ours Model Level
28.34
13.38
46.51
35.95
Ours Layer Level
27.18 80004
49.79
50. 50.6440.4
42.68
Ours Attn/MLP Level
27.33
55.86
49. 80.80
42. +0.28
Ours Head/MLP Level
13.08
14.17:1.4
41. 11.34
8.16.130
Llama-2-13B
Ours Model Level
31.27 20.43
18,90
15.75 +700
50.68
Ours Layer Level
34.47
66.08
52.42
50.80
Ours Attn/MLP Level
35. #0.29
66. 1011
52.51 +0.26
51.05 20.16
Ours Head/MLP Level
12.91
30.17:193
13.71
22.59
of decomposition granularity on our approach. In addition to the Layer-Level and Attention/MLP Level configurations previously illustrated, we also apply our method the Model-Level, which serves as a baseline setting without any decomposition, as well as at the Head/MLP-LLvvl, where attentions are further vertically divided into individual heads.
We also attempted to directly apply our method for model merging the model level. The results Table indicate that, most cases, failed to yield reasonable outcomes. A visualization of the parameters calculated at the model level can be found in Table The failure at the model level can be attributed to the lack of sufficient linear correlation between the weight differences caused by fine-tuning and the differences in output features resulting from fine-tuning. The absence of linearity prevents us from leveraging feature-level computations to reasonably merge parameters, ultimately leading unpredictable results. This implies that method is only effective under linear conditions.
Results at the Model Level.
Results at the Head/MLP Level. Despite exhibiting good linearity the Head level, the merging with our method at this level reveals unstable outcomes, shown Table 3 We also provide a visual example of the parameters during actual Head level merging in Figure
For the Head/MLP Peea//MLPllevel.. due to the architecture of the transformer, the output features corresponding to single head cannot he directly obtained. Therefore, we employed certain techniques to achieve this in practice. Please refer to Appendix A.3.5 for additional results and analysis.

---

Published as a conference paper at ICLR 2025
We offer conjectural explanation for the occurrence of this situation here. Many studies (Wang jet al., 2023; Chen et al. [2244] Zhang et al.] 2024) indicate that each head in the model serves dif- ferent functions during inference. After tumeng various tasks, certain heads may take distinct roles, while our objective function may ask the head to simultaneously average close to multiple heads with different functionalities. This requirement can be quite challenging for individual head, ultimately leading to a collapse of functionality of the merged head.
However, this may not indicate that the linearity of the head is unhelpful for the merging process. Considering the specific functionality of the head (or the particular task associated with that head) during the merging to redefine a more reasonable merging objective could be valuable direction for future research.
5 RELATED WORK
Weight Interpolation and Task Arithmetic. In recent years, the increasing scale of large language models has significantly heightened the demand for data and training costs associated with fine- tuning multi-task models. To merge the capabilities of various existing single-task models into unified framework, several model merging techniques (Daheim al., 2024) Jin et al.) [023; Wan
et al. 2024; Yang et al., 2024b) have been developed. The simplest and most intuitive approach
is weight interpolation Frankle et al. 2020 [zmailov et al. 2018 Ramé et al. 2023 2022). which has been applied to enhance model generalization Wortsman al., 2022b), improve specific single- task performanced Wortsman et al.] 2022a), and boost multi-task effectiveness (Iharroo et al.) [202]]
Li et al., 2022). Subsequently, Task Arithmetic(IIharco et al.] (023) was introduced, enabling the merging of multiple models through a weighted operation on the parameter difference 7. Many
subsequent methods( (aatood(yaaggaano et al. 2024c| Yu et al.) [2024] Yadav al. 2023 Zhang al. 2023) have been proposed based on the principles of Task Arithmetic.
Linearity and Task Arithmetic. The research conducted by Ortiz-Jiménez et al. (2023) begins
with the relevant theories of the Neural Tangent Kernel (NTK) et al. (018) Chizat et al. 2019)
to explore the connection between linearity and task arithmetic. Although experimental validations indicate that fine-tuned models do not comply with NTK's s conclusion regarding the approximate one-order Taylor expansion of models, (OrtizJJiénnz et al. (2023) observes that linearity still can aid task arithmetic. To enhance model linearity, Ortti--imééezz et al. (2023) proposes constraint on the parameter update space during fine-tuning, thereby fine-tuning a model that achieves linearity, ultimately improving the efficiency of model merging. To expedite this training process, various studies have attempted to linearize only a subset of parameters during training. For instance, Jin let al. (2024) suggest linearly fine-tuning solely the linear layers in the attention modules. Tang
jet al. 2024) combine linearly fine-tuming with Earammter-Efffcieett
(PEFT) techniques by linearizing only the Adapter modules, thereby reducing computational costs. Liu al. (2024) derives a closed-form linearized solution for efficiently fine-tuning Transformer networks.
In conclusion, modifying fine-tuning methods to achieve more linear model can reduce interference during model merging. However, these methods still require training data and retraining, which is often challenging in practical scenarios.
CONCLUSION
In this study, we performed a statistical analysis investigate the linearity of model-level and de- composed submodule-Ievel components. Our findings indicate that the missing linearity in full models can be found in the submodules. This observation motivated us to develop an innovative, training-free approach to enhance task arithmetic performance. Specifically, we first decompose the model into its constituent submodules and then compute the optimal merging weights for each module based on a closed-form solution derived from linear properties. Subsequently, we perform a linear merging of all submodules using the corresponding merging weights. Our experimental re- sults demonstrate that this approach significantly outperforms various baseline techniques, including standard task arithmetic methods, across different model scales and diverse task scenarios. These findings underscore the advantages of leveraging the linearity of submodules and present novel perspective for investigating effective and practical solutions for multi-task model merging.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by NSFC No. 62222117.
ETHICS STATEMENT
We proposed a novel approach to enhance model merging performance in large language models (LLMs). Our methodology utilizes publicly available language datasets and leverages pre-trained language models for experimental validation. We do not believe that our code or method is inher- ently subject to concerns such as discrimination, bias, fairness, inappropriate potential applications, impact, privacy and security issues, legal compliance, or research integrity concerns. However, lan- guage datasets and models may possess intrinsic biases that could be inherited by models merged using our approach.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our approach, we provide key information from the main text and Appendix as follows.
Algorithm. We provide our algorithm in Algorithm
!
Experimental Details. We provide our experimental details in Section 4 Moreover, we provide
detailed experiment settings in Appendices A.311 A.3.2 A.3.3 and A.3.5
Moreover, are committed providing the source code of approach, if accepted.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models mod- ulo permutation symmetries. In The Eleventh International Conference on Learning Representa-
tions.
Zeyuan Allen-Zih Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint 143111 2023.
Sahil Chaudhary. Code alpaca: An instructionn-folloiing llama model for code generation. https: 2023.
// wit.hb..ccmm/stst www.ccc...aa.aaacccccccccc.cc..a
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Kierbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. CoRR, abs/2107.03374, 2021. URL https::/p:/ee... http://axxvv... org/ abs/ 221/2/17..02 03374
Wei Chen, Zhen Huang, Liang Xie, Binbin Lin. Hougiang Li, Le Lu, Xinmei Tian, Deng Cai. Yong- gang Zhang, Wenxiao Wang, Xu Shen, and Jieping Ye. From yes-men to truth-tellers: Addressing sycophancy in large language models with pinpoint tuning. In Forty-first International Confer- ence on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, htttt:::///peereieeee net forum? d2OONN90RR
Lénaïe Chizat, Edouard Oyallon, and Francis R. Bach. On lazy training in differen- tiable programming. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' Alchh-BBue,, Emily B. Fox, and Roman Garnett (eds.), Advances Neural In-
11

---

Published as a conference paper at ICLR 2025
formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp. CC pttpppp/ee////aatt... http://ceo....oaoooappe/////////
2933 2943, 2019. URL
ae61 4c55 3e13555774666661 355353333..111222222222222533333
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14166. 2021. URL
org/ Cettss://axxo http///aw.....//aa.../././// 2110. 14168
Nico Daheim, Thomas Möllenhoff, Edoardo M. Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty -based gradient matching. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL httLhttts///ppereeeiee
net
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3259-333333 3269., PMLR, 2020. WRRhhtp:: pttt:::/pproeeeding/ mlr. wwwpppwwwsss.eeeeeeeee (at.11///annee22aaa html
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: machine read- ing comprehension with contextual commonsense reasoning. In Kentaro Inui, Jing Jiang, Vin- cent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing, EMNI.P-ICNNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 2391- 2401. Association for Computational Linguistics, 2019. doi: 10.18653/V1/D19-1122 URL. /tt.:/...rrrr111.1333//1////11/1 1111116/////11911225
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Si- mon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpo- lating weights. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural In- formation Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28. De- cember 9, 2022, 2022. URL ht tp: http:::pppppr......ss.sss www.. wwppppppppsccpppp..ppppppppppp.e paper_ files/ wttees/papr///a/ 2022/ c0
hash:/bc6ccdeed555222222 (00.:e22255522000000066066626aa.
Ceaetttooofeeeeee htm.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL U0.33.RLLtttgeee https:://ppererre.ew nwt. www.:etet..ofmmmmm d= jri
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fourre Conference on Uncertainty Artificial In- telligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 876-885, AUAI Press, htt.://////opcceennss....ooo//// pppppeeeeiinnssppppp////11/ 2018. URL httt:t::/// //wpai//aaaaa/a//aaa/eeeeaaae..a ott.:///sss.aall..oooooooooooerp
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman. Nicolò Cesa-Bianchi. and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580 8589, 2018. URL I neurip.....e....... cc/ppappr/211//hast// html
Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie J. Su, and Li Shen. Fine-tuning linear layers only is a simple yet effective way for task arithmetic. CoRR, abs/2407.0700 2024. doi: 10.48550/ AR ARXIV.2407.0m 070099.. UtI/ttps:: //oo/ http:/...855//axiiiii www.:/wwwww.............
12

---

Published as a conference paper at ICLR 2025
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pies Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview..net, 2023. URL htttps://ppeneeviie
net forumrrrmii (oorum?id==CCnoouRGAA
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merge:: Embarrassingly parallel training of expert language models. CoRR, 0333,, 2022. doi: 10.48550/ARXIV.226....0 03306. URL (ttps::: //oo:///oooo org/
10. (1488500/aaXa0 2208 03306
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. In MRQA @ EMNLP, 2019.
Tian Yu Liu, Aditya Golatkar, and Stefano Soatto. Tangent transformers for composition, privacy and removal. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRevvew.net, 2024. URL httpt: ptttps://operreviiee net ferrm?? 1d=VLF (0002555555
Michael Matena and Colin Raffel. Merging models with fisher-wiggtte averaging. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neurul Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http::/ppperr..oot httpp///eeeaapppeeeeaa////////// 70c26937 fbr 3d4
http::/0-Seeeaeoaoaaaaaaooooooae htmll
Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neurul Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023. New Orleans. LA, USA, December 10 16, 2023, 2023. URL http:: wwp.errs.ipiiiooo nips. wwwpsicppppppppeomme files httpp//aappee/////////p /atppppppp_LLss/pppee//33//33 haash/ html
Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks. arXiv preprint arXiv:2410. 13025, 2024.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URthttps:// //wwn/.m
github. hwwplllogggwn..5g30 qwen2. 57
Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Galli- nari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- fion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December ni nips. http:ppppccccppppeeeaeaaaaaa//pa
9. 2022, 2022. URL 46108d807b50ace 4ateaad4444445555555555330 oaw.::/eeeee-tttttaaateoooooeeee
h html
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez- Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Re- .mw.. press/
search, pp. 28656 28679. PMLR, 2023. 2222/Tame2 html
URL
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8732 8740. AAAI Press, 2020. doi: 10.1609/AAAL V34105.6399 URL https: doi http://oo...ggg/////// 1 1609 aaai. v34105.
6399
13

---

Published as a conference paper at ICLR 2025
George Stoica, Daniel Bolya, Jakob Bjomer, Pratik Ramesh, Taylor Hearn and Judy Hoffman. Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.0305. 2023.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. 2024.
Mixing expert Ilms
into a mixture-of-experts Ilm. arXiv preprint arXiv:2403.07880
Anke Tang, Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. In Forty-first International Conference on Machine Learning.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. multi-task model fusion with partial linearization. In The Twelfth Interna- tional Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OeenReview.net, 2024. URLpttpss /ttp:///ppeeeeieieeee net www./eeetoo///e htt///orrmmiidie
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta. Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.092883 2023. doi: 10.48550/ARXIV.230... 099888877........... URL https: wrg/10.......................... 48550 (a..S///axivv.. 2307. 09288
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
ltttps://ppeneviiee wwt//eeoouunnneee wwr.m?ormmmmmmmm ?id JiDsk12qcz
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. In- terpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Fleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
net forum?id
NpsVSN6o4ul
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models im- proves accuracy without increasing inference time. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore. Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 33965-23995 PMLR, 2022a. URL https: //proceedings. mlr. wwwpsreesvvvveee. /vaass/v666 62 http://ortsaaaaaa.cmmmmm
html
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 7949-7961. IEEE, 2022b. doi: CVPR5268B 2022 00780
URL https: //te::////o/../ org 1109/
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a. htRp:ttss: /ttp:a:://///eeeeeeeeee net
6XXntP
14

---

Published as a conference paper at ICLR 2025
Zhengqi Xu, Ke Yuan, Huiqiong Wang, Yong Wang. Mingli Song, and Jie Song. Training -ree pretrained model merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5915-5925, 2024b.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties- merging: Resolving interference when merging models. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023. URL http:/ http:::/ppppp...ee ni 1644c9af28ab 0TA11228807711177666666266626882 -ttt wwnferenneeeennno html
nis.. ps. wwtpps.ccppppppppppepepppppppppp ff pttpe//ppp///////////m/
Abstract
An Yang, Baosong Yang. Binyuan Hui. Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong. Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui. Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, milms, and beyond: Methods, theories, applications and oppor- tunities. CoRR, abs/2408.07666, 2024b. doi: 1044555/ARXI.2408..oo
URL https:
48554445550048 / 2408 07666
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024c. http:tttsss//ppprre//e
net/ wow.mmiidnnnaa D3QY
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models free lunch. In Forty-first International Con- ference on Machine Learning, ICML 2024. Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htt.htttps/oppeeeee
d= net. forum? d= fuN U8EE
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficcent modules with arithmetic operations. CoRR, abs/2306. 14870, 2023. doi: 10.48550/ARXII.2306..64 URL htttttt:///oo.oog//
4ww50//aXX3. arxiv. 2306. 14870
Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping
Ye. Interpreting and improving large language models in arithmetic calculation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
net wwtpeeefwwtmmieeeeeeeeeeeeeoeeee attee//ooooomntttttteeeeee
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence of cross- task linearity in pnetranningfieetuiinng paradigm. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: ptops://ope httt::://openreieen net for.m?id=ggaA
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MORE RESULTS
A.1.1 MORE RESULTS IN ANALYZING THE LINEARITY OF SINGLE FINE-TUNED MODELS
To ascertain whether the previously mentioned observations regarding linearity remain applicable within the context of model merging, followed the methodology outlined in Zhou et al. 2024)) We analyzed two types of lincarity metrics introduced in their work, cosine similarity and projection distance in feature space.
Specifically, we analyze the cosine similarity between two sets feature deltas: the differ- ences between the output features of the merged module (derived from multiple fine-tuned mod- els) and those of the pre-trained module, denoted as (x 0b Σ a(T)) E atrt)) ences in output features from the fine-tuned models, represented as Σi=1 af (x; + 7/)
f(a; 0j + ()), as well as the corresponding weighted combination of the differ-
E. (f (z; 0 + r!) (1; 0a)). This similarity is denoted by the following equation:
cosine (z; a
A/(15
Σ alr
Σ
a
AS
-t)
(15)
and
cosine merine (af....r E cosinemerge a....r)
(16)
Following Zhou al. (2024), we compare these cosine similarities against a natural baseline: the average cosine similarity of the feature deltas induced by fine-tuning across all possible pairs of the tuned models, denoted as:
E cosine EREH T(T 1) ISKS'T
cos(Af (z: + Ti) Af (z: 06 ++))
(17)
J1 E. cosine cosinef...
I1 Excosine
Kusnine
11
assine
0.5
Model
Layer
Layer
(x) Cosine Simillarit Lhme--27b
(b) Coxine Simiilarity Lluma-2-13b
Figure 4: Comparison of Cosine Similarity in full models and submodules within the context of model merging. A lower value indicates better linearity. For each backbone, we computed the results across all possible combinations of three fine-tuned models, alongside the corresponding merging weights and a2 (and Qz3). In the case of merging two models, there are three combinations of fine- tuned models, yielding 25 possible configurations for merging weights a₁ [0.2, 0.4, 0.6, 0.8. 1] where t E [1, 2]. For merging three models, there's only one combination of fine-tuned models, resulting in configurations for merging weights ai E [0.3, 0.5,0...... 0.7] with t [1,2,3] Error bars represent standard deviations.
Results of Cosine Similarity. As illustrated in Figure (a) and (b), we observe cosine similarity re- sults consistent with Zhou et al. 2024), revealing a significant disparity between 1 - E, cosine merge and the baseline cosine Furthermore, a greater gap and reduced values noted at the mod- ule level compared to the model level, suggesting more favorable condition for linear merging.
A.1.2 MORE RESULTS IN MERGING MODULES WITH LINEARITY
In this section, we report more detailed results of Tables D and 2 in Tables and 5 as well as the outcomes of approach additional models and datasets. The results of model merging the
16

---

Published as a conference paper at ICLR 2025
Qwen-2-0.5B Yang et al.
2024a) model, along with datasets Gsm8k(Cobbel al., 2021). Cosmos
QA Huang et al. 2019).
Ropes Lin et al., 2019), and Winogrande (Sakaguchi et al. 2020). are
presented in Table 6 We also report result on Qwen-2-7B Qwen Team 2024 model. During
training, we adopt the prompt template from FastChat (Zheng et al. 2023 fine-tuning the models
for 3 epochs with batch size of 128 and learning rate of 10
Table 4: Detailed experimental results on LLama-2-138 2-13B.
Method
Merging Two Models Avg
Llama-2-1aB
Coding
Math
Translatc
Avg
Fine-tuned Model
47.91
20.12
34.02
47.91
82.40
65.16
Weight Avg
42.76
24.39
33.57
45.11
84.04
64.57
DARE
41.55
24.39
32,97
40.94
83.99
62.47
Task Arithmetic
45.41
23.78
34.60
45.87
83.58
64.72
Ours Model Level
48.12aa39
14,43
31.27a04:
0.20
37.60a49i
18.90
Ours Layer Level
16.37
22.561uce
34,47
48.42 m2
83.75.1om
66.08 junt
Ours Attn/MLP Level
46.85gmne
23.37
35.1 1an
48.55_mm
66.11 1nm
Ours Heedd/MLP Level
15.87
1.96
12.91 11500
16.27
44.07=
30.17
Method
Merging Two Models
Merging Three Models Coding
Llama-2-13B
Coding
Translate
Avg
Math
Translate
Avg
Fine-tuned Model
20.12
82.40
51.26
47.91
20.12
82.40
50.14
Weight Avg
20.12
84.43
52,28
33.13
23.78
83.95
46.95
DARE Task Arithmetic
24.39
84,27
54.33
34.87
23.78
83.88
47.51
21.95
84.48
53.22
38.36
22.56
83.80
48.24
Ours Model Level
0.00=a0
31.50+ss.19
15.75
47.08
23.58
81.38
50.68emu
Ours Layer Level
2025
84,31
52.42
45.2 1,02
23.37
83.8 1sam
50.80
Ours Attn/MLP Level
20.73=a50
84.29=004
52.51=026
45.59
23.78+0.0
83.78
51.05=a.s
Ours Hcad/MLP Level
0.00=
27.43-710
13.71
14.99
8.13+tm1
44.66-29a
22.59
Table 5: Detailed experimental results on LLama-2-7B.
Method
Merging Two Models Avg
Llama-2-7B
Math
Coding
Math
Translate
Avg
Fine-tuned Model
43.97
20.73
32.35
43.97
81.26
62.62
Weight Avg
31.99
18.29
25.14
25.02
82.38
53.70
DARE
21.15
82.14
50.77
Task Arithmetic
37.83
17.07
2745
27.07
82.24
54.65
Ours Model Level
43.47
13.21
28.34
2.27-a034
24.48-um
13.38
Ours Layer Level
25.25
19.11
27.18
17.77
81.82=00
49,79 49.79
Ours Attn/MPP Level
35,96=0m
18.70=029
27.33+8.0
30.00=00
81.73-as
55,86=044
Ours Head/MLP Level
17.41=nm
8.74
13.08
0.58
27.77-151
14.17
Method
Merging Two Models
Merging Three Models
Llama-2-7B
Coding
Translate
Avg
Math
Coding
Translate
Avg
Fine-tuned Model
20.73
81.26
50.99
43.97
20.73
81.26
48.65 68.65
Weight Avg
15.85
82.64
49.25
17.66
15.24
81.99
38.30
DARE
17.68
82.62
50.15
15.24
82.13
39,00
Task Arithmetic
19,51
82.70
51.11
22.29
14.63
81,86
39,59
Ours Model Level
11.79_n7
81.23_mm
46.51 1m1
41.572mm
14.431om7
51.841151
35.95_051
Ours Layer Level
18.90-1xx
82.38
50,6440u
30.50=us4
15.85
81.68 tous
42.68-ua
Ours Attn/MLP Level
17.28
82.33
49.80
32.02.ma
14.841cs:
81.791oom
42.88
Ours Head/MT.P Leel
16.06_21
66.22
41.14gm
0.00
0.001.00
24.4811.:
8.16
A. 1.1.3 ABLATION STUDY ON NUMBER OF MODEL TO BE MERGED
To demonstrate the advantages of method in scenarios where more models merged simul- tancously, we increase the number of task-specific models merged simultancously gradually and calculate the average performance of the merged models.
17

---

Published as a conference paper at ICLR 2025
Table 6: Detailed experimental results Qwen-2-0.5B.
Method Qwen- 2-05B
Merging Three Models
gsn8k
cosmis_pp
ropes
gsm8k
cosmor_qui
winogrande
Avg
Fine tuned Model
31.61
65.01
45.65
47.42
31.61
65.01
$6.59
51.07
Weight Avg
25.09
51.91
37.50
38.17
17.74
49.31
51.82
39.62
DARE
34.57
40.09
41.61
38.76
31.24
45.14
51.34
42.57
Task Arithmetic
26.54
50.83
37.56
38.31
22.52
48.63
51.82
40.99
Level
31.35
22.71-211
46.95
33.67
14.56.a20
47.95
52.96
Ours Layer Level
30u52asu
51.76=01e
38. 38.07=a2
40.12
27.71
51.00
52.31
43.67
Ours Atte/MPP Level
30.55
22107
37 22
40
27 27.48
31.13
52.55
43.72
Ours Hcad/MLP Level
29. 19 19sus
50.07
37.5 a.
38 93
15.05
39.77
50,57
35.13
Mathod
Merging
Three Models winogrande
Qwen-2-0.5B
gsm8k
winogrande
ropes
Avg
cosmos.ga
ropes
Avg
Fine-tuned Model
56.59
45.65
44.62
65.01
55.75
Weight Avg
10.84
50.93
49.09
36,96
51.42
44.36
45.23
47.00
DARE Tisk Arithmetic
33.97
52.39
48.85
45.07
51.82
42,91
46.20
46.97
15.54
$2.06
48.67
38.76
51.90
44.84
45.35
47.36
Ours Model Level
7.77
53.68
31.34
30.93
52.39
48.00
30.95
43.78
Layer Level
24.30
51.4610u
41.52
some
6027
18.
Ours Atta/MPP Level
24.18
31 31.94auu
48.25-12a3
41.46
$1.62amu
50.64:20s
42.48
48.25
Ours Hcad/MLPP Level
24.00aa
51.58amm
47.98
11.18
50,93anm
50.22
43.21
48. 12.500
Table 7: Experimental results on LLaMA-2-7B with More Dataset: Winogrande, Cosmos qa and Ropes
Method
Merging Two Models
Ilama 2-7h
Winogrande
Cosmos qa
Winogrande
Ropes
Cosmos
Ropes
Avg
Fine-tuned Mode]
82,26
84.72
83.49
82.26
75,42
78.84
84.72
75,42
80.07
72.22
58.75
66.
Ours Atta/MPP Level
68.42
84.18
76.3
69.47
69.98
69.72
73.07
68.59
70.83
Method
Merging Three Models
Llama-2-7b
Winogrande
Cosmos qa
Ropes
Avg
Fine-tuned Model Task Arithmetic
82.26
84.72
75.42
80.8
71.25
64.13
$5.85
63.7433
Ours Attn/MLP Level
71.49
68.70
56.70
65.63
In the table 10 we observe that the number of models increases, method gradually sur- passes Task Arithmetic and the performance gap becomes larger, which demonstrates the efficiency of our method particularly when more models merged simultaneously.
A.1.4 ABLATION ON THE NUMBER OF DATA
From the table 11 we can see that our method shows consistent performance across different data quantities, sometimes achieving good results with as few as 3 data per task. As result, if one wishes to use our method in a situation absolutely no data, we believe that simply synthesizing a few data for the corresponding task through manual methods or with the help of LLMs is feasible.
A.2 MORE DETAILS ABOUT NON-LINEARITY SCORE
Overall, we achieve the practical implementation of the Non-linearity Score through discretization and using the Euclidean distance as distance metric in the feature space.
Given a fine-tuned model 0, interpolate between this model and its corresponding pre-trained model 8 using & E {0, 2, N 1, N}. This results in N + 1 models 02 = $ - 0g = 0 + & (0 - Bu). If 8 satisfies linearity in Definition under a distance metric D in the feature space, N). ,, the following approximation holds:
then for any j E (0, 1,2,
(i ((22
D(f(r;
(18)
N
18

---

Published as a conference paper at ICLR 2025
Table 8: Experimental results -wen--2.-7BB
Methol
Merging Two Models
Merging Three Models
Ques-2.5-78
Math
Coding
Math
Transiate
Coding
Translate
Math
Coding
Translate
Fine-tuned Model
71.49
60L36
71.49
72.56
Taik Arithmeric
75.13
67.68
71.40
78.69
86.43
82.56
64.02
86.30
75.16
78.92
65.24
86.56
76,90
Ours Layer Level
78.01
64.02
71.01
77.93
86.55
82.24
64.63
86.59
75.61
81.04
64.02
86.65
77.23
AIt/MP
86.64
77.51
Table 9: Experimental results with more baseline
Merging
Merging
Llama-2-70
Math
Coding
Avg
Math
Transate
Aug
Coding
Translate
Avg
Math
Coding
Translate
Avg
Fine-tueed Model
43.97
20.73
32.35
43.97
K1.26
62.62
20.73
81,26
50.99
43.97
30.73
81.26
48.65
Weight Avg
18:29
25.02
82.38
82.64
17.66
81.99
38.3
DARE
21.15
17.07
1941
87.14
50.77
17.68
19.64
82.13
Task Arithmetic
31.83
17.07
27.45
27.07
82.24
54.65
19.51
82.7
51.11
22.29
14.63
81.86
39.59
Breaderumhs
18.11
18.00
18.51
16.22
K2.24
49.23
19.51
82.26
50.89
7.96
18.90
82.78
36.55
TIES
82.11
42.12
23.21
17.68
22.44
25.39
53.93
18.29
26.38
82.36
Our Layer Level
35.25
19.11
27.18
17.77
81.82
49.79
18.90
82.38
30.64
30.5
15.85
81.68
42.68
Ours AItMMP Level
35.96
18.70
27.33
30.00
81.73
55.86
17.28
82.33
49.8
32.02
14.84
81.79
42,88
Since fæ; 0) is a vector, we utilize the Euclidean distance as distance metric D, to measure the distance between features. To assess the discrepancy between the actual situation regarding 0 and the Eq 18 the Non-linearity Score is defined as follows:
0.), [(;;
1 - m N
Non-linearity Score
Σ
Σ
(19)
(O....NN) (O.......
((f(:::)).(((;0.1)))) is shown in Figure "(
We set N = 10 in this paper. Some visualization
We can also extend the Non- Non-linearity Score for analyzing the linearity of merging multiple fine-tuned models, maintaining the same functionality as the Projection Distance metric discussed in Section Due to the limitations of our compact computational and storage budget, we employ Projection
2.3
Distance in Section 2.3 for its higher efficiency.
A.3 MORE DETAILS ABOUT OUR PROPOSED METHOD
A.3.1 THE DERIVATION OF THE CLOSED-FORM SOLUTION
Objectives Function. For module located at position i (which could be a layer/attention/MLPP let 8 represent the weights this module from model and f(x; 8)) denote the output of this module given input x. The merging parameters aj az..... .... ar for 0 .f,....,6, 07 can be obtained by
T
T
aj 0g of arg min Σ
E E
f
x; 0 +
Eaini
Σ airti
f(z;
(20)
where = 8, Hi represent the set of input features module corresponding to D,. this paper, H; obtained by inputting all E D, into the model 0g and collecting the input features corresponding to iodule i.
If module satisfies linearity, by applying Property
2
we have
19

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study on Number of Model be Merged
Arg Are
Collig & TI Traniiee Coiling Transarc & Clonoo.qu qp Codling & Tronstate Comtos Royes Coolig & Transttt Coamis Ripe Wingraide
Tint Arthirett
04990
03382
03460
2.0
19
+1895
Table 11: Ablation on the Number of Data
Data Num per Task
50
Math
29.92 1.53
0.33
0.28
10.50=0.544
30.8 2
Layer Level Merge Three
Coding
0.29
Translate
81.78+0.10
8157 = 0.02
$1.72
81.68 0113
81,80
Math
12.17
32.78 0.66
0..8
32.02 = 0.31
32.68 22.88:10000
Merge
Coding
+.550
0.57
Tranilate
81.44 0.05
81.74 0.04
81,722006
81.79 0103
81.79
3; 0
Σ
ag [13; 04)
===1
Σ
de (ʃ (1; 03 + +z) -(;; - - (1 +T,) -(x; s(x;
(21)
1'=1,'
Σ df (z: 61,) +(a - 1) (1;
1=1,//t
where Aʃ (1; 02) (x; 01) fxx; 0))
Consider f(x; () as a d-dimensional vector. We have
T Σ
ag.. af min
EZEHi
Σ
df (2:6%) + (af - 1) (z; 6i)
1=1
km
e'ml,t'
(22)
Taking the derivative of aÿ and setting the derivative equal to zero, we have
T
d
T
2 Σ
BRED
Af (z:
Σ
df (x;6,,, + - (I;
(=1,6
k=
P'=1,t'ft
(23)
+2 EPED,
Σ
Af (2;0)),
Σ af (x:0%), + - 1)1f 1aa (z;
=0
k=1
Let
Babe EnED_ Σ
As (2;0))) AS AS (2;00)
(24)
k=1
We have
Σ
Σ
a Bij + (aj B)jt
(=1(f)
1'=1,14
(25)
T
+
Σ
at + (a²) - 1)B3 =0
#=1,1*3
Then have
20

---

Published as a conference paper at ICLR 2025
Σ
E
a Bew +
B aa +
E
dBB + B
Σ
Beje + B
t=,t7]
1=1,1/7
(26)
We have
Σ
Σ
Bijm Bujj Bmjm Bjjmmann +(
Σ
Bij + B
Σ
Bija
m=1,mfj 1=1,
(=,,f)
(27)
Then
Σ
Bejm)
Bija
(28)
Therefore, by differentiating with respect to each af. where [I,..., 7), and setting the derivative equal zero, obtain:
T
Beom )ar
Bun
m=1
T
Betm )an
Bert
(29)
m=1
t=1
T Σ BarmJa
T
Brrt
have
Ala az...,, aT! =
(30)
where
T
Aj, Σ Bijk, A C TxT
(31)
bj
Σ
B Bijt, b CCR
1=1
Finally we have
[a] 02 ...
O77
A
(32)
In the practical implementation, we utilize np. linall............. sol to solve Eq 32
A.3.2 BALANCE THE INFLUENCE OF EACH SAMPLE
In practical applications of the formula for calculating merging parameters, we observed discrep- ancies in biases present in data from different tasks. For instance, the average token length varies across tasks (e.g., code data exhibits a greater average length than translation data), and there are dif- fering token distributions (c.g., common token distributions differ between mathematical and code datasets). Consequently, this leads to varying norms of the output feature differences A (z; 12) of the models and modules.
21

---

Published as a conference paper at ICLR 2025
To balance the influence of each sample within the formula, we introduce denominator for each data-related term:
(x:18 + Ei= abrb)) f2;; 03)
a. 02-- af arg min
E Dr
(33)
7 + | 0J + -/2
Finally, obtain that
El=1 Af (x; 8() A (x;88))
Babe E D.
(34)
E |||||| (1;; 0g )
The remaining content identical that of the previous section.
A.3.3 GATHER INPUT FEATURE SET FOR SURMODULE LEVEL ANALYZING AND MERGING
When analyzing the linearity modules and calculating the merging parameters for module level merging (including layer level, attn/mlp level and head/mlp leve), We should collect the feature set Hj corresponding to the input position each module first.
In practice, we input the data from dataset De into the pre-trained model On all at once, performing a single forward inference to collect all intermediate features, which serve as the input for each module at the corresponding position.
Especially, in layer level's calculation, the input 2th layer should be the output of the (i 1)th layer in 8g. In attn/mlp level's calculation, the input of 7t attn should be the output of the (i - 1)th layer in Pn. and the input of 7th mlp should be the output of the (i)th attn together with output of (i 1 layer in fn- In head level's calculation, the input of head in layer 7th should be the output of the (i - 1)th layer in do.
A.3.4 MORE DETAILS ABOUT THE BASELINE
For Task arithmetic, We explored the hyperparameter merging weights E [0.1, 0.2, ... .09...,...99.... 1.0] and selected the best results for reporting the table.
For DARE, following the recommendations in the paper, we examined various combinations of the hyperparameter dropout probability drop_rate ratio E [0.6...., and merging weights a [0.6, 0.8, 1.0). reporting the best results in the table.
The optimal hyperparameters corresponding to the best results in Table [nd obtained in practice are reported in Table 12
Table 12: The optimal hyperparameters corresponding the best results obtained in practice for baslines.
Math & Code
Math
Coding & Translate
Math Coding & Translate
Hyper-parameters
& Translate
Llama-2-7B
DARE
0.8 1.0
0.8 1.0
0.8 1.0
0.8 1.0 0.5
Task Arithmetic
0.6
0.5
0,4
Llama-2-13B
DARE
0.7 1.0
0.7 1.0
0.8 1.0
0.7 1.0
Task Arithmetic
0.6
0.6
0,4
0.5
A.3.5 MORE DETAILS THE HEAD/MLP LEVEL
For the Head/MLP level, due to the architecture of the transformer, the output features corresponding to single head cannot be directly obtained: The о_pro module will gather the intermediate features of each head and map them into a comprehensive attention module output, which can not be directly divided by the arrangement of the heads.
22

---

Formula Form Published as a conference paper at ICLR 2025
Therefore, employed certain techniques achieve this in practice: For each layer, first collect the input of o.proj. Subsequently, we divid o.pro into n_h linear layers according to the pa- rameter relationships associated with each head. The corresponding segments of the collected input are then fed into these corresponding linear layers, producing outputs that serve as the final output features for each head, which will be used for subsequent calculations of the merging parameters.
A.4 VISUALIZATION RESULTS
Table 13: The actual parameters obtained at the model level. This is one of the results from three random trials.
Math Code
Math
Coding
Math & Coding & Translate
Method
& Translate
& Translate
Llama-2-7B
Ours Model Level
[0.85] 0.259]
[3.43] -15537]
[0.237 0.882]
[0.758 0.480 -0. 180]
Llama 2-13B
Ours Model Level
[1.039 0.132]
[-1.083 1.652]
[-0.463 1.923] [0.562 0.422 0.062
A.5
MORE DISCUSSIONS
A.5.1
DIFFERENCES BETWEEN THE LINEARLY PROPERTIES STUDIED THIS PAPER AND TRADITIONAL LINEAR PROPERTIES
The linearity properties studied in this paper differ from the Traditional Linear Properties, as illustrated in the following table:
Table 14: Differences between the Linearly Properties Studied this Paper and Traditional Linear Properties
Traditional Linear Properties
Linearity Properties Studied in This Paper
Linear relationship between the difference in output features of different input fe(x) - fo())) and the difference in input zg- foO denotes modul(mo with
Lincar relationship between the difference in output features of the model (module) before and after fine-tuning fo (x:) fo, (ar) and the difference in parameters before and 0g- fo(x)) fa, (z) 2 C(0 0o)
Definition
parameter 0
after fine-tuning
fa(z) fe(zu) z C(z zn)
where C' is constant determined by O.
where C constant determined by 0g and æ.
Determined by both the alteration introduced by fine-tuning process and the model (module) architecture of fp.
Determined by the model (module) architecture of
Determinants
We also present an example futher illustrate that there no particularly intuitive connection between the linearity of the module and the linearity of its submodules:
Consider two contiguous submodules, 0 and 02. Let (01; 02) denote the composed module.
If 0, and 02 both satisfy the Traditional Linear Properties, it is obvious that the module {01; 02) which formed by these submodules will also adhere to the Traditional Linear Properties.
However, if 02 and Og both satisfy the linearity property studied in this paper, for an arbitrary x, the final output results after applying perturbations 20, and Ad2 to the two modules is:
Ipptt
Output the first submodule
Ad,
(a) Co A0 Jo, (2) Co A8) Cor (1)4Co, on, Atly (..(1)
Output of the second submodule 8g + A8,
23

---

Published as a conference paper at ICLR 2025
100
0-033 0.9 9:57
0334 0:35 034
0.00
042 0.2
0:33 035 0:2
0.24 2.0 0:1
0:4 335
0.45 040 0:30
0.7 031
052
040 020
0.28
02 252 0.32
030 330
000 0:7 6:27
0:00 376
030 0.20
033
0.5 0:97
60 300 0.55
0:56
0.13
51 6/42 022
0.3
230 0.28
030
3:0
0.52 0:6 6:06
0. 6:07 027
50
0:00 0.0
0:55
0.7 6.2
0.51
09 0
0.5 3:9
075
055
51
0.48
0.50 043 6:25
52 8-40 0:7
52 0:4 220
0:70
0:46
6:22
84
345 0.26
60
041
921
055
0:42
0:46 038 8:00
0.50
8.37 0:00
055
0/0 026
8:41 1:0
4,40
0.21
356
0.70
0.04
0:0
4.00
9:30
30
0.5
0.7 0:31
0.53
0.35 0:2
30
255
0:30 0:0
13
0:45
0.30 6/0
0.70
2.22 0.25
5.65
0.0
3.65
32
12
0:57
035 0:3
55
0.34 0:3
12
2.50
0:30 030
32
0.54
0.31 040
12-45 02-0
3TH
0.33
0.32
0.64
0:37
33
14
035
022 9.4
14
55
025 4.4
54
0.00
0:3 0:30
34
0:5
0.54
8.44
16
0.7
247
0.25
18
5.
020
3.50
022
16-09 0.62
0.33 0:32
16
5
3:00
16-15
044 0.3
16
0.50
0.84
0.34
16
2.65
2.61
0.22
16
0.7
0:00
3.40
38
003
0.99
0:55
19-
0252
0.50
3:35
18
0.50
0:0 0.51
38
063
0.55
0:00
10
0.52
0.02
0.30
10
0.75
065
3:22
Laye
Lawe
Layer
Layer
0.50
0.55
0.44 0:31
20
56
6:45
0:5
20
055
0:48
823
30
0:44
0.44
0:25
20
0000
3.02
0.37
20
0272
056
3.35
22
22
055
0.38 0:30
22
0252
0:07
0:31
22
857
0:4
9.34
22
0:22
0.36
0:42
22
0.68
0:29
0.29
22
0056
0:09 3:20
24
0:54
045 0.20
24
0252
0:44 020
24
8559
049
6:30
24
0.36
0,49
0.39
24
9:37
380
0.22
24
0:67
059
320
25-
0.50
043 8:34
20
0:45
0.30
330
28
8:5
0:0
28
0.0 0:39
28
26
0.34
0.56
044
2:
057
0.28
251
0.27
0:27
3.0
28-
0.50
0.35 0.30
28
3550
021
350
28
9.00
839
630
28
048
0.56
28
0.40
0.20
28
3.0
0.50
0.3 6:25
49
520
341
055
829
0.32
031
055
330
-0.50
30
0.0
30
3.4
30
230
830
620
30
0.42
30
0.00
320
055
041 0.34
40
03
350
2.5
050
820
91
047
022
-0.2
0.0
0.24
3.0
32
034
049 0.3L
32
333
0
0.3
32
030
838
220
32
033
035
82
0.37
3.0
0.54 047 020
555
0:45 027
2.54
035
520
30
0.52
0.7
87
0.70
035
8.00
24
035
229
96
037
033 043 0:30
305
0:44 3:0
0.50
951
0:00
25
3.04
2
0.83
2:50
0.52
0:00
3.45
9:31
6:00
37 0:45 041 8:33
45
9:00
37
8:88
37
064
043
D:3
37
0.29
342
0.43
5.27
0000 330
0:31 0:21 8:7
20
0.27 0:42
0.37 0:43 6:00
0:75
0.55 0.28
20
0.43 341 0.32
30 9:62 0.10 3:40
-1.00
Layer Level
ARH Level
Hip Level
Head Level
Head 3 Level
Head 2 Level
Figure 5: A visual representation of the merging parameters obtained at different levels during the actual merging process is presented. Here, we showcase a result from the merging of three fine- tuned models. Additionally, only portion of the head's merging parameters is displayed, while the distribution of the merging parameters for the other heads is similar to those presented.
Co.e denote constant determined by 0 and x. Now, if (0;; as whole satisfies linearity property studied in this paper, then its output should be fe, (fo, (x)) +( (A0;; A02] .2.(() when input
We can observe a significant difference between formulas (1) and (2) indicating that there is no particularly intuitive connection between the linearity of the module and the linearity of its submod- ules, which also necessitates further experimental and observational analysis, as what we do in this paper.
24

---

Published as a conference paper at ICLR 2025
XCM
Cnte
ATV
CANA
6.11
CAV0
AJH
441
AMK
SUM
CAA
CATA
6TM
EXIT
64
140
00M
C31
631
C411
636
CI1
63A
6C3
6312
6JM
AXII
46-
5200
130
538
003
6:0
6317
C3H
631
630
6111
E30
X221
340
C318
6317
050
630
6:30
6702
6311
6311
6312
6111
6301
3421
1PM
160
639
C227
6216
670
6471
5311
E314
6311
0ph
Figure 6: A sample of Visualization of the D(((=))) The image is sourced from the linearity validation of the Llama-2-131 model after fine-tuning for mathematical tasks. From left right, displays the model-level linearity validation, the layer-level validation at the 9th layer, and the MLP- level validation at the 9th layer. The latter validations already very close to ideal linearity.
A.5.2 COMPARISON WITH WEMOE AND CAT
Table 15: Comparison with WEMoE and CAT.
Coieaoelcs
Oues
BTX Sakahanter et al postsad MoE)
WESSE lang et
CAT Prabbakar et NES
Samt pretats
model
Saile yrurds
Samr pectrin
Requicenets the Model Merged
fin-uming
ing-uming with
diffcret back
inc-uning
diftores
different, talk.
with
Which Put to be Merged
modeles
exeegt
modries
excepe MLF. The MLP will be marged during
All LibA's
Panuneter
Welghex
Hyperpanmetrv Raquied Merging
subnoodaless
Reater Netweek
Netwonk
Waight Martis
Layst/Atts/MLL Cloeed Korm
Optimization Methods
Training
Trannig
Training
MrE
MIs. Anditectur
Amditectam Marad
Merael
mip's
Marsed
Weights)
Merging
Weighhi))
In our approach, merging is performed directly, which may result in different merging outcomes due to the selection of varying data. Our method requires only minimal data and does not necessitate training, yielding a model with standard architecture. In comparison to our method, WEMoE con- structs MoE architecture model and employs Router during inference to determine the merging weights for each MLP based on the current input, which also utilizes a linear merging form. As a result, WEMoE may get better merging weights and achieve better results but incurs additional training and inference costs.
A.5.3 DISCUSSION ON SECOND-ORDER INFORMATION
In our initial setup, considering that the original form of Task Arithmetic is first-order linear, we did not give much consideration the involvement of second-order information for consistency. As mentioned in the comments, we recognize that further considering the incorporation of second- order information is very meaningful direction. If we can find methods to utilize second-order information, may be able to further refine our definitions of linearity and formula expressions, and estimate the linearity of each module with lower error.
Additionally, we might consider iteratively calculating the weight of each module; for example, first calculate the weight of the first layer, then use the fused output features of the first layer to guide the weight calculation of the second layer. This might further exploit the potential connections between modules in different positions. By utilizing second-order information, we might be able to design loss function that merges all the weights to be optimized into simultaneous optimization
25

---

Published as a conference paper at ICLR 2025
In future work, we will explore more applications of second-order information in method.
process.
A.5.4 COMPARISON WITH GIT RE BASIN, ZIPIT AND MUDSC
Table 16: Comparison with Git Re Basin, Zipit and MuDSC
Comparison Table
Applicable Module
The Type of Linearity Relled upon durlng the Merging
Key methed for merging
Layer (fell
Traditional Lincarity
Combinatorial optimization
Git Ainsworth
connected layer)
Propecties
Network
Lincar Layer (fuil
Traditional Lincarity
Align features by similarity in the activation
Zipic! Stoica et al. 2023
layer)
Properties
Network
Align by similarity both parameter and
Lincar Layer (full connected layer)
Traditional Linearity Properties
MuDSC Xu et al. 2024b
Network
activation space
layer, attn and MLP)
level (including
Lineurity Properties Studiod This
Closed form solution optimal merging
Ours
that exhibit Lincarity
introduced in this
Paper
weights
paper
As shown in the table, all these methods relies on the Traditional Linearity Property, and can only apply to linear layers. In summary, we believe that our method has several novel contributions that make it distinct from Git Re Basin (Ainsworth et al.). Zipit!(Stoica et al. 2023) and MuDSC(Xu ct al. 2024b).
A.5.5 DISCUSSION ON FINELY-GRAINES COMPONENTS
Further decomposing the model into more finely-grained components is indeed an intuitive and promising direction for exploration. In the paper, we conducted some related experiments. Based on the structure of the LLM model, we further divided the attention layer into individual heads, inde- pendently estimating their linearity and ultimately testing their performance, but revealing unstable outcomes at this level (as shown in Table 3).
Here we propose a potential explanation for the occurrence of this situation. Numerous studies (Wang al. 2023 Chen et al., 2024, ZZhang al. 2024) suggest that each head in the model performs different roles during inference. Following fine-tuning across various tasks, certain heads may adopt distinct functions. However, our objective function may necessitate a single head to simultaneously average among multiple heads with diverse functionalities. This requirement can be quite challenging for an individual head and may ultimately result in a collapse of functionality in the merged head.
However, this may not imply that the lincarity of the head or more finely-graine components is unhelpful for the merging process. As mentioned above, the relationship between the linearity of the module and the lincarity of its submodules is not straightforward: therefore, further experi- ments and analyses required. We will further explore whether there better ways address the aforementioned issues in future work. For instance, based on the architecture of the LLM, we could further decompose each module into individual linear layers (decompose attention into qkvo projection linear layers, and MLP into two linear layers). Alternatively, we could incorporate inter- pretability methods Wang et al. 2023) to improve understanding and identify which head is most relevant to specific task.
A.5.6 DISCUSSION ABOUT ACTIVATION FUNCTIONS
In this paper, we did not make any changes to the architecture of any part of the model, and all nonlinear activation functions remained unaltered. Regarding the achievement of such linearity without modifying the activation functions, we believe this is due to the many differences between the Traditional Linear Properties and the Linearity Properties Studied in This Paper discussed above.
The connection between nonlinear activation functions and Traditional Linear Properties has been widely studied, but the connection between nonlinear activation functions and the Linearity Proper- ties Studied in This Paper still awaits exploration. We believe that activation functions are likely to play crucial role in the emergence of linearity.
26

---

Published as a conference paper at ICLR 2025
Currently, the vast majority of LLMs (including Qwen and Llama-2) use the SwiGLU activation function. Compared to other activation functions such as ReLU and GELU, SwiGLU more com- plex and nonlinear, yet it also enhances performance. We hypothesize that replacing SwiGLU with more linear activation functions could improve overall linearity, which would be beneficial for our method. However, we must carefully consider the balance between the immediate performance drop caused by the replacement and the performance gains achieved through the merging, which requires further experiments and analysis.
However,it is difficult to find other identical LLMs, differing only in their activation function, for fair comparison due to the enormous cost of training LLM. We plan conduct experiments related to activation functions in a controlled environment as described in Allen-Zhu & Li (2023) in future work.
A.5.7 DISCUSSION ABOUT How TO LEVERAGE LINEARITY AT THE HEAD LEVEL
1. Our initial proposal involves collaborating with interpretability methods Wang et al. 2023 Initially, can use interpretability methods determine which head actually more relevant to a particular task. By obtaining weights through relevance, we can further adjust the originally explored weights make full use of the capability of each head.
2. The second proposal involves modifying our loss function: our original loss is quite simple and only explores the output feature level. We can refer other approaches (Stoica al. 2023 Xu et al. 2024b) to also explore parameter space, making the explored weights more reasonable and enhancing the use of linearity.
3. The third proposal is to use our linearity to analyze which task given head is actually more related (since our linearity introduced by fine-tuning). This might potentially become new method of interpretability.
A.5.8 DISCUSSION ABOUT THE EFFECTS OF MODEL DEPTH
For fair and convincing comparison, we utilized existing backbones like Qwen and Llama to examine how different depths impact linearity, using the Non-linearity Score as an indicator, which introduced in our paper and with smaller score indicating closer adherence ideal linearity.
Table 17: Non-lincarity Score for Different Models
Level
Qwen-2-0.5B
Qwen-2.5-7B
Llama-2-7B
Llama-2-133
Model
2.1130 = 0.5250
4.2559 = 0.7037
2.7637 + 0.3442
2.1929 + 0.2271
Layer
0.4430 = 0.7331
0.1412 + 0.3019
0.1645 0.4329
0.1675 + 0.3760
Attn
0.2439 + 0.1631
0.0649 0.0592
0.0229
0.0107
0.0250 + 0.0115
MLP
0.3112 + 0.8353
0.0810 + 0.2068
0.2030 + 0.5655
0.1622 + 0.4295
From the table,
observe:
1. At the model level, linearity does not seem to have clear relationship with model depth. For the Qwen model, the linearity of the 0.5B model is better than the deeper 7B model, while for Llama, linearity is greater in the 7B model compared to the deeper 13B.
2. At the Layer/Att//MLP level, within the transformer architecture of decoder language models, the depth of each Layer/Attn/MLLP across different model sizes can be regarded as the same, with variations occurring in the width. Overall, wider modules tend exhibit slightly better linearity, which aligns with conclusions in NTK-related works such as Jacot et al. (2018) and Chizat et al. 20199.
A.5.9 DISCUSSION ABOUT THE PHENOMENON THAT QWEN-2.5-7B HAS MUCH HIGHER NON-LINEARITY SCORE THAN OTHERS
In order provide possible explanation for the phenomenon that the non-linearity score for Qwen- 2. is much higher than others, we conducted further experiments based on Property from the paper.
27

---

Published as a conference paper at ICLR 2025
Specifically, we first computed the parameter difference before and after fine-tuning for each model, denoted as 7 = - 8g. We then used parameter a to weight this parameter difference and added back the pre-trained model On to obtain the interpolated model 8, + ar. The L2 distance between the features output by this model 0g + oT and the features output by the fine-tuned model was computed and is presented in the table below. (When = O, the interpolated model becomes the pre-trained model dg. When a = 1, the interpolated model becomes the fine-tuned model 0, the distance will be All distances normalized. The Non-linearity Score directly proportional to the difference between the value for the specific model and the value for ideal linearity in the table.)
Table 18: Model Linearity and Corresponding Non-linearity Score
Modil
D0000
a
0.2
0=03
o =-4.4
m-0.5
a U.1
0.31
127
Non-lieariiyy Score
Kdaal Limearity
1,00
0090
0300
0.200
0.00
0,500
0.00
0,300
0.200
0.00
0.000
Qwen-2.5-76
1,000
0911
0.79
0.61
0.85
0.731
0.499
0.777
0.169
0085
0,000
4,2559
0.835
LImna-377
0.57
Llmma-2-136
1000
0305
0200
0.633
0331
0.35
0339
0247
0150
0013
0.000
2.1929
For model that satisfies linearity, the distance between the interpolated model's output features and the fine-tuned model's output features should decrease linearly as a increases (as shown the first row of the table). However, we observe from the table that the Qwen-2.5-7B model experiences very steep decline between a = 0.5 and 0.6, which does not occur with other backbones. Such steep decline does not satisfy linearity well, resulting in the non-linearity score for Qwen-2.5-7B model level being much higher than others.
This is very interesting phenomenon. We find that the performance of Qwen-2.5-7B is very similar to sigmoid function: when the parameter difference (or a)) is far from the threshold, the change in its output features is relatively slow, whereas the change is rapid when the parameter difference (or a approaches the threshold.
We suspect that this phenomenon is related to the model's foundational capabilities, Qwen-2.5- 7B is a model with significantly stronger performance in various aspects compared to other models (according official website data). To explore the specific reasons for this intriguing phenomenon, we will conduct more experiments and analyses in our future work.
28


Reviews:
[{"review_id": "PMdqFVqWIk", "reviewer": "Reviewer_SeC4", "summary": "This paper shows that by identifying submodules that exhibit linearity, more granular and effective merging can be achieved. After defining these submodules, they approximate the true objective with a linear interpolation, which holds when the submodules behave linearly. The objective is defined as minimizing the loss on the joint dataset by adjusting the weighting coefficients $\\alpha$. This approach outperforms existing parameter merging methods, such as Task Arithmetic. They also perform insightful ablations to determine the optimal level of merging granularity—whether at the model, MLP/attention, or head level.", "questions": "An interesting direction for improvement could be incorporating second-order information into the optimization process rather than updating each weighting parameter independently. Currently, the approach adjusts the coefficients linearly, assuming each weighting parameter can be optimized in isolation. However, second-order information, such as Hessian-based adjustments or other curvature-aware methods, could potentially capture interactions between weighting parameters, especially when submodules exhibit interdependencies. This is why routing methods could be more powerful, as by iteratively optimizing each layer they can better account for submodule interdependency.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear presentation, and interesting proposition of understanding which set of weights are more amenable to being merged. Novel idea of relaxing the original optimization with a linear approximation, which provides a fast and closed form solution. Good experimentation, and valuable ablation study on granularity of merging. The paper also shows impressive results with only 30 datapoints per task, making it highly efficient in terms of data and computational demands. It performs well across different scales like Llama-2-7B and Llama-2-13B, highlighting scalability for both small and large models. The approach is flexible, enabling merging at different granularities such as layer-level and attention/MLP-level, and delivers consistent performance across these structures.", "weaknesses": "The authors overlook referencing Mixture of Experts (MoE) merging approaches, such as in https://arxiv.org/abs/2402.00433, where merging also involves linearly weighting submodules. However, in traditional MoE, this weighting occurs at the activation level, rather than directly on model parameters. Since the authors allow for optimization of the weighting coefficients on the joint dataset, then the idea in this paper can be a useful baseline to run: https://arxiv.org/pdf/2410.13025. \n\nThat said, it’s unclear why the proposed method would surpass a router-based approach in performance, as a trained router can achieve task-specific specialization without approximations to the objective. While the linear interpolation method is efficient, a router's task-based dynamic routing could potentially allow finer-grained adjustments in response to specific inputs or task demands, leveraging the submodule structures without relying on assumptions of linearity. Adding comparisons or addressing these differences in the paper would clarify the potential advantages and trade-offs between the approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TiH3DiftFu", "reviewer": "Reviewer_F2K5", "summary": "This paper conducted a statistical analysis to examine the linearity of model-level and decomposed submodule-level components. The authors discovered that submodules (e.g., layers, self-attentions, and MLPs) demonstrate significantly higher linearity compared to the overall model. Based on this finding, they introduced a new model merging strategy that independently merges these submodules. Experimental results indicate that the proposed method surpasses the standard task arithmetic approach and other baselines across various model scales and tasks.", "questions": "Please see the Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is elegantly written and easy to comprehend. The motivation, proposed method, and experiments are all well elucidated. \n2. The authors focus on an important issue in model merging, and proposed a principled approach to address this issue.\n3. The authors offer detailed explanations of the proposed method and experimental outcomes in the appendix, which is highly commendable.", "weaknesses": "1. The observation that submodules exhibit higher linearity than the overall model may be deemed rather obvious. Since the overall model is constructed from these submodules, the nonlinearity of each submodule contributes to the overall model's nonlinearity. Consequently, the comparison results presented in Figure 1 and Figure 2 are unsurprising. The linearity of the model is fundamentally inferior to that of its submodules.\n\n2. Building upon the first comment, the proposed method could be interpreted as a straightforward extension of previous global linearization approaches. The authors introduce a layerwise linearization version, which has been explored in prior studies, such as Git Re-Basin [1], Zipit! [2], and MuDSC [3]. It would be beneficial if the authors could provide comparisons with these works.\n\n3. Furthermore, it is conceivable to decompose the model into more finely-grained components, such as different groups, channels, or even neurons. I posit that the linearity of these components is more pronounced than that of layer-level submodules. However, this would necessitate a greater number of merging parameters, similar to the method proposed in this paper. Could the authors engage in discussions regarding this aspect?\n\n4. The three properties outlined in Section 2 are somewhat redundant and could benefit from simplification.\n\n5. The performance improvement is not so impressive. As the proposed method adopts a layerwise merging strategy, it is expected to be much more performant than others. However, experiments demonstrate that the proposed method is still outperformed by prior methods. For example, coding & translate, math & coding in Table 1 and Table 2.\n\n\n### References\n[1] Git Re-Basin: Merging Models modulo Permutation Symmetries, ICLR 2023.\n\n[2] Zipit!: Merging Models from Different Tasks without Training. NeurIPS 2023.\n\n[3] Training-Free Pretrained Model Merging, CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAkn9IxWY4", "reviewer": "Reviewer_g3he", "summary": "The paper tackles task vector-based model merging, following a line of approaches that obtain a multi-task model by adding to a pretrained model the differences between fine-tuned models and their pretrained base.  In particular, the work builds on top of previous work relating the effectiveness of task vectors to a particular kind of linearity such that if a model is taken on the line connecting pretrained and fine-tuned, its output features should be on the line connecting the features of the pretrained and those of the fine-tuned. The treatment starts by defining a non-linearity score that measures how much this linearity property is violated, showing this score to be much lower for individual layers than for the whole model. The authors then leverage this finding and propose merging individual layers, deriving a closed form for the layer weights based on the linearity requirement.  Experiments show the approach to be slightly beneficial for two LLAMA-based architectures when merging a model with math, coding, and translating capabilities.", "questions": "- The requirement of a small set of data per task is reasonable in practice, but what if no data is available? Can you still do something?\n- Maybe add a few more compatible tasks so that merging more than three models reveals some interesting trends as the number of tasks increases. Maybe your framework works even better compared to the baselines when the number of tasks is raised? Or perhaps the advantage becomes less pronounced?\n- Minor suggestion: In Property 2, I would really avoid using footnotes over equations as these look like powers instead.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally clear and easy to read. The experiments make intuitive sense and are well explained, with enough details to be reproduced.\n- The layer-wise linearity perspective is novel. Overall, I find layer-wise approaches to better leverage the structure of the network when compared with standard vanilla task vectors. The proposed non-linearity score is novel, well-motivated, and intuitive.", "weaknesses": "- First and foremost, I am not convinced that the results warrant the added complexity: overall, the performance of the approach is in the same ballpark as vanilla task arithmetic while not inheriting its intuitiveness and simplicity. While it exhibits some slight improvements in a portion of the (limited) evaluation settings, these are not substantial nor consistent enough.\n- Still on the evaluation, the paper only considers two LLAMA-based architectures and only a subset of the evaluation settings considered by the previous works. For instance, Task Arithmetics also considers the GLUE benchmark and a suite of computer vision benchmarks, while DARE considers GLUE and OpenLLM. An exhaustive evaluation is necessary to rule out possibly lucky choices of models and datasets that do not guarantee the effectiveness of the approach in general.\n- The baselines are also somewhat limited, some possible works to consider are *e.g.* [1], [2] and [3].\n- The main finding of the paper, that is the fact that modules show more linearity than the whole model, obtained as a composition of all those intermediate modules, is not surprising. In fact, if the intermediate modules have non-zero non-linearity scores, the overall non-linearity score will grow as these ones accumulate.\n- Experiments have shown that merging three models exhibits a lower projection distance than merging just two, highlighting the potential of merging even more models. However, the experiments only merge up to three models. Can the trend be confirmed? More experiments would help clarify this aspect.\n\n[1] Yadav, Prateek, et al. \"Ties-merging: Resolving interference when merging models.\" *Advances in Neural Information Processing Systems* 36 (2024).\n\n[2] Davari, MohammadReza, and Eugene Belilovsky. \"Model breadcrumbs: Scaling multi-task model merging with sparse masks.\" ECCV 2024.\n\n[3] Wang, Ke, et al. \"Localizing Task Information for Improved Model Merging and Compression.\" *Forty-first International Conference on Machine Learning*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ypuscEoQKp", "reviewer": "Reviewer_KQV8", "summary": "This work is inspired by models demonstrating linearity that can improve the performance of task arithmetic, and hypothesize that linearity already exists within the model’s submodules. Linearity refers to the linear relationship between the differences in model weights and the differences in output features caused by fine-tuning. A model framework that independently merges these submodules is proposed to enhance task arithmetic and hence lead to superior multi-task models. The author justify their motivation by comparing the non-linearity scores between the full model and the submodules. Using submodules instead full models requires computation of the optimal merging weights after splitting the models into submodules that exhibit linear characteristics. Experiments are conducted on fine-tuned models based on Llama-2 and the results indicate superior merging performance with task arithmetic.", "questions": "1. Merging weights in head level appears to give unstable outcomes in Table 3, and the hypothesis here is that the representation functionalities can be averaged out. Is there any potential method to avoid that?\n\n2. The fine-tuned models are based on Llama-2, and the 7B model tends to show stronger non-linearity than the 13B model. Have you considered the influence of the varying depth of the model and data size, and the implications with repect to universal approximation theorem?\n\n3. Is there any family of activation function more suitable for your model framework than others (e.g. ReLU, GELU, Swish) and what is their impact on the linearity of submodules and overall merging performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper propses an interesting idea of decomposing the models into submodules that exhibit linear characteristics. The non-linearity score is proposed and merging the weights this way is novel. In particular, Figure 1 and Figure 3 clearly illustrate the motivation behind this paper and the weight merging is well-defined. The results and the ablation study support the author's claim well. The contribution of Layer level and Attention/MLP level weight merging is convincing.", "weaknesses": "One relevant topic concerning linearity is the non-linear activation function. This paper focused on the submodules and weight merging, but lacks a thorough discussion on activation functions. Figures show that the non-linearity score for all different submodeules are near zero. Why is that? Have you discarded all non-linear activation functions in submodules? If that is not the case, how do they affect the linearity assumptions and why do they not contribute to a larger non-linerarity score? Please include a discussion of activation functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)