Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Authors: Runxi Cheng
Feng Xiong
Yongxian Wei
Wanyun Zhu
Chun Yuan
Venue: ICML 2025 poster (2025)
Keywords: Model Merging
Data-Free
Multi-task Learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: Guiding Data-Free Model Merging via Task Vectors

Abstract: Model merging seeks to integrate task-specific expert models into a unified architecture while preserving multi-task generalization capabilities, yet parameter interference between constituent models frequently induces performance degradation. Although prior work has explored many merging strategies, resolving interference without additional data for retraining or test-time computation remains challenging. In this paper, we theoretically demonstrate that the task vectors of the linear layer constitute an approximate linear subspace for its corresponding input. Therefore, we can minimize interference under the guidance of task vectors. Based on this insight, we propose **WUDI-Merging** (**W**hoever started the interference sho**U**ld en**D** **I**t), a simple yet effective model merging method that eliminates interference without any additional data or rescaling coefficients. Comprehensive empirical evaluations across vision and language benchmarks demonstrate our method's superiority, achieving state-of-the-art performance in data-free model merging scenarios (average 10.9% improvement versus baseline methods)  while even outperforming mainstream test-time adaptation approaches by 3.3%, and only very few computing resources are required. The source code and implementation details are available at https://github.com/nathanielyvo/WUDI-Merging.

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Runxi Cheng Feng Xiong? 2
Yongxian Wei!
Wanyun Zhu
Chun Yuan
Abstract
1. Introduction
With the widespread adoption of the pre-training and fine- tuning paradigm, large number of pre-trained and fine- tuned checkpoints have been released in open-source com- munities. However, directly applying multiple individual models for multi-task problems incurs significant storage costs. While multi-task learning has been employed to ad- dress this issue, typically requires costly training and poses potential privacy risks. Recently, model merging has emerged as a solution, enabling the integration of multiple expert models into single unified multi task model without the need for expensive retraining multi-task datasets.
Model merging seeks to integrate task-specific expert models into a unified architecture while preserving multi-task generalization capabilities, yet parameter interference between constituent models frequently induces performance degrada- tion. Although prior work has explored many merging strategies, resolving interference without additional data for retraining or test-time com- putation remains challenging. In this paper, we theoretically demonstrate that the task vectors of the linear layer constitute an approximate linear subspace for its corresponding input. Therefore, we can minimize interference under the guidance of task vectors. Based on this insight, we pro- pose WUDI-Merging (Whoever started the in- terference shoUld enD It), simple yet effective model merging method that eliminates interfer- ence without any additional data or rescaling co- efficients. Comprehensive empirical evaluations across vision and language benchmarks demon- strate method's superiority, achieving state- th-the-art performance in data-free model merg- ing scenarios (average 10.9% improvement ver- sus bascline methods) while even outperforming mainstream test-time adaptation approaches by 3.3%, and only very few computing resources are required. The source code and implementation de- tails are available https: / / gitgtuutbb tubb. com/ www.hhnielwwo/wuuI---
However, due to interference among the expert models, the merged model exhibits a performance gap when compared to its corresponding expert models on specific tasks. Nu- merous model merging approaches have been proposed to address this issue. Test-time adaptation model merging methods, such as AdaMerging (Yang et al., 2024b) and Surgery (Yang et al., 2024a), leverage unlabeled test data to resolve interference through reweighting or model editing. MoE-like (Mixture-of-Experts model merging methods, ex- emplified by EMR-Merging (Huang et al., 2024), retain additional task sasc-ipe parameters, such as task task-specific masks, to prevent interference when integrating diverse ex- pert models into unified model. While these approaches ef- fectively mitigate the interference, they require extra storage or access to test data. Conversely, data-free model merging methods, like Task Arithmetic (Ilharco et al., 2023), enable model merging without additional data or storage require- ments, which is particularly advantageous when data privacy or availability is a concern. However, due to the absence of data, current data-free model merging techniques still exhibit performance gap compared to the aforementioned methods and multi-task learning.
Equual contribution 'Tsinghua Shenzhen International Graduate School, Tsinghua University "Guangdong Provin- cial Key Laboratory of Novel Security Intelligence Tech- Runxi
In this paper, we revisit the update process of fine-tuning phase. Consider linear layer whose parameters are updated through gradient descent: each neuron's 's weight adjustment derives from the product of the learning rate, the gradient of its output, and the corresponding input vector. Crucially, since the learning rate and total optimization steps always remain constrained during standard fine-tuning procedures, the corresponding inputs for individual samples remain con-
nologies 'CUHK-SSenzher" Cheng <crx23 mails. maillstsingguu.cdu .singhua.cdu.cn>, Chun Yuan Duuanc@sz.tsinghuue edd.
Correspondence
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Task Vectors Ty
Corresponding Input
Coeffieient
introduced Fisher Merging, performing weighted model merging by utilizing the Fisher information matrix to as- sess the importance coefficient of expert models. Jin et al. (2023) addressed model merging by minimizing prediction differences between the merged model and the expert mod- els. Recently, Ilharco et al. (2023) demonstrated that we can effectively edit models by simply applying arithmetic operations to task vectors. However, the interference among expert models remains significant challenge. Yadav et al. (2023) proposed Ties-Merging to resolve this challenge by three novel steps. Yu et al. (2024) introduced DARE, which resolves the merging interference by dropping parameters and unscaling operations. Wang et al. (2024) introduced Consensus Merging, which enhanced the overall perfor- mance of existing model merging methods by removing the selfish and catastrophic weights. Furthermore, Xiong et al. (2024); Wei et al. (2025) proposed to align the loss be- tween the merged model and expert models. Although these methods alleviate interference between experts to some ex- tent, a gap still exists between data-free model merging and test-time adaptation model merging methods.
ai
Approximate Lincar Subspace The Corresponding Input
Xn
Figure 1:
The task vector constitude approximate linear subspace of its corresponding input.
sistent across successive update iterations. The temporal consistency indicated that the cumulative update to each neuron can be approximated as weighted summation of fixed imput vectors, with coefficients determined by the prod- uct of learning rates and corresponding gradient magnitudes. This implies that the task vectors of a linear layer constitute an approximate linear subspace of the input, allowing us to implicitly utilize training data information solely through the task vectors. In this context, the task vectors are com- puted by calculating the difference between the weights of the expert models and those of the pre-trained model.
Test-time adapta- tion model merging leverages a portion of unlabeled test data to resolve interference among expert models. Yang et al. (2024b) proposed AdaMerging, which utilized entropy min- imization on unlabeled test samples as a heuristic objective function to learn the merging coefficients. In addition, Yang et al. (2024a) introduced Representation Surgery, which reduces representation bias by minimizing the distance be- tween the representations of the merged model and those of individual models. Despite these methods are promising, critical limitation of these methods lies in their reliance on access to test data, which may not be feasible in practice.
Test-time adaptation Model Merging:
Based on this insight, we propose WUDI-Merging (Whoever started the interference shoUd enD It), a simple yet effective data-free model merging method. We evaluate our method both vision and natural language processing tasks, where our method significantly surpasses recent data- free model merging techniques and outperforms mainstream test-time adaptation model merging methods. In Summary, our contributions are summarized as follows:
Through detailed theoretical analysis, we have demon- strated that the task vectors within the linear layer ap- proximately form a linear subspace of the input space.
MoE-like Model Merging: MoE-like model merging has emerged as a promising paradigm for leveraging multiple specialized models while storing task-specific knowledge in multi-task learning scenarios. Huang et al. (2024) proposed EMR- Meeg-nggging, which elect a unified model and uses task- specific masks for multi-task learning problems. Lu et al. (2024) introduced Twin-Merging, incorporating a router to dynamically merge shared and exclusive knowledge based on the test inputs. While these methods effectively mitigate interference issues in model merging, they present certain practical limitations. The requirement for storing multiple task-specific components increases memory overhead.
We propose WUDI-Merging, a simple yet effective data-free model merging method that minimizes inter- ference to the task vector without requiring any addi- tional data, extra storage, or rescaling coefficients.
Extensive evaluations have demonstrated that WUDI- Merging achieves bate-of-the-at performance in data- free model merging, surpassing mainstream test-time adaptation model merging methods.
2. Related Work
Although test-time adaptation and MoE-like methods have achieved remarkable results in resolving interference among expert models, their practical applicability is limited. This limitation arises from challenges such as data privacy con- cerns, the need for extra storage, and the lack of parallelism in unified models. Therefore, in this paper, we primarily focus on the data-free model merging.
Current research primarily falls into three categories: Data- free, Test-time adaption, and MoB-like methods.
Data-free model merging aims to combine different expert models without any additional data for retraining. Wortsman et al. (2022) proposed Simple Averaging, which constructs the merged model by averaging the parameters across all models. Matena & Raffel (2022)
Data-Free Model Merging:
2

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
RESISO15
84
03
44
Depdi Layer
AMagnitule
AMoynitok
AMggniuue
64
ADirection SVINN
ADievtion GTSRB
ADircction MNIST
ADirevtion
DTD
1.2
AMaggindd
AMagnitude
Depbh of Laaeer
64
ADDreccton
ADirection
Airrectio
ADirection
Figure 2: The input consistency between the pretrained model and the fine-tuned model for ViT-B/32
3. Methodology
Interference of the Linear Layer. Let Oi,, Om. denote the parameters for the linear layer of expert model for task i and merged model, respectively. Similarly, let Ts and Tm,, represent the corresponding task vectors associated with o,, and fm.... Additionally, we define corresponding input as æ4,1 Then, we can formally define the interference of the I-th linear linear for task as follows (Jin et al., 2023; Fang et al., 2024; Zeng et al., 2019):
3.1. Preliminary of Model Merging
Notation.
Formally, let 0 denote the parameters of the pretrained model, and let 0, represent the parameters of the expert model for task i, fine-tuned from 0. Following Task Arithmetic (Ilharco et al., 2023), we define the task vector т; for task as the difference between the expert model and the pretrained model:
J((Tm.) E (pp(; 1 m, 0 0,,
(3)
0:
Ea ()
T T
(4)
We then define the merged task vector Ti, and the final parameters of the merged model as O.. = 0 Ti
= Em 10,111
(5)
The impact of the Linear Layer's Task Vector. As demon- strated by Jin et al. (2023); Xiong et al. (2024), the task vector in the linear layer encapsulates most of the capa- bilities of the expert models. As shown in Figure A.3, expert model utilizing only the task vector of the linear layer achieves performance comparable to that of the full expert model. Therefore, we primarily focus on the linear layers of the model.
3.2. Towards Understanding the Task Vector
In practical fine-tuning scenarios, the process typically em- ploys small learning rate and executed over limited number of iterations and we assume the model Lipschitz continuous (Kim et al., 2021; Latorre al., 2020; Fazlyab et al., 2019). Based these, propose Lemma 1 to analyze the consistency of the input for the linear layer I.
Lemma 1 (Input consistency). Let xt æt denote the input of the linear layer after and (p >> iterations during fine-tuning, respectively. Denote vi as the learning rate of iteration Assume that the model before layer is Ct- Lipschitz continuous, and the gradient of loss bounded in zz-norm by Gr. then:
Interference Vector. Empirical observation (Ilharco et al., 2023) reveals that task vectors for a certain pretrained model and task exhibit consistent directional characteristics within the parameter manifold. This directional consistency sug- gests that task vectors converge to local optima. Therefore, for task i, we defined the rest of the weight in Tm as inter- ference vector 5,, which is formulated as:
|x xf112 S CG9
(6)
6, = T Ti
The proof is in Appendix A.I.I.
3

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
According Lemma 1, the input of individual sample to linear layer remain highly consistent across different iterations when both the learning rate and the number of iterations are small. We further validate this conclusion through an experimental study. Specifically, we compute the input consistency between the pre-trained model and the fine-tuned model, where the input of the pre-trained model can be seen as =f and the input of the expert model can be seen the ua-
an approximate linear subspace of its associated inputs. By leveraging this property, we can exploit the training data information solely through the task vectors.
3.3. Methodology
In this section, we introduce WUDI-Merging (Whoever started the interference shoUd enD It), simple yet effective Data-Free Model Merging method that operates without the need for any additional data or rescaling coefficients. Building upon the insights from Section 3.2, we observe that the task vector constitutes an approximate linear subspace for its corresponding inputs. This property motivated us to reconstruct the inputs by the task vectors:
ADirection 1 cos (expert Fpretrain)
AMagnitude [exprrr 2 preemain 2
(8)
preetrain
Where the Texpert and Teretrain represent the input of the ex- pert model and the pretrained model, respectively. The result is shown in Fig. 2. Most layers exhibit high consistency, with direction changes less than 0.4, indicating that the in- puts remains highly consistency between the pretrained and fine-tuned models. With this observation, we reconsider the update process of the task vector, which can be formulated as follows:
Ti,i -k (Lu())) + +e(a,)) Ti. E Di,
(13)
Where af,, (a:.,) are the reconstruction coefficients of ai,t and exx,,) represents the reconstruction error. Then, we can reformulate the interference of the linear layer / as follows:
IT(Tm,)) E,,, 16 (===1 of (T.,)
OC(0'-1)
+ (14)
1= 17,-
(9)
T 1=1 1/} Σ n=
80(0'-1)
a(0
(10) Theorem (Upper bound of interference). Denote æi,
Based on these discussions, we propose Theorem 1:
T
N Σ
8C(0--1
11
(11) as the linear layer I's input of task i, 5ū, and Tı, as the linear layer I's interference vector and task vector of task i, respectively, then:
t1
1/t
n=1
00
cnefficient
where + denote the task vector of neuron & in linear layer 1, and 01,- denote the parameters of neuron in linear layer 1 time t. According to Eq. 11, each neuron in the linear layer can be interpreted a weighted sum of the inputs across different iterations. Given that the inputs remain highly consistent across iterations, as previously established, we propose Proposition 1:
110,,, S wh, 1(0,,(TT))
+ + 116.,113 (15)
Where wh, and w², is the reconstruction constant. The proof is in Appendix A.1.3.
According to Theorem we can minimize the upper bound of interference for task by:
Proposition 1 (Approximate Linear Combination). Let of denote the task vector of neuron k in linear layer 1. Con- sider N impul samples {z let dh,, denote the corresponding inpul to layer 1 after iteration for sample on during fine-tuning. Assume the gradient of the loss with respect to the product Tia is bounded by Dt. Then, the following inequality holds:
min wh, (I....(T...) TIP + wf,, 118 wh, 11....1? wh,
(16)
min [[ 82
(17)
However, the reconstruction constant is related to the actual input, which is inaccessible. As substitute, we can use an empirical coefficient w. Consequently, we consider the following optimization objective:
7 - -M 8k (xT
Φ. 1EE Ei neTl:)
(12)
where T = N.C. GT The proof is in Appendix A.1.2.
T and Bh,
ΣE=
1tt
OC
min 8..
(T.),((T.)) + w 115,,
(18)
Consequently, each neuron in the linear layer can be approx- imated as a weighted sum of its corresponding inputs. That to say, for linear layer the task vector Tú, constitutes
Given that task vectors of different scales can tolerate vary- ing degrees of deviation, we adjust our optimization accord- ingly. Specifically, we weight the loss of each task based

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
on the scale of its task vector. By weighting each loss with the square of the F-norm, we balance the interference of different tasks. Therefore, the final objective function can be formulated as:
Algorithm 1 WUDI-Merging
1: Input: pretrained model parameters O; task vectors
T = { solution steps N; learning rate Ç.
2: Output: merged multi-task model parameters 9m-
min Ee Ielll (1 w. ((uul()))
3: & Initialize merged task vector: T
TT. Ti,--
(6.,}
4: for linear layer I E {1..... . , do
min Tmdd
E, b ([((m - TT, + t.. -/ 1()) (19)
5: for n E {1..... [[] do
Calculate loss by merged task vector and task vectors:
7:
Li --=E, Ei | [[ (TT)TT
Then, get the close-form solution for Eq. 19,
8:
D. Update the merged task vector mm,
Tm, (2 ITall (T. + - (20)
? Teu(TT + wf),
9: m. m, 5 V Tmm L1 (Tm))T1))
10: end for
11: end for
The detailed derivation is in Appendix A.1.4. However, selecting general regularization coefficient suitable for different tasks is challenging. To address this issue, we propose an alternative approach. Revisiting Eq.18, which can be seen as form of ridge regression, and motivated by prior work (Smith et al., 2021; Wang al., 2022; Zou al., 2021) which have demonstrated that gradient descent methods such as SGD and Adam induce implicit regulariza- tion and shown better generalization in such problems, we propose to solve the problem by using the gradient descent. Therefore, we can eliminate the regularization term, which allows us to avoid the need to search for an appropriate co- efficient. Accordingly, the loss function for gradient descent formulated as follows:
12: & Assemble the merged task vectors from all linear
layers:
13: Tum = m1
14: & Calculate the Merged multi-task model parameters:
15: 0m = 0 +
We primarily compared WUDI-Merging with recent data-free model merging methods, including Weight Averaging (Wortsman et al.. 2022), Fisher Merging (Matena & Raffel, 2022), RegMean (Jin et al., 2023), Task Arith- metic (Ilharco et al., 2023), Ties-Merging (Yadav et al., 2023), Consensus Merging (Wang et al., 2024), and PCB Merging (Du al., 2024). Additionally, we com- pared WUDI-Merging with mainstream test-time adapta- tion methods, namely Adamerging (Yang et al., 2024b) and Surgery (Yang et al., 2024a).
Baselines:
(21) Since the optimization of each linear layer is independent, we can sequentially solve the problem layer by layer, thereby avoiding significant computational overhead. The algorith- mic flow is detailed in Algorithm 1.
1 [(Tm.I-TW(TT))) C = Σ. I
Experiment Details. Our method only uses two hyperpa- rameters. In practice, we used Adam optimizer and set the learning rate to Ic-5, The number of iterations is set 300, Following (Jin al., 2023; Xiong et al., 2024), we only applied our method to the linear layer in the model. We provide more details in A.8
4. Experiment
4.1. Experimental Settings
Datasets. For vision tasks, following (Yang al., 2024b; Wang al., 2024), we investigate multi-task model merging across eight image classification datasets. For discriminative language tasks, we employ the GLUE (Wang et al., 2018) to assess our method. For generative language tasks, follow- ing Yu et al. (2024), we evaluate our method on instruction- following, mathematical reasoning and code-generation.
4.2. Main Results
Results of Visual Tasks. The main experimental results of vision tasks are presented in Tables 1, 2, and 10. These findings illustrate that a performance gap generally persists between Test-Time Adaptation (TTA) methods and current data-free methods. Compared to the aforementioned ap- proaches, our proposed method notably outperforms all existing methods without utilizing additional data or re- quiring extra storage. Specifically, on the ViT-B/32 model, WUDI-Merging achieves 8.9% performance improvement over the sota data-free method PCB- Mer-gngg and surpasses the leading TTA method AdaMerging++ by 4.1% For the VIT-L/14 model, WUDI-Merging exceeds PCB- Mer-ing by 5.1 1% and outperforms AdaMerging++ by 1.6%. Notably,
Models. For vision tasks, we employ the ViTs (Dosovit- skiy et al., 2020) derived from CLIP (Radford al., 2021), including VIT-B/32, ViT-B/16 and VIT-L/14. For discrimi- native language tasks, we employ the RoBERTa-Base (Liu et al., 2019) and RoBERTa-Large for evaluation. For gener- ative language tasks, we employ the Llama2 (Touvron et al., 2023) for evaluation.
5

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Table 1: Multi-task performance when merging ViT-B/32 models on 8-task vision benchmark.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Non-merging Methods
Pretrained
62.3
59.7
60.7
45.5
31.4
32.6
48.5
43.8
48.0
Individual
79.2
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.8
Traditional MTL
73.9
74.4
93.9
98.2
95.8
98.9
99.5
77.9
88.9
Test-time Adaptation Methods
AdaMerging
64.5
68.1
79.2
93.8
87.0
91.9
97.5
59.1
80.1
AdaMerging++
68.3
82.2
94.2
89.6
98.3
Representation Surgery
63.8
59.9
83.3
97.9
87.0
87.0
98.6
69.4
80.9
Data-ffree Methods
Weight Averaging
65.3
63.4
71.4
71.7
64.2
52.8
87.5
50.1
65.8
Fisher Merging
68.6
70.7
72.9
51.1
87.9
RegMean
65.3
63.5
75.6
78.6
78.1
67.4
93.7
52.0
71.8
Task Arithmetic
55.2
54.9
66.7
78.9
80.2
69.7
97.3
50.4
69.1
Ties-Merging
59.8
58.6
70.7
79.7
86.2
72.1
98.3
54.2
72.4
Consensus Merging
65.7
63.6
76.5
77.2
81.7
70.3
97.0
57.1
73.6
PCB Merging
66.7
65.5
78.5
79.3
86.4
77.1
98.2
59.1
76.3
WUDI-Merging (Ours)
71.1
71.0
85.7
95.6
94.2
94.7
99.5
69.7
85.2
Table Multi-task performance when merging ViT-L/14 models 8-task vision benchmark.
Method
SUN397
Cars
RESISC45
EuroSAT SVHN
GTSRB
MNIST
DYTD
Avg Acc
Non-merging Methods
Pretrained
66.8
77.7
71.0
59.9
58.4
50.5
76.3
55.3
64.5
Individual
82.3
92.4
97.4
100
98.1
99.2
99.7
84.1
94.2
Traditional MTL
80.8
90.6
96.3
96.3
97.6
99.1
99.6
84.4
93.5
Test-time Adaptation Methods
AdaMerging
79.0
90.3
90.8
96.2
93.4
98.0
99.0
90.8
AdaMerging++
79.4
90.3
91.6
97.4
93.4
97.5
99.0
79.2
91.0
Representation Surgery
75.7
84.4
93.1
98.8
91.3
93.4
99.1
76.1
89.0
Date-free Methods
Weight Averaging
81.6
82.6
91.9
78.2
70.7
97.1
62.8
Fisher Merging
69.2
88.6
87.5
93.5
80.6
74.8
93.3
70.0
82.2
RegMean
73.3
81.8
86.1
97.0
88.0
84.2
98.5
60.8
83.7
Task Arithmetic
73.9
82.1
86.6
94.1
87.9
86.7
98.9
65.6
84.5
Ties-Merging
76.5
85.0
89.3
95.7
90.3
83.3
99.0
68.8
86.0
Consensus Merging
84.3
95.6
88.3
PCB Merging
76.8
86.2
89.4
96.5
88.3
91.0
98.6
73.6
87.5
WUDI-Merging (Ours)
81.0
91.0
94.2
99.2
96.3
98.1
99.6
81.2
92.6
our method lags behind supervised multi-task learning by only 4.7% and 0.9% on the ViT-B/32 and ViT-L/14 models. Results of Language Tasks. Table 5 summarizes the results for discriminative language tasks. In comparison to prior studies, WUDI-Merging demonstrates substantial perfor- mance improvements. Specifically, when evaluated against the advanced data-free approach Ties-Merging, WUDI- Merging achieved 19.7% improvement on the RoBERTa- Base model and 16.0% improvement on the RoBERTa- Large model. These findings strongly indicate the robust generalization capabilities of WUDI-Merging in language tasks. Table presents the results for generative language tasks. We observed 4.3% improvement over Ties-Merging
(with DARE). However, regarding the results for code tasks, our performance remained at an average level without achieving significant advantage. Our analysis suggests that this may be attributed to substantial confficts between the math and code tasks, leading considerable decline in one task's 's performance when the other task excelled. This phenomenon is also reflected in the results from alternative methods. Nevertheless, our overall average score remains significantly ahead of the current state-of-the-art
Results of Merging LoRA Fine-tuned Models. To further demonstrate the generalizability of our method LoRA fine-tuned models, we supplemented the experiments on
6

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Table 3: Performance of merging decoder-based WizardLM-13B (LM), WizardMath-13B (Math), and 1lama-2-13b codealpaca (Code) on all the datasets, we reported average normalized score.
Method
AlpacaEval
GSM8K
MATH
HumanEval
MBPP
Avg.
FT
100.0
100.0
100.0
100.0
100.0
100.0
Task Arithmetic
102.7
91.0
70.5
50.0
87.7
80.4
TIES MEE-Megginng
98.1
97.4
68.1
60.0
89.4
82.6
Task Arithmetic DARE)
103.1
88.0
72.5
63.3
92.9
84.0
MMrging DARE)
107.9
80.0
92.4
87.2
WUDI-Merging (Ours)
105.5
105.9
103.3
58.3
84.7
91.5,4.4
Table Experimental results merging Flan-T5- base (LoRA fine-tuned) models on all eight tasks.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
Individual
69.1
82.7
85.5
90.9
84.4
92.9
87.4
84.6
Ties- Ties-Merging
68.3
56.3
79.4
89.8
83.7
79.4
91.6
71.2
77.5
AdaMerging++
69.1
60.3
78.4
90.0
83.6
79.1
91.6
74.1
78.3
WUDI-Merging (Ours)
68.6
79.0
77.7
87.2
83.1
75.8
93.2
85.0
81.2
Table Multi-task performance when merging RoBERTa models 8-task GLUE benchmark. Following Lu et al. (2024), reported average normalized score.
this arises from the random initialization of LoRA, which causes the task vector to accumulate gradients from nonzero and thereby introduces small amount noise. This noise degrades the task vector's ability reconstruct the corre- sponding input, leading to the observed performance gap. Nevertheless, when evaluated under the LoRA-finetume model, WUDI still attains SOTA performance.
Method
RaBsRTa-Base
RoBERTa-Large
Pretrained
41.7
38.2
Individual
100.0
100.0
Weight Averaging
52.6
53.3
Task Arithmetic
67.8
70.9
4.3. Ablation Study
Ties-Merging
64.7
72.4
Balanced Weight. In Eq. 19, introduce dynamic weighting mechanism designed to balance the loss contri- butions across multiple experts. The experimental results presented in Table demonstrate that the application of bal- anced weights significantly enhances generalization perfor- mance. This improvement further suggests that task vectors with larger magnitudes exhibit greater resilience to higher levels of interference.
Task Arithmetic (w/ DARE)
63.7
70.9
Ties-Merging (w/ DARE)
65.6
WUDI-Merging (Ours)
85.7
18.8
Table 6: Experimental results merging Qwen-14B (LoRA fine-tuned) models on all four tasks.
Method
MMLU TruthfulQA BBQ CNN
Avg.
Stabilization Analysis on Solution Steps. As shown in Fig. 4, we evaluate our method across range of solution steps. The experimental results indicate a consistent in- crease in accuracy across all datasets and models over it- erations, with convergence typically achieved within 100 to 200 iterations. These results underscore the generaliza- tion ability of method and further highlight that only limited number of steps are required to reach a solution, thereby demonstrating the efficiency of our approach.
Individual
68.35
53.34
93.53 19.46
58.67
Task Arithmetic
67.56
52.33
78.38
20.54
54,70
Ties Merging (w/ DARE)
69.38
52.03
81.06
15,91
54.62
WUDI-Merging (Ours)
69.17
55.71
80,56
17.33
25.69..0.90
Flan-T5-base and Qwen-14 14. For merging LoRA, We first re- store LoRA matrix back into the task vector (T; = B;4:). then apply WUDI-Merging directly to т, to obtain To. and merging it into O. The experimental results obtained from merging Flan-T5-base (LoRA fine-tuned) models and Qwen- 14B (LoRA fine-tuned) shown in the Table 4 and Table 6. We observed 0.99% improvement over Ties Ties-Merging (with DARE) in Qwen-148 14B models, and 2.99% improvement over Ada.erging+++ in Flan-T5 models. Experimental results indicate that WUDI does not yield as large an improvement as observed in conventional settings. We hypothesize that
Stabilization Analysis on Task Numbers. To further in- vestigate the impact of task number on the performance of WUDI-Merging, we conducted experiments by varying the number of tasks. Following the methodology outlined in (Xiong al., 2024; Wei et al., 2025), sampled eight random subsets from the complete task set and calculated the average accuracy of the merged model across these sub-

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Interference of Each Layer(V)T-B/322
Interference of Each Layer(ViT-B/16)
Interference Each Layer
0.5
Wirlie Talk Artheste
WITDI Marghs Teak Antheek
64
Thik Artkmeete
0.4
Relariee Ertor
Releivve
Reletive
02
Lover
Agyel
Layer
Love'
Layer
Loved
Love?
Layer
Lower
Laye to Layer
Love!
Loven
Laye
Lowe?
Laye
Laye
Lowe 10
Laye
Layers
Layers
Layers
Figure 3: The interference of Task Arithmetic and WUDI-Merging for different layers.
VITE3077
Differee Neeioes ofTiis
Table 7: The impact of applying balanced weighting to the loss function. We report the performance when merging RoBERTa models on 8-task GLUE benchmark.
60
Method
RoBERTa-Base
RoBERTa-Large
Dandal
35
Task Arithmetic
67.8
70.9
70
Ours (wo Balance)
77.4
82.4
2001
710
Ours (w/ Bulance)
85.3
88.8
Figure 4: Figure (a) presents the results obtained at different stages of the solution process; Figure (b) presents the result of varying the number of tasks.
sets. As shown in Fig. 4, all methods exhibit decline in average accuracy as the number of tasks increases, indi- cating heightened conflicts and interference among expert models. Baseline methods such as Task Arithmetic and Ties-Merging experience more pronounced performance drops, while WUDI-Merging consistently maintains higher accuracy the number of tasks increases. This finding highlights the exceptional robustness of WUDI-Merging and its effective mitigation of task interference in multi-task learning scenarios.
of Differest Repulatization Coeffliiens
of Different Merping Coeflisient
IAS
Ied 3-3 Ind
361
12
1.
4.4. Further Analysis
(a)
(b)
Comparison between Adam Optimization and Closed- Form Solution. While WUDI-Merging requires very few GPU resources for optimization, we also present a closed- form solution for scenarios where GPU resources are not available. Specifically, beginning from Eq. 19, we derive the closed-form expression outlined in Eq. 20, which re- fer to WUDI-Merging- CFS. Furthermore, we investigate the influence of the regularization term on the closed-form solution. As shown in Fig. 5, WUDI-Mergine demonstrates greater stability by utilizing implicit optimization through the Adam optimizer. In contrast, WUDI-Merging-CER CCF is sensitive to the choice of regularization coefficient; more- over, its overall performance typically inferior to that achieved with the Adam optimizer. Therefore, suggest that employing gradient descent via Adam optimization is more effective strategy for solving Eq. 19. This assertion is further corroborated by previous studies (Smith et al., 2021; Wang et al., 2022; Zou et al., 2021).
Figure 5: Figure (a) displays the results obtained using the closed-form solution with varying regularization coef- ficients; Figure (b) shows the results for different merging
coefficients applied during the merging process.
evaluated WUDD-Merging'' 's performance across range of scaling coefficients to analyze the effect of the rescaling coefficient on WUDI-Merging. The rescaling coefficient is defined as On 0 + t T
Where e is the rescaling coefficient. As shown in Fig. 5, WUDI-Merging achieves optimal performance for the mod- els ViT-B/32, ViT-B/16, and ViT-L/14 when e approx- imately 1. The results suggest that method did not necessitate the tuning of the rescaling coefficient. In other words, WUDI-Merging is rescaling-free.
The Interference of Each Layer. The evaluation of layer- wise interference is presented in Fig. 3. The results indicate that interference accumulates progressively as the depth of
Analysis on Rescaling Coefficient, In this section, we
8

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
ViT-B/32
VIT-B/16
VIT-L/14
42
10
Accraay
Accurscy
Lina Fakepice Talk Vectes Rander Liwa Sahpar
1ime Salapace Vedte
Linu Sulquee of Vestes
Render Live Sehapac
Ronden Liner Sulgee
65
-0
60
63
55
601 0.2
0.1
02
0.1
0.1 Ratio
0.1
0.85
Ratio
Ratio
Figure 6: Performance comparison when utilizing subset of task vectors or random vectors for optimizing the loss function.
Table 8: Computational time and gpu memory requirements.
The corresponding loss is computed as:
Model
Solving steps
Solving Time
GPU Memory
Sub
(Tm Ti) (T!Th) T
(25)
ViT-B/32
300
1min54s 8min23s
3.03GB
ViT-I/144
300
4.02GB
The experimental results are presented in Fig. The find- ings indicate that the random linear subspace yielded poor performance. Conversely, we observed that utilizing complete task vector consistently led to improved results. Notably, when the reconstruction was performed using the linear subspace formed by the complete task vector, we achieved the best performance. These results underscore the necessity of employing the entire task vector for effective reconstruction.
Table 9: Comparison of different merging methods in terms of accuracy, time, and GPU memory usage under the setting of ViT-B/32.
Method
Accuracy
Time
GPU Memory
Ties Merging
72.4
4s
0GB
Adamerging
81.1
127min
17.1GB
WUDI-Merging-CTF (CPU)
84.4
Ss
0GB
Computation Resource. To assess the computational ef- ficiency of our method, we measured the solving time and GPU memory usage on the ViT-B/32 and ViT-L/14 archi- tectures. The number of solving steps was fixed at 300, and the Adam optimizer was employed with the loss optimized sequentially, layer by layer. shown in Table 8, executing our method the ViT-B/32 architecture requires approx- imately 2 minutes and 3 GB of GPU memory. Even for the larger ViT-L/14 architecture model, it only necessitates about 8 minutes and 4 GB of GPU memory. These results indicate that our method is highly computationally efficient.
WUDI-Merging-CFS (GPU)
84.4
2s
1,8GB
WUDI-Merging
85.2
1min54s
4.0GB
the network increases. Specifically, task arithmetic intro- duces a relative error of nearly 50% in the final output. In contrast, our proposed method significantly mitigates this in- terference compared to task arithmetic and effectively slows the rate of interference accumulation.
Selection of Linear Subspace. Reconsidering Eq. 13, we investigate whether the entire task vector linear subspace is optimal for reconstruction in data-free scenario. To verify this point, we employed both random vectors and subset of the task vectors to optimize the loss. For the random vectors, we sample them from Gaussian distribution where the mean and standard deviation are computed from the original task vectors:
5. Conclusion
In this paper, we demonstrate that the task vectors constitute approximate linear subspace of the corresponding input for the linear layer. Therefore, we can minimize the interfer- ence among expert models guided by the task vector. Based on the insight, we propose WUDI-Merging, a simple yet effective model merging method that resolves interferences without requiring any additional data or rescaling coeffi- cients. Extensive empirical evaluations demonstrate the ef- fectiveness of our method. We believe that this work is one step toward simple and penerra-purpose data-free model merging technique. Further research may consider a more detailed way to resolve the conflict based on our method, which has the potential to reach the performance of multi- (24) task learning the data-free model merging paradigm.
random ~ N(u,,
(22)
where #4 mean(T.) o, std(T.). The loss given by:
Crandom
Σ T (Tm Ti) (random))
(23)
For the subset of task vectors, we sample random subvector from the original task vector as follows:
sub = T (rand index, :]
T

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Acknowledgments
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv: 2404.4455.. 2024.
This work supported by the National Key R&D Pro- gram of China (202XYBB47010044701004)) SSTIC Grant IKJ2D20230093111100001 KJZDD202309211149 16032,,,, GJHZ20240218811660000000 Beijing Key Lab of Networked Multimedia.
Fang, J., Jiang, H., Wang, K., Ma, Y., Wang, X., He, X., and Chua, T.-s. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprini arXiv:2410.02355, 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Fazlyab, M., Robey, A., Hassani, H., Morari, M., and Pap- pas, G. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in neural information processing systems, 32, 2019.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266, 2019.
References
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski,
H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., al.
Program synthesis with large language models. arXiv 2021.
preprint arXiv:2108.077
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv 2021.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P.D.O. Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et Evaluating large language models trained on code. arXiv preprint 1IXv:22077.03374 2021.
preprint arXiv:2103.00::
Huang, C., Ye, P., Chen, T., He, T., Yue, X., and Ouyang, W. Emr-merging: Tuning- free high- gegh-performance model merging, arXiv preprint arXiv:2405. 17461, 2024.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10)11865-1888 2017.
2017.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. Proceedings of the Eleventh International Conference on Learning Representations, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint arXiv: 20.1::2110.111. 14168, 2021.
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights language models. In Proceedings of the Fleventh International Conference on Learning Representations, 2023.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene un- derstanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Kim, H., Papamakarios, G., and Mnih, A. The lipschitz constant of self-attention. In International Conference Machine Learning, pp- 5562-5571, PMIR, 2021.
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3d object rep- resentations for fine-grained categorization. In Proceed- ings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. 2020.
Latorre, F., Rolland, P., and Cevher, V. Lipschitz constant estimation of neural networks via sparse polynomial opti- mization. arXiv preprint arXiv:2004.0866 2020.
Du. G., Lee, J., Li. J., Jiang, R., Guo, Y., Yu. S., Liu, H., Goh, S. K., Tang, H.-K., He, D., et al. Parameter competition balancing for model merging. arXiv preprint arXiv:2410.0239% 2024.
LeCun, Y., Bottou, L.,, Bengio, Y., and Haffner, Gradient- based learning applied to document recognition. Proceed- ings of the IEFE, 86(1 11:2278-2321 1998.
10

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. 2019.
Wei, Y., Tang, A., Shen, L.,, Xiong, F., Yuan, C., and Cao, X. Modeling multi-task model merging as adaptive projec- tive gradient descent. arXiv preprint Siiv:2501.012300 2025.
Lu, 7.,, Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin-merging: Dynamic integration of modular expertise in model merging. 2024.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998. PMLR, 17-23 Jul 2022.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. In Proceedings of the International Conference on Neural Information Processing Systems, volume 35, pp. 17703-17716, 2022.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Xiong, F. Cheng, R., Chen, W., Zhang, Z., Guo, Y., Yuan, C., and Xu, R. Multi-task model merging via adaptive weight disentanglement. arXiv preprint urXiv:2411.1877 2024.
Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. Ties- Res-meeginn: resolving interference when merging models. In Proceedings of the International Conference on Neural Information Processing Systems, volume 36, pp. 7093-7113 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. 2021.
Smith, S. L., Dherin, B., Barrett, D. G., and De, S. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101. /2176, 2021.
Yang. E., Shen, L., Wang. Z., Guo, G., Chen, X., Wang, X., and Tao, D. Representation surgery for multi-task model merging. In Proceedings of the 41st International Con- ference on Machine Learning, volume 235, pp. 56332- 56356. PMLR, 2024a.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algo- rithms for traffic sign recognition. Neural networks, 32: 323-332, 2012.
Yang, E., Wang, Z., Shen, L.., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. In Proceedings of the Twelfth International Conference on Learning Representations, 2024b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprini 230::2232..1977 2023.
Yu, L., Yu. B.. Yu, H., Huang, F., and Li, Y. Language models are super mario: Absorbing abilities from ho- mologous models as a free lunch. In Proceedings of the International Conference on Machine Learning. PMIR, 2024.
Wang, A., Singh, A., Michael, J., Hill, F. Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analy- sis platform for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop BlackboxNILP: Analyzing and Interpreting Neural Networks for NLP. pp- 353-355, November 2018. doi: 10. 11.11166/11111--555
Zeng. G., Chen, Y., Cui, B., and Yu, S. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364 372, 2019.
Wang, B., Meng, Q., Zhang, H., Sun, R., Chen, W., Ma, Z.- M., and Liu, T.-Y. Does momentum change the implicit regularization separable data? Advances in Neural Information Processing Systems,
Zou, D., Wu, J., Braverman, V., Gu, Q., Foster, D. P., and Kakade, S. The benefits of implicit regularization from sgd in least squares problems. Advances in neural infor- mation processing systems, 34:5456-5468, 2021.
2022.
Wang, K., Dimitriadis, N., Ortiz-Jimenz,, G., Fleuret, F., and Frossard, P. Localizing task information for improved model merging and compression. In Proceedings of the 41st International Conference on Machine Learning, vol- ume 235. pp. 50268-50222 PMLR, 2024.

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A. appendix
A.1. Proof theoretical results
A.1.1. PROOF OF LEMMA
Lemma (Input consistency). Let ai æi denote the input of the linear layer after p and q (p q) iterations during fine-tuning, respectively. Denote пi as the learning rate of iteration Assume that the model before layer I is C--iipccitzz continuous, and the gradient of loss bounded Ez-norm by Gv then:
|| 2/112 12
C
Σ
7
(26)
i=++1
Proof. Let f=; 0~1)) denote the mapping from the input x to the input of layer I, parameterized by 0~1, which includes all parameters up to but not including layer Then,
= f=1(x;;0 H'me)
(27)
The parameters are updated using gradient descent:
o 0²,, Th4t Vo C()'),
(28)
where 7) is the learning rate iteration i, and ((O",) is the loss function evaluated 0', From Eq. 28, the cumulative change in parameters is:
p
0%
0
Σ
Vo-
th
0%1 0%
Σ 7h V -000-((00
(29)
1=1
The difference between x. and up can be formulated
xf 2 = fmi(a; (x;
(30)
To analyze the magnitude of the change, we consider the l norm of the output difference.:
| 2 (> OLL) (~r( 0L))lla
(31)
Assume that the model before layer is Cj- Ci-Lipschitz continuous, and the gradient of loss is bounded in f--norm by 9/ then:
b - xill2 = fl.f (x; )% - f~r(2;; 112
(32)
S 0 1
(33)
9
-ill(0%)
m V o² (((--1)) -00
Σ
11 V 0
(34)
Cill
Σ
Th V 8' L(
(35)
1=+1
= Ci Σ
1 V ن L( L
(36)
i=9+1
2 C Σ n1)
(37)
i=9+1
where the Eq.33 follows from the Ci-Lipschitz continuity of f-1, and Eq. 37 uses the assumption that the gradient is bounded in & norm by Gt.
12

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A. 1.2. PROOF OF PROPOSITION I
Proposition (Approximate Linear Combination). Let Ti denote the task vector of neuron k in linear layer l. Consider N input samples {I, \. an= let E', denote the corresponding input to layer after t iteration for sample nn during fine-tuning. Assume the gradient of the loss with respect the product 0 a bounded by k. Then, the following inequality holds:
Σ
3 Bh
S 11,
nm%s)
(38)
i=!
where 11,A N. C. G: ii and Bh, Ei=
OC Tt 010 m
Proof. The task vector is calculated as the accumulation of gradients:
T
AL(0'-1)
(39)
00
N
00
(40)
a0
1=1
1.
N
0
(x)
(41)
0(8
x
1,6
Let Th,, denote the gradient of the loss with respect to the product , x that is, Mh.
a(0 ".."" then:
BC(0'' 1
7
Inikk
(42)
T
N
1Tt
t
Indlk ((x (1 - (TTT)T T (T)
(43)
n=1
Rearranging, we obtain:
T
N
(wT))T
1/t
Inkk (1xT (xTT.)T)
n.l
(44)
n=1
n=1
Taking the norm of both sides, we have:
N
TJt
miLk
(I
=
Σ
Ind ((x (xTT1)) T)
(45)
R=1
2
t=1
n=1
N
h
(ar (IxT..)
(46)
Assume 2
is bounded by TıA then:
1-1
nt
makk
(T
Σ
m Σ
Tia
(T (x_h))
(47)
13

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Applying Lemma.l Eq.47, then:
N
7
mlkk
(2
[x
S
Σ T Σ
Ti
C99
mp)
(48)
2
= N -iA C. 9i-
Σ
7
(49)
1=1
i=t
=N-Ti C1
70%)
(50)
1=1 i=!
Denote -
N. =N = Et= TI - and D1 =
C. Gr Pi,e, then we can reformulate Eq.50 as follows:
N
gh n. (xx T T nill
I 1
m
(51)
n=1
2
14

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A. 1.3. PROOF OF THEOREM 1
Theorem (Upper bound of interference). Denote ar as the linear layer I's input of task dū and a as the linear layer I's interference vector and task vector of task respectively. then:
Emg 1 S wh, [To.(Te)) + wº,1- -16,;1
(52)
Where wh, and w is the reconstruction constant.
Proof. According to Lemma 2, each neuron of the task vector associated with linear layer can be interpreted as weighted sum of the corresponding input samples. Consequently, the entire task vector forms an approximate linear subspace of the input space, which motivates us to reconstruct the input using the task vector:
K Zi,1 Σ
(x.,))))))) + =((,)), Ii,1 E Dia
(53)
where D is the input domain of layer for task af, (x.,,) are the reconstruction coefficients of T (T(,) are k-th neuron of task vector Ti,,, and e(x,,)) represents the reconstruction error. The expectation of interference is formulated as follows:
E Pe. 115,,111111 =
|10,,0011112 p(x...) dx;,.-
(54)
Reconstructing æi, by 5q.53, then:
K
t (TT + e((r.)) [[ p(ac.)) dan..-
Ex Pe.. Dii
6i Σ
(55)
Scaling Eq.55 by the triangle inequality, then:
K
8,
Σ
(Zr,))))) + +
p(ar,1) dai,
(56)
K
Σ
a (x:,) + ),
p(a(.)) daii
(57)
6.e(0r.)) placa) daya.
(58)
JaLIEDE
k=1
Σ (
(,.(( Tl1, + [|0 p(x,,)) dx,,,.
(59)
A==
Applying the Cauchy-Schwarc inequality the part in brackets, then we can get:
K
2
(x.,))
+16.
(60)
K
2
(x
-|0,((T(.) + 1--15,0(22:))))
(61)
o (w))) 2 ( 1122)
Σ ( T1 1 (16.0(021)11
(62)
15

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
Applying Eq.62 to Eq.59, then:
(2;,)) (( 1l2 + [10,e( p(æ,,1) dæy,
(63)
K
(0;,() +1
Σ (5(() ) 11 [ plaar) daaa (64)
K
( + +12
||o,(T(.))
1 T + 11 [[( p(z,,1) dxy,
(65)
K Consider that Σø= (, |6,((T.)) and let (x,()),)) denote Σ | (z..) 2 + ).). then we can
reformulated Eq.65 as follows:
(e.,,)(( ,,,,,,,1112 - /
(.,())))) (TT)) 11. + 118.112. le(aw)))) p(zo,,) does
(66)
ED,,
-( ((1(1)))(( da ||o,(TA))
(67)
+ VeLEER (alla p(x.,)) dxys 110:11
(68)
(Barropar, (10: + (Ber, [ ||8:11
(69)
Considering that when Tip and æu are fixed, the reconstruction error and reconstruction coefficient are fixed, therefore Bearop(an.)) Ou, and (ee.:ppee)))) ( is constant. Denote E and Be. (2v.)) ||=( as wh, and w respectively. Therefore we can get:
(xp(e)))) 1,,,1001111 S wh, [[).(T))) + w/, 11
(70)
16

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A. 1.4. THE DETAILED DERIVATION FOR THE CLOSE-FORM SOLUTION.
Considering the objective with the regularization term. The objective can be formulated follows:
min obj = Ei || 1 [[(mm,-TTa)) -(a)T || + w. ||Tmm T.,
(71)
min Ei | ([ TouTT || +w ||Tma T Tma
(72)
Calculate the gradient of TmuITir T..Tiil!
0
|mi
T.. - 2(Tm,IT - TI)TT
(73)
ÖTm,l
Calculate the gradient of I|. Ti,1l17
0
[[ Tu,l 2(Tm.. T(a))
(74)
mmm.
Combine the gradients:
Tm, obj = Σ ||:,,||
1 (2(TmiTTH T.uTTl)a + 2w(Tm. - T(.))
(75)
To find the minimizer Tm,J, set V V obj =
Σ ITull ((TmIT - TT,TT,)) w(Tm.) - T(ii)
(76)
The equation can be written in the form:
Tm, = B
(77)
where:
Σ
(T.TT
(78)
TTull
B Σ
|| (FuTTTiTT) + wT..)
(79)
Assuming invertible, the optimal m, is:
Tm. = BA-1
(80)
Substituting back A and B:
mm.
Σ Tepll (TuTTTTT wT..)
Σ
|| (TITTS)
(81)
Σ T
(Taa(T + wl))
Σ
(TTiTe) + wl))
(82)
17

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A.2. Towards understanding the conflict in Model Merging
According 3.2, the conflict among different experts arises from the interaction between the interference vector and the input. Moreover, this conflict may be exacerbated by the overlap of imput domains. This phenomenon is further evidenced by the results of the vision and language tasks. Specifically, the domain overlap among various visual tasks is limited, whereas the inputs across different language tasks exhibit a larger overlap. Consequently, the current model-merging approach allows for a high degree of recovery in vision tasks, while there remains notable disparity between the performance of the merged model and that of individual experts in language tasks. To address this limitation, it may be necessary to adopt a data processing perspective aimed at reducing the overlap of input domains across different language tasks.
A.3. The impact of linear layer
Performence of Fine-Tuned VIT Models
Performance of Fine-Tuned RoBERTa Models
Cagh-
Weglh Ouly
100
93
70-
60-
SUN099
Carx
DT3
REMISCAS
TunSAT
SVIN
GrSka
MNIST
COLA
MNLI
MINC:
00"
ITII
STS-B
Figure The impact of linear layer
Motivated by (Jin et al., 2023), we only apply our method to the linear layer in the model. To further demonstrate the effectiveness, we only apply the task vector the linear layer to the pretrained model. The result is shown in Fig.7. The result shows that only using the task vector of the linear keeps most of the ability of the model. This means focusing on the linear layer is the key the model merging problems.
18

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A.4. Additional Experiments
In this subsection, we provide the result of ViT-B/16 and the detailed result of discriminative language tasks in the following table, which further demonstrates the generalization of our method.
Table 10: Multi-task performance when merging ViT-B/16 models on eight tasks.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Non-merging Methods
Pretrained
63.8
51.7
55.0
Individual
81.8
86.8
96.9
99.7
97.8
99.1
99.7
82.0
92.9
Test-time Adaption Methods
TW AdaMerging
64.4
64.2
75.4
86.7
86.3
86.7
97.6
46.9
76.0
AdaMerging
70.2
94.8
91.6
84.9
Representation Surgery
68.3
72.3
88.7
97.7
91.0
89.5
98.9
72,9
84.9
Date-free Methods
Weight Averaging
67.7
70.0
75.3
79.5
74.9
60.1
94.4
43.8
70.7
Fisher Merging
73.2
94.5
50.7
71.7
RegMean
69.1
71.6
77.6
88.8
83.7
70.2
96.9
54,6
76.6
Task Arithmetic
61.1
65.9
74.0
76.2
88.0
73.9
98.4
53.0
73.8
Ties-Merging
69.1
72.5
80.5
84.0
85.0
71.5
98.1
54.9
77.0
Consensus Merging
69.8
71.4
80.8
86.5
88.0
71.1
98.4
57.0
77.9
WUDI-Merging (ours)
75.7
90.7
74.7
89.1
Table 11: Multi-task performance when merging RoBERTa models on 8-task GLUE benchmark. We report normalized score (Lu al., 2024).
Method
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
MNLI
RTE
Avg.
Pro-trained
0.0
53.8
85.0
4.0
37.5
53.1
37.1
71.2
41.7
Individual
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Weight Averaging
0.0
59.2
85.8
47.0
45.4
63.9
48.0
71.2
52.6
Task Arithmetic
8.4
89.6
32.8
82.0
85.4
80.4
Ties- Ties-Merging
31.8
88,9
86.2
10.9
61.1
85,9
83.0
69.6
64.7
Task Arithmetic (w/ DARE)
0.0
88.1
86.6
30.2
84.3
79.1
64.0
77.2
63.7
Ties-Merging (w/ DARE)
11.8
95.5
85.8
9.4
86.8
88.7
83.1
63.6
65.6
WUDI-Merging (ours)
81.8
98.3
78.7
60.5
92.7
90.5
93.3
86.4
85.3 A19.7
Table 12: Multi-task performance when merging RoBERTa-Large models on 8-task GLUE benchmark. We report normalized score (Lu et al., 2024).
Method
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
MNLI
RTE
Avg.
Pre-trained
0.0
51.5
40.9
20.9
36.4
56.0
37.6
62.4
Individual
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Weight Averaging
7.4
55.1
84.2
46.3
56.7
73.8
35.8
66.7
53.3
Task Arithmetic
7.4
86.1
86.8
78.0
90.7
77.0
73.3
67.6
70.9
Ties-Merging
42.7
85.2
Task Arithmetic (w/ DARE)
4.1
85.2
85.8
71.6
91.3
85.6
75.2
68.1
70.9
Ties-Merging (w/ DARE)
2.9
90.4
86.8
75.4
92.4
86.4
79.0
69.1
72.8
WUDI-Merging (ours)
82.2
98.7
87.3
81.4
94.6
96.6
93.4
77.1
88.8
19

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A.5. Consistency of the input between pretrained model and fine-tuned model
In this section, we provide the imput consistency between the pretrained model and the expert models, as shown in Fig.8 and Fig.9. It can be seen that the large model shows less consistency, however the change of the direction and magnitude still very small, where the change of the direction of the most layer is less than 0.4. Consider that he task vector is the weighted sum of the input from the start to end during fine-tuning, the task vector has better consistency than the consistency between the pretrain model and the fine-tuned model.
SUN397
Cars
RESISCA5
EarSAT
4Magniuule
AMMggiitue
4
AMagniuuds
04
AMagnitde
Depti of Layer
ADirection SVHN
ADirection
ADircction MNIST
ADirection
Depnh of Layer
AMaggniude
AMogsiuuk
Axfanitude
ADirection
ADirvction
ADirection
ADirection
Figure 8: The input consistency between the pretrained model and the fine-tuned model for ViT-B/16
SUN397
Cars
RESISC45
EuroAAT
AMagnitule
AMagnitdi
AMagnitue
04
AMagnitue
Desch of Layer
0.4
ADirection SVIIN
ADirection
ADirection
ADirection
GTSRB
MNIST
DTD
4Magniude
AMognitule
44
@Magnitude
04
Magnitdde
Depth of Layer
ADirection
ADircction
ADircttin
AlDrcctionn
Figure 9: The input consistency between the pretrained model and the fine-tuned model for ViT-L/14
20

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A.6. The detailed results of interference in different layers and tasks.
To further demonstrate our method's capability eliminate layer-wise interference, we use relative error for evaluation. Specifically, for given merged vector Tm the relative error is calculated follows:
N [z 0 + Tm) f(z,; Ti)l
1 N
Relative Error
(83)
[T 0 T:)1
Fig. 10, 11, and 12 respectively present the relative errors of different models with varying numbers of layers across different datasets. Fig. 3 summarizes the average relative errors across these datasets for the models with different layer counts.
SUN397
RESISCAS
EAT
Steghe
Talk
935
MM
SVIIK
GTSRB
MINSST
DID
Rolanto Erme
Ceerte
Cove
Layes
Layes
Layes
Layes
Figure 10: The interference for different layers and tasks for ViT-B/32.
SUN397
Clas
RESISCAS
EureSAT
WILSE Hearg
6.4
Jecaaive Eree
towil Low
lowel Coww
Lewrrt
Layes
Layens
Lasen
Layen
SVIN
GTSRB
MNIST
Tull dilaii
Solthie Suio
Roliiie Einer
Figure 11: The interference for different layers and tasks for ViT-B/16.
21

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
SUN297
Clars
RESISC4S
EuroAAT
WEE serenn
Mencee
Layen SVHN
Layen
Lapex MNIST
Lagers
GTSRB
Tal didinns
Tell Acderii
R-daiing Emno
2
Figure 12: The interference for different layers and tasks for ViT-L/14.
A.7. Performance comparison when using different steps for the solution process.
Fig. 13 presents performance comparison for models of various sizes, including ViT-B/32, ViT-B/16, and ViT-L/14, when using different steps for the optimization process.
ViT-H/32
VIT-8/16
VIT-L/14
100
80
80
90
Accuracy (%5)
SUN397
SUND07
SUNG97
40
Cas
RUSISCAS
Cars
Cirs
RINBUSS
EareAAl
EundAl
20
GTSLD
MINIST
XINISC
20
MNIST
DTD
DED
DTD
Iterationn
Iteration
Iteration
Figure 13: The results obtained at different stages of the solution process.
22

---

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
A.8. Details of the experiment.
1. DATASETS
Vision tasks. Following (Ilharco al., 2023; Yadav et al., 2023; Yang al., 2024b), we use SUN397 (Cordts et al., 2016). Cars (Krause et al., 2013), RESISC45 (Cheng et al., 2017), EuroSAT (Helber al., 2019), SVHN (Netzer al., 2011), GTSRB (Stallkamp et al., 2012), MNIST (LeCun et al., 1998), DTD (Cimpoi et al., 2014)
NLP tasks. For discriminative language lasks, we use GLUE benchmark (Wang et al., 2018). For generative language tasks, we use AlpacaEval (Dubois et al., 2024), GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021).
A.8.2. CALCULATE RESOURCES AND ENVIRONMENT.
All of our experiments were conducted on NVIDIA A100 40GB, Python 3.10, PyTorch 2.4.0, and the CUDA 11.8 toolkit.
A.8.3. HYPERPARAMETERS
There are only two hyperparameters in our method. In practice, we set the learning rate to le-5 and the number of iteration to 300. Following (Jin et al., 2023; Xiong et al., 2024), we only applied our method to the linear layer in the model.
23


Reviews:
[{"review_id": "AHuOqh2aO1", "reviewer": "Reviewer_VcLV", "summary": "This paper proposes WUDI-Merging, a data-free model merging method where the merged model weights are optimized via SGD using the Adam optimizer. The optimization objective leverages the insight that task vectors form an approximate linear subspace of the corresponding input space. Additionally, the authors provide a closed-form alternative (WUDI-Merging-CFS) for scenarios with limited computational resources. Empirical evaluations demonstrate that WUDI-Merging achieves state-of-the-art performance across vision and NLP tasks and is rescaling-free.\n\n## update after rebuttal\n\nI would like to thank the authors for the rebuttal. While I think the performance of WUDI is impressive and the new results on Flan-T5 and Ewen LoRAs are reassuring, I believe authors should make a huge effort for improving the presentation of the paper. As I have read authors' responses to also other reviewers, I found some of the responses were not very informative or persuasive. For example, as for the linear claims, I think using Lipschitz constant to explain it is too superficial. Also, Reviewer 7sbW's questions about how authors run other methods that require data are valid, but not answered. WUDI would be a great addition to the community if these problems are addressed and I strongly encourage the authors to handle the requests properly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers’ valuable feedback and have addressed each point as follows:\n\n---- \n\n### W1: Results on More Models and LoRA\n\nTo further demonstrate the generalizability of our method on different models and LoRA, we supplemented the experiments on Flan-T5-base and Qwen-14B. For merging LoRA, We first restore BA back into the original matrix $(\\tau_i=B_iA_i)$ , then apply WUDI-Merging directly to $\\tau_i$ to obtain $\\tau_m$, then merging it into $\\theta_{base}$.\nThe experimental results obtained from merging Flan-T5-base (LoRA fine-tuned) models and Qwen-14B (LoRA fine-tuned) are shown in the table below:\n\n**Tab 1.1:** Experimental results of merging Flan-T5-base (LoRA fine-tuned) models on all eight tasks.\n\n| Method| CoLA | MNLI | MRPC | QNLI | QQP  | RTE  | SST2 | STSB | Avg.  |\n|-----|----|---|----|----|---|----|------|------|--|\n| **Individual**            | 69.1 | 82.7 | 85.5 | 90.9 | 84.0 | 84.4 | 92.9 | 87.4 | 84.6  |\n| **Ties-Merging**          | 68.3 | 56.3 | 79.4 | 89.8 | 83.7 | 79.4 | 91.6 | 71.2 | 77.5  |\n| **AdaMerging++** | 69.1 | 60.3 | 78.4 | 90.0 | 83.6 | 79.1 | 91.6 | 74.1 | 78.3  |\n| **WUDI-Merging(Ours)**                  | 68.6 | 79.0 | 77.7 | 87.2 | 83.1 | 75.8 | 93.2 | 85.0 | **81.2(+2.9)**  |\n\n**Tab 1.2:** Experimental results of merging Qwen-14B (LoRA fine-tuned) models on all four tasks.\n| Method | MMLU    | TruthfulQA | BBQ     | CNN-DailyMail | Avg.   |\n|---|---|--|---|---|---|\n| **Individual**| 68.35|53.34|93.53|19.46|58.67|\n| **Task Arithmetic**           | 67.56   | 52.33      | 78.38   | 20.54         | 54.70  |\n| **Ties-Merging (w/ DARE)**    | 69.38   | 52.03      | 81.06   | 15.91         | 54.62  |\n| **WUDI-Merging(Ours)**|  69.17 | 55.71 | 80.56 | 17.33 | **55.69(+0.99)**  |\n\nWhile there is a slight degradation relative to individual models, our method demonstrates SOTA performance on merging LoRA fine-tuned models.\n\n----\n\n### W2: Potential Limitation\n\nWe think that merging heterogeneous models may be the potential Limitation, as such models might not adhere to the assumptions underlying WUDI-Merging and current merging methods. Addressing these challenges is a key direction for our future research.\n\n----\n\n### Q1: On Using Only the Task Vector of the Linear Layers\n\nIn the context of homogeneous model merging, we think that **\"using only the task vector derived from the linear layers is usually enough\"**. Removing the nonlinear task vector from the fine-tuned model can be viewed as applying a small offset to a limited number of parameters. For a $C$-Lipschitz continuous model, the change in the output is bounded by:\n\n$\n||f_{\\theta+\\tau_i}(x) - f_{\\theta+\\tau_i^{linear}}(x)||\\le C·||\\theta+\\tau_i - \\theta+\\tau_i^{linear}|| = C·||\\tau_i^{non-linear}|| \n$\n\nConsidering that the parameters of the nonlinear layer accounting for only a small fraction (Qwen-14B$\\approx$0.007%, LLama3.1-8B$\\approx$0.003%) and has a small offset. The affect brought by only using the task vector of the linear layer is small. Therefore, we think that using only the task vector derived from the linear layers is usually enough. In all our other experiments, we found that utilizing only the task vector of the linear layer achieves performance comparable to that of the full expert model, which also confirms this point.\n\nHowever, if the parameter's offset are large,our method may have limitations in use, which is also a major limitation in the current field of model merging [1]. Current model merging methods need to limit the change of fine-tuned expert models to base model in a relatively small range. We will explore this limitation in our future work.\n\n----\n\n### Q2: Results on Different Optimization Strategies\n\nWe evaluate the effects of applying different optimizers and learning rates in our method, which is shown in Tab 1.3 and Tab 1.4. Tab 1.3 shows that Adam (85.2%) and SGD (85.1%) achieve similar performance, which suggest that our method is not sensitive to kind of the optimizers. Tab 1.4 demonstrate that a smaller learning rate can better ensure the stability of optimization and enhance the performance of results.\n\n**Tab 1.3:** Experimental results of applying different optimizers in our method, the average accuracy of ViT-B-32 are reported.\n| | Adam|SGD|\n|-|-|-|\n|Acc.| 85.2| 85.1 |\n\n**Tab 1.4:** Experimental results of applying different learning rate in our method, the average accuracy of ViT-B-32 are reported.\n| 1e-5|1e-4|1e-3|1e-2|Ada-Merging|\n|-|-|-|-|-|\n| 85.2 |  84.3  |  83.9 | 83.5| 80.9 |\n\n\n----\n\n### Q3: Typo\n\nWe appreciate the reviewers' attention to detail and will correct the identified typographical errors in the final revision.\n\n---\n\nWe thank the reviewer again for their constructive feedback and hope that our detailed responses address the concerns. We look forward to further discussion.\n\n[1] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I have reviewed the theoretical claims in the main paper. Lemma 1 regards input consistency, Proposition 1 approximates task vectors as a linear combination of inputs, and Theorem 1 provides an upper bound on interference. The proofs appear to be correct.", "experimental_designs_or_analyses": "Both vision and language model merging experiments follow the standard designs of previous model merging work. The analyses in Section 4.2, 4.4 are sound and valid.", "relation_to_broader_scientific_literature": "This work may inspire researchers to further explore SGD-based optimization strategies for data-free model merging, potentially shifting the focus from traditional closed-form solutions or heuristic-based approaches.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths\n\n- The optimization objective is theoretically motivated and can be efficiently approached using Adam within the current framework (e.g., PyTorch). Corresponding computation cost is discussed.\n- An alternative solution, WUDI-Merging-CFS, is also provided for scenarios with limited GPU resources, along with a sensitivity analysis on regularization coefficient.\n- WUDI-Merging is rescaling-free, as demonstrated in Figure 4 (b).\n- A comprehensive benchmark evaluation on both vision and language models support the superiority claims of proposed method.\n- The experiments are well-designed and analysis provided further understanding into WUDI-Merging.\n\n### Weaknesses\n\n- The experiment for generative language models is limited to merging three Llama2 models, which is relatively insufficient. Expanding the evaluation to include merging *more models*, and *more model architectures* (e.g., encoder-decoder, decoder-only) would provide a more comprehensive understanding of the general applicability and potential limitations of WUDI.\n- Experimental study of \"Input consistency\" and results of model merging performance are based on fully fine-tuned models. The paper does not explore scenarios where task vectors are derived from PEFT methods (e.g., LoRA). Further analysis on this would provide a clearer understanding of method's applicability.\n- The potential limitations of proposed method are not well-discussed.", "other_comments_or_suggestions": "- The analysis stems from the assumption/literature observation that \"the task vector in the linear layer encapsulates most of the capabilities of the expert models. As shown in Figure A.3, an expert model utilizing only the task vector of the linear layer achieves performance comparable to that of the full expert model. Therefore, we primarily focus on the linear layers of the model.\" I assume this may be model-dependent observation and thus I am particularly interested in the case when using only the task vector of the linear layers is not enough? Will WUDI be less effective in those cases?\n- While WUDI-Merging is described as hyperparameter-efficient, it would be helpful to provide a more detailed discussion on how different learning rates or optimization settings affect the quality of the merged model.\n- Please proofread the manuscript once again as there are several typos/mistake throughout. e.g. line 133 \"linear linear\", line 993 \"magnitude still very small\", etc.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d1A0An9XRj", "reviewer": "Reviewer_kYu3", "summary": "This paper introduces WUDI-merging, a new data-free model merging method. The authors provide theory-backed idea that task vectors for a linear layer represent a linear subspace corresponding to its inputs. They use this knowledge to construct a merging method that aims to minimize the inference of the merged model's weights on their corresponding tasks. They also introduce a per-task weighting corresponding to the Frobenius norm of the task vector. When compared to other methods on a variety of benchmarks, WUDI-merge obtains a signficantly higher normalized average multi-task score than other data-free merging baselines and even test-time adaptation methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers’ valuable feedback and have addressed each point as follows:\n### W1: Reconstruction Error\n\nFor calcuating the reconstruction error in Equation (13), we first obtain the input for each layer from a set of samples and then compute the reconstruction coefficients using the least squares method with its corresponding task vector. The reconstructed vector, $x_{\\text{recon}}$, is derived using these coefficients along with the task vector. Then we calculate the Relative Reconstruction Error (RRE) for the sample from task $i$ as follows:\n\n$\n\\text{Relative Reconstruction Error (RRE)} = \\frac{\\|x - x_{\\text{recon}}\\|}{\\|x\\|}, \\quad \\text{where}  \\quad x_{\\text{recon}} =\\tau_i\\(\\tau_i^T\\tau_i)^{-1}\\\\tau_i^T x .\n$\n\nTable 2.1 shows the result of different layers and tasks. which demonstrate that the relative reconstruction errors across different layers are extremely small, which further validates our theoretical analysis.\n\n**Table 2.1**: Reconstruction Error Results on eight tasks of different layers\n\n| Task    | SUN397   | Cars     | RESISC45 | EuroSAT | SVHN    | GTSRB   | MNIST   | DTD     |\n|---------|----------|----------|----------|---------|---------|---------|---------|---------|\n| Layer 1 | 1.3e-5   | 1.3e-5   | 1.3e-5   | 3.3e-3  | 7.1e-3  | 3.6e-3  | 5.8e-3  | 1.3e-5  |\n| Layer 3 | 1.1e-5   | 1.1e-5   | 1.2e-5   | 1.4e-5  | 1.6e-5  | 1.3e-5  | 1.3e-5  | 1.1e-5  |\n| Layer 6 | 1.0e-5   | 1.1e-5   | 1.1e-5   | 1.3e-5  | 1.2e-5  | 1.2e-5  | 1.2e-5  | 1.1e-5  |\n| Layer 12| 9.8e-6   | 1.1e-5   | 1.2e-5   | 1.1e-5  | 1.2e-5  | 1.1e-5  | 1.2e-5  | 1.0e-5  |\n\n---\n\n### W2: Resource Consumption Comparison\n\nWe report the computational time and GPU memory usage of different method on ViT-B-32 tasks. In comparison to the Adamerging method, our approach not only improves performance but also significantly reduces computational cost. The details are summarized in the following table:\n\n**Table 2.2**: Detailed computational time and gpu memory requirements on ViT-B-32 tasks.\n| Method                        | Accuracy (%) | Time     | GPU Memory (GB) |\n|-------------------------------|--------------|----------|-----------------|\n| Ties Merging                  | 72.4         | 4 s      | 0               |\n| Adamerging                    | 81.1         | 127 min  | 17.1            |\n| WUDI-Merging-CFS (CPU)        | 84.4         | 5 s      | 0               |\n| WUDI-Merging-CFS (GPU)        | 84.4         | 2 s      | 1.8             |\n| **WUDI-Merging**              | **85.2**     | 1 min 54 s | 4.0           |\n\n\n### Q1 & Q2: How to Use Random Vectors or Subsets of the Task Vector for Optimization\n\n**For the subset of task vectors**, we randomly sample a subvector from the original task vector as follows:\n\n$\\tau^{\\text{sub}}_i = \\tau_i [\\text{rand\\\\_index}, :]$\n\nThe corresponding loss is computed as:\n\n$\n\\mathcal{L}_{\\text{sub}} = \\sum\\_{i=1}^{n} \\frac{1}{\\|\\tau_i\\|\\_F^2}\\ \\delta_i(\\tau^{\\text{sub}}\\_i)^\\top = \\sum\\_{i=1}^{n} \\frac{1}{\\|\\tau_i\\|_F^2}\\ (\\tau_m - \\tau_i) (\\tau^{\\text{sub}}_i)^\\top$\n\n\n**For random vectors**, we sample from a Gaussian distribution whose mean and standard deviation are computed from the original task vectors:\n\n$\n\\tau^{\\text{random}}_i \\sim \\mathcal{N}(\\mu_i, \\sigma_i^2), \\quad \\text{where} \\quad \\mu_i = \\text{mean}(\\tau_i) \\quad \\text{and} \\quad \\sigma_i = \\text{std}(\\tau_i)\n$\n\nThe loss in this case is given by:\n\n$\n\\mathcal{L}_{\\text{random}} =  \\sum\\_{i=1}^{n} \\frac{1}{\\|\\tau_i\\|_F^2}\\delta_i(\\tau^{\\text{random}}_i)^\\top =  \\sum\\_{i=1}^{n} \\frac{1}{\\|\\tau_i\\|_F^2}(\\tau_m - \\tau_i)(\\tau^{\\text{random}}_i)^\\top\n$\n\n**All reported results are averaged over 5 sampling runs**. We thank the reviewers for their suggestions and will include these additional details in the revision.\n\n---\nWe thank the reviewer again for their constructive feedback and hope that our detailed responses address the concerns. We look forward to further discussion.", "claims_and_evidence": "The claims are supported by evidence. The core observation, namely that task vectors for a linear layer represent a linear subspace corresponding to its inputs, and the derivation of the optimization problem are backed by theory. WUDI-merging is compared to a variety of data-free merging methods and TTA methods on a variety of benchmarks, where it achieves SoTA performance. Sufficient ablation studies are conducted on their method.\n\nOne potential caveat, however, is that experiments are missing that explicitly test some of the theoretical results in the paper. This would include seeing how close the inputs to a linear layer are to the subspace corresponding to its task vector (i.e. experimental exploration of the reconstruction error in equation (13)). The theoretical analysis, while sound, involves the use constants bounding the norm of gradients, so the bound on the error might be large if those constants are large. Since the main purpose of these theoretical results was to motivate the construction of WUDI-merging, this does not significantly detract from the main contribution of this paper.", "methods_and_evaluation_criteria": "Evaluation of average normalized accuracy on multi-way merges for vision models, discriminative text models, and generative text models makes sense for evaluation of a merging method.", "theoretical_claims": "I skimmed the proofs and did not find any issues. However, I did not examine them in detail.", "experimental_designs_or_analyses": "I didn't review anything in detail, but the experimental designs and analysis makes sense to me.", "relation_to_broader_scientific_literature": "The paper categories existing merging methods into data-free, test-time adaptation, and MOE-like merging. The differences and advantages/disadvantages of these methods are explained, and the categorization of WUDI-merging as a data-free method makes sense.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Strengths:\n- Theoretical results are used to motivate the method.\n- Method is non-trivial and novel.\n- Method is data-free and relatively computational cheap.\n- Writing is clear.\n\nWeaknesses:\n- Computation times and memory requirements are not provided for baseline merging methods.", "other_comments_or_suggestions": "Typos:\n- line 175, the \"Where\" should not be capitalized\n- Table 2, \"date-free\" should be \"data-free\"\n- line 994: \"that he task vector\" should be \"that the task vector\"", "questions_for_authors": "In the \"Selection of Linear Subspace\" paragraph of section 4.4:\n1. How exactly are a combination of task vectors and random vectors used to optimize the loss? Do you replace some neurons in the task vector with random vectors and run WUDI-merging as normal?\n2. How are the random vectors selected?\n\nIt would be good to include this information in at least the appendix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIMjjJbSX5", "reviewer": "Reviewer_7sbW", "summary": "The paper offers very satisfying results and method in a nice writeup that could be communicated much better.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers’ valuable feedback and have addressed each point as follows:\n### Concern 1: The Use of Validation Data\n1. **Whether task arithmetic is data-free**: We summarize TA as a data-free method as it can select empirical parameters for merging without requiring data. Although it can be improved by seaching rescaling coefficient, it is reasonable to summarize TA as a data-free method.\n2. **How we run other methods**: We run other methods in the same way as described in their paper and report the results accordingly in the paper if the experimental settings are same.\n3. **Whether we use validation data: Our proposed method is entirely data-free**, and did not require any validation data. This is because our approach is rescale-free and solely relies on our data-free loss.\n### Concern 2: Focus on Multi-Task Learning\nOur work focuses on model merging rather than model editing. In current model merging field, multi-task learning is the primary goal to pursue [1,2,3,4,6,7], where TA and task vector is a foundamental concept for analysis the interference and conflict[1,2,6] .\n### Concern 3: Related References\n1. **Linearity claims**: Our work focued on the linear layer but not linearizing the whole model. Therefore, these linearization works are not highly relevant to us.\n2.  **Reconstructing data**：Our work used the reconstruction as an intermediate step in our derivation and did not use reconstruction data in our method. Therefore, the data reconstruction attacks works methoded are not highly relevant to us. \n3.  **Other works**: We appreciate the references you provided and will incorporate their discussion in our revision. The conclusions and methods in these works do not significantly overlap with ours. Therefore, we consider them to be \"Related References Not Discussed\" rather than \"Essential References Not Discussed\"\n### W1: The Claim of Local Optima and Definition of Interference Vector\n1. **The Claim of local optima** What we want to express is that the task vectors for same task are at similar positions, which indicated that the models are fine-tuned to convergence, so we think they are optimized to local optimal. But these task vectors may not be the optimal solution in the entire parameter space, so using local optimal is a more rigorous statement. Also we have not stated that this local optimal is unique, and this has no significant connection with our method.\n2. **The Definition of Interference Vector: We do not claim that any changes to task vector constitute interference**, we define the interference as the difference between outputs of merged model and expert model, which is composed of $\\delta x$, if $\\delta$ is orthogonal to $x$, then $\\delta$ is not cause interference. Similar concepts are adopted in methods like Eq.7 in Alpha-Edit [5] and Eq.1 in Regmean[4]. Furthermore, the success of TA and other methods doesn‘t mean \"you can get disentangled vectors that do not interfere\" since the current model merging method is far from allowing model merging without reducing any performance.\n### Q1: Convex Hull\nNot falling in convex hull is not unique to our method, as other methods do not constrain the sum of rescaling coefficients equal to 1. The reason why our method effective compared to previous data-free methods is our method implicitly leverages input data information via task vectors.\n### Q2: The Choice of RoBERTa and LoRA Results\nWe selected RoBERTa because it's a commonly used benchmark, as used in works such as [3]. We further provide results for other widely adopted LoRA benchmarks [1][2][3]. Due to character limitation, you can view the results in **Tab1.1 and Tab1.2 in our rebuttal to reviewer VcLV**.\n### Q3: The Least Similar Problem to Our Work\n1. **Least Similar Problem** Most existing methods focus on reducing the interference and conflicts [1,2,3,4,6,7], so it's hard to say what is the least similar problem to our work. \n2. **Combine with other methods** Since our objective is a simple optimization problem, we think different initialization will only provide limited influence. The result in Tab.3.1 demonstrated that using different methods as initialization yields similar results.\n   \n**Tab 3.1** Results of using different methods as initialization\n|Method|Ours|Ties+Ours|Adamerging+Ours|\n|-|-|-|-|\n|Acc.|85.2|84.9|85.0|\n\n---\nWe thank the reviewer again for their valuable feedback and hope our detailed responses address the concerns. We look forward to further discussion.\n\n[1] TIES-Merging: Resolving Interference When Merging Models\n\n[2] Parameter Competition Balancing for Model Merging\n\n[3] Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging\n\n[4] Dataless Knowledge Fusion by Merging Weights of Language Models\n\n[5] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models\n\n[6] Task Singular Vectors: Reducing Task Interference in Model Merging\n\n[7] DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling", "claims_and_evidence": "You refer to Task Arithmetic as data-free which is mostly true, except that it reweights the models with data. Classic merging does not and Ties also has a scaling factor but it is less sensitive to it so the default is more likely to be just fine.", "methods_and_evaluation_criteria": "The paper mainly focuses on multitask learning (not generalization, new capabilities etc.) somewhat awkwardly it focuses on task arithmetic quite heavily although this is not even the idea behind task arithmetic (there the idea is compositionality of tasks not orthogonality and lack of interference which are closer to the first merging works, to ties, to https://proceedings.neurips.cc/paper_files/paper/2023/file/d28077e5ff52034cd35b4aa15320caea-Paper-Conference.pdf etc.).", "theoretical_claims": "There are proofs (and I commented also on some things that were unclear to me). In general, this takes a meaningful part of the paper, but I don't feel this is a main contribution of the paper.", "experimental_designs_or_analyses": "The baselines on NLP are outdated, RoBERTa is a better hyperparameter tuned (slightly larger) BERT, the first model to pretrain (parallel to Elmo), a lot has changed since.\n\nBeyond this, the method performs extremely well, it is compared with a lot of baselines and (while for some reason not highlighted) almost reaches multitask learning, all without data, this is a massive result.", "relation_to_broader_scientific_literature": "The paper states well its relations to previous merging methods. It explains well what are the main changed and difficulties in preexisting methods and covers many baselines.\n\nI do wonder how does this method work on LoRA and or whether KnoTS would be irrelevant here or will prealigning be still useful even thought you learn all the weights, but it is just something I am curious about and not a strong lack the paper has.\n\nYou mention not using external data, and that others do not, but other methods do (especially task arithmetic). Usually, they use a validation set to tweak some sizes. (noteworthy and very recent is arxiv.org/pdf/2310.02575 in that regard)\nIt is unclear how exactly did you run those methods and what you did yourself (did you use some validation set? where?).\nAlso recent, but not as recent as the above is KnoTS, which I wonder if, because you are assuming all kinds of linear stuff is further improving your method like it does to ties. i.e. is it complementary or conflicting.", "essential_references_not_discussed": "The linearity claims remind me of works like https://arxiv.org/abs/2303.09435 https://arxiv.org/abs/2303.08112 from interpretability and of works like https://arxiv.org/abs/2407.15845 on reconstructing data from weights or of memebership attacks.\n\nThe claim in 140 should be referred to https://arxiv.org/abs/2302.04863, this is the paper that deals with this (while Task arithmetic is doing many nice things, this is not the main source of knowledge regarding that claim).\n\nMaybe useful to cite the ones proposing model merging when you first use the term? I think you did cite (most?) of them somewhere else (https://arxiv.org/abs/2203.05482 https://arxiv.org/abs/2111.09832 https://arxiv.org/abs/2204.03044 )\n\nThe moerging lit. is not essential, but it is quite lacking (maybe arrow and phatgoose are the clearest cases to cite? there is also a survey if it helps (https://arxiv.org/abs/2408.07057)", "other_strengths_and_weaknesses": "The paper is generally clear, but there are many small missing pieces that are left for the reader to reinvent, this makes following the paper very hard. I tried to highlight those cases (what is an input, what distribution are we discussing etc.) and explain where is the confusion, but at the current state it looks really well when skimming but makes a really hard life for following along.\n\nIt is unclear from the figure\\intro what do you mean as input vector. for an LLM there is the overall input for all of the training (but it is not a vector), the input to a given layer or the input to the whole network (looks like the second).\n\nI do not understand the claim in l140 \"This directional consistency suggests that task vectors converge to local optima\". Local and not global? (that is obvious) optima and not just anywhere? Well this what makes convergence. Are you trying to say the local optima is the same and unique? This is only partially true (see Guetta et al. 2023 I also shared in the related work section). Moreover, the fact that models converge to similar solutions doesn't mean that the differences between them are noise (interference they are for sure not, as the literature thinks of interference as places where the different Task Vectors ruin for each other, regardless if it is good for the original task). By the definition of the delta, it seems like you are thinking of the interference as *any* change from the task vector, this is obviously not true, the whole point of task arithmetic was that you can \"add\" things together, merging methods also show cases where you can get disentangled vectors that do not interfere (although they do change) to some extent.", "other_comments_or_suggestions": "constitude-> constitute\n\"Crucially,\nsince the learning rate and total optimization steps always\nremain constrained during standard fine-tuning procedures,\nthe corresponding inputs for individual samples remain consistent across successive update iterations.\" Can't parse this sentence, the inputs are consistent? In what sense? \nl61 indicate\"s\"?\n\" As shown in Figure A.3\" A.3 is an App.~ not a figure. Also the appendix has only subsections and no sections? why?\nl137 (right col) what is p? probability over the distribution of inputs of the training data in i? all tasks? something else?\nl160 (right) \"lemma1\" missing space same sentence \"a individual\" wrong det.\nl179 \"highly consistency\"\nl194 eq.1 missing space \nl198 why does preposition has a . instead of a space? (there are many typos, you should pass over it, grammarly might find some of them as well) Also in Theorem.1, if the thing is that you want space and don't want it to jump lines, use ~ instead of a regular space (it is a space that counts as a regular character and not a word boundary)\nl243 \"the\" unnecessary?\nl256 (right) \\citet not \\cite or \\citep when using the citation as part of the sentence.\nl262 right I am not sure I would call it analysis, maybe just evaluation or performance? We don't learn anything, just compare.\nWhat are the red delta numbers in the tables? Difference from what?", "questions_for_authors": "In your method you relearn all of the weights, so in some essence you have no reason to fall in any convex hull of the original models (like AdaMerge for example). Is that an important feature of the method? Do you think that is the main gain?\nMore understanding of what this optimization procedure actually finds might be illuminating (both on what should be still improved, and what is being done here successfully).\n\nIs the choice of RoBERTa related to what you could train yourselves? because if so, maybe this can help? https://huggingface.co/Lots-of-LoRAs this is quite a large set of LORA models all trained similarly, if you use it because the method doesn't work with LORAs, this is something you should state much more clearly.\n\nWhat methods do you see as addressing the least similar problem to yours?\nYour method starts from a merged model, what if you started from a better merged model (of a dissimilar method?) would that result in another boost in performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper got mixed reviews feedback, i.e., 2 accept and 1 weak reject rating.

Before rebuttal reviewers suggested the strengths of the papers are: 1) The optimization objective is theoretically motivated and can be efficiently approached using Adam within the current framework; 2) Theoretical results are used to motivate the method; 3) Method is non-trivial and novel; 4) Method is data-free and relatively computational cheap.

weaknesses are: 1) experimental results are limited; 2) Computation times and memory requirements are not provided for baseline merging methods; 3) as mentioned by reviewers 7sbW, writing needs to be improved otherwise it is hard to follow. 

After rebuttal, all the reviewers confirmed they had read the authors' response and would update review if needed. In. the AC-reviewer discussion phase, reviewer VcLV mentioned most of their concerns are fixed, but some parts like the linear claim was not addressed. Also reviewers 7sbW suggested their concerns are not addressed.

Given these AC decided to give weak accept rating. But suggested authors carefully addressed reviewers' comments in the final version.
Decision: Accept (poster)