Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration
Authors: Wenju Sun
Qingyong Li
Wen Wang
Yang Liu
Yangliao Geng
Boyang Li
Venue: NeurIPS 2025 poster ()
Keywords: Model Merging
Vision Transformer
Task Arithmetic
Primary area: general_machine_learning

Abstract: Multi-task model merging aims to consolidate knowledge from multiple fine-tuned task-specific experts into a unified model while minimizing performance degradation. Existing methods primarily approach this by minimizing differences between task-specific experts and the unified model, either from a parameter-level or a task-loss perspective. However, parameter-level methods exhibit a significant performance gap compared to the upper bound, while task-loss approaches entail costly secondary training procedures. In contrast, we observe that performance degradation closely correlates with feature drift, i.e., differences in feature representations of the same sample caused by model merging. Motivated by this observation, we propose Layer-wise Optimal Task Vector Merging (LOT Merging), a technique that explicitly minimizes feature drift between task-specific experts and the unified model in a layer-by-layer manner. LOT Merging can be formulated as a convex quadratic optimization problem, enabling us to analytically derive closed-form solutions for the parameters of linear and normalization layers. Consequently, LOT Merging achieves efficient model consolidation through basic matrix operations. Extensive experiments across vision and vision-language benchmarks demonstrate that LOT Merging significantly outperforms baseline methods, achieving improvements of up to 4.4% (ViT-B/32) over state-of-the-art approaches. The source code is available at https://github.com/SunWenJu123/model-merging.

Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration
Wenju Sun 1.2 Qingyong Li,,,,, Wen Wang! Yang Li Yangliao Gengl.2; Boyang Li
Key Laboratory of Big Data & Artificial Intelligence Transportation, Beijing, China School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China Frontiers Science Center for Smart High-Speed Railway System, Beijing Jiaotong University. Beijing, China
*College of Computing and Data Science, Nanyang Technological University, Singapore (sunwenju, liqy, wangwen, yliucit, gengyla)&bjtu tu. edu. cn boyang wewelii.eeeeee edu.
Abstract
Multi-task model merging aims to consolidate knowledge from multiple fine-tuned task-specific experts into unified model while minimizing performance degrada- tion. Existing methods primarily approach this by minimizing differences between task-specific experts and the unified model, either from a parameter-level or a task-loss perspective. However, parameter-level methods exhibit significant per- formance gap compared the upper bound, while task-loss approaches entail costly secondary training procedures. In contrast, we observe that performance degradation closely correlates with feature drift, i.e., differences in feature represen- tations of the same sample caused by model merging. Motivated by this observation, we propose Layer-wise Optimal Task Vector Merging (LOT Merging), a technique that explicitly minimizes feature drift between task-specific experts and the unified model in a layer-by-layee manner. LOT Merging can be formulated as a convex quadratic optimization problem, enabling to analytically derive closed-form solutions for the parameters of linear and normalization layers. Consequently, LOT Merging achieves efficient model consolidation through basic matrix operations. Extensive experiments across vision and vision- lasion-language benchmarks demonstrate that LOT Merging significantly outperforms baseline methods, achieving improve- ments of up to 4.4% (ViT-B/32) over state-of-the approaches. The source code is available /tttt::////e gi thub. httpt//coo//tnneenn113//////// http://////mmmmmmmmmmmee
Introduction
Pretrained foundational models [24 4] encapsulate rich, transferable knowledge, which has facilitated their widespread use for fine-tuning on downstream tasks, yielding superior performance. However, paradigm has resulted in proliferation of fine-tuned
the predominant use of the cestraining-finetunne models, substantially increasing storage and maintenance costs for deployment. This challenge has driven the development of model merging, effective strategy that consolidates the knowledge from multiple fine-tuned models into single model, thereby climinating the need for costly retraining.
Existing model merging methods typically focus on minimizing parameter differences between task- specific models and the merged model, such as arithmetic averaging of model parameters [12 20 or generating masks based on heuristic factors 38 5 34 These approaches aim to identify important
"Correspondence to: Yangli-ao Geng ggengylabbju.ccdug Qingyong <liqy @bjue.buuuccuccc>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

parameters across different task models, attempting to preserve these key parameters during the merging process through weighting or masking strategies. While these methods are simple and efficient, their performance still lags behind the upper bound. Other approaches incorporate loss functions to train merging weights [41] or use adapter modules [40] during the merging phase. These techniques achieve impressive performance but are computationally expensive due to the need for secondary training.
In contrast to the aforementioned meth- ods, our study investigates the performance degradation of merged models from the per- spective of feature drift, i.e., differences in feature representations of the same sample caused by model merging- As illustrated in Figure aa), we observe a strong corre- lation between feature drift (measured by cosine distance) and the performance degra- dation associated with merging. Further- more, Figure lb) demonstrates that feature drift becomes more pronounced as network depth increases, as small perturbations in the initial layers are progressively ampli- fied through the layers of the network.
60-
Accuracy Decroase
Cosino Distunce
0.2
40
G 8
0.1
20
Data Points
0
Lincer Ft:
01
0,0
0.2
0.4
0.6
10
Cosine Distance
Blocks
(0)
(b)
Figure 1: An illustration of feature drift (measured the cosine distance between features extracted by task-specific expert models and the merged model) is presented using Task Arithmetic [12] on eight vision tasks. (a) The feature drift in the last layer shows linear correlation with the accuracy decrease, with data points collected for each task under varying merging coefficients. (b) Feature drift becomes more pronounced as the network depth increases.
These findings motivate our proposal of Layer-wise Optimal Task vector Merg- ing Merging, a method that explic- itly minimizes feature drift across layers. Specifically, LOT Merging uses squared error to measure the feature difference be- tween the merged model and task susk-specific models, which can be formulated as a convex quadratic optimization problem. Solving this problem analytically yields closed-fone solutions for parameters in both linear and normalization layers, enabling efficient model consolidation via basic matrix operations. Additionally, we provide an intuitive explanation of LOT Merging through theoretical analysis in two extreme cases, illustrating its ability to adaptively adjust the merging strategy based on task dependencies. In summary, this paper makes the following contributions:
We formulate the model merging task as a convex quadratic optimization problem by minimizing layer-wise feature drift, which enjoys a closed-form solution and such inspire an efficient model consolidation method referred to as LOT Merging.
We provide theoretical analysis of LOT Merging, revealing its resilient merging capabilities that effectively account for correlations among different task feature spaces.
We conduct extensive experiments on a variety of visual and vision-language benchmarks, showing LOT Merging's 's superiority over state-of state-of-the-arr art methods while maintaining robust- ness with limited exemplars.
2
Related Work
Model merging aims integrate the knowledge of multiple fine-tuned models into a single model. Early work primarily focused on weighted averaging strategies, such as employing the Fisher information matrix (20), or minimizing the feature distance between the merged model and individual models [13]. Recent research on multi-task model merging is typically built upon "task vectors" [12). defined as the parameter differences between the pre-trained model and each fine-tuned counterpart. Task Arithmetic [1] demonstrates that simple operations on task vectors- vectors-such as addition can be used to edit or merge knowledge effectively. Subsequent methods like Ties-Merging [8 and PCB.errgigg [5] improve upon this idea by pruning low-magnitude components from task vectors. Extensions to parameter-efficiee
fine-tuning settings include PEFT [42 and MoLE (35), which adapt Task Arithmetic to LoRA-based modules [10]. Additionally, some model merging methods rely on test-time training techniques. For instance, AdaMerging 44I learns a set of layer-wise merging coefficients via gradient descent; Surgery [40] introduces auxiliary adapter module to

---

align intermediate representations; WEMoE [32 and Twin Merging [18] adopt mixture-of-expeer frameworks and train router select among experts; and Localize-and-Stiee 7 constructs binary mask that determines which parameters to merge. Since the training requires constructing computation graphs to achieve gradient descent, the memory and computation overhead of such methods tend to be expensive when applied to large-scale foundation models.
This paper introduces layer-wise merging strategy for task vectors in parameter space, aiming to mitigate feature drift during model merging. By analyzing three parameter types in transformer architectures linear weights, scaling factors, and bias terms we derive closed-form solutions that enable efficient and principled integration. While our solution for linear weights parallels RegMean [T3]. our formulation operates on task vectors rather than original parameters, marking a fundamental distinction (see Section 5.2). Moreover, our method extends beyond RegMean by addressing additional parameter types and offering a deeper understanding of the mechanisms behind LOT merging. Compared to RegMean, our approach yields improvements of 10.9% (ViT-B/32) and 6.8% (ViT-L/144, and remains effective in dataless scenarios. Additionally, our method shares the goal of mitigating feature drift with Surgery [40] and CAT Merging [29], but differs in approach. Surgery introduces test-time alignment module, whereas our method is entirely training-free. Unlike CAT Merging, which constructs bases or masks to trim task vectors, we derive direct closed-form
solution for optimal merging, resulting in further performance improvements.
3 Preliminary
3.1 Model Merging
Problem setup. Consider a pre-trained model characterized L-layer neural network pre Weeee= where W pre represents the parameters of the 1-th layer. Starting from W pr: fine-tune independently on K down-stream tasks, resulting in task-specific expert models (WI.... Wx The objective is to merge these fine-tuned experts into single unified model Wmt without the need for redundant retraining.
Task Vector, To facilitate model merging, Task Arithmetic [2]
introduces the concept of task vector, defined as the parameter difference between the fine-tuned expert Wa. and Wpre:
T W W
Wree:
(1)
where the arithmetic operations over parameter sets are applied layer-by-layer, i.e., Wx pre {W M/ Task Arithmetic has demonstrated that performing simple arithmetic operations over task vectors can effectively integrate knowledge into the pre-trained model:
W mil
W pre
TE.
k=1
where the addition is performed layer-wise and À is manually defined scaling factor.
Resource limitations. In practice, is often infcasible to access the full downstream training sets, yet most existing merging techniques still depend some form of data-driven calibration. For instance, Fisher Merging [20]. TATR [30], and CAT Merging [2 require a subset of labeled examples to estimate parameter importance, while approaches such Task Arithmetic and Ties- Mees-ngg rely on a validation set to select hyperparameters. In addition, test-time adaptation methods require access to unlabeled examples at inference time to adjust the merging weights train additional modules. These methods typically require iterative backpropagation, making them impractical under constrained GPU resources to merge large-scale models. In contrast, our method is entirely training-free, incurs only a handful of forward passes with a small sample set.
3.2 Negative Transfer
One major challenge model merging is the negative transfer [3 30
which occurs when the
knowledge acquired by individual models conflicts or interferes with one another. Mathematically, negative transfer can be quantified by the degradation in performance across tasks resulting from the merging process. Let (-(-) denote the loss for task k, the negative transfer introduced by the merging
3

---

task vector T' (where for task arithmetic, T x Tx) can then be defined as follows:
ACs La (W pre + T) Lx (Wx).
However, analyzing this degradation is often challenging due the hierarchical structure inherent in deep neural networks. Instead, this work examines layer-wise negative transfer in the form of feature drift. Let f (W&) be the representation produced by the layer of the model W on the samples of task k. Then, the layer-specific feature drift caused is formulated as:
Afk fk (Wpre + T*) f/h (W&).
where W pre + T denotes the model parameters with only the 1-th layer merged, i.e., Wpee pee T (W1 pre:--.. Wi-1 W pre T W
pre .... Wp
pre
Sun al. [29] have derived that the feature drift of every layer contributes the overall negative
transfer, yielding the following upper bound of knowledge conflict:
L II 7m) |A I=1 m=1+1
ACC S
(S)
where La is assumed B-Lipschitz continuous with respect to the network's final output, and the function implemented by each layer is 7t-Lipschiz continuous with respect to its input (i.e., the output of layer within the merging region. For detailed proof, please refer to Section B
Method
Motivated by Eq. (5) our approach aims to mitigate knowledges conflicts by minimizing the feature drift || flll for each layer Specifically, we pursue an optimal shared task vector T by solving the following optimization problem:
K
Tdd arg mii T
arg min Σ | k=1
I + -
(6)
T
The computation of T depends on the specific form of f'(.). In transformer- based architectures, all such operations be categorized into the following three types:
Matrix multiplication, corresponding to the weight parameters of linear layers;
Element-wise (Hadamard) products, corresponding to the scaling factors in normalization layers;
Element-wise addition, corresponding to the bias parameters of linear layers and the shifting factors normalization layers.
In our analysis, treat each minimal computational unit independently. For example, the weight and bias components of a linear layer are considered as separate layers to facilitate analysis. Similarly, in complex modules such attention blocks, the computations of queries, keys, and values (QKV) are disentangled and analyzed as independent layers. It is also worth noting that the convolutional operation can be equivalently expressed as matrix multiplication (3I] Therefore, do not treat it as a separate category in our analysis. In the following, we discuss these three types of operations and present the closed-form solution r for each case.
4.1 Solution for Matrix Multiplication
Suppose W E xdd+ corresponds to the weight of a linear layer, which transforms features through matrix multiplication. Specifically, for task k given the input feature matrix X E g extracted by the first layers, the transformed feature representation is expressed as:
fL(W) X(W!

---

Substituting Eq.
into Eq. O6 yields the following objective:
T arg min T
X (W +
(W
(8)
K
arg min T
(2 - 1
arg min T
trace( - T2) x x -1)))
This defines a convex quadratic optimization problem. Consequently, the optimal solution T can be derived in closed form follows:
T"d
Σ x x/ Ext X
(9)
where denotes the Moore-Penrose inverse
99
Solution for Element- Wise (Hadamard) Multiplication
Next, consider the where W E R44 represents the scaling factors in normalization layer, i.e.,
((W) x W [..., o W ......".
(10)
where denotes the element-wise product and orf represents sample feature in X Under this setting, the objective can be formulated as
T²== arg min ΣΣ o pre +T*) of (W arg min
add (T
(11)
where ærdd] represents the d-th dimension of x. By setting the derivative of the objective with respect to T [d] to zero, we obtain the closed-form solution:
ΣK
k= Ent +k(d] Tfldd"T((a
7d [d]
(12)
k= En, s//dd
4.3 Solution for Element- Wise Addition
Suppose W Rd represents the bias coefficients, which are added element-wiss X
(()) = x + W [.... W "......]
(13)
Thus, the optimal solution can be derived as follows:
K
T arg min
Σ
l (W!l pre +T + W2
arg min
I
1
T
T
K
1=1
(14)
Now, we have derived the optimal vectors for all three types of operation. As summarized in Algorithm we first perform forward pass over small exemplar set (16-64 samples per task) to extract each layer's input features for every expert. After obtaining I*t we integrate it into the pre-trained parameters using predefined weight A:
Wlot Wo + XI" md
(15)
Empirically, setting À = 1 already yields competitive performance. However, further tuning of À on a validation set, similar to the approach Task Arithmetic, can lead to additional improvements (see Section 6.3 for a detailed sensitivity analysis).

---

Discussion
5.1 Exploring the Mechanism LOT Merging
We present theoretical analysis of the LOT Merging mechanism, focusing the behavior of merging matrix- maatiiiiuuiiliiaaiionn parameters. Specifically, we apply singular value decomposition (SVD) to the input feature matrices at given layer I, where the representation for task is given by X = V To gain insight into the behavior of LOT Merging, we analyze two extreme cases: (1) the ideal case in which task-specific features are mutually orthogonal, and (2) the worst case where task-specific features are fully collinear.
V.
Ideal case. Assume that for any pair of distinct tasks k + j. their right singular vectors satisfy the orthogonality condition V V = 0. Under this condition, the optimal solution in Eq. ( simplifies to summation of the task vectors:
T ideal Σ 1
Σ
VI V
V
- E(v
v2 VV M VIVE''
(16)
Here, the term V T corresponds to a projection of T onto the subspace spanned by the singular vectors Vg This projection retains only the components T aligned with the corresponding feature space X effectively filtering out irrelevant directions (3I]. layer-wise feature drift:
As result, LOT Merging introduces no
X (T ideal - 1221
Σ
M VVV='' -
121
(17)
K
Σ
u vgvi up THl =0
Worst case. Now assume the opposite extreme condition that all task features share common group of singular vectors, i.c., V/ = VL, Vk. Under this condition, the optimal solution in Eq. becomes a weighted average within the shared subspace spanned by yl:
T worst
Σ
Σ r' VI TM
(18)
1 E
Vl
xx
V
p 2
p
T
Σ
V
Σ Σ
pl ViTT!
Normalized Weight
While feature drift is unavoidable in this setting due to collinearity, the formulation ensures that the deviation is minimized through weighted averaging.
These two extreme cases highlight the adaptability of LOT Merging. In realistic settings where task representations are partially aligned, LOT Merging balances task specificity with shared structure, projecting task-specific transformations into relevant subspaces while aggregating shared patterns. This fiexibility allows it to perform robustly across range of multi-task learning scenarios.
5.2 Task Vector Merging vs. Parameter Merging
In this work, we aim to derive an optimal task vector for merging. A natural question arises: why not directly solve for the optimal merging parameters as done in ?13] ? To illustrate this, consider the direct merging of linear weights via the following objective:

---

Table 1: Multi-task performance when merging VIT-B/32 models eight vision tasks. The best performance among training-fiee free methods is highlighted with bold. The "#best" column represents the number of datasets where the training-free method performs the best.
Method
SUN397
RESISC4S
EaroSAT
SVHN
GTSRB
MNIST
Avg
#hest
Basic baseline methods
Pre-trained
62.3
59.7
60.7
45.5
31.4
32,6
48.5
43.8
48.0
Individual
77.7
97.5
98.7
79.4
90.5
Traditional MTI.
73.9
74.4
93.9
98.2
95.8
98.9
99.5
77.9
88.9
Training free methods
Weight Averaging
65.3
63.4
71.4
71.7
64.2
52.8
87.5
50.1
65.8
Fisher Merging
68.6
69.2
70.7
66.4
72.9
51.1
87.9
59.9
68.3
2
RegMean
Task Aritmetic
55.2
54.9
66.7
78.9
80.2
69.7
97.3
50.4
69.1
Ties-Merging
59.8
58.6
70.7
79.7
86.2
72.1
98.3
54.2
72.4
TATR
62.7
59.3
72.3
82.3
80.5
72.6
97.0
55.4
72.8
Ties Merging & TATR
66.3
65.9
75.9
79.4
79.9
68.1
96.2
54.8
73.3
Consensus Merging
65.7
63.6
77.2
81.7
70.3
97.0
57.1
73.6
AWD Merging
63.5
61.9
72.6
84.9
85.1
79.1
98.1
56.7
75.2
PCB Merging
62.0
87.5
78.5
58.4
CAT Merging
68.1
65.4
80.5
89.5
85.5
78.5
98.6
60.7
78.3
Merging (ours)
67.5
93.4
98.7
63.6
(xxt x Σx xtW
W arg min W
[ W -x
(19)
While Eq. 19 provides straightforward solution, it implicitly modifies the knowledge contained in the pre-trained weights. To make this explicit, we establish connection with our proposed T'":
Wi
Σ x x Σ
+
Σ x Σ
x
X
(20)
k
Modifying Pre-trained Knowlodge
T
This decomposition highlights a critical issue: when X is meinkeeficient-wwaiee often occurs practice due to lim- ited exemplar data-the projection term tends to discard useful pre-trained knowledge. The resulting distortion can lead to catastrophic forgetting [6] of previously learned
Avg Ace Decrease (%))
Merg. Task vector
40
Modi. Pre-trained Weight
20-
representations. To quantify this effect, we compare the accuracy of merged models using W against individual expert models under varying exemplar budgets. As shown
Figure 2 performance degrades significantly as the number of exemplars decreases, where the majority of this degradation stems from the alteration of the pre-trained weights.
16
32
64
Number of exemplars
Performance (Avg ACC %) degradation of direct parameter merg- ing compared to Individual under differ- ent numbers of exemplars when merging ViT-L/14 models. The error is decom- posed into the impact of merging task vectors (red) and modifying pre-trained weights (orange).
Figure 2:
Consequently, methods based on Eq. (19). such [13] typically require large exemplar sets (e.g., 1600 samples per task) maintain competitive performance [7]. In con- trast, our method achieves superior accuracy with as few as 16 to 64 exemplars per task. This substantial reduc- tion in data requirement highlights the effectiveness of our merging strategy in preserving pre-trained knowledge while flexibly adapting to new tasks.

---

Table 2: Multi-task performance when merging ViT-L/14 models eight vision tasks.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DID
Avg
#best
Basic baseline methods
Pre-trained
66.8
77.7
71.0
39.9
58.4
50.5
76.3
53.3
64.5
Individual
82.3
92.4
97.4
100.0
98.1
99.2
99.7
84.1
Traditional MTL
80.8
90.6
96.3
96.3
97.6
99.1
99.6
84.4
93.5
Training free methody
Weight Averaging
72.1
81.6
82.6
91.9
78.2
70.7
62.8
79.6
Fisher Merging
69.2
88.6
87.5
93.5
80.6
74,8
93.3
70.0
82.2
RegMean
81.8
88.0
84.2
98.
83.7
Task Arithetic
73.9
82.1
86.6
94.1
87.9
86,7
98.9
65.6
84.5
Ties- Merging
95.7
90.3
83.3
99
TATR
74.6
83.7
87.6
93.7
88.6
88.1
99.0
66.8
85.3
Ties- Ties-Merging TAAR
76.3
85.3
83.8
94.4
90.8
88.7
99.2
68.8
86.5
Consersus Merging
75.0
84.3
89.4
95.6
88.3
82.4
98.9
68.0
85.2
AWD Merging
76.2
85.4
88.7
96.1
92.4
92.3
99.3
69.4
87.5
PCB Merging
86.0
95.9
89,9
92.3
71.4
87.6
CAT Merging
78.7
88.5
91.1
96.3
91.3
95.7
99.4
75.7
89.6
Merging (ours)
98.7
97.1
95.7
99.5
76.4
Table 3: Multi-task performance when merging BLIP models on six vision-language tasks.
Method
COCO Caption
Flickr30k Caption
Textcaps
OKVQA
TexaVQa
ScienceQA
#best
Metrie
CIDEr
CIDEr
CIDEr
Accuracy
Accuracy
Accuracy
Pre-trained
0.07
0.03
0.05
42.80
21.08
40.50
Individual
1.17
0.65
50.84
29.79
76.89
Task Arithmetic
0.86
0.50
0.39
17.71
0.49
40.10
Ties-Merging
0.53
0.27
0.22
27.95
0.57
40.35
TATR
0.46
0.31
0.21
28.30
14.74
42.98
PCB Merging
0.71
0.52
0.30
36.04
1.88
43.01
Merging
19.69
46.36
LOT Merging (ours)
0.91
0.54
0.44
38.35
20.82
48.24
Experiments
6.1 Settings
Benchmarks.
Our experiments cover vision and vision-language tasks. For the vision tasks, we
follow [12] and utilize eight image classification datasets: SUN397 [36], Cars (14), RESISC45 (2).
EuroSAT [ SVHN (22]. GTSRB (28). MNIST [33,, and DTD 3]. For the lision-language tasks,
focus three captioning datasets (COCO Caption (I], Flickr30k Caption 233. Textcaps (26))
and three Visual Question Answering (VQA) datasets (OKVQA [9), TextVQA 27 and ScienceQA
Baselines. We select several training-free ffrc model merging methods as the primary comparison
baselines, including weight averaging, Fisher Merging [20], RegMean [3, Task Arithmetic [22]
Ties-Merging (38) TATR 30). Ties-Merging & TATR [30 Consensus Merging 34), AWD Merging
37. PCB Merging 55 and CAT Merging (29]. We also present three baseline methods for reference: Pre-trained model performance, Individual fine-tuned model performance, and Traditional Multi- Task
Learning (MTL) performance.
Metrics. For both classification and VQA tasks, we employ accuracy as the evaluation metric. For captioning tasks, we select CIDEr the evaluation criterion. To minimize potential performance fluctuations arising from the selection different exemplar sets, we rerun LOT Merging with three randomly selected exemplar sets and report the average performance.
Implementation details. For vision tasks, following [2], utilize the vision encoder of CLIP 24 as the pre-trained model, including both ViT-B/32 and ViT-L/14 versions. The task vectors obtained from the official repository of [12] When merging vision-language tasks, task vectors are generated by fine-tuning the VQA version of BLIP (25) for 6000 steps per task. The BLIP architecture consists of an image encoder, text encoder, and text decoder, with all model weights fine-tuned during training. Further details of the experiments can be found in our supplementary appendix and code. All exemplar sets are randomly sampled from the training set and are kept strictly separate from the test set used for evaluation.

---

6.2 Comparison Results
Merging ViT-B/32 models on vision tasks. We first compare multi-tass performance when merging ViT-B/32 models eight vision tasks (Table I.) Among the training-free frree methods, Fisher Merging achieves the highest accuracy on SUN397 and Cars, while our approach performs the best on all the remaining six tasks. Notably, our approach achieves the highest average accuracy of 82.7%, suppressing the second-best training-free methods with 4.4%.
Merging ViT-L/14 models on vision tasks. Next, we evaluate the multi-task performance when merging ViT-L/14 models on eight vision tasks (Table 2). Our method also achieves the highest average accuracy of 90.5%, higher than the second-best performance by 0.9%. Furthermore, our method performs best seven tasks, confirming its robustness.
Merging vision-language tasks. Finally, we compare multi-task performance across six vision- language tasks when merging BLIP models (Table Our method leads the performance with the highest CIDEr scores on COCO Caption, Flickr30k Caption, and Textcaps, as well as the highest accuracy on TextVQA and ScienceQA. total, our method delivers the best performance in five tasks, showcasing its adaptability in vision-language tasks.
6.3 Sensitivity Analysis
Sensitivity analysis of exemplar ber. The number of exemplars impacts LOT Merging by influencing the accuracy of x x As shown in Figure (a), the performance of LOT Merging steadily improves as the number of exemplars in- creases. Notably, when the number of sam- ples reaches only 16 per task, performance stabilizes with respect to the exemplar num- ber. This demonstrates the robustness of LOT Merging in data-less scenarios (c.g., 16 samples per task). Based on our exper- iments, we empirically recommend using 64 samples per task.
Avg Ace (%)
80
ViT-B/32
ViT-L-14
16
32
64
(a) Number exemplars
Avg Avg Ace (%)
90
VIT-B/32
1.0 1.1 1.2 1. 1.4
1.5 1.6 (b)
1.7 1.8 1.9
2.0
Sensitivity analysis of À (scaling factor in Eq. (15)). The scaling factor controls the contribution of task vectors to the merged model. As illustrated in Figure 3 (b), per- formance remains relatively stable for A values in the range of 1.0 to 1.5. For larger values of A. performance gradually decreases. Based on our empirical analysis, we set À 1.2 for merging ViT-B/32 and 1.5 for merging ViT-L/14.
Figure Average accuracy (% on eight vision tasks with various numbers of exemplars (a) and X (scaling factor Eq. (5)).
Conlusion
In this paper, we propose Layer-wise Optimal Task Vector Merging (LOT Merging), a novel approach to mitigate the problem of feature drift in model merging. By adaptively merging task-specific knowledge at the layer level, LOT Merging reorganizes the merging process as an optimization task that minimizes the discrepancy between the representations of individual models before and after merging. This adaptive strategy ensures that task task-specific information retained while minimizing interference across tasks. LOT Merging requires no retraining and is suitable for scenarios with limited samples, making it highly efficient and applicable in resource-come cooourceconsttrinee environments. Our experiments both vision and llaguaele benchmarks demonstrate that LOT Merging significantly outperforms baseline methods, achieving up to 4.4% improvement on vision tasks and showing strong performance across various vasion-language tasks.
While LOT Merging achieves strong empirical results with minimal data, still requires access to small exemplar set, which may not be feasible in strictly data-free scenarios. In addition,

---

although the method supports mainstream transformer architectures, extending it to models with more complex operations (e.g.. layers involving exponential functions) may require additional adaptation. Future work could explore how feature drift interacts across layers to better disentangle conflicting knowledge and broaden the applicability of the approach.
Acknowledgments
This work supported, in part. by the Fundamental Research Funds for the Central Universi- ties under Grants 2025JBRC004, 2023JBZY037, and 2022JBQY007, by the Beijing Natural Sci- ence Foundation under Grant L231019, by the Hebei Natural Science Foundation under Grant F2025106045, by the National Natural Science Foundation of China under Grants 62276019, 62306028, 62501043, and U22B2004, by the Shenzhen Science and Technology Program Project under Grant 10Z2D202409031022222555 by the RIE2025 Industry Alignment Fund - Industry Col- laboration Projects (IAF-ICP) (Award 12301 administered by A*STAR, by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL), and by the Nanyang Associate Professorship and National Research Foundation Fellowship (NRFFI3-2021- 0006), Singapore.
References
[1] Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft captions: Data collection and evaluation server. arXiv preprint 2015.
[2] Gong Cheng, Junwei Han, and Xiaoqjang Lu. Remote sensing image scene classification: Benehmark and state of the art. Proceedings the IEEÉ, 185(10):1855-1883, 2017.
[3] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE Conference Computer Vision and Pattern Recognition. June 2014.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers image recognition seale. In International Conference on Learning Representations, 2021.
Guodong DU, Junlin Lee, Jing Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing Hc, and Min Zhang. Parameter competition balancing for model merging. In Advances in Neural Information Processing Systems, 2024.
[6] lan Goodfellow, Mchdi Mirza, Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 1rXiv::312..2111, 2013.
[7] Yifei He, Yuzheng Hu, Yong Lin. Tong Zhang, and Han Zhao. Localize-and- Efficient model merging via sparse task arithmetic. Transactions on Machine Learning Research, 2025.
[8] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217 2226,226622222222 2019.
[9] Roger Horn and Charles Johnson. Marrix Analysis. Cambridge university press, 2012.
[10] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen -hu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.
[11] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high-performance model merging. In Advances in Neural Information Processing Systems, volume 37, pages 122741-122769, 2024.
[12] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference Learning Representations. 2023.
[13] Xisen Jin, Xiang Ren, Daniel Preotiuc-Piette and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations, 2023,
10

---

[14] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graines categorization. In IEEE International Conference Computer Vision Workshops, June 2013.
[15] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
[16] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
[17] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, 2022.
[18] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. In Advances in Neural Information Processing Systems, 2024.
[19] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozheh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[20] Michael S Matena and Colin Raffel. Merging models with fisher weightee averaging. In Advances in Neural Information Processing Systems, volume 35, pages 17703-17714 2022.
[21] Niklas Muennighoff, Nouamanc Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Association for Computational Linguisticx, pages 2014 2014-2037, May.
[22] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4, 2011.
[23] Bryan A. Plummer. Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer maage-to-sentence models. In IEEE International Conference on Computer Vision, December 2015.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, volume 139, pages 9748-8763. 18-24 Jul 2021.
[25] Yujun Shi, Li Yuan, Yunpeng Chen, and Jiashi Feng. Continual learning via bit-level information preserving. In IEEE Conference Computer Vision and Pattern Recognition, pages 16674 16664,,66,,, June 2021.
[26] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: A dataset for image captioning with reading comprehension. In European Conference Computer Vision, pages 742-758. 2020.
[27] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang. Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqu models that can read. In IEFE Conference on Computer Vision and Pattern Recognition, June 2019.
[28] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. In International Joint Conference on Neural Networks, pages 1453 1460, 2011.
[29] Wenju Sun, Qingyong Li, Yangli-ao Geng, and Boyang Li. Cat merging: A training-free approach for resolving conflicts in model merging. International Conference on Machine Learning, 2025.
[30] Wenju Sun, Qingyong Li. Wen Wang, Yangli ao Geng, and Boyang Li. Task arithmetic in trust region: A training- free model merging approach navigate knowledge conflicts. arXiv preprint 1rXiv 2025.
[31] Wenju Sun, Qingyong Li, Jing Zhang, Wen Wang, and Yangli-ao Geng. Decoupling learning and remembering: A bilevel memory rramework with knowledge projection for task-incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, June 2023.
[32] Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi task models via weight-ensembling mixture of experts. In International Conference Machine Learning, 2024.

---

[33] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. 2019.
[34] Ke Wang, Nikolaos Dimitriadis, Guillemmo frtiz-Jimenez, François Fleuret, and Pascal Frossard. Local- izing task information for improved model merging and compression. International Conference on Machine Learning, 2024.
[35] Xun Wu, Shaohan Huang, and Furu Wei. Mixture IoRA experts. In International Conference on Learning Representations, 2024.
[36] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba. Aude Oliva. Sun database: Exploring large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016.
[37] Feng Xiong, Runxi Cheng, Wang Chen, Zhanqiu Zhang, Yiwen Guo, Chun Yuan, and Ruifeng Xu. Multi-task model merging via adaptive weight disentanglement. arXiv preprint arXiv:2411.18729 2024.
[38] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems, 2023.
[39] Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338 2025.
[40] Enneng Yang, Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, Dacheng Tao. Representation surgery for multi-task model merging. In International Conference on Machine Learning, 2024.
[41] Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference Leurning Representations, 2024.
[42] Jinghan Zhang, shiqi chen, Junteng Liu, and Junxian He. Composing parameter-efficien modules with arithmetic operation. In Advances in Neural Information Processing Systems, volume 36, pages 12589- 12610, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction accurately represent the paper's core contributions, including the motivation behind reducing feature drift, the intro- duction of LOT Merging, along with detailed board analysis and experimental evaluations.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitations of the proposed method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: All key formulas in the paper are numbered and accompanied by appropriate derivations explanations. Formal proofs are provided where necessary, such the upper bound derivation for feature drift in Section
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section 6. and Section C comprehensively describe the experimental configu- rations, while the code is made available in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The source code for all experiments is included in the supplemental materials. All datasets used are well-established benchmarks that are publicly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
publlc//giide//Cdelebbbiiss wwt.aaammmiiioooooooooooooooeiie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //ips.cc/puut pctpss/cccclllcgguudeeeese guides wwtp/oeeeiiiiiiinnnooooooooo Ppllccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Section 6.1 and Section C comprehensively describe the experimental configu- rations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section D.5 provide results with random runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section C.I details the computational resources employed in experiments,
while Section D.6 presents a comprehensive analysis of execution time complexity.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The authors have read the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper advances machine learning methodology through theoretical inno- vation, with experimental validation conducted on established public datasets. The research focuses algorithmic improvements rather than direct societal applications.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles. surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Our work develops machine learning methodologies that architecture- agnostic and evaluated on conventional datasets. As we do not release pretrained models, scraped datasets, generative systems, the question of misuse potential does not directly apply to the current contribution.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our paper provides comprehensive citations of related work in the field.
The accompanying source code will be released in full compliance with all applicable open-source licenses, with clear attribution given to any incorporated external code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes)
Justification:
We will follow these instructions.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: An LLM was used only for grammar and wording improvements. It played no role in the research methodology or scientific contributions.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy hbtpgee//eaarrrpeccc/////ffeeere for what should or should not be described.
19

---

A Pseudo Code
Algorithm Noutinnes the proposed LOT merging process. Given a pre-trained model and set of task vectors, we first extract input features from exemplar sets using task-specific adapters. Then, for each layer, we compute the optimal merged task vector by minimizing the discrepancy between the adapted and reference features. The final model is obtained by linearly combining the pre-trained weights with the optimized task vectors.
Algorithm 1: The model merging process
Input: Pre-trained model W Task vectors (Ti..... Tk]: Exemplar set (M...... Mk}
Output: Merged model Wilo
T {T
T // Initial
// Collect the input features
for k == 1 to K do
Initialize task inputs: X = M2 do
I
L x = [(x; W + )
// Compute optimal task vector for each layer for | to I. do // Refer to
14
T arg milly - (W + T²) - (W pro + 1
// Merging
W mtl
W pre + XI"
return Weltt
B Proof of Eq. 5
Let Wt the parameters for task k, and T task vector. The feature dirt at layer caused by T; is defined as:
At = (W pre +T!) (.W)).
We aim to prove the following bound:
AC S
Σ II 7m) |لlMl
(21)
where C is assumed 3-Lipschitz continuous with respect to the network's final output, and each layer 1 is Lipschhi continuous with respect to its input (i.e., the output of layer 1 - 1) within the merging region.
Proof.
The feature shift at layer I can be decomposed as:
fW pre +T) - fl((W)) 1 (Wpre + T);; W Wore + T²) - -
SAC (( + T); W Wore + TD) - (( W + +T) (22)
fN((Ft'((a)) Wpre Th) - (
(WW): W.).
Under the norm form-induced triangular incquality, we have:
(W pre T) flWll < -W pre T): W pre T (((ff---WWa) W pre + T'
+ ((L(t-((W++:: pre Th) - (WWe)
(23)
20

---

For the first part, we apply the -ipisschitz continuity of fk::
II (W +): W pre - (W&); W pre +T salls
- (Wpre+T) (17T)))--(WW)
For the second part:
( (Wi):: Wpre + +T*) fll -
Combining these two terms, have:
( pre + T) fl(W))l S f Unfolding this recursive inequality from to 1 and accumulating the error gives:
( (Wpre pre + - ft +
Il (Wpre pre + T) (WWIl & Σ II 7m IIA
(24)
mmi+1
Using the assumption that &. is B-Lipschitz continuous with respect the network's final output, conclude:
S 3 Σ II 7m)
(25)
[=1 m=i+1
C
Experimental Setup
This section provides an overview of the experimental setup, including details about the computational environment, datasets, and the baseline models employed in the experiments.
C.1 Computational Resources
All experiments described this paper were performed on workstation running Ubuntu 16.04. The system configuration includes dual Intel Xeon 2.60GHz CPUs, 256 GB of RAM, and six NVIDIA RTX 3090 GPUs. The code was implemented in Python 3.8 and executed on this hardware platform to ensure consistency across all experiments.
C.2 Datasets
Our experimental procedure follows the guidelines outlined Task Arithmetic 112 utilizing eight commonly used image classification datasets, which are summarized below:
SUN397 [3]]
A large-scale dataset comprising 108,754 images, organized into 397 cate- gories. Each category contains minimum of 100 images, making this dataset comprehen- sive benchmark for scene classification tasks.
Stanford Cars [14] fine-grained dataset with 16, 185 images of 196 distinct car models. The dataset is split evenly into training and testing sets, providing reliable resource for evaluating car model recognition systems.
RESISC45 ? This remote sensing dataset consists of 31,500 images representing 45 different scene categories. Each category contains around 700 images, covering broad range of geographical and structural themes.
EuroSAT [8] A satellite image dataset containing 27,000 labeled and geo-referenced images. The dataset is divided into categories, including forests, urban areas, and agricultural fields, designed for land-use classification tasks.
SVHN 22 This dataset, derived from real-world street view images, consists of 73,257 training and 26,032 test images of digits, distributed across 10 classes. An additional 531,131 samples are included for extended training purposes.
21

---

The German Traffic Sign Recognition Benchmark, which includes over 50,000 images across traffic sign categories. This dataset is well-known benchmark for traffic sign recognition systems.
GTSRB [28]:
MNIST [5 A foundational dataset for handwritten digit classification, consisting of 60,000 training images and 10,000 test images, distributed across 10 digit classes. DTD B
A dataset designed for texture classification, containing 5,640 images across 47 categories, with approximately 120 images per category. is commonly used for evaluating texture recognition algorithms.
In addition, we also utilize six Assionlanguagg datasets, which are detailed below:
COCO Caption M: A large image captioning dataset derived from the MS COCO collec- tion. It contains over 330,000 images, each annotated with five different captions, aimed at training models to generate natural language descriptions for images.
Flickr30k Caption 23 This dataset consists of 31 1,000 images from Flickr, with each image paired with five descriptive sentences. It is used for both image captioning and retrieval tasks.
TextCaps [26]: A challenging dataset for image captioning, where the captions require reasoning over both visual and textual information present within the image. The dataset includes 145,000 image-caption pairs.
OKVQA 19 : A knowledge-barc based visual question answering dataset that includes over 14,000 questions requiring external knowledge to answer. The dataset is designed to assess reasoning beyond visual content alone.
visual question answering dataset that emphasizes reading and interpreting text embedded within images. It contains over 45,000 questions across 28,000 images, necessitating both visual and textual reasoning.
Te [7
ScienceQA 17
A multi-modal dataset for scientific question answering, consisting of over 21 1,000 multiple-choice questions paired with images and textual explanations across various scientific disciplines, including biology, chemistry, and physics.
C.3 Baseline Methods
In this study, we compare our approach with several baseline methods. Below, we provide description of each baseline:
Pre-trained: This bascline uses a pre-trained model to perform tasks across multiple domains. Since it does not leverage any task-specific fine-tuning, it typically results in suboptimal performance on downstream tasks.
Individual: In this method, separate models are fine-tuned for each task individually. Although this avoids task interference, it is limited by the inability to perform multi-task learning simultaneously. represents reference for the best possible performance, or upper bound, for merging methods.
Traditional MTL: This approach combines the training data from all tasks and trains a single multi-task model. It serves traditional method for joint task learning.
The following are compression-bes based approaches:
EMR Merging [1]: This method applies lightweight task-specific masks and rescalers compress task vectors.
WEMOE 32 Upscales MLP layers into input-dependent Mixture-of -foeturr-of-Eppertt modules to dynamically integrate shared and task-specific knowledge during model merging.
The following are test-time training- based approaches:
AdaMerging 41 This method adapts the model to new tasks by using an unlabeled test
set to learn adaptive merging coefficients both the task- and layer-level, as applied Task Arithmetic.
22

---

AdaMerging++ 41 An improved version of AdaMerging, which integrates the principles further enhance the adaptive merging process.
of Ties-nerging 38
Surgery :40: Surgery introduces feature transformation module that aligns the features from different tasks during the merging process. In our experiments, we utilize the basic version of Surgery alongside Task Arithmetic for evaluation.
Localize-and-Stice 7 This method optimally combines the strengths of several fine-tuned models by identifying and localizing essential regions within each model before merging.
The following methods do not require training:
Weight Averaging: This technique averages the parameters from the models of different tasks to form single multi-task model without any additional training steps.
Fisher Merging 20 This approach uses the Fisher information matrix to assess the relative importance of model parameters, merging them based on their significance.
RegMean RegMean adjusts and combines rows from weight matrices based on
statistical information gathered from the training data, refining the model's parameters.
Task Arithmetic [12]
Task Arithmetic introduces the concept of "task vector," which is defined as the difference between fine-tuned model parameters and pre-trained model parameters. By combining multiple task vectors and adding them to the pre-trained model, it enables multi-task learning.
Ties-Merging removes insignificant parameters from the task vectors and resolves sign conflicts, reducing interference during the merging of task vectors.
Ties-Merging [8]
TATR 30): TATR improves upon Task Arithmetic by restricting the merging of task vectors to a defined trust region, which reduces knowledge conflicts between tasks.
TATR & Ties-Merging [30 38]: This method combines the trust region restriction of TATR with Ties-Merging to further enhance task vector merging.
Consensus Merging 34 This method computes a of masks for each task vector to
minimize the distance between the merged model and each fine-tuned model in the parameter
space.
AWD Merging 37:
AWD Merging generates redundant vectors such that subtracting them from the original task vectors leads to increased orthogonality in the remaining vectors.
PCB Merging 5 PCB Merging trims components of the task vector that have small magnitudes and are not strongly correlated with other tasks.
CAT Merging 29 CAT Merging trims components (using projection or mask) of the task vector that may cause knowledge conflicts.
D Additional Experiments
D.1 Additional Performance Comparision
In this section, we compare LOT Merging with both compression-based methods and test-time adaptation-base approaches. Compression-based methods rely on techniques such as masking (I]) or singular value decomposition (SVD) 32 to compress task vectors, and typically require manually specified or trained routers during inference to select the appropriate task vector. On the other hand, test-time adaptation-based methods leverage unlabeled test data to train merging weights certain modules, which introduces additional computational overhead and data requirements.
It is important to note that comparing LOT Merging with these methods is inherently unfair, as LOT Merging is training-free approach and does not require any adaptation or auxiliary modules at inference time. Moreover, the merged model of LOT Merging retains the original network architecture without introducing any structural modifications. As shown Table 4 while LOT Merging lags behind comppression-based methods in terms of average accuracy, it achieves comparable or even superior performance to several representative test-time adaptation approaches. This highlights the strength of LOT Merging in competitive performance across diverse vision tasks.
23

---

Table 4: Multi-task performance when merging eight vision tasks. Results of Localize-and-Sie with "F" stem from the original paper, where only the performance on ViT-B/32 is provided.
Method
SUN399
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Ace
VII-B/32
Compression based methods
WEMOE EMR-Merging
74.1
77.4
93.7
99.1
96.2
98.9
99.6
76.4
89.4
75.2
72.8
93.5
99.5
96.9
98.1
99.6
74,4
88.7
Test-time Adaption hased methods
TW AdaMerging
58.0
53.2
68.8
85.7
81.1
84,4
92.4
44.8
71.1
TW AdaMerging++
60.8
$6.9
73.1
83.4
87.3
82.4
95.7
50.1
73.7
AdaMerging
64.5
79.2
93.8
87.0
91.9
97.5
80.1
LW AdaMerging++
68.3
82.2
94.2
89.6
89.0
81.1
Surgery Merging
63.8
59.9
83.3
97.9
87.0
87.0
98.6
69.4
80.9
Localize-and S
67.2
68.3
81.8
89.4
87.9
86.6
94.8
62.9
79.9
Training-free free methods
LOT Merging (ours)
67.7
67.5
85.7
94.9
93.4
89.8
98.7
63.6
82.7
ViT-L/14
Compression based methods
WEMOE
81.4
92.6
95.4
99.4
97.7
99.3
99.7
83.7
93.6
EMR- EMR-Mcrging
83.2
90.7
96.8
99.7
97.9
99.1
99.7
82.7
93.7
Test-time Adaption based methods
AdaMcrging
79.0
90.3
90.8
96.2
93.4
98.0
99.0
79.9
90.8
AdaMerging++
79.4
90.3
91.6
97.4
93.4
97.5
99.0
79.2
91.0
Surgery Merging
75.7
84.4
93.1
98.8
91.3
93.4
76.1
89.0
Training- free methods
LOT Merging (ours)
88.6
91.7
98.7
97.1
95.7
76.4
90.5
Table 5: Multi-task performance when merging ViT-B/16 models on eight tasks.
Method
SUN397
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
Avg Acc
Pre-trained
63.8
65.7
54.5
52.0
43.3
51.7
45.1
55.0
Individual
81.8
86.8
96.9
99.7
97.8
99.1
99.7
82.0
92.9
Weight Averaging
67.7
70.0
75.3
79.5
74.9
60.1
94.4
43.8
70.7
Fisher Merging
68.5
75.2
80.4
73.2
61.2
94.5
50.7
71.7
RegMean
69.1
71.6
77.6
88.8
83.7
70.2
96.9
54.6
76.6
Task Arithmetic
61.1
65.9
74.0
76.2
88.0
73.9
98.4
53.0
73.8
Ties-Merging
69.1
72.5
80.5
84.0
85.0
71.5
98.1
$4.9
77.0
77.9
81.7
87.6
77.2
98.3
77.0
AWD Merging
67.8
72.7
78.7
88.5
83.6
98.9
57.1
79.8
CAT Merging
72.9
75.9
83.1
92.8
88.2
82.7
98.8
62.7
82.1
LOT Merging (ours)
71.0
76.2
87.6
95.8
96,5
91.9
99.2
67.0
85.7
D.2 Comparison on VIT-B/16
Table 5 presents the multi-task performance of various model merging techniques on eight diverse tasks using ViT-B/16. As can be seen, LOT Merging outperforms all baselines, achieving the highest average accuracy (85.7%). Notably, excels in almost all tasks, demonstrating ability to balance task-specific feature retention and shared information utilization. These results highlight the effectiveness of LOT Merging multi-task learning scenarios.
D.3 Analysis of Exemplar Number
presens the multi tast performance of LOT Merging with varying numbers of exemplars. The results are shown for both the VIT-B/32 and ViT-L/14 models with different numbers of exemplars used for merging, ranging from 4 to 64. As the number exemplars increases, we observe consistent improvement in performance across all datasets for both ViT architectures. Notably, LOT Merging already achieves state-of-the-arr performance with only exemplars per task, demonstrating that LOT Merging is particularly well-suited for dataless adaptation settings.

---

Table 6: Multi-task performance of LOT Merging with various numbers of exemplars. The "#exem- plar" column represents the number of exemplars used for merging.
#exemplar
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
VIT-B/32
4
64.2
64.3
69.1
84.4
88.6
77.9
96.5
51.1
74.5
8
65.4
66.3
79,4
87.6
92.8
85,5
98.2
54.1
78.7
16
66.4
67.1
82.6
90.9
91.7
86.
98.5
58.1
80.2
32
67.4
67.5
83.3
92.4
93.2
89.6
98.6
60.6
81.6
64
67.7
67.5
85.7
94.9
93.4
89.8
98.7
63.6
82.7
VIT-L/14
4
87.5
86.6
92.6
95.3
99.1
66.5
86.7
8
75.4
88.4
95.9
96.4
93.9
68.3
88.2
16
75.7
89.0
98.2
96.6
94.0
71.3
89.0
32
91.2
98.3
97.0
74.1
90.0
64
76.7
91.7
98.7
97.1
95.7
99.5
76.4
90.5
Table 7: Ablation Study of LOT Merging.
Linear Weight
Scaler
Bias
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
VET-B/32
61.9
57.4
60.4
36.2
30,0
54.3
42.3
49.8
67.8
67.9
85.9
92.6
93.3
87.5
98.7
62.0
81.9
67.9
84.6
64.0
67.7
67.5
85.7
94.9
93.4
89.8
98.7
63.6
82.7
VIT-1/14
67.3
78.5
72.3
64.6
59.5
50.4
76.7
55.6
65.6
76.7
88.4
92.4
98.4
97.0
94.8
99.5
75.3
90.3
88.3
99.5
75.3
76.7
88.6
91.7
98.7
97.1
95.7
99.5
76.4
90.5
D.4 Ablation Study
In this ablation study, investigate the impact of merging pre- pre-trained and learned parameters the performance of Vision Transformers (ViT). Specifically, we examine three types parameters: linear weight, scaling factors, and bias coefficient. For each type, replace the merged parameters with the pre-trained ones and evaluate the effect on model performance.
The results, shown in Table 7 demonstrate that merging each type of parameter leads to consistent improvements across the board. Among the three parameter types, merging the Linear Weight yields the largest performance boost, represents the majority of the model's parameters.
Table 8: Multi-task performance of LOT Merging with three random exemplar sets.
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Ace
VIT-B/32
Random exemplar set
85.3
98.7
63.9
82.6
Random exemplar set
67.6
67.5
85.8
94.6
93.7
89.2
98.6
63,9
82.6
Random exemplar set
67.7
67.7
86.1
95.6
93.3
90.5
98.7
63.1
82.8
Average
67.7
67.5
85.7
94.9
93.4
89.8
98.7
63.6
82.7
VIT-L/14
Random exemplar set
91.5
98.7
97.0
95.7
99.6
76.3
90.5
Random exemplar set
76.8
88.3
91.6
98.8
97.1
95.5
99.5
76.5
90.5
Random exemplar set
76.6
88.6
91.9
98.4
97.0
95.9
99.4
76.3
90.5
Average 76.7
88.6
91.7
98.7
97.1
95.7
99.5
76.4
90.5
D.5 Robustness Analysis of Exemplar Sets
demonstaatssss the robustness of LOT Merging across three random exemplar sets. For both
ViT-B/32 and ViT-L/14, the average accuracy remains highly stable (S 0.2% variance), with minimal fluctuations across datasets. The stronger ViT-L/14 backbone further enhances stability, maintaining a consistent 90.5% average accuracy. These results confirm that LOT Merging highly robust to exemplar selection, ensuring reliable performance across diverse tasks.
25

---

Table 9: Computational complexity comparison (in seconds) for merging ViT-B/32 and ViT-L/14 models across eight vision tasks, measured on a single RTX 3090 GPU.
Method
Surgery
AdaMerging
TATR
PCB Merging
CAT Merging
LOT Merging (ours)
VIT-B/32
12621 36826
8276
176
43
46
44
VII-1/144
16299
283
131
150
161
D.6
Analysis of Computational Complexity
The computational overhead of LOT Merging is reasonable and practically efficient. The procedure consists of two primary stages:
Feature extraction: This step is highly efficient, requiring only small number of unlabeled examples (typically 16 16-64 per task). As it avoids training or gradient-based updates, the computational cost remains minimal;
This step relies on a closed-form solution, eliminating the need for iterative optimization. For linear task representations, this involves matrix inversion operation. While matrix inversion can be associated with higher theoretical complexity (O(d")) we mitigate this overhead through parallelized implementation modern GPUs. In practice, the computation is highly efficient and introduces negligible latency.
Optimal task-vector computation:
Empirical runtime measurements, summarized in Table (recorded single NVIDIA RTX 3090 GPU), confirm that LOT Merging achieves significantly lower wall-clock time compared to training- based baselines such as TA with Surgery [40] and AdaMerging 41. These results highlight the practical efficiency of our approach, making it highly suitable for scenarios where rapid adaptation across tasks required.
Table 10: Consistency comparison for merging ViT-B/32 and ViT-L/14 models across eight vision tasks.
Fisher Merging
RegMean
Task Arithmetic
PCB Merging
CAT Merging
LOT Merging(ours)
ViT-B/32 VIT-I/114
13.78 6.79
8.46 6.86
8.95
6.85
6.21 2.51
4.58
5.11
3.49
2.64
D.7 Analysis of Consistency
While LOT Merging achieves the highest average performance Tables I and 2 it does not consistently deliver the best accuracy on every individual dataset. This variation can be attributed to methods like Fisher merging, which tend to yield uneven performance across tasks. To quantify such inconsistency, we compute the standard deviation of accuracy drops drefnedefined as the difference between the task-specific and merged model accuracies-aue across all tasks. As shown in T LOT Merging exhibits low variance, indicating more balanced integration of task knowledge without favoring specific tasks.
Table 11: Results (%) on seen and unseen Tasks
RESISCAS
SVIN
GTSRB
Avg.
(Unseen)
EuroSAT (Unseen)
Ang (Unseen))
Arithmeni Merging
63.3
75.1
80.4
70.6
77.2
46.2
59.6
Merging
93.7
EunSAA
MNIIST
Avg. Acc
(Unseea)
SVHI (Unseen)
Ang (Unseen)
Arithmenic
05.7
73.0
53.3
$1.1
Merging
71.3
49.2
Merging
79.7
56.3
56.6
55,5
D.8 Analysis of Generalization Ability
We first evaluated our method by merging models trained on six out of eight tasks and then testing on the remaining two unseen tasks. As shown the tables] 11 LOT Merging consistently outperforms both Task Arithmetic and Ties-Merging- tasks. These results demonstrate the moderate generalization ability of our approach. However, the performance gap does become narrower out-of-domain tasks compared to in-domain tasks.
only on seen tasks, but also on unseen (out-of-domain)
26

---

Table 12: Average accuracy with Gaussian blur kernel
Method
Gaussian Blur Kernel
ViT-B/32 (%)
ViT-L/14 (%)
Task Arithmetic
69.1
84.5
LOT Merging
kernel_sise size = 3.
82.2
90.3
LOT Merging
kernel size a
80.0
89.5
LOT Merging
kernel size 7,
78.3
88.7
Table 13: Performance when using similar distributed exemplars
Method
COCO Caption
OKVQA
ScienceQA
Task Arithmetic
0.53
40.37
51.55
LOT Merging
0.87
44.25
60.08
LOT Merging w/ surrogate exemplars
0.62
40.15
53.54
This observation motivates us further explore ways enhance LOT Merging 's effectiveness on out-of-domain tasks in future work.
Next, we explored the use of domain-simille auxiliary data by applying Gaussian blur to the original task exemplars. As shown in Table 12, LOT Merging consistently outperformed Task Arithmetic across all blur levels, although performance gradually declined blur strength increased.
To investigate whether similar distribution exemplars would benefit, we merged BLIP models fine- tuned on COCO Caption, OKVQA, and ScienceQA, but replaced each task's exemplars with samples from semantically related datasets: Flickr30k for COCO Caption, AOKVQA for OKVQA, and IconQA for ScienceQA. As shown in Table 13 LOT Merging with surrogate exemplars outperforms Task Arithmetic, though the improvements are task-dependent--uustaattt for COCO Caption, but more limited for the other two.
D.9 Robustness Analysis with Data Corruption
To assess the robustness of LOT Merging under noisy conditions, we adopted the corruption protocol introduced in [41], applying seven types of corruption both exemplars and test sets. We evaluated the performance by merging eight ViT-B/32 models on each corrupted dataset and compared LOT Merging to Task Arithmetic. Table 14 shows that LOT Merging maintains consistent improvement over Task Arithmetic under all corruption types.
Table 14: Average accuracy under different corruption types
Corruption Type
Task Arithmetic
LOT Merging (f A)
Clean (no corruption)
69.1
82.7 (t 13.6)
Motion Blur
47.3
59.9 (f 12.6)
Impulse Noise
51.1
60.8 (1 9.7)
Gaussian Noise
50.6
60.1 (1 9.5)
Pixelate
48.9
61.8 (1 12.9)
Spatter
65.0
79.2 (1 14.2)
Contrast
38.5
45.9 7.4)
JPEG Compression
25.9
28.1 (1 2.2)
D.10 Results on Merging Language Models
We conducted experiments using RoBERTa [16] as the backbone on the GLUE benchmark [33]. which comprises eight diverse NLP tasks spanning both classification and regression (e.g., stsb). For evaluation, we report accuracy classification tasks and the average of Pearson and Spearman correlations for the regression task. The results are presented in Table 15 LOT Merging achieves the highest average performance compared the baseline methods, demonstrating its effectiveness for language model merging.
27

---

Table 15: Performance of Different Algorithms on GLUE Tasks
Algorithm
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS B
Average
#best
Task Arithmetic
6.68
78.46
78.62
72.69
53.43
83.49
27.10
58.34
Bies-Merging
9.46
59.34
74.71
65.93
41.29
47.29
72.13
9.21
47.42
PCB Merging
11.40
50.85
77.63
78.22
55.78
60.29
75.57
67.01
59.59
CAT Merging
33.20
72.33
68.22
82.92
76.05
62.82
89.33
15.57
62.56
LOT Merging
17.13
73.04
78.27
77.22
78.81
65.35
89.74
25.50
63.13
Table 16: Performance Comparison when merging two Qwen models.
Method
ArguAna
ArXiv Hierarchical Clustering S2S
Individual
53.62
0.6377
Task Arithmetic
30.86
0.6318
LOT Merging
47.36
0.6374
We also conduct experiments large language models. We select Qwen3-0.6B 39 the pretrained backbone, and two official fine-tuned variants- Senss---enee-E-neeinn-n-000
and Swenn-Reranker-O.. Although Qwen3-0.6B has a relatively small parameter count, it features deep architecture with 28 Qwen3DecoderBlocke each comprising attention module (four projection linear layers) and an MLP module (three projection linear layers), for total of 196 linear layers.
To reflect their respective strengths embedding and ranking, we selected two tasks from the MTEB (English, v2) benchmark (21): ArguAna retrieval task requiring strong ranking capabilities; ArXiv Hierarchical Clustering S2S a clustering task requiring high-quality embeddings. Evalu- ation metries were top-1 accuracy for ArguAna and V-measure for ArXiv Hierarchical Clustering. Results in
that LOT Merging substantially outperforms Task Arithmetic on Ar- guAna, indicating improved cross-task transfer for ranking, while maintaining competitive clustering performance. These results confirm that our method remains effective on deep LLMs such as Qwen3,
Table 17: Pearson Correlation across Different Tasks
COCO Caption
Flickr30k Caption
TexCCaps
OKVQA
TextVQA
ScienceQA
Pearson Correlation
0.51
0.60
0.30
0.82
0.77
0.20
D.11 Analysis of Feature Drift Multi-Model Tasks
To assess the impact of feature drift vision-language settings, we measured the Pearson correlation between feature drift and loss change in BLIP models across several multimodal benchmarks after Task Arithmetic merging. The results in 17 demonstrate that feature drift remains a positively correlated factor.
While the correlation strength refiesi-ef inherent complexities such as reasoning demands and modality interactions our experiments consistently show that reducing feature drift leads to performance gains, even in vision-language tasks. This suggests that, despite the greater complexity in multimodal models, minimizing feature drift remains an effective and broadly applicable merging objective.
D.12 Robustness Analysis under Real-Time Data Distribution
We further study the impact of real-time data on model performance, particularly in scenarios where exemplars are available for only subset classes, while evaluation conducted on the complete test set. The results in Table 18 show that although LOT Merging's performance declines with fewer exemplar classes, still consistently outperforms Task Arithmetic.
28

---

Table 18: Effect of Exemplar Classes on Average Accuracy
Backbone
Method
Exemplar Classes
Avg. Acc.
VIT-B/32
Task Arithmetic
69.1
ViT-B/32
LOT Merging
All classes
82.7
VIT-B/32
LOT Merging
First 1/2
81.8
VIT-B/32
LOT Merging
First 1/3
81.0
VIT-B/32
LOT Merging
First 1/4
79.6
ViT-B/32
LOT Merging
First 1/5
77.6
VIT-L/14
Task Arithmetic
84.5
VIT-L/14
LOT Merging
All classes
90.5
VIT-L/14
LOT Merging
First 1/2
90.0
VIT-L/14
LOT Merging
First 1/3
89.6
VIT-L/14
LOT Merging
First 1/4
88.9
VIT-L/14
LOT Merging
First 1/5
88.3
29


Reviews:
[{"review_id": "bbMMk6jW5U", "reviewer": "Reviewer_ZqXr", "summary": "The paper titled “Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration” introduces a training-free approach called Layer-wise Optimal Task Vector Merging (LOT Merging) to mitigate feature drift in merged models. The method frames the merging process as a closed-form convex quadratic optimization problem, aiming to minimize representation differences at each layer. It does this at a task-vector level instead of parameter level avoiding the issue with modifying the pretrained weights which may lead to catastrophic forgetting.", "questions": "- While the method requires access to a small amount of task-specific data to generate the necessary activations, it would be interesting to explore whether task-agnostic auxiliary data could serve as an effective alternative.\n- It remains unclear whether the performance gains observed on in-domain data would generalize to out-of-domain scenarios—an area where traditional merging methods typically excel. This concern is especially relevant since the proposed method focuses on minimizing feature drift specifically using in-domain data.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The work proposes a novel technique to reduce layer-wise feature drift using a closed-form solution at the task-vector level, rather than the parameter level—a direction that has not been previously explored.\n- It demonstrates strong performance, significantly closing the gap with jointly trained models, and benchmarks favorably against a wide range of training-free model merging baselines.\n\n### Weaknesses\n- The paper lacks an evaluation of the merged model's generalization performance—an aspect where merging often excels, sometimes outperforming individual task-specific models.\n- It does not assess the method on large language models (LLMs), a domain where merging techniques are widely adopted. Such an evaluation could broaden its relevance to more popular and practical use cases.\n- The method is described as “data-free,” though this may be misleading. It would be more accurate to position it as suited for data-scarce scenarios rather than completely data-free.", "was_revised": true, "final_justification": "I'd like to thank the authors for their response. I'm happy with the additional evaluation to test generalisation and evlauations in the language domain and would like to maintain my positive rating.", "rebuttal": "### Response to **ZqXr**\n\nWe sincerely thank you for your critical assessment and constructive suggestions. \n\n**Q: Evaluation of the generalization performance.**\n\nThank you for your thoughtful suggestion. To address concerns about generalization, we evaluated our method by merging models trained on six out of eight tasks and then testing on the remaining two unseen tasks.\n\nAs shown in the tables below, LOT Merging consistently outperforms both Task Arithmetic and Ties-Merging—not only on seen tasks, but also on unseen (out-of-domain) tasks. These results demonstrate the moderate generalization ability of our approach. However, the performance gap does become narrower on out-of-domain tasks compared to in-domain tasks. This observation motivates us to further explore ways to enhance LOT Merging’s effectiveness on out-of-domain tasks in future work.\n\n|  | SUN397 |  Cars | RESISC45 | DTD | SVHN | GTSRB | **Avg Acc (Seen Tasks)** | MNIST (Unseen) | EuroSAT (Unseen) | **Avg Acc (Unseen Tasks)** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Task Arithmetic | 63.3 | 62.4 | 75.1 | 57.8 | 84.6 | 80.4 | **70.6** | 77.2 | 46.2 | **61.7** |\n| Ties-Merging | 67.8 | 66.2 | 77.2 | 56.7 | 77.1 | 70.9 | **69.3** | 75.9 | 43.3 | **59.6** |\n| LOT Merging | 69.1 | 69.1 | 89.7 | 65.6 | 94.1 | 93.7 | **80.2** | 81.5 | 45.4 | **63.5** |\n|  | SUN397 |  Cars | GTSRB | EuroSAT | DTD | MNIST | **Avg Acc (Seen Tasks)** | RESISC45 (Unseen) | SVHN (Unseen) | **Avg Acc (Unseen Tasks)** |\n| Task Arithmetic | 64.0 | 64.0 | 75.2 | 87.7 | 57.0 | 95.7 | **73.9** | 52.3 | 44.9 | **51.1** |\n| Ties-Merging | 68.0 | 67.1 | 67.7 | 78.4 | 56.5 | 92.8 | **71.8** | 58.7 | 49.2 | **53.9** |\n| LOT Merging | 68.8 | 69.9 | 84.8 | 95.8 | 61.5 | 97.5 | **79.7** | 56.3 | 54.6 | **55.5** |\n\n**Q: Evaluation on large language models (LLMs).**\n\nA: Thank you for your suggestion. To address this point, we followed CAT Merging [1] and conducted experiments using RoBERTa as the backbone on the GLUE benchmark, which comprises eight diverse NLP tasks spanning both classification and regression (e.g., stsb). For evaluation, we report accuracy for classification tasks and the average of Pearson and Spearman correlations for the regression task. The results are presented in the table below. LOT Merging achieves the highest average performance compared to the baseline methods, demonstrating its effectiveness for language model merging.\n\n| **Algorithm** | **cola** | **mnli** | **mrpc** | **qnli** | **qqp** | **rte** | **sst2** | **stsb** | **Average** | #best |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Task Arithmetic | 6.68 | 66.23 | **78.46** | 78.62 | 72.69 | 53.43 | 83.49 | 27.10 | 58.34 | 1 |\n| Ties-Merging | 9.46 | 59.34 | 74.71 | 65.93 | 41.29 | 47.29 | 72.13 | 9.210 | 47.42 | 0 |\n| PCB Merging | 11.40 | 50.85 | 77.63 | 78.22 | 55.78 | 60.29 | 75.57 | **67.01** | 59.59 | 1 |\n| CAT Merging | **33.20** | 72.33 | 68.22 | **82.92** | 76.05 | 62.82 | 89.33 | 15.57 | 62.56 | 2 |\n| LOT Merging | 17.13 | **73.04** | 78.27 | 77.22 | **78.81** | **65.35** | **89.74** | 25.5 | **63.13** | 4 |\n\nIn addition, following [2], we conducted preliminary experiments (due to time constraints) on merging two LLaMA-2-13B-based models: WizardLM-13B, fine-tuned for instruction following (evaluated on AlpacaEval 2.0), and WizardMath-13B, specialized for mathematical reasoning (evaluated on the MATH dataset). As shown in the table below, LOT Merging outperforms Task Arithmetic on both benchmarks, although the margin is relatively modest.\n\n|  | AlpacaEval 2.0 | MATH | Average |\n| --- | --- | --- | --- |\n| Task Arithmetic | 10.91 | 12.21 | 11.56 |\n| LOT Merging | 11.02 | 12.40 | 11.71 |\n\n[1] Sun W, Li Q, Geng Y, et al. CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging[C]//ICML2025.\n\n[2] Yu L, Yu B, Yu H, et al. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement[J]. arXiv preprint arXiv:2408.03092, 2024.\n\n**Q: The description of “data-free” may be misleading.**\n\nA: Thank you for raising this point. We would like to clarify that we do not describe our method as “data-free” in the paper. Instead, we refer to it as “training-free,” indicating that it does not require any gradient-based optimization. We will make this distinction more explicit in the revised manuscript to avoid any potential confusion.\n\n**Q: Exploring the benefit of task-agnostic auxiliary data.**\n\nA: Thank you for this interesting suggestion.\n\nTo investigate the potential of task-agnostic auxiliary data, we first experimented with using ImageNet samples as exemplars for all tasks. However, when merging eight ViT-B/32 models, we observed a noticeable performance drop compared to Task Arithmetic, as shown in the table below:\n\n|  | Task Arithmetic | LOT Merging w/ 64 ImageNet Exemplars | LOT Merging w/ 128 ImageNet Exemplars | LOT Merging w/ 256 ImageNet Exemplars |\n| --- | --- | --- | --- | --- |\n| Avg. Acc. (%) | 69.1 | 65.6 | 65.1 | 65.3 |\n\nNext, we explored the use of *domain-similar* auxiliary data by applying a Gaussian blur to the original task exemplars. In this setting, LOT Merging consistently outperformed Task Arithmetic across all blur levels, although performance gradually declined as blur strength increased:\n\n|  | Gaussian Blur Kernel | Avg. Acc. when Merging ViT-B/32 (%) | Avg. Acc. when Merging ViT-L/14(%) |\n| --- | --- | --- | --- |\n| Task Arithmetic | - | 69.1 | 84.5 |\n| LOT Merging | $kernelsize=3, \\sigma=1.0$ | 82.2 | 90.3 |\n| LOT Merging | $kernelsize=5, \\sigma=2.0$ | 80.0 | 89.5 |\n| LOT Merging | $kernelsize=7, \\sigma=3.0$ | 78.3 | 88.7 |\n\nThese results suggest that fully task-agnostic data can be detrimental to performance, whereas *semantically similar* data—even when degraded—can still facilitate effective merging. This insight could help extend LOT Merging to more practical or data-constrained scenarios."}, {"review_id": "RSUc26H0Dw", "reviewer": "Reviewer_eNpX", "summary": "This paper introduces Layer-wise Optimal Task Vector Merging (LOT Merging) , a training-free approach to model merging that minimizes feature drift between task-specific expert models and the resulting merged model, operating layer by layer. In the context of Transformer-based models, operations are categorized into three types: matrix multiplication, element-wise product, and element-wise addition. And tailored merging strategies for each type of operation are proposed. LOT Merging is evaluated across a range of vision tasks as well as vision-language tasks.", "questions": "Please refer to Weaknesses. Overall I think this is a nice paper, I have some concerns as raised in Weaknesses. My initial rating is BA.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper is well-structured and clearly written, making it easy to follow.\n* The core idea is intuitive and supported by theoretical analysis.\n* The method is validated across a variety of tasks, accompanied by thorough ablation studies that provide a comprehensive understanding of its behaviour.\n\nWeaknesses:\n* LOT Merging is based on the assumption that feature drift has a significant impact on the performance of merged models. This seems reasonable for relatively simple unimodal tasks, such as those in vision-only settings demonstrated in Task Arithmetic (e.g., Figure 1). However, I question whether this relationship remains as clear-cut for more complex tasks, such as vision-language tasks or image generation, where interactions between modalities may introduce additional challenges and obscure straightforward patterns of feature drift.\n* Related to task complexity, LOT Merging relies on examplars to estimate feature drift. When dealing with highly complex tasks, the limited number of exemplars may fail to accurately represent the overall data distribution, potentially leading to unstable performance, this is somewhat reflected in the results on OKVQA in Table 3. Additionally, it would be helpful to clarify what the Individual results refer to in that table.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns. Having read other reviews, I've decided to keep my original positive assessment of this work, and increase the score to 5.", "rebuttal": "### Response to **eNpX**\n\nWe sincerely thank you for your critical assessment and constructive suggestions. \n\n**Q: Does feature drift still have an impact on multimodal tasks?**\n\nA: Thank you for raising this important point.\n\nTo assess the impact of feature drift in vision-language settings, we measured the Pearson correlation between feature drift and loss change in BLIP models across several multimodal benchmarks after Task Arithmetic merging. The results below demonstrate that feature drift remains a positively correlated factor:\n\n|  | COCO Caption  | Flickr30k Caption | Textcaps | OKVQA | TextVQA | ScienceQA |\n| --- | --- | --- | --- | --- | --- | --- |\n| Pearson Correlation | 0.51 | 0.60 | 0.30 | 0.82 | 0.77 | 0.20 |\n\nWhile the correlation strength varies—reflecting inherent complexities such as reasoning demands and modality interactions—our experiments (see Table 3 in the manuscript) consistently show that reducing feature drift leads to performance gains, even in vision-language tasks. This suggests that, despite the greater complexity in multimodal models, minimizing feature drift remains an effective and broadly applicable merging objective.\n\n**Q: Can the limited number of exemplars lead to unstable performance?**\n\nA: Thank you for the thoughtful comments.\n\nTo assess whether limited exemplars affect merging performance, we varied the number of exemplars per task from 1 to 64 and report LOT Merging accuracy on both vision and vision-language tasks. As shown below, increasing the number of exemplars generally improves performance, and most tasks reach near–peak performance by 32 samples. This indicates that the number of exemplars alone cannot fully explain the marginal gain on OKVQA in Table 3.\n\n| Num. of Exemplars | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| 1 | 62.0 | 59.9 | 67.0 | 57.1 | 78.7 | 66.9 | 87.3 | 47.0 |\n| 2 | 63.2 | 60.5 | 74.2 | 71.4 | 87.5 | 72.9 | 95.3 | 47.5 |\n| 4 | 64.2 | 64.2 | 69.1 | 84.4 | 88.6 | 77.9 | 96.5 | 51.1 |\n| 8 | 65.4 | 66.3 | 79.4 | 87.6 | 92.8 | 85.5 | 98.2 | 54.1 |\n| 16 | 66.4 | 67.1 | 82.6 | 90.9 | 92.9 | 86.1 | 98.5 | 58.1 |\n| 32 | 67.4 | 67.5 | 83.3 | 92.4 | 93.2 | 89.6 | 98.6 | 60.6 |\n| 64 | 67.7 | 67.5 | 85.7 | 94.9 | 93.4 | 89.8 | 98.7 | 63.6 |\n\n| Num. of Exemplars | COCO Caption | Flickr30k Caption | Textcaps | OKVQA | TextVQA | ScienceQA |\n| --- | --- | --- | --- | --- | --- | --- |\n| Metric  | CIDEr | CIDEr | CIDEr | Accuracy | Accuracy | Accuracy |\n| 2 | 0.83 | 0.39 | 0.39 | 27.95 | 13.04 | 47.14 |\n| 4 | 0.83 | 0.46 | 0.41 | 31.86 | 14.70 | 47.68 |\n| 8 | 0.88 | 0.48 | 0.43 | 36.22 | 18.43 | 47.47 |\n| 16 | 0.89 | 0.53 | 0.45 | 37.34 | 20.12 | 47.37 |\n| 32 | 0.91 | 0.54 | 0.44 | 38.35 | 20.82 | 48.24 |\n\nRegarding the OKVQA results in Table 3, we would like to clarify that LOT Merging only underperforms CAT Merging on this particular dataset. Importantly, LOT Merging improves upon Task Arithmetic across *all* vision-language tasks, while CAT Merging degrades performance on Textcaps. We therefore attribute CAT’s large gain on OKVQA to task-specific overfitting, rather than to generally superior alignment. A similar observation also applies to our vision-only benchmarks (see Section D.7).\n\n**Q: Clarify the individual results of Table 3.**\n\nA: Thank you for the suggestion. We will include the individual results in the final version. For reference, the detailed results are as follows:\n|  | COCO Caption | Flickr30k Caption | Textcaps | OKVQA | TextVQA | ScienceQA |\n| --- | --- | --- | --- | --- | --- | --- |\n| Metric  | CIDEr | CIDEr | CIDEr | Accuracy | Accuracy | Accuracy |\n| **Individual** | 1.17 | 0.65 | 0.65 | 50.84 | 29.79 | 76.89 |\n| Task Arithmetic | 0.86 | 0.50 | 0.39 | 17.71 | 0.49 | 40.10 |\n| CAT Merging | 0.91 | 0.53 | 0.36 | 44.07 | 19.69 | 46.36 |\n| LOT Merging | 0.91 | 0.54 | 0.44 | 38.35 | 20.82 | 48.24 |"}, {"review_id": "k24xk1dzyF", "reviewer": "Reviewer_NbB2", "summary": "The paper presents LOT Merging, a training-free model merging approach to tackle feature drift in multi-task scenarios. It reveals that performance degradation during merging correlates with feature drift, which is amplified in deeper network layers. LOT Merging mitigates this via layer-wise optimization, formulating model merging as a layer-wise feature drift minimization problem to derive closed-form solutions. It achieves up to 4.4% accuracy gains on vision tasks (leading in 6–7 out of 8 datasets for ViT models) and excels in 5 out of 6 vision-language tasks among the selected baselines.", "questions": "- Where the exemplar sets are selected from? Are they sampled from the training, validation, or test sets?\n- Since the method optimizes an upper bound and decomposes the problem into layer-wise objectives, how does merging effectiveness scale with the number of layers? For instance, does LOT merging exhibit improved or degraded performance when applied to models with deeper architectures (e.g., 96-layer LLMs like Qwen)?", "limitations": "The method relies on optimizing an upper bound, reducing the problem to layer-wise optimization. It depends critically on two key assumptions: the Lipschitz continuity and linear operations within different neural network layers and requires exemplars to estimate feature drift, which may pose practical challenge with modern architectures and noisy datasets. The applicability on LLMs and NLP tasks are also unverified.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths \n- The paper provides an algorithm that can work with few exemplars (16–64 samples per task) to mitigate the merging conflicts. \n- The paper provides detail theoretical derivation in view of layer feature drifts.\n- The performance of the proposed method is strong across vision-related benchmarks. \n- The comparion of direct parameter merging and task vector merging, along with their dependency on samples size in Section 5.2, is interesing.\n\n### Weaknesses\n- The paper primarily employs the theoretical framework from Cat Merging [1] but does not explicitly discuss the relationship between the two methods, which may affect its originality. I am also interested in an analysis of why Cat Merging underperforms compared to LOT Merging, given that their primary difference lies in the optimization objective.\n- The method performance may highly related to the quality of exemplars. While the authors report average performance across 3 randomly sampled exemplar sets, including the standard deviation would better illustrate consistency.\nFurthermore, I think futher analysis of the robustness of LOT Merging against distributional differences is valuable. How does the method perform when exemplars/test sets are noisy (e.g.,  corruption data in Ada-Merging Figure 5 [1]),  or tasks involve complex patterns (e.g., NLP datasets with different instructions[2] ). In situations where access to exemplar sets is limited, or when real-time test data diverges from the distribution of exemplars, the applicability of LOT Merging may be constrained.\n- Limited evaluation for larger models and NLP tasks. The experiments focus on vision tasks (ViT, BLIP), but omit testing on LLMs (e.g., Qwen, LLaMA) or complex benchmarks (e.g., GSM8K, MMLU). For large language models, does the $O(n^3)$ inverse operation efficient with high-dimensional layers (e.g., Qwen’s 12,288-dim FFN)? Does the layer-wise optimization perform well with deep architectures? Furthermore, The paper does not address attention mechanisms or SwiGLU activations, which are prevalent in modern LLMs.\n- Derivation Errors:  In Equation 22-23 of Section B, should $f_k^{-1}$ instead be $f_k^{l-1}$ ?\n\n[1] CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging. ICML, 2025\n\n[2] Adamerging: Adaptive model merging for multi-task learning. ICLR,2024\n\n[3] Wizardlm: Empowering large pre-trained language models to follow complex instructions. ICLR,2024", "was_revised": true, "final_justification": "The addtional experiments solve my concerns about the differences with CAT Merging, the robustness of exemplars. \nThe effectiveness on LLMs and the theoretical treatment of nonlinear operations (like attention/SwiGLU) still can be improved.", "rebuttal": "### Response to **NbB2**\n\nWe sincerely thank you for your critical assessment and constructive suggestions. \n\n**Q: Comparison between CAT Merging and LOT Merging.**\n\nA: Thanks for highlighting this point. While both CAT Merging and LOT Merging address layer-wise conflicts, they differ fundamentally in their treatment of task vectors:\n\n- **CAT Merging** uses **trimming** (projection and masking) to remove “conflict-prone” components. While this suppresses harmful interactions, it may also discard critical task-specific directions, particularly when those directions contain information valuable for other tasks.\n- **LOT Merging**, by contrast, **directly minimizes layer-wise feature drift** via a convex quadratic formulation. This closed-form solution preserves all informative subspaces and optimally balances contributions from each task. In Section 5.1, we show that in both the ideal (orthogonal features) and worst (collinear features) cases, LOT Merging achieves the optimal consolidation of task vectors—something trimming cannot guarantee.\n\nEmpirically, CAT’s trimming sometimes over-prunes useful information, whereas LOT Merging shows more consistent improvements (in Table 10, Section D.7). We will clarify these distinctions in the revised manuscript.\n\n**Q: The robustness of LOT Merging under corruption exemplars/test sets.**\n\nA: We appreciate the reviewer’s insightful comments regarding the robustness of LOT Merging.\n\nTo assess the robustness of LOT Merging under noisy conditions, we adopted the corruption protocol introduced in Ada-Merging [1], applying seven types of corruption to both exemplars and test sets. We evaluated the performance by merging **eight** ViT-B/32 models on each corrupted dataset and compared LOT Merging to Task Arithmetic. The following results show that LOT Merging maintains consistent improvement over Task Arithmetic under all corruption types. \n\n| Corruption Type | Task Arithmetic | LOT Merging (↑ Δ) |\n| --- | --- | --- |\n| Clean (no corruption) | 69.1 | **82.7 (↑ 13.6)** |\n| Motion Blur | 47.3 | **59.9 (↑ 12.6)** |\n| Impulse Noise | 51.1 | **60.8 (↑ 9.7)** |\n| Gaussian Noise | 50.6 | **60.1 (↑ 9.5)** |\n| Pixelate | 48.9 | **61.8 (↑ 12.9)** |\n| Spatter | 65.0 | **79.2 (↑ 14.2)** |\n| Contrast | 38.5 | **45.9 (↑ 7.4)** |\n| JPEG Compression | 25.9 | **28.1 (↑ 2.2)** |\n\n**Q: Results on NLP tasks & the robustness analysis of LOT Merging with different instructions.**\n\nA: Thanks for your suggestion. We follow CAT Merging and conduct experiments using RoBERTa as the backbone model on the GLUE benchmark, including eight diverse NLP tasks spanning both classification and regression (e.g., STS-B). For evaluation, we report accuracy for classification tasks and the average of Pearson and Spearman correlations for the regression task. The results are summarized in the table below. Our proposed LOT Merging achieves the highest average performance compared to baseline methods.\n\nNotably, we observe that LOT Merging exhibits superior robustness on tasks with more complex or nuanced instructions, such as:\n\n- rte demands precise logical inference between sentence pairs.\n- qqp involves paraphrase detection, relying on intent and semantic understanding rather than surface patterns.\n- sst2 requires fine-grained sentiment classification, sensitive to subtle linguistic cues.\n\nIn contrast, some baseline methods tend to overfit tasks with simpler instructions, such as stsb, leading to less balanced performance. These results suggest that LOT Merging is more robust across tasks with varying levels of instruction complexity.\n\n| **Algorithm** | **cola** | **mnli** | **mrpc** | **qnli** | **qqp** | **rte** | **sst2** | **stsb** | **Average** | #best |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Task Arithmetic | 6.68 | 66.23 | **78.46** | 78.62 | 72.69 | 53.43 | 83.49 | 27.10 | 58.34 | 1 |\n| Ties-Merging | 9.46 | 59.34 | 74.71 | 65.93 | 41.29 | 47.29 | 72.13 | 9.210 | 47.42 | 0 |\n| PCB Merging | 11.40 | 50.85 | 77.63 | 78.22 | 55.78 | 60.29 | 75.57 | **67.01** | 59.59 | 1 |\n| CAT Merging | **33.20** | 72.33 | 68.22 | **82.92** | 76.05 | 62.82 | 89.33 | 15.57 | 62.56 | 2 |\n| LOT Merging | 17.13 | **73.04** | 78.27 | 77.22 | **78.81** | **65.35** | **89.74** | 25.5 | **63.13** | 4 |\n\n**Q: The robustness of LOT Merging under real-time data distribution.**\n\nWe further study the impact of real-time data on model performance, particularly in scenarios where exemplars are available for only a subset of classes, while evaluation is conducted on the complete test set. The results below show that although LOT Merging’s performance declines with fewer exemplar classes, it still consistently outperforms Task Arithmetic.\n\n| Backbone | Method | Exemplar Classes | Avg. Acc. |\n| --- | --- | --- | --- |\n| ViT-B/32 | Task Arithmetic | - | 69.1 |\n|  | LOT Merging | All classes | 82.7 |\n|  | LOT Merging | First 1/2 classes | 81.8 |\n|  | LOT Merging | First 1/3 classes | 81.0 |\n|  | LOT Merging | First 1/4 classes | 79.6 |\n|  | LOT Merging | First 1/5 classes | 77.6 |\n|  |  |  |  |\n| ViT-L/14 | Task Arithmetic | - | 84.5 |\n|  | LOT Merging | All classes | 90.5 |\n|  | LOT Merging | First 1/2 classes | 90.0 |\n|  | LOT Merging | First 1/3 classes | 89.6 |\n|  | LOT Merging | First 1/4 classes | 88.9 |\n|  | LOT Merging | First 1/5 classes | 88.3 |\n\n**Q: Effectiveness for larger models.**\n\nA: Good question. To assess computational feasibility, we empirically measured the time required to compute the Moore–Penrose inverse of a randomly initialized 12,288×12,288 matrix across several platforms:\n\n| Device | Xeon Gold (cpu) | Xeon Platinum (cpu) | RTX3090 (gpu) | A100 (gpu) |\n| --- | --- | --- | --- | --- |\n| Time (second) | 87.20 | 36.51 | 79.02 | 22.91 |\n\nThese results confirm that the inverse operation is tractable in practice. Even on the Xeon Gold, performing such inversion across 96 layers requires approximately 139.5 minutes. Given the one-time cost and the absence of any retraining, we consider this overhead acceptable at LLM scale. Furthermore, our method only invokes the pseudo-inverse on exemplar-aggregated covariance matrices, which are typically low-rank in practice, further reducing computational burden.\n\nIn addition, following [1], we conducted preliminary experiments (due to time constraints) on merging two LLaMA-2-13B-based models: WizardLM-13B, fine-tuned for instruction following (evaluated on AlpacaEval 2.0), and WizardMath-13B, specialized for mathematical reasoning (evaluated on the MATH dataset). As shown in the table below, LOT Merging outperforms Task Arithmetic on both benchmarks, although the margin is relatively modest.\n\n|  | AlpacaEval 2.0 | MATH | Average |\n| --- | --- | --- | --- |\n| Task Arithmetic | 10.91 | 12.21 | 11.56 |\n| LOT Merging | 11.02 | 12.40 | 11.71 |\n\n[1] Yu L, Yu B, Yu H, et al. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement[J]. arXiv preprint arXiv:2408.03092, 2024.\n\n**Q: How LOT Merging tackle attention mechanisms or SwiGLU activations.**\n\nA: Thank you for your question. Our approach decomposes complex modules into their fundamental parameterized linear operations:\n\n- **Attention Mechanism:** The Q, K, V projections and the output projection are each treated as distinct linear layers and merged independently. Our experiments with attention-based models such as ViT and BLIP demonstrate the effectiveness of this decomposition.\n- **SwiGLU Activation:** The gate and linear projections in SwiGLU are treated as two separate matrix multiplications, which can be merged independently as well.\n\nThis modular treatment allows LOT Merging to naturally extend to attention mechanisms and SwiGLU activations, without requiring any additional modifications.\n\n**Q: Where are the exemplar sets selected from?**\n\nA: All exemplar sets are randomly sampled from the training set and are kept strictly separate from the test set used for evaluation.\n\n**Q: Writing issues (standard deviation and equation error in Section B).**\n\nA: Thank you for highlighting these issues. We will thoroughly revise the manuscript, correct the notation, and include the standard deviation in the experiments in the final version."}, {"review_id": "uJSBBRj7Rv", "reviewer": "Reviewer_VNUV", "summary": "The paper studies the problem of negative transfer in model merging. The authors argue that the degradation of merged model compared with individual model partly comes from feature drift. To solve this problem, they propose LOT merging, which minimizes such drift in a per-layer manner. The solution improves performance compared with baselines.", "questions": "1. [1] finds that when the original training set is unavailable, surrogate data from similar or related datasets can be useful for their localization step. I wonder whether it would also be the case for feature alignment? Would using similar data from other distribution be helpful for LOT merging? If so, it would widen the applicability of the approach.\n\n[1] Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic. He et al, 2025.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The paper is well written, with claims supported by empirical and theoretical evidence.\n2. The comparison with RegMean is clear and highlights the difference, strengthening the motivation.\n3. The ablation studies clearly demonstrate the effect of different parameters.\n\nWeakness\n1. Similar to RegMean, the closed-form solution only applies to linear cases. Although such computation can be applied to each of the computation module separately, it is unclear why the aggregation would work for general transformer models, which are highly non-linear. It would be better to provide more intuition and explanation about this restriction.", "was_revised": true, "final_justification": "The proposed approach is effective and technically sound. The rebuttal address the concerns, and the inclusion of surrogate data analysis extends the applicability of the approach.", "rebuttal": "### Response to **VNUV**\n\nWe sincerely thank you for your critical assessment and constructive suggestions. \n\n**Q: How does the linear feature drift minimization affect the merging of nonlinear models?**\n\nA: Thank you for your thoughtful question. You are correct that the classical closed-form solution is strictly applicable to linear systems. However, our approach leverages the modular structure of modern deep neural networks, which are composed of sequences of linear modules interleaved with nonlinearities. While the overall model is highly nonlinear, we show that it is still possible to **bound the output drift of nonlinear modules** in terms of the drift in their underlying linear components.\n\nTo illustrate this, consider the attention module, a central building block in transformer models. Its computation can be written as:\n\n$Att(Q, K, V) = \\mathrm{Softmax}\\left(\\frac{QK^\\top}{\\sqrt{d}}\\right) V,$\nwhere $d$ is the feature dimension. We establish the following proposition (with a proof sketch provided below):\n\n---\n\n**Proposition 1.**\n\nLet $\\Delta Q, \\Delta K, \\Delta V$ denote arbitrary perturbations to the input matrices. Then:\n\n$\\|Att(Q, K, V) - Att(Q +\\Delta Q, K +\\Delta K, V + \\Delta V)\\|_F $\n\n$\\leq \\Delta_{linear} + \\Delta_{non-linear}$\n\n\nwhere $\\|\\cdot\\|_F$ denotes Frobenius norm and\n\n$$\n\\Delta_{linear}=\\frac{\\|V\\|_F \\|Q\\|_F}{\\sqrt{d}} \\|\\Delta K\\|_F+ \\frac{\\|V\\|_F \\|K\\|_F}{\\sqrt{d}} \\|\\Delta Q\\|_F + \\|\\Delta V\\|_F\n$$\n\n$$\n\\Delta_{non-linear}=\\frac{\\|V\\|_F}{\\sqrt{d}} \\|\\Delta Q\\|_F \\|\\Delta K\\|_F\n$$\n\n---\n\n**Proposition 1** shows that the total drift in the attention output—caused by perturbations to $Q$, $K$, and $V$—is bounded by a **sum of linear and nonlinear terms**. Crucially, when the perturbations $\\Delta Q, \\Delta K, \\Delta V$ are small (which is precisely the regime targeted by LOT merging), the nonlinear term becomes negligible, and the bound is dominated by the linear terms. Furthermore, the coefficients of the linear terms are either constant or normalized by $\\sqrt{d}$, ensuring that the overall drift remains controllable.\n\nThis bound provides a theoretical justification for our decomposition approach: **by controlling the drift in the internal linear layers ($\\Delta Q, \\Delta K, \\Delta V$), we can effectively bound the output drift of the entire attention module—even though the module itself is nonlinear**. This insight extends to other nonlinear modules in deep networks, provided we can analyze their dependence on internal linear computations.\n\n---\n\n**Proof Sketch of Proposition 1:**\n\nLet\n\n$A = \\frac{QK^\\top}{\\sqrt{d}}$\n\n$A' = \\frac{(Q + \\Delta Q)(K + \\Delta K)^\\top}{\\sqrt{d}}$\n\n$\\Delta A = A' - A$\n\nBy the triangle inequality,\n\n$\\|Att(Q, K, V) - Att(Q + \\Delta Q, K + \\Delta K, V + \\Delta V)\\|_F \\leq \\|\\mathrm{Softmax}(A)V - \\mathrm{Softmax}(A')V\\|_F+\\|\\mathrm{Softmax}(A')\\Delta V\\|_F .$\n\nSince each row of $\\mathrm{Softmax}(A')$ is a probability vector:\n\n$\\|\\mathrm{Softmax}(A')\\Delta V\\|_F \\leq \\|\\Delta V\\|_F$.\n\nBy the Lipschitz continuity of softmax [1] and sub-multiplicativity of norms:\n\n$\\|\\mathrm{Softmax}(A)V - \\mathrm{Softmax}(A')V\\|_F \\leq \\|\\mathrm{Softmax}(A) - \\mathrm{Softmax}(A')\\|_F \\|V\\|_F \\leq \\|\\Delta A\\|_F \\|V\\|_F.$\n\nNext,\n\n$\\Delta A = \\frac{Q \\Delta K^\\top + \\Delta Q K^\\top + \\Delta Q \\Delta K^\\top}{\\sqrt{d}}$\n\nso\n\n$\\|\\Delta A\\|_F \\leq \\frac{\\|Q\\|_F \\|\\Delta K\\|_F+\\|K\\|_F \\|\\Delta Q\\|_F+\\|\\Delta Q\\|_F \\|\\Delta K\\|_F\n}{\\sqrt{d}}$\n\nSubstituting back yields the stated bound.\n\n---\n\n**Reference:**\n\n[1] Gao, B., & Pavel, L. (2017). On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805.\n\n**Q: Would using similar data from other distributions be helpful for LOT merging?**\n\nA: Thank you for raising this point.\n\nTo investigate this, we merged BLIP models fine-tuned on COCO Caption, OKVQA, and ScienceQA, but replaced each task’s exemplars with samples from semantically related datasets: Flickr30k for COCO Caption, AOKVQA for OKVQA, and IconQA for ScienceQA. As shown below, LOT Merging with surrogate exemplars outperforms Task Arithmetic, though the improvements are task-dependent—substantial for COCO Caption, but more limited for the other two:\n\n|  | COCO Caption | OKVQA | ScienceQA |\n| --- | --- | --- | --- |\n| Task Arithmetic | 0.53 | 40.37 | 51.55 |\n| LOT Merging | 0.87 | 44.25 | 60.08 |\n| LOT Merging w/ surrogate exemplars | 0.62 | 40.15 | 53.54 |\n\nWe hypothesize that this variability is due to differences in semantic similarity between the surrogate and original exemplars. To systematically test the effect of degraded similarity, we conducted additional experiments using the 8-task vision model merging setup, applying progressively stronger Gaussian blur to the exemplars.\n\nWe observe that LOT Merging continues to outperform Task Arithmetic even as the exemplars are heavily degraded, though accuracy decreases as blur increases:\n\n|  | Gaussian Blur Kernel | Avg. Acc. when Merging ViT-B/32 (%) | Avg. Acc. when Merging ViT-L/14(%) |\n| --- | --- | --- | --- |\n| Task Arithmetic | - | 69.1 | 84.5 |\n| LOT Merging | $kernelsize=3, \\sigma=1.0$ | 82.2 | 90.3 |\n| LOT Merging | $kernelsize=5, \\sigma=2.0$ | 80.0 | 89.5 |\n| LOT Merging | $kernelsize=7, \\sigma=3.0$ | 78.3 | 88.7 |\n\nTogether, these findings suggest that **LOT Merging can indeed benefit from surrogate data, but the effectiveness depends on the semantic similarity to the original task**. This insight extends the application of LOT Merging when original data is unavailable."}]

Author rebuttal:

Meta-review: The paper received mostly positive reviews initially. After the rebuttal, the reviewers' concerns have been addressed and agreed to accept the paper. The final decision of AC is consistent with the reviewers.
Decision: Accept (poster)