Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent
Authors: Yongxian Wei
Anke Tang
Li Shen
Zixuan Hu
Chun Yuan
Xiaochun Cao
Venue: ICML 2025 poster (2025)
Keywords: Model Merging
Task Vector
Gradient Projection
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data. Existing methods attempt to alleviate task conflicts by sparsifying task vectors or promoting orthogonality among them. However, they overlook the fundamental target of model merging: the merged model performs as closely as possible to task-specific models on respective tasks. We find these methods inevitably discard task-specific information that, while causing conflicts, is crucial for performance. Based on our findings, we frame model merging as a constrained optimization problem ($\textit{i.e.}$, minimizing the gap between the merged model and individual models, subject to the constraint of retaining shared knowledge) and solve it via adaptive projective gradient descent. Specifically, we align the merged model with individual models by decomposing and reconstituting the loss function, alleviating conflicts through $\textit{data-free}$ optimization of task vectors. To retain shared knowledge, we optimize this objective by projecting gradients within a $\textit{shared subspace}$ spanning all tasks. Moreover, we view merging coefficients as adaptive learning rates and propose a task-aware, training-free strategy. Experiments show that our plug-and-play approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent
Yongxian Wei Anke Tang²
Li Shen 3
Zixuan Hu Chun Yuan
Xiaochun Cao
Abstract
paradigm (Muqeeth al., 2024). However, the indepen- dent deployment of multiple fine-tuned models increases storage costs. While traditional multi-task learning (MTL) can mitigate these issues, they typically require concurrent training across multiple task-specific datasets, which incurs significant training overhead and potential privacy risks (Wei et al., 2024a). Consequently, there growing interest in merging multiple expert models into a unified model with- out accessing their original data (Yang et al., 2024a; Huang et al., 2024). Model merging is performed directly at the parameter level and maintains only one final model dur- ing inference. In recent years, numerous pre-trained and fine-tuned checkpoints have been released on open-source communities like GitH or Hugging Face, making it easy to obtain expert models from diverse domains. These rich model repositories underscore the value of model merging.
Merging multiple expert models offers a promis- ing approach for performing multi-task learning without accessing their original data. Existing methods attempt to alleviate task conflicts by spar- sifying task vectors or promoting orthogonality among them. However, they overlook the fun- damental target of model merging: the merged model performs as closely as possible to task- specific models on respective tasks. We find these methods inevitably discard task-specific in- formation that, while causing conflicts, cru- cial for performance. Based on findings, we frame model merging constrained optimiza- tion problem (i.e., minimizing the gap between the merged model and individual models, sub- ject to the constraint retaining shared knowl- edge) and solve via adaptive projective gradi- ent descent. Specifically, we align the merged model with individual models by decomposing and reconstituting the loss function, alleviating conflicts through data-free optimization of task vectors. To retain shared knowledge, we opti- mize this objective by projecting gradients within a shared subspace spanning all tasks. Moreover, we view merging coefficients as adaptive learn- ing rates and propose a task-aware, training-free strategy. Experiments show that our plug-and- and play approach consistently outperforms previous methods, achieving state-of of-thh-art results across diverse architectures and tasks in both vision and NLP domains. Our code is available here.
One popular approach, Task Arithmetic (TA) (Ilharco et al., 2023), combines task vectors through arithmetic operations for model merging. A major challenge is addressing con- flicts that emerge when multiple task specifcc models coexist within a single model. Ties-Merging (Yadav et al., 2023) proposes pruning redundant parameters, resolving sign con- flicts, and merging sparse models, while AdaMerging (Yang et al., 2024c) applies test-time adaptation techniques to ad- just merging coefficients in the weight space. Most recently, AWD (Xiong et al., 2024) finds that orthogonality among task vectors is key to model merging and introduces adaptive weight disentanglement to improve orthogonality. However, these methods overlook the fundamental requirement of model merging: ensuring the merged model performs com- parably to task-specific models on their respective tasks.
Revisiting multi-task model merging, we make the follow- ing findings: (i) As the number of tasks increases, existing methods inevitably discard task-specific information that, while causing conflicts, crucial for performance. (ii) Task vectors are inherently close to orthogonal. Further promot- ing orthogonality results in the loss of shared knowledge, especially when tasks are similar. (iii) Merging coefficients share similarity with learning rates in MTL, considering task vectors actually represent accumulated gradients.
1. Introduction
Fine-tuning pre-trained foundational models to ad- dress downstream tasks has become an effective
'Tsinghua University "Wuhan University 'Shenzhen Campus of Sun Yat-sen University "Nanyang Technological University. Correspondence to: Li Shen <mathshenlic @ @gaiil com'> Chun Yuan <'uann @ sz. www.innhaa..uu.c......
Based on our rethinking, we frame model merging as a constrained optimization problem (i.e.. minimizing the gap between the merged model and individual models, subject
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Deseent
to the constraint of retaining shared knowledge) and solve it via adaptive projective gradient descent (DOGF) method. Specifically, we measure the gap between the merged model and individual models in task-specific losses, and decom- pose it into a data-free objective using the first-order Taylor expansion. To alleviate conflicts, we introduce a modifi- cation vector A (i.e., redundant parameters) to each task vector. This data-free objective aims to achieve optimal average performance across multiple tasks by optimizing A. For the modification vector, task vectors still compete to minimize the loss on their own tasks. Therefore, we construct a shared subspace based on all task vectors and optimize the problem within this subspace. The gradient of A can divided into two components: one projected onto the shared subspace and the other orthogonal to it. We only take gradient steps in the direction orthogonal to the shared space, effectively constraining task vector optimization. As the former represents movements of parameters within the shared subspace, and the latter maintains shared knowledge while minimizing the gap for each task. Moreover, we deter- mine aware, training-fie free merging coefficients based on the norm of task vectors to mitigate the dominance of any single task's gradient influence.
the parameter level and consolidates multiple models into single final model for inference. Existing model merging methods are categorized into two paradigms: pre-merging and during merging (Yang et al., 2024a). Pre-merging meth- ods aim to create favorable conditions for merging, such as using linearized fine-tuning to achieve weight disentangle- ment (Ortiz-Jimenee et al., 2023; Tang et al., 2024c).
During-merging methods focus on developing techniques to merge given models and can be broadly categorized into data-free and test-time adaptation (TTA) approaches. TTA methods assume access to unlabeled test datasets and are often considered a form of transductive learning. For ex- ample, AdaMerging (Yang et al., 2024c) learns merging coefficients by minimizing entropy as a surrogate loss on test data, while Representation Surgery ( (Yang et al., 2024b) calibrates biases and aligns the merged model's 's represen- tations with those of the original task-specific models. In contrast, our approach designs a fully data-free objective to resolve task conflicts without relying on test data.
Data-free methods depend solely on the pre-trained and fine- tuned model weights for merging (Choi et al., 2024). Ties- Merging (Yadav et al., 2023) prunes redundant parameters by magnitude, resolves sign conflicts, and merges sparse models. Concrete Merging (Tang et al., 2023) adopts meta- learning framework to learn concrete mask that suppresses conflicting parameters. MAP (Li et al., 2025a) examines task vector magnitudes and leverages a second-order Taylor expansion to approximate loss-based metrics, providing a formal bound on the remainder term and using linear regression to estimate the Hessian.
We conduct experiments on diverse vision and NLP tasks, including classification and generation, using various fully fine-tuned and LoRA fine-tuned architectures. Our plug- and-play approach achieves up to 11.6% gains over TA and 5.8% over AdaMerging. Simple task-aware À provides a 2.8% performance boost. Furthermore, experiments on unseen tasks and nut-of-distribution test sets demonstrate its generalization and robustness. Extensive ablation studies clarify the mechanisms of each component.
Calculating the loss gap has been reflected in some stud- ies: MetaGPT (Zhou et al., 2024) formally defines the loss difference and derives a closed-form solution for the merg- ing coefficient X. TATR (Sun et al., 2025) introduces the concept of knowledge conffict between tasks by modeling the loss gap as the product of gradients and task vectors. Other relevant works explore merging within subspaces. TSV (Gargiulo et al., 2024) aggregates task vectors using low-rank approximation and whitening to minimize inter- ference, while KnOTS (Stoica et al., 2025) aligns repre- sentation spaces between LoRA models via SVD to enable compatible merging. These approaches, like ours, recognize the inherent low-rank structure of parameter updates and perform merging within subspaces. We focus optimizing task vectors via gradient descent while constraining it within shared subspace to retain shared knowledge.
In summary, our main contributions are three-fold:
We rethink model merging from a multi-task learning perspective, and model it as a constrained optimiza- tion problem that aims to mitigate task conflicts while retaining shared knowledge.
We propose adaptive projective gradient descent, a novel approach that optimizes a data-free objective within shared subspace and incorporates task-aware, training-free merging coefficients.
We conduct comprehensive experiments and discus- sions; our empirical results demonstrate a significant improvement over previous methods.
2. Related Work
Multi-task learning. Existing MTL research addresses the issue of negative transfer (Jiang et al., 2023) from two principal perspectives: architecture and optimization. From the architectural perspective, negative transfer is mitigated through strategies like modularization (Lu et al., 2024), spar-
Model merging. Model merging (Crisostomi et al., 2024; Wang et al., 2024b; Daheim et al., 2024; Chen et al., 2024; Maldonado et al., 2024) climinates the need for raw train- ing data or expensive computations. operates directly at
2

---

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Deseent
sification (Sun et al., 2020), or soft sharing of the backbone. From an optimization perspective, it is widely recognized that tasks sharing similar underlying structures can bene- fit from being trained together. Gradient alignment meth- ods (Yu et al., 2020; Shi et al., 2022) emphasize maintaining consistency in gradient directions or signs to resolve con- flicts, which projects one task's gradient onto the normal plane of another task's gradient reduce forgetting (Saha et al., 2021). Our approach enhances multi-task perfor- mance by aligning the merged model with each individual model and utilizing adaptive merging coefficients.
86
Accurayy 1
90
86
80
75
+
Meigru
HAD
70
2
Number of Tasks
Figure 1. The effect of task numbers on average accuracy for ViT- B/32, with error hars representing the 95% confidence interval. As the number of tasks increases, negative transfer becomes more pro- nounced. Although
3. Revisit Model Merging
method initially performs lower than other methods, its performance decreases more slowly, demonstrating superior robustness when handling larger number of tasks.
In this section, we first introduce the problem setup and notations for model merging, followed by rethinking of model merging from multi-task learning perspective.
SES BAT B4C SRUVENT
3.1. Preliminary
ginal
aet
aed salepeen Raace
We begin with pre-trained model f. parameterized by Oo. which has been trained on a large-scale dataset. This model is paired with a collection of n downstream tasks, denoted as (D:) Subsequently, the pre-trained model f is fine- tuned individually for each downstream task Di resulting in a series of fine-tuned models, each with its unique param- eters 0z. To isolate task specific information, we define the task vector as Ti 0,- Oo.3 a concept introduced by Ilharco et al. (2023). The set of these task vectors is represented enabling focused analysis of the task-specific characteristics. Model merging aims to compose a multi- task model 0* to approximate the optimal solution:
gloal prdient PC.
sever whnt oxt K
prjjctess ofFG, Faace
pojjetiss IfR Yeare
Figure 2. (a) Cosine similarity matrices of task vectors for ViT- B/32. (b) A schematic representation of the subspace spanned by the task representations. depicted as a two-dimensional plane.
as (T, )
the merged model performs closely as possible task- specific models on respective tasks. As shown in Fig. 1, increasing the task numbers leads to a continuous perfor- mance decline across methods. This is because more tasks result in increased negative transfer, causing the discard of valuable conflict-related task-specific knowledge. There- fore, we propose explicitly modeling the gap between the merged model 0* and individual models 0.. This transforms conffict resolution into an optimization problem that can be solved using gradient descent.
pppt == 0- = A(lo, TI," ,T))
(1)
Here, A represents an arbitrary merging algorithm. For instance, in Task Arithmetic, 0- = 0o + + Σ Ti-
3.2. Rethinking Model Merging for MTL
How to resolve conflicts among parameters? Resolving conflicts among tasks is key challenge in model merging- Unlike MTL, which can mitigate conflicts during training with access to original data, model merging operates en- tirely in the parameter space. Existing methods mainly address conflicts by sparsely adjusting parameters, either by dropping conflicting parameters based on signs (Yadav et al., 2023) or importance scores (Du et al., 2024). Other methods promote orthogonality among task vectors, either by fine-tuning models in the tangent space (Ortiz-Jimenez et al., 2023) or directly optimizing task vectors (Xiong et al., 2024). While these methods alleviate conflicts, they in- evitably discard task-specific information that contributes to conflicts, resulting in performance degradation. However, they overlook the fundamental target of model merging:
Is shared knowledge retained? In addition resolving confficts, MTL should also encourage shared representa- tions a crucial aspect overlooked by existing methods. Ex- periments reveal that sparsely retaining parameters across tasks results in disjoint parameter dimensions, causing a loss of shared knowledge, Fig. 2(a) shows the cosine similarity between task vectors, which is inherently small, consistent with the theorem that high-dimensional vectors tend to be almost orthogonal (Vershynin, 2018). This explains the success of methods like TA. However, further increasing orthogonality to mitigate conflicts can exacerbate shared knowledge loss. Parameters between similar tasks are share- able (e.g.. applying the MNIST task vector improves ac- curacy on SVHN). Therefore, we propose constructing a
3

---

Modeling Multi-Tass Model Merging as Adaptive Projective Gradient Deseent
CLINMAN
the modification of 0" minimize the loss.
600030
In Sec. 4.1, we introduce an optimizable modification vector A using gradient descent to reduce the gap. In Sec. 4.2, we construct the shared subspace Sahare and project the objective into this subspace for optimization. Finally, in Sec. 4.3, we introduce the adaptive merging coefficient A;
600125
606830
Vecurta
4.1. A Data-Free Objective
Figure 3. An illustration of element magnitudes in the task vector, inspired by (Shen et al., 2024). Best viewed when zoomed in.
Considering the fundamental target that the merged model should perform comparably to its respective task-specific model for each task, we follow Zhou et al. (2024) define the objective for resolving model merging
shared subspace Sshare to preserve common representations (see Fig. 2(b)). This involves constraining task vector opti- mization to reduce updates along Sahare.
2
c, - n j=1
T2
What is the role of X? A critical observation is the im- portance of the merging coefficient A. In methods like TA, unified À is searched on the validation set. Ideally, X values should be task- and layer-specific. However, when dealing with a large number of tasks and layers, traditional methods such as grid search or combinatorial optimization search (Liu et al., 2020) become impractical. TTA meth- ods require training using unlabeled test data, which also presents limitations. A statistical analysis of task vector values reveals that tasks and layers exhibit different magni- tudes (see Fig. 3). Modern adaptive optimizers (e.g., Adam) dynamically adjust learning rates based on gradient history, which is often more effective than a global learning rate. These optimizers suppress parameters with large gradients and reward those with small gradients, smoothing gradient fluctuations. Similarly, task vectors T; represent cumulative gradients for each task, and À can be viewed as learning rate balancing gradients across multiple tasks.
min
(, -
(3)
where (O) denotes the loss for task j with model param- eters O. This objective requires that the merged model's performance each task closely matches the performance achieved using only the corresponding task vector TY
Multi-task conflicts often arise during model merging, as ex- pert models encapsulate diverse and sometimes conflicting knowledge. Therefore, we introduce a modification vector A to each task vector, aiming to alleviate conflicts by opti- mizing A. Previous work (Xiong et al., 2024) has shown that eliminating redundant components from task vectors can help reduce interference between tasks. In this context, A can be understood as the shared redundant portion of task vectors. However, directly optimizing Eq. (3) requires task- specific data to compute C,. which is unavailable as we only have access to model parameters. To overcome this limita- tion, we apply Taylor expansion around the pre-trained model parameters On (Ortiz-Jimenez et al., 2023):
4. Methodology
Based on above findings, we frame model merging as a constrained optimization problem (i.e, minimizing the gap while the position in the subspace remains unchanged):
n
2
min
Co + X Σ (T.+ A)) C,0 + +T,))
1=1
a Σ
n (0)( - (V),, E(T, +1)
min C(D: T() = Σ Gap(O", 0.).
min
(T,
j=l
1=1
(2)
- L,(0)) (VoL,(0)).T]) -,,)
s.t. Saharc 0 0, Σ Tx) 7x)= =0
(VoL, A Σ (T. (T++A)-))) -
Here, A is initialized as a zero tensor with the same shape as the task vector. The function Gap(s-) measures the distance between two sets of parameters, while (--)) denotes the distance within the shared subspace. Then, we solve it via adaptive projective gradient descent:
min
Similarly, calculating the gradient (eC) of the pre- trained model for task j requires access to data Dj. which is typically unavailable. As an alternative, we approximate this gradient using the task vector - -Tj, since the task vector can be interpreted as an accumulation of gradients. Under the Neural Tangent Kernel assumption (i.e., fine-tuning oc-
A = A g, where g Projis ((AC(A:), ).).
uses adaptive >, for different tasks, projects the gradient orthogonal to Sahrre to satisfy the constraint, and optimizes

---

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Deseent
curs in linear regime), (00) can be estimated as KTy with k < 0. Here, ; = d ----00- = - - 0g VeVo, (Ot)), where ae is the learning rate and T is the number of up date steps. Given that parameters remain near Qg. we have Vo. (0.)) (0)), Thus, we obtain (0)) - / Ei= 0r- Consequently, the data-free objective can be approximated as:
Algorithm 1: Adaptive Projective Gradient Descent
0u
Input :Pre-trained model 0:: Fine-tuned models (0. (0.)==:
Subspace basis size k; Global scaling factor 1/-
Output :Merged multi-task model 0"
// Task-Wise Preparation.
for i + to n do
Compute task vector T; < 0: 0o
Compute merging coefficients X + lT!
Perform SVD on T: T. = U.
R E T A) T3) A))))
B; <- the first columns of U,
min
(5)
// Construct the Shared Subspace
Sahre +- the first k columns of U from SVD SVD( (IB....., B,))
// Optimiza 2c A in the Subspace
The set of task vectors {T,}2 is known, and Eq. (5) repre- sents data-free objective that optimizes the modification vector A based on model parameters. This can be solved us- ing optimizers such as gradient descent, enabling the merged model to achieve enhanced performance on specific tasks. Next, we illustrate how to perform optimization within a shared subspace through gradient projection.
for iteration +- 1 to I do
12 ((A) + Σ Tb
(
11
(T(+++))) - 7,)²
j=1
VaL + VaL - Projs Sane (VaC)
Update A via gradient descent
return 0- + Ou + Σ (T)+
4.2. Shared Subspace Optimization
Model merging promotes multi-tasking capabilities within single model, which inevitably leads to parameter compe- tition across tasks. For the modification vector A, each task competes to minimize the loss of the merged model on its own task. Towards this end, we construct a shared subspace for all tasks to retain shared knowledge.
parameter updates À E (Tx + A) within the shared sub- space; (ii) the other component lies in the direction orthogo- nal to Sahare when learning Eq. (5). Notably, this component optimizes A without altering the shared knowledge, while minimizing the gap for task j. Thus, before taking a gra- dient step, the new gradients V are first projected onto Samer The projected components are then subtracted from the new gradient, leaving only the components orthogonal to S The updated gradients are calculated as:
Let S, = span {B represent the subspace spanned by the task vector J, where B; = (u.,....., x, is the basis matrix for Sj, consisting of A basis vectors extracted from task vector Tj. For any matrix A with suitable dimensions, its projection onto subspace Sy is defined as:
VAL= V&C - Projssme (VaL).
Projs, (A) = BB, A.
(6)
Compared to optimizing A in the original parameter space, our approach explicitly constrains the gradient directions the optimizer can take. By taking gradient steps in the direction orthogonal to the shared subspace, we narrow the gap with the task-specific model. This effectively mitigates task conflicts while retaining shared knowledge.
We utilize Singular Value Decomposition (SVD) to extract the rank-k subspace for the task vector. Specifically, the first k singular vectors from the left singular matrix are selected as By, forming an orthogonal basis that efficiently captures the primary information within the task-specific Tj- Once the subspaces for all tasks are established, they are com- bined into a shared subspace Sahare = span { [ [ B,)]. However, Sature includes overlapping singular vectors, indi- cating redundant parameters in the weight space across tasks. Such overlaps challenge the orthogonality requirement of basis vectors and lead to inaccuracies during projection onto the shared subspace. To mitigate this, we perform another SVD on Sahore to deduplicate it further, resulting in refined Sahure that effectively preserves shared knowledge.
4.3. Task-aware Training-free À
The sensitivity to À may arise from potential conflicts or intricate relationships among tasks, making the merging pro- cess highly dependent on the choice of this coefficient. To address this, we propose a direct method for computing task- aware X based solely on task vectors, thereby climinating the need for training or additional data. Building on rethink- ing of the role of X, we derive the following layer-wise, adaptive X calculation:
Eq. (5) can be projected onto the shared subspace, which al- lows the gradient to be decomposed into two distinct compo-
X
1ISL,
(8)
nents: (i) component projected onto Sature, which induces
5

---

Modeling Multi-Task Model Merging Adaptive Projective Gradient Deseent
Table J. Multi-task performance when merging ViT-B/32 models 8-task vision henchmark.
Method
SUN397
Cars
RESISC45
EuroSAT Non-Merging Methods
SVHN
GTSRB
MNIST
DTD
Avg.
Pre-trained
62.3
59.7
60.7
45.5
31.4
32.6
48.5
43.8
48.0
Individual
79.2
97.5
99.7
79.4
90.8
Traditional MTI.
73.9
74.4
93.9
98.2
95.8
98.9
99.5
77.9
88.9
Data- Free Methods
Task Arithmetic
55.2
54.9
66.7
78.9
80.2
69.7
97.3
50.4
69.1
Tee--Merging
59.8
70.7
86.2
98.3
54.2
72.4
Consensus Merging
65.7
63.6
76.5
77.2
81.7
70.3
97.0
57.1
73.6
AWD TA
63.5
61.9
72.6
84.9
85.1
79.1
98.1
56.7
75.2
PCB-Merging
66.7
65.5
78.5
79.3
86.4
77.1
98.2
59.1
76.3
Concrete
62.5
76.0
95.7
91.0
81.9
98.5
51.9
77.3
DOGE TA (Ours)
67.7
70.1
82.0
90.3
86.3
86.8
98.3
64.0
80.7
Test- Adaption Methods
AdaMerging
64.5
68.1
79.2
93.8
87.0
91.9
97.5
59.1
80.1
AdaMerging++
66.6
82.2
89.6
98.3
60.6
Representation Surgery
63.8
59.9
83.3
97.9
87.0
87.0
98.6
69.4
80.9
AWD AM
68.1
71.4
83.4
94.8
87.7
93.6
97.9
66.1
82.9
Concrete AM
67.8
70.0
87.5
96.0
91.6
96.7
98.7
63.8
84.0
DOGE AM (Ours)
70.5
74.8
88.7
94.1
91.6
95.7
98.8
72.5
85.9
where represents the number of layers, and 1y is hyper- parameter that sets the global magnitude. The computed X takes into account the differences between tasks, balancing single 7,,
simply defined as the rank of each task vector divided by the number of tasks (i.e., 8). Following Ties-Merging (Ya- dav al., 2023), retain only the top 30% of parameters with the largest magnitudes. We report Spearman's 's p for STSB and the standard average accuracy (%) for other tasks. Additional information on the experimental setup for model merging can be found in App. B.
the scale of the task vectors. By focusing we can replace the traditional task-wiss and layer-wise À search, reducing the risk of dominance by any single task.
To conclude, we concisely outline the pipeline of the pro- posed framework in Alg.
Compared baselines. We categorize the baselines into three main groups: Non-Merging methods, Data-Free meth- ods, and Test-Time Adaptation methods. The non-merging category includes individually fine-tuned models and a tra- ditional multi-task learning approach. The traditional MTL trains the base model on all tasks simultaneously, serving as an upper bound for multi-task model merging. The data-free methods we evaluate include Task Arithmetic (Ilharco et al., 2023), Ties-Merging (Yadav et al., 2023), Consensus Merg- ing (Wang et al., 2024b), AWD TA (Xiong et al., 2024), PCB-Merging (Du et al., 2024), and Concrete TA (Tang et al., 2023). Lastly, we include TTA methods such as AdaMerging (Yang et al., 2024c) (layer-wise) and Repre- sentation Surgery (Yang et al., 2024b). Further details about these baseline methods are provided App. C.
5. Experiments
In this section, we first describe our experimental setup. Then, we present our main results. We also provide ablation studies and discussions for thorough analysis.
5.1. Experimental Setup
Datasets and pre-trained models. For vision tasks, we use the ViT-B/32 and ViT-L/14 models, originally derived from CLIP (Radford et al., 2021). The downstream tasks encompass a variety of challenges, including SUN397 (Xiao al., 2016), Stanford Cars (Krause et al., 2013), RE- SISC45 (Cheng al.,, 2017), EuroSAT (Helber et al., 2019), SVHN (Netzer et al., 2011), GTSRB (Stallkamp et al., 1), MNIST (LeCun, 1998), and DTD (Cimpoi et al., 2014). For NLP tasks, we use the Flan-T5-base and Flan- Ta-large models (Chung et al., 2024), evaluated on eight tasks from the GLUE benchmark (Wang et al., 2019). Fur- ther details are provided App. A.
5.2. Main Results
Vision tasks. Tabs. and 2 present the results for the VIT- B/32 and ViT-L/14 architectures, respectively. Methods like Concrete Merging and Ties-Merging address parame- ter conflicts by eliminating certain neurons during model merging, outperforming baselines such TA. AdaMerging and AdaMerging++ automatically learn layer-wise merging coefficients on the test set in an unsupervised manner, also demonstrating strong performance. However, despite these
We perform 400 iterations of learning A with learning rate of 1e 4. The global mag- nitude of the merging coefficient is set to 0.07 for vision tasks and 15 for NLP tasks. The subspace basis size le is
Implementation details.
6

---

Modeling Multi-Task Model Merging Adaptive Projective Gradient Deseent
Table 2. Multi-task performance when merging ViT-I/14 models on 8-task vision benchmark.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg.
Non- Mon-Merging Methods
Pre-trained
66.8
77.7
71.0
59,9
58.4
50.5
76.3
55.3
64.5
Individual
99.2
Traditional MTL.
90.6
96.3
96.3
97.6
99.1
99.6
84.4
93.5
Data Free Methods
Task Arithmetic
73.9
82.1
86.6
94.1
87.9
86.7
98.9
65.6
84.5
Ties-Merging
76.5
85.0
83.3
Consensus Merging
75.0
84.3
89.4
95.6
88.3
82.4
98.9
68.0
85.2
AWD TA
76.2
85.4
88.7
96.1
92.4
92.3
99.3
69.4
87.5
PCR-Merging
76.8
86.2
89.4
96.5
88.3
91.0
98.6
73.6
87.5
Concrete TA
86.2
66.9
96.7
93.4
89.0
74.6
93.6
87.4
DOGR TA (Ours)
76.7
87.7
91.6
96.2
94.4
93.4
98.9
71.6
88.8
Test-Time Aduption Methods
AdaMerging
79.0
90.3
90.8
96.2
93.4
98.0
99.0
79.9
90.8
AdaMerging++
90.3
97.4
93.4
97.5
79.2
Representation Surgery
75.7
84.4
93.1
98.8
91.3
93.4
99.1
76.1
89.0
AWD AM
79,8
90.6
91.8
97.0
93.9
98.4
99.2
81.1
91.5
Concrete AM
77.8
91.2
92.1
97.0
94.4
97.9
99.0
79.5
91.1
DOGE AM (Ours)
79.7
91.6
94.4
96.7
96.5
98.6
99.0
84.1
92.6
Table 3. Multi-task performance when merging Flan-T5 -a-bee (LoRA fine-tuned) models on all eight tasks.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
Individual
69.1
82.7
85.5
90.9
84.0
84.4
92.9
87.4
84.6
Data-Free Methods
Weight Averaging
69.7
59.7
78.9
90.1
83.8
80.5
91.2
72.0
78.2
Task Arithmetic
68.8
55.2
78.7
89.8
83.7
79.1
91.5
72.4
77.4
Ties-Merging
68.3
56.3
79.4
89.8
83.7
79.4
91.6
71.2
77.5
Concrete TA
69.1
78.4
89.9
83.5
79.4
91.6
73.4
78.0
DOGE TA (Ours)
69.1
71.9
80.9
90.3
83.5
79.8
92.5
71.1
79.9
Test Time Adaption Methods
AdaMerging++
69.1
78.4
90.0
83.6
79.1
91.6
74.1
78.3
Concrete AM
69.0
59.4
80.1
89.9
82,9
79.1
91.7
75.4
78.5
advances, all existing model merging methods still show noticeable gap compared to individually fine-tuned models. AWD also optimizes but focuses on increasing orthogo- nality among task vectors, neglecting the performance gap with individually fine-tuned models. In contrast, our pro- posed DOGE orthogonal to existing merging methods and can complement them. When applied Task Arithmetic and AdaMerging, significant performance improvements are observed. For instance, on VIT-B/32, Task Arithmetic's s accuracy improves from 69.1 1 to 80.7% with DOGE. For the test-time adaptation method AdaMerging, accuracy in- creases from 80.1 % 85.9%. On ViT-L/14, AdaMerging achieves 92.6% accuracy after incorporating DOGE, nearly matching the 93.5% achieved by Traditional MTL.
TS-large models. Given that pre-trained I LLMS already ex- hibit strong multitasking capabilities, the potential for sub- stantial improvement via specialized methods inherently limited. Nevertheless, our approach achieves the highest performance, even outperforming TTA methods under data- free conditions. On -5-are- our data-free method achieves accuracy of 88.0%, closely approaching the per- formance of individually fine-tuned models at 89.6%. These results highlight the superior generalization ability of our method across diverse models and tasks.
5.3. Ablation Studies
Generalization and robustness evaluation. To further assess the generalization and robustness of our approach, we conduct experiments on unseen tasks and corrupted test sets (i.e., -illo--ddittibution).. Tab. 5 presents generalization results on two unseen tasks. On in-domain tasks, our ap- proach (under data-free conditions) performs comparably to AdaMerging, which leverages the test set for adaptation. Notably, on unseen tasks, where no corresponding task vec- tors were merged, our method outperforms AdaMerging by
Language tasks. We extend our approach to language models and LoRA fine-tuned models to evaluate its gen- cralizability (Li et al., 2023). Unlike classification tasks, text-to-text generation requires generating coherent outputs rather than merely projecting hidden representations log- its, introducing additional complexity (Li et al., 2025b). Tabs. 3 and 4 present the results Flan-T5-base and Flan-

---

most vital, causing notable performance drops of 8.8% and 3.5% in vision tasks, and 2.0% and 0.9% in language tasks, Modeling Multi-Task Model Merging Adaptive Projective Gradient Deseent
Table 4. Multi-task performance when merging Flan-T5-large (LoRA fine-tuned) models all eight tasks.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
Avg.
Individual
80.2
88.5
89.2
94.4
87.2
91.7
95.2
90.9
89.6
Data Free Methods
Weight Averaging
74.6
84.3
84.1
92.8
86.3
87.4
94.8
88.0
86.5
Task Arithmetic
76.9
85.4
85.3
93.9
85.8
88.1
95.2
87.8
87.3
Mees-Mggging
77.1
85.1
86.3
93.9
86.0
87.7
95.1
88.0
87.4
Concrete TA
76.6
86.4
86.0
93.9
85.9
88.4
95.2
87.9
87.5
DOGE TA (Ours)
78.4
88.1
86.5
93.8
86.3
87.7
95.1
87.7
88.0
Test-Time Adaption Methods
AdaMerging++ Concrete AM
76.7
87.6
84.8
93.8
85.9
88.1
95.2
88.6
87.6
76.1
87.7
85.5
93.8
85.9
88.1
95.4
87.1
87.5
Table 5. Generalization results
unseen tasks when merging ViT-B/32 models six tasks.
Seen Tasks
Unseen Tasks
Method
SUN397
Cars
RESISC45
DTD
SVHN
GTSRB
Avg.
MNIST EuroSAT
Avg.
Pre-trained
63.2
59.9
60.6
43.9
23.5
30.4
46.9
47.6
45.6
46.6
Task Arithmetic
64.3
63.0
73.2
54.9
84.7
79.5
69.9
75.5
42.6
59.1
Ties-Merging
68.3
65.5
76.9
54.9
75.4
72.0
68.9
73.1
47.3
60.2
AdaMerging
68.4
71.9
87.9
69.1
92.2
93.8
77.7
47.3
62.5
DOGE (Ours)
69.8
72.6
86.6
67.6
90.8
91.6
79.8
81.3
48.2
64.8
an average 2.3%, demonstrating superior generalization. By contrast, TTA methods rely the test set, which strains their ability generalize. Furthermore, Tab. 11 in Appendix evaluates each method's robustness corrupted test sets, designed to simulate real-world scenarios where input data may be noisy or corrupted. The results underline our approach's overall strength and efficacy, particularly in handling noise and -iitriibbititiooo data.
Subspace Woo Shared Subspace
1000
Explained Standerd Deviation D9
80
Shared
Aug. Accuracy
78-
60
76-
220
74
100
30
40
60
100
Preserved Rank Ratio (%)
Effects of each module. Tab. 6 evaluates the con- tribution of each module to overall performance. We start with DOGE TA and remove component at time, reporting the performance for full model merging (VIT-B/32) and for merging PEFT models (TS-base on GLUE). Removing A optimization corresponds to using the task-aware on TA, underscoring the effective- ness of the data-free objective applied to task vectors, which reduces conflicts between tasks. In cases where the shared subspace is removed, A optimization occurs in the original parameter space. This demonstrates that optimizing within the shared subspace enables the merged model to capture shared knowledge across multiple tasks. When task-aware is removed, we utilize a uniform merging coefficient of 0.3. Tab. 13 Appendix further presents the task-wise and laser-wise improvements over TA. Tab. shows that each component is crucial for achieving optimal performance: particularly, A optimization and the shared subspace are
Table Effects the proposed modules.
Figure 4. The average accuracy changes corresponding to different rank ratios in the subspace under ViT-B/32 architecture.
Model
VIT-B/32
T5-base
Task Arithmetic
69.1
77.4
DOGE TA
80.7
79.9
A Optimization
71.9
77.9
Shared Subspace
77.2
79.0
respectively. With all modules included, we achieve the hest performance, boosting TA by 5% % and demonstrating the complementarity of these components.
Task-Aware
79.2
79.8
Effects of the subspace. Since the effectiveness of method hinges the decomposition of the subspace, explore the impact of the rank (k of Sahare) on merging per- formance. Fig. 4 displays the performance with varying rank ratios, alongside the explained standard deviation (i.e., the ratio of preserved singular values a to the total sum of singular values E). Updates performed orthogonally the subspace direction have shown positive results, with the optimal rank identified between 10%-30% where the ex- plained standard deviation already exceeds 40%. Preserving higher rank introduces noise, resulting in a high volume of constraints in the gradient space. Updates along the direc- tion of the shared subspace also slightly outperform those in the original parameter space, due to the allowance for
8

---

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Deseent
Table? 7. Different gradient projection directions in the subspace when merging ViT-B/32 models.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Pre-trained
62.3
59.7
60.7
45.5
31.4
32.6
48.5
43.8
48.0
Individual
79.2
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.8
1. Shared Suhspace
67.7
70.1
82.0
90.3
86.3
86.8
98.3
64.0
80.7
w/o Shared Suhspace
63.3
67.1
74.9
85.2
86.9
83,9
98.2
57.9
77.2
w/ Shared Subspace
62.2
66.6
74.7
87.3
88.7
84.7
98.3
57.5
77.5
Table 8. Sensitivity analysis for the global scaling factor 7j-
Table 10. Normalized scores are computed relative to individual models when merging Wizardl M MM-13B 1-1 Cosstuctioo-Folloiinng WizardMath-133 13B (Math), and
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
ViT-B/32
79.5
80.3
80.6
80.9
81.0
80.8
80.7
80.2
79.8
(Code).
Methed
AlpacaFoal
GSMSK
MATH
HamanFyal
MRPP
Avg.
Table 9. The computational time and GPU memory requirements for optimizing A in the subspace.
Individual
100.0
100.0
100.0
100.0
TA
102.7
91.0
70.5
50.0
87.7
80.4
TIES
98.1
97.4
68.1
60.0
89.4
82.6
Model
Solving Time
GPU Memory
103.1
88.0
72.5
TIES DARE
107.9
90.3
65.6
80.0
92.4
87.2
VIT-B/32
121s 311s
729MB
Ours
107.5
105.0
94.4
56.7
86.5
90.0
VIT-L/14
2448MB
once the beginning, with a computational complexity of (min(mmn² m²n)). These findings highlight the near- universal scalability of our method on devices equipped with modern GPUs.
learning personalized subspaces. Tab. 7 reports the specific performance across eight tasks at the same rank. Com- pared to w/o or updates only along the subspace direction, we observe significant improvements on the DTD dataset but decreased performance on the SVHN dataset. This is attributable to DTD's requirement for rich textural and ge- ometric features, which are well-preserved in the shared subspace. Conversely, SVHN (Street View House Numbers) differs significantly in visual representation from other tasks, making the primary components in the shared subspace less suitable for SVHN. This is further evidenced by the gap from the pre-trained model to individual performance: SVHN shows the lowest pre-trained performance 31.4%, yet finetuning results peak at 97.5%, indicating a need for task-specific features. In summary, this demonstrates that our method effectively preserves shared knowledge across multiple tasks, achieving optimal overall performance.
Generative language tasks. We further extend our method to LLMs and conduct experiments following standard set- tings (Yu et al., 2024). The merging process is completed in just 58 minutes on single A100 GPU. We report normal- ized scores relative to the performance of individual mod- els when merging WizardLMM-33 Instruction-Followwe WizardMath-13B (Math), and 2-mma-2-13b-oode alpca (Code). As shown in Tab. 10, our method achieves the highest average performance across tasks, demonstrating its effectiveness and scalability in generative language tasks.
6. Conclusion
Existing merging methods often prioritize mitigating task conflicts, neglecting a critical requirement of model merg- ing: achieving performance comparable to task-specific models. In this paper, we rethink model merging from a multi-task learning perspective, treating it as a constrained optimization problem. We introduce an adaptive projective gradient descent method that optimizes a data-free objective within a shared subspace and includes adaptive merging coefficients. Extensive experiments validate the superior generalization and robustness of our approach, highlighting its effectiveness across various benchmarks.
Hyperparameter sensitivity. Additional sensitivity anal- ysis for the global scaling factor 77 is provided in Tab, 8. Evaluations across 17 values from 0.01 to 0.09 show that performance remains stable, even reaching higher values at certain points. (We did not conduct exhaustive grid search; this range was chosen because the computed was close to 0.03.) This consistent performance across different values demonstrates the robustness of our approach and highlights the practicality of task-aware coefficients.
Computational requirements. As illustrated in Tab. 9, our approach involves optimizing A within the subspace across 8 vision tasks over 400 iterations. The results demon- strate that our method incurs minimal computational over- head across different model variants and requires only mod- erate GPU memory. This efficiency is achieved through laser-wise optimization and fast convergence via gradient descent. Notably, the SVD operation is performed only
A Acknowledgmentss
This work is supported by the National Key R&D Pro- gram of China
SSTIC Grant (K/ZD202309221115100002 KJZD20230923I 14916032. GJHZZ202402181 11100400091111)) Beijing Key Lab of Networked

---

Modeling Mask Model Merging as Adaptive Projective Gradient Deseent
Multimedia, the Shenzhen Basic Research Project (Natu- ral Science Foundation) Basic Research Key Project (NO. (1112224122112440000 National Natural Science Foun- dation of China (No. 62025604).
Du, G., Lee, J., Li, J., Jiang, R., Guo, Y., Yu, S., Liu, H., Goh, S. K., Tang, H.-K., He, D., et al. Parameter competition balancing for model merging. In NeurIPS, 2024.
Gargiulo, A. A., Crisostomi, D., Bucarelli, M. S., Scar- dapane, S., Silvestri, F,, and Rodolà, E. Task singular vectors: Reducing task interference in model merging. In CVPR, 2024.
Impact Statement
The use of large-scale image datasets often involves privacy, labor, and ethical challenges, limiting research opportunities. As result, the research community is turning towards lever- aging pre-trained models. Model merging offers a novel approach to multi-task learning by utilizing the abundant expert models made available by the open-source ethos. With more than a million models accessible on Hugging Face, this strategy leverages the community's vast resources. This shift enables the creation of multi-task models by di- rectly merging independently trained expert models without needing original training data, presenting a new paradigm.
Guan, Y. Cheng, R., Liu, K., and Yuan, C. Enhancing logits distillation with plug&play kendall's ranking loss. arXiv preprint arXiv:2409. 17823, 2024.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Top- ics in Applied Earth Observations and Remote Sensing, 2019.
Horoi, S., Camacho, A. M. O., Belilovsky, E., and Wolf, G. Harmony in diversity: Merging neural networks with canonical correlation analysis. In ICMI,, 2024.
References
Chen, M., Jiang, M., Zhang, X., Dou, Q. Wang, Z., and Li, X. Local superior soups: A catalyst for model merging in cross-silo federated learning. In NearlPS, 2024.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 2017.
Huang, C., Ye, P., Chen, T., He, T., Yue, X., and Ouyang, W. EMR-merging Tuning-free free high- seth-perffrmacce model merging, In NeurlPS, 2024.
Choi, J., Kim, D., Lee, C., and Hong. S. Revisiting weight averaging for model merging. arXiv preprint 13X1v:2412.12/53, 2024.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In ICLR, 2023.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang. X., Dehghani. M., Brahma, S., et al. Scaling iestruction-iietunee language models. JMLR, 2024.
Jiang, J., Chen, B., Pan, J., Wang, X., Liu, D., Long, M., al. Forkmerge: Mitigating negative transfer in auxiliary- task learning. In NeurlPS, 2023.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine-grained categorization. In ICCVW, 2013.
Cimpoi. M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR, 2014.
LeCun, Y. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Crisostomi, D., Fumero, M., Baieri, D., Bernard, F., and Rodolå, E. 33: Cycle-consistent multi-model merg- ing. In NewrlPS, 2024.
Li, L., Zhang, T., Bu, Z., Wang, S., He, H.. Fu, J., Wu, Y., Bian, J., Chen, Y., and Bengio, Y. MAP: Low-compute model merging with amortized pareto fronts via quadratic approximation. In ICLR, 2025a.
Daheim, N., Möllenhoff, T., Ponti, E., Gurevych, I., and Khan, M. E. Model merging by uncertainty-baat based gradi- ent matching. In ICI.R, 2024.
Li, W., Peng, Y., Zhang, M., Ding, L., Hu, H., and Shen, L. Deep model fusion: survey. arXiv preprint arXiv:2309, AXiv:2309.16698, 2023.
Dong, S., Lu, F., Wu, Z., and Yuan, C. Dfvsr: Directional frequency video super-resolution via asymmetric and en- hancement alignment network. In IJCAI, 2023a.
Li, Z.-Z., Zhang, D., Zhang, M.-L., Zhang, J., Liu, Z., Yao, Y. Xu, H., Zheng, J., Wang, P.-J., Chen, X., et al. From system I to system 2: A survey of reasoning large lan- guage models. arXiv preprint arXiv:2502. /74/9, 2025b.
Dong, S., Wu, Z., Lu, F., and Yuan, C. Enhanced im- age deblurring: An efficient frequency exploitation and preservation network. In ACM MM, 2023b.
10

---

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Deseent
Liu, J., Moreau, A., Preuss, M., Rapin, J., Roziere, B., Tey- taud, F., and Teytaud, O. Versatile black-box optimization. In GECCO, 2020.
Sun, W., Li, Q.. Wang, W., Geng, Y-a., and Li, B. Task arithmetic in trust region: A training-fiee model merging approach to navigate knowledge conflicts. arXiv preprint arXiv:2501.1506. 2025.
Lu, Z.,, Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin-merging: Dynamic integration of modular expertise in model merging. In NeurlPS, 2024.
Sun, X., Panda, R., Feris, R., and Saenko, K. Adashare: Learning what to share for efficient deep multi-task learn- ing. In NeurIPS, 2020.
Maldonado, H. M., Möllenhoff, T., Daheim, N., Gurevych, I., and Khan, M. E. How to weight multitask finetuning? fast previews via bayesian model merging. arXiv preprint arXiv:2412.088 02. 2024.
Tang, A., Shen, I.,, Luo, Y., Ding, I.., Hu, H., Du, B., and Tao, D. Concrete subspace learning based interference elimination for multi-task model fusion. arXiv preprint arXiv:2312.06173, 2023.
Matena, M. S. and Raffel, C. A. Merging models with weight- averaging. In NeurlPS, 2022.
Tang, A., Shen, L., Luo, Y., Hu, H., Du, B., and Tao, D. Fu- sionbench: A comprehensive benchmark of deep model fusion. arXiv preprint (XXx:2446.00288 2024a.
Mugeeth, M., Liu, H., Liu, Y., and Raffel, C. Learning to route among specialized experts for zero-shot generaliza- tion. In ICML, 2024.
Tang, A., Shen, L.,, Luo, Y., Yin, N., Zhang, I.., and Tao, D. Merging multi-task models via weight-ensembling mixture of experts. In ICML. 2024b.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NewrIPSW, 2011.
Tang, A., Shen, L., Luo, Y., Zhan, Y., Hu, H., Du, B., Chen, Y., and Tao, D. Parameter-effficeet multi-task model fusion with partial linearization. In ICLR, 2024c.
Ortiz-Jimenez, G., Favero, A., and Frossard, P. Task arith- metic in the tangent space: Improved editing of pre- trained models. In NeurlPS, 2023.
Vershynin, R. High-dimensional probability: An introduc- tion with applications data science. Cambridge univer- sity press, 2018
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML, 2021.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benehmark and analysis platform for natural language understanding. In ICLR, 2019.
Saha, G., Garg, I., and Roy, Gradient projection memory for continual learning. In ICLR, 2021.
Wang. K., Dimitriadis, N., Favero, A., Ortiz-limenez, G., Fleuret, F., and Frossard, Lines: Post-training layer scaling prevents forgetting and enhances model merging. arXiv preprint arXiv:2410.171444 2024a.
Shen, I.., Tang, A., Yang, E., Guo, G., Luo, Y., Zhang, L., Cao, X., Du, B., and Tao, D. Efficient and effec- tive niight-ensembling mixture of experts for multi-task model merging. arXiv preprint arXiv:2410.22 1804, 2024.
Wang. K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F,, and Frossard, P. Localizing task information for improved model merging and compression. In ICML, 2024b.
Shi, G., Li, Q. Zhang, W., Chen, J., and Wu, X.-M. Recon: Reducing conflicting gradients from the root for multi- task learning. ICLR, 2022.
Wei, Y., Hu, Z., Shen, L., Wang, Z., Li, Y., Yuan, C., and Tao, D. Task groupings regularization: Data-fre free meta- learning with heterogeneous pre-trained models. In ICMI., 2024a.
Stallkamp, J., Schlipsing, M.. Salmen, J., and Igel, C. The german traffic sign recognition benchmark: multi-class classification competition. In IJCNN, 2011.
Stoica, G., Bolya, D., Bjorner, J., Ramesh, P., Hearn, T., and Hoffman, J. Zipit! merging models from different tasks without training. In ICLR, 2024.
Wei, Y., Hu, Z., Wang, Z., Shen, L., Yuan, C., and Tao, D. FREE: Faster and better data-free meta-learning, In CVPR, 2024b.
Stoica, G., Ramesh, P., Ecsedi, B., Choshen, L., and Hoff- man, Model merging with svd to tie the knots. In ICLR, 2025.
Wei, Y., Hu, Z., Shen, L., Wang, Z., Yuan, C., and Tao, D. vocabbuoyy customization from CLIP via data-free knowledge distillation. In ICLR, 2025.

---

Modeling Mask Model Merging Adaptive Projective Gradient Descent
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H Farhadi, A., Carmon, Y., Korbblith, S., and Schmidt, L. Model soups: averaging weights of multiple tne-tuned models improves accuracy without increasing inference time. ICML, 2022.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring large collection of scene categories, IJCV, 2016.
Xiong, F., Cheng, R., Chen, W. Zhang, Z.,, Guo, Y., Yuan, C., and Xu, R. Multi-task model merging via adaptive weight disentanglement. arXiv preprint arXiv:2411. 18729, 2024.
Yadav, P.. Tam, D., Choshen, L., Raffel, C. A., and Bansal, mers-ng Resolving interference when merging models. In NeurIPS, 2023.
Yang, E., Shen, L.,, Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. Model merging in lims, milms,, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666 2024a.
Yang, E., Shen, L., Wang, Z., Guo, G., Chen, X., Wang, X., and Tao, D. Representation surgery for multi-task model merging. ICML, 2024b.
Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024c.
Yu, L., Yu, B., Yu, H., Huang, F, and Li, Y. Language mod- els are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. In NeurlPS, 2020.
Zhang, F. Z., Albert, P., Modriguez-Opaazz C., van den Hengel, A., and Abbasnejad, E. Knowledge composition using task vectors with learned anisotropic scaling. In NeurlPS, 2024a.
Zhang, Q., Liu, X., Li, W., Chen, H., Liu, J., Hu, J., Xiong, Z., Yuan, C., and Wang. Y. Distilling semantic priors from sam to efficient image restoration models. In CVPR, 2024b.
Zhang, Q. Qi, Y., Tang, X., Yuan, R., Lin, X., Zhang, K. and Yuan, C. Rethinking pseudo-label guided learn- ing for weakly supervised temporal action localization from the perspective of noise correction. arXiv preprint arXiv:2501.11122 2025.
Zhou, Y., Song, L., Wang, B., and Chen, W. Metagpt: Merging large language models using model exclusive task arithmetic. In EMNLP. 2024.
12

---

Modeling Multi-Tass Model Merging as Adaptive Projective Gradient Deseent
A. Model Details
For vision tasks, we employ pre-trained models from CLIP (Radford et al., 2021), fine-tuning them using the AdamW optimizer with a weight decay of 0.1 and learning rate of 1 x 10-5, The downstream tasks encompass a variety of challenges. SUN397 (Xiao al., 2016) is large-scale scene recognition dataset comprising over 100,000 images across 397 indoor and outdoor scene categories. Stanford Cars (Krause et al., 2013) contains 16, 185 images of 196 car models and is commonly used for fine-grained image classification. RESISC45 (Cheng et al., 2017) consists of 31,500 remote sensing images evenly distributed over 45 scene categories, supporting research in acrial scene classification. EuroSAT (Helber et al., 2019) is based on Sentinel-2 satellite images and includes 27,000 samples covering 10 land use and land cover classes. SVHN (Netzer et al., 2011) 1) is a real-world digit recognition dataset with over 600,000 images of house numbers captured from Google Street View. GTSRB (Stallkamp et al., 201 1) comprises more than 50,000 images of 43 traffic sign categories, serving as a benchmark for traffic sign recognition tasks. MNIST (LeCun, 1998) is a well-known dataset for handwritten digit classification,, featuring 70,000 grayscale images of digits from 0 to 9. DTD (Cimpoi et al., 2014) is texture dataset with 5,640 images organized into 47 -esaanddssribbble categories, designed for studying texture perception and classification. We measure the models' performance using top-1 accuracy as the primary metric (Horoi et al., 2024; Stoica et al., 2024; Wei et al., 2025; Guan et al., 2024).
For NLP tasks, our pre-trained model is Flan-T5 (Wang et al., 2024a). We deploy Flan-T5 on eight tasks from the GLUE benchmark (Wang et al., 2019), including CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST2, and STSB. To ensure consistency and reproducibility, we use the same parameter-efficient models following FusionBench (Tang et al., 2024a). The Flan-T5 models, which are encoder-decoder Transformer models, undergo LoRA fine-tuning with hyperparameters r 16 and 32 (Hu et al., 2022). We maintain constant learning rate of 4 x 10 and a uniform batch size of 16 across all tasks, fine-tuning for 2000 steps per task. Adapting to the text-to-text framework, we have restructured the initial inputs accordingly. Performance is evaluated using exact match accuracy for all tasks, except for STSB where we report Spearman's p.
B. Implementation Details
The experiments in our study were conducted a consistent hardware setup, utilizing NVIDIA GTX 4090 GPUs equipped with 24GB of memory. We performed 400 iterations of learning A with a learning rate of 1e 4 using the Adam optimizer. The global magnitude of the merging coefficient v/ is set to 0.07 for vision tasks and 0.15 for NLP tasks. We did not perform a specialized grid search. This setting was chosen because the calculated average À was close to 0.3, which is beneficial scaling coefficient for the Task Arithmetic method, demonstrating that our approach is not tricky. The subspace basis size k is simply defined as the rank of the task vector divided by the number of tasks (i.e.., 8), with the shared subspace basis size set at the rank divided by 6. Following Ties-Merging (Yadav et al., 2023), we retain only the top 30% of parameters with the largest magnitudes. We only apply our method the linear layer in the model. For the implementation of our experiments, we employed Py PyTorch version 2.5 with Python 3.10.
C. Compared Baselines
Pre-trained: Uses a pre-trained model for each task without integrating task-specific information. Serves as a basic benchmark for comparison.
Individual: Fine-tunes separate model for each task, ensuring no task interference and providing ideal baseline for task-specific performance.
Traditional MTL: Trains single base model on all tasks simultaneously, representing the upper bound for multi-task learning.
Weight Averaging (Wortsman et al., 2022): Simply averages the weights of models fine-tuned on different tasks without considering task- ssecifpei dynamics.
Task Arithmetic (Ilharco al., 2023): Computes task vectors for individual tasks and sums them up construct multi-task vector. This vector is scaled by a coefficient (A) and added to the pre-trained model's 's initial parameters.
Fisher Merging (Matena & Raffel, 2022): Uses the Fisher information matrix to assess parameter importance, guiding the merging process to retain critical parameters for each task.
Ties-Merging (Yadav et al., 2023); Combines steps like trimming, parameter sign determination, and disjoint merging to
13

---

Modeling Multi-Task Model Merging Adaptive Projective Gradient Deseent
produce merged task vector The final model is defined 0 λr, where À tuned using validation set.
Consensus Merging (Wang al., 2024b); Improves traditional merging methods by removing "selfish" and "catastrophic" parameters beneficial only to specific tasks but detrimental to others.
weights-ppramettes
AWD (Adaptive Weight Disentanglement) (Xiong et al., 2024): Enhances orthogonality among task vectors to minimize interference and improve multi-task merging.
PCB-Merging (Du et al., 2024): Combines intra- bara-balancing, which evaluates the significance of parameters within individual tasks, and bnter-balan,ing, which measures parameter similarities across tasks. Parameters with low importance scores pruned, while the remaining parameters rescaled to create the final merged model.
Concrete Merging (Tang et al., 2023): Introduces meta-learning framework to generate concrete mask for mitigating task interference.
AdaMerging (Yang et al., 2024c): Learns wise or layer-wir wise merging coefficients adaptively using entropy minimization on unlabeled test data surrogate objective.
AdaMerging++ (Yang al., 2024c): Extends AdaMerging by incorporating task vector adjustments from Ties-Merging. removing parameter redundancies. and resolving sign confticts.
Representation Surgery (Yang et al., 2024b): Aligns the representation of the merged model with independent models while calibrating biases to ensure task compatibility.
D. Experiment Results
Table 11. Robustness the test data distribution VIT-B/32.
Method
Cars EuroSAT RESISC45 GTSRB Avg.
Cars EuroSAT RESISC45 GTSRB Avg.
Clean Test Set
Corrupted Test Set (Motion Blur)
Fisher Merging
66.0
92.7
83.7
78.7
80.3
60.7
57.6
81.7
78.4
69.6
Task Arithmetic
64.6
91.8
80.2
74.8
77.9
62.4
59.2
78.5
63.3
65.9
Ties-Merging
65.2
83.3
78.1
67.4
64.4
57.1
62.9
AdaMerging
75.2
94.3
87.6
96.7
88.5
72.4
72.7
94.3
81.2
DOGE TA
72.5
95.6
86.4
90.3
86.2
70.7
71.7
85.0
82.7
77.5
DOGE AM
77.3
96.4
91.5
97.6
90.7
74.7
79.1
89.5
95.7
84.8
Corrupted Test Set (Impluse Noisc)
Corrupted Test Set (Gaussian Noise)
Fisher Merging
61.5
50.0
74.7
52.6
59.7
61.6
48.1
76.0
51.3
59.3
Task Arithmetic
59.8
53.3
72.3
45.0
57.6
61.5
$2.5
75.0
50.1
59.8
Ties-Merging
60.2
45.6
69.8
38.3
61.8
42.3
56.1
AdaMerging
69.2
40.0
79.6
83.3
68.0
70.0
53.3
82.1
80.0
71.4
DOGE TA
66.7
57.2
79.2
61.0
66.0
68.7
50.9
81.7
64.1
66.4
DOGE AM
68.6
25.7
79.6
86.5
65.1
71.2
50.7
86.2
83.2
72.8
Corrupted
Test Set (Pixelate)
Corrupted
Test Set (Spatter)
Fisher Merging
2.2
34.0
17.0
63.2
29.1
61.4
64.2
74.6
47.3
61.9
Task Arithmetic
2.3
33.2
19.1
65.6
61.0
62.5
57.0
63.3
Ties-Merging
3.3
31.8
18.0
58.5
27.9
61.3
52.9
70.3
48.1
58.2
AdaMerging
1.3
52.9
21.0
91.0
41.5
68.4
55.9
78.3
92.3
73.7
DOGE TA
3.4
39.9
21.9
84.6
37.5
68.4
63.9
78.9
75.3
71.6
DOGE AM
1.4
55.9
25.8
93.4
44.1
71.0
54.5
83.9
93.4
75.7
Corrupted
Test Set (Contrast)
Corrupted Test Set (JPEG Compression)
Fisher Merging
63.8
58.4
75,5
70.4
67.0
66.3
67.6
82.6
58.9
68.8
Task Arithmetic
62.3
55.7
75.3
70.8
66.0
63.9
66.1
80.1
61.0
67.8
Ties-Merging
64.2
52.4
74.8
63.5
63.7
65.0
59.5
77.9
53.2
63.9
AdaMerging
73.1
67.4
83.0
96.2
79.9
72.9
70.7
86.3
90.6
80.1
DOGE TA
70.2
66.3
82.1
86.8
76.4
71.8
76.4
86.3
76.9
77.9
DOGE AM
75.1
73.5
87.9
96.9
83.4
75.0
78.1
92.4
83.9
14

---

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Deseent
Robustness. To evaluate our approach's 's robustness real-world variations, where data characteristics can significantly differ, we conducted extensive ablation studies across diverse data distributions. These studies specifically assessed the model's performance out-of-distribution (OOD) data (Zhang et al., 2024a;b; Dong et al., 2023a;b; Zhang al., 2025). To simulate real-world conditions, we introduced various types of noise into the test data following the procedure outlined by Yang et al. (2024c). Fight distinct noise types were used-motion blur, impulse noise, Gaussian noise, pixelation, spatter, contrast, and JPEG compression- to reflect a wide range of potential distortions encountered in practical applications.
The test sets included both clean and corrupted conditions to emulate distribution shifts. As shown in Tab. 11, while each strategy exhibited varying levels of robustness to different distortions, our approach consistently achieved the highest accuracy across most scenarios, often by notable margin. Notably, DOGE AM demonstrated exceptional resilience under severe conditions such as pixelation and spatter, significantly outperforming other methods. This consistent performance across diverse corruptions underscores DOGE AM's robustness and adaptability, making it particularly effective for challenging OOD environments in real-world applications.
We conduct experiments evaluating generalization on three unseen tasks when merging five other tasks. The results in Tab. 12 reveal that SUN397, DTD, and Cars datasets pose challenges for ViT models, while MNIST/EurOSAT show limited generalization to these complex tasks (Wei al., 2024b). Despite this, our method consistently outperformed other model merging approaches by significant margin.
Table 12 Generalization results on three unseen tasks when merging ViT-B/32 models five tasks.
Seen Tasks
Unseen Tasks
Method
RESISC45
SVHN
GTSRB MNIST
EuroSAT
Avg.
SUN397
Cars
DTD
Avg.
Pre-trained
60.6
23.5
30.4
47.6
45.6
41.5
63.2
59.9
43.9
55.6
Task Arithmetic
52.8
83.9
71.1
97.7
61.9
73.5
27.9
25.0
26.4
26.4
Ties- Ties-Merging
74.6
89.1
81.8
97.7
73.7
83.4
57.5
51.9
38.7
49.4
AdaMerging
73.5
76.0
81.5
97.4
69.4
79.6
42.3
37.8
32.0
37.4
DOGE TA (Ours)
82.6
89.4
89.0
98.6
92.3
90.4
58.7
54.3
41.4
51.5
Effects of X. Tab. 13 compares our two proposed variants of task-aware and layer-wise with the baseline Task Arithmetic. We observe that applying task-wise À provides a noticeable improvement over the baseline, boosting the average accuracy from 69.1% 11 to 70.7%. Further refining the granularity to layer-wise À achieves new highest average accuracy of 71.9%.
Table 13.
awake and training-fret free x combined with Task Arithmetic.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg.
Task Arithmetic
55.2
54.9
66.7
78.9
80.2
69.7
97.3
50.4
69.1
Task-wise A
61.4
62.5
70.0
82.8
71.3
66.4
95.1
56.1
70.7
Layer-wise
62.6
63.9
71.0
86.8
73.2
65.2
95.9
56.4
71.9
More task numbers. Tab. 14 illustrates the robustness of our approach when handling a larger number of tasks. Following Wang et al. (2024b), we evaluate its performance as more tasks are merged. In addition to the previously used 8 tasks, the 14-task scenario incorporates CIFAR100, STL Flowers 102, OxfordliITPet, PCAM, and FER2013. The 20-task scenario further adds six tasks: EMNIST, CI F0od101, FashionMNIST, RenderedSST2. and KMNIST. Our approach exhibits increasingly significant performance advantages as the number of tasks grows, demonstrating its effectiveness in mitigating negative transfer through gradient descent while preserving task-specific knowledge.
Comparisons with dynamic merging. As shown in Tab. 15, merging multiple models into a single model presents notable challenges. DOGE is static, plug-and-play merging method (similar to Task Arithmetic) that maintains the standard model size and supports parallelized inference. In contrast, dynamic merging approaches (Tang et al., 2024b; Huang et al., 2024; Lu et al., 2024) offer greater flcxibility by dynamically selecting task-specific modules but typically require additional storage and encounter scalability considerations during inference. These methods often rely on either dynamic I/O loading of modules or maintaining all components in GPU memory. For instance, some methods train routing networks using validation data to guide module selection.
15

---

Modeling Multi-Task Model Merging Adaptive Projective Gradient Deseent
Table 14. Average accuracy (%) when merging models larger number of tasks.
ViT-B/32
ViT-L/14
Method
tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Pre-trained
48.4 4
$7.3
$6.1
64.4
68.0
65.1
Weight averaging
66.5
64.4
61.1
79.4
76.6
71.5
Data- Free
Task Arithmetic
70.8
65.4
60.6
84.8
79.3
74.0
TIFS
75.1
68.0
63.4
86.9
79.5
75.7
Consensus Consensus DOGE TA (Ours)
75.0
70.4
65.4
86.2
82.2
78.9
74.8
67.7
63.2
86.9
81.5
76.8
80.7 (45.7%)
77.9 (+7.5%)
72.5 (+7.1%)
88.8 (+2.6%)
87.1 (+4,9%8)
81.0 (+2.1%)
Table 15. Distinction based parameters, data requirements, and computational costs.
Method
Parameters
Router
Data
Parallel Performance
Task Arithmetic
1x
static
69.1
AdaMerging
1x
unlabeled test dataset
static
80.1
DOGE
1x
static
(+11.6)
DOGE AM
1x
unlabeled test dataset unlabeled test dataset
static
85.9 (t5.8)
Representation Surgery
> 1x
static
80.9
EMR merging
4x
perfect router
dynamic
Twin merging
2.25x
trained router
labeled validation dataset unlabeled test dataset
dynamic
86.
WEMoE
5x
trained router
dynamic
89.4
Traditional MTL
88.9
Multiple Models
8x
90.8
Potential limitations A potential limitation is the lack of consideration for heterogencous model merging, which requires transformation when task vectors have inconsistent shapes or layer numbers.
16


Reviews:
[{"review_id": "Q28rZ5z8GY", "reviewer": "Reviewer_B1Xi", "summary": "This paper proposes a new perspective on model merging—treating it as a multi-task learning problem rather than merely a parameter-level combination of multiple expert models. The main idea is to preserve each task’s strong performance while reconciling the potential conflicts that arise when unifying several task-specific models into a single, merged network. To that end, the authors introduce an approach they call “Adaptive Projective Gradient Descent” (DOGE). This method formulates the model-merging goal as a constrained optimization problem that minimizes the “performance gap” between the merged model and each of the individual expert models, while explicitly retaining cross-task shared representations.\n\nThe procedure has three core steps. First, it refines (or “modifies”) each task-specific vector so that merging doesn’t simply discard conflict-ridden parameters that might actually be performance-critical. Second, it projects the gradients of these task modifications onto a shared subspace to maintain overlapping knowledge across tasks rather than forcing all task vectors into near-orthogonality. Third, it adapts the merging coefficients in a “training-free” way that is reminiscent of how adaptive optimizers dynamically adjust the learning rate; effectively, the magnitude of each merging coefficient is scaled inversely by the norm of the corresponding task vector. The overall pipeline is then shown to achieve strong performance across diverse architectures (vision and language) and tasks (classification, generation) without requiring access to the original datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed comments. We hope the following discussion can address your concerns!\n___\n>Q1: Some relevant work that tackle model merging on subspace need to be discussed.\n\nA1: Thanks for suggesting additional relevant work. We will discuss them in related work: *TSV [1] aggregates task vectors within their subspaces via low-rank approximation and whitens matrices to minimize interference. KnOTS [2] aligns representation spaces between LoRA models using SVD, enabling the application of merging methods. Both these methods and ours recognize parameter low-rankness and implement merging within subspaces.*\n___\n> Q2: Theoretical Rigor: The rationale behind using $−τ_j$ as a proxy for the gradient, could benefit from a more rigorous treatment.\n\nA2: Under the Neural Tangent Kernel assumption (i.e., fine-tuning often occurs in a linear regime), which has been validated in prior work [3,4], $\\nabla_ {\\theta}\\mathcal{L}_ j(\\theta_ 0)$ can be estimated as ${k\\tau_ i}$ where $k < 0$. Here, $\\tau_ j = θ_ T - θ_ 0 = -\\sum_ {t=1}^T\\alpha_ t\\nabla_ {\\theta_ t}\\mathcal{L}_ j(\\theta_t)$, where $\\alpha_t$ represents the learning rate and $T$ denotes the update iteration. Given the linearity of parameters $θ_0$ in the vicinity, we have $\\nabla_ {\\theta_t}\\mathcal{L}_ j(\\theta_t) = \\nabla_ {\\theta_0}\\mathcal{L}_ j(\\theta_0)$. Therefore, we derive $\\nabla_ {\\theta}\\mathcal{L}_ j(\\theta_0)=-\\frac{τ_j}{\\sum_{t=1}^T\\alpha_t}$.\n___\n> Q3: Hyperparameter Sensitivity: The method involves several hyperparameters (e.g., the subspace basis size $k$ and global scaling factor $\\eta$) whose selection may critically affect performance.\n\nA3: We have conducted experiments on the subspace basis size $k$ in Figure 4, which displays performance with varying rank ratios alongside the explained standard deviation. We also investigated the relationship between different projection directions and basis sizes. Additional sensitivity analysis for the global scaling factor $\\eta$ is supplemented as follows:\n| η |0.01|0.02|0.03|0.04|0.05|0.06|0.07|0.08|0.09|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ViT-B/32| 79.5 | 80.3 | 80.6 | 80.9 | **81.0** | 80.8 | 80.7 | 80.2 | 79.8 |\n\nThe evaluation of  across values from 0.01 to 0.09 demonstrates that performance remains stable and even achieves higher results. (We did not conduct a specialized grid search, this setting was chosen because the calculated $\\lambda$ was close to 0.3). This consistency across different $\\eta$ values verifies the robustness of our approach and highlights the practicality of applying task-aware coefficients.\n___\n> Q4: Computational Overhead: A deeper analysis of the additional computational costs (e.g., due to SVD and projection operations) would enhance understanding of the method’s scalability.\n\nA4: We have reported training time and memory usage in Table 10 of the Appendix, showing an efficient total training time of only 121 seconds and memory usage of 729MB. The SVD operation only needs to be executed once at the beginning, with a computational complexity of $O(\\min(mn^2, m^2n))$. We appreciate your reminder and will relocate this to the main text. Moreover, we supplement the final version with convergence loss curves for 8 and 20 tasks, showing that convergence is typically achieved within 100 to 200 iterations.\n___\n> Q5: Clarity in Derivations: A step-by-step explanation with more intuition would improve readability. For example, ∥θ∗ − θi∥_Gap apprears in Eq. (2) without introduction.\n\nA5: We apologize for any confusion caused. Eq. (2) is a brief mathematical summary presented before the detailed methodology. We have revised it and explained each symbol's meaning. Combined with the proof presented above, this will enhance the overall clarity of the derivation.\n___\n> Q6: It would be beneficial for the authors to include a discussion on potential limitations and directions for future research.\n\nA6: Thanks for your suggestion. A potential limitation is the lack of consideration for heterogeneous model merging, which requires transformation when task vectors have inconsistent shapes or layer numbers. Regarding future research, we are extending our work to LLMs by merging WizardLM-13B, WizardMath-13B, and llama-2-13b-code-alpaca, achieving SOTA performance. For detailed table results, please refer to our response to Reviewer `QBR6`.\n___\n> Q7: Should I expect gradient-based MTL methods can outperform task arithmetic.\n\nA7: Yes. Task arithmetic implements MTL in a training-free manner and can be viewed as a post-transfer approach for existing models, while MTL methods typically serve as performance upper bounds that we aim to approach.\n___\n[1] Task Singular Vectors: Reducing Task Interference in Model Merging. CVPR 2025.  \n[2] Model Merging with SVD to Tie the KnOTS. ICLR 2025.  \n[3] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS 2023.  \n[4] A Linearized Framework and A New Benchmark for Model Selection for Fine-Tuning. ArXiv 2021.", "claims_and_evidence": "- Improved performance on merged models: DOGE is claimed to achieve higher accuracy than previous data-free merging approaches by better preserving task-specific information while retaining shared representations.\n- Effectiveness of gradient projection: By projecting gradient updates orthogonally to the shared subspace, the method aims to resolve task conflicts without sacrificing common knowledge.\n- Task-aware coefficient design: The adaptive (training-free) merging coefficients based on task vector norms provide a natural way to balance gradient contributions across tasks.\nThe evidence supporting these claims comes from extensive experimental results on multiple benchmarks in both vision and NLP domains. - Quantitative comparisons across a variety of baselines (non-merging methods, data-free approaches, and test-time adaptation techniques) and detailed ablation studies show significant improvements. Although the experimental evidence is robust, one might wish for more discussion regarding statistical variability (e.g., more error bar analysis or significance testing) to further solidify the claims.", "methods_and_evaluation_criteria": "The proposed method is well-motivated and methodologically sound. Key components include:\n\n- Data-Free Objective: Derived via a first-order Taylor expansion, the objective minimizes the loss gap between the merged model and each individual model by approximating the unavailable gradient with the task vector.\n- Shared Subspace Construction: SVD is used to extract task-specific subspaces, which are then combined and refined to form a shared subspace that guides the gradient projection.\n- Adaptive Merging Coefficients: Interpreting task vectors as cumulative gradients leads to a natural formulation where merging coefficients play a role akin to adaptive learning rates.\n- The evaluation criteria—such as average accuracy across tasks (or Spearman’s ρ for STSB in NLP) and performance on out-of-distribution or unseen tasks—are appropriate for demonstrating the effectiveness and generalization of the method. The comprehensive experiments across different architectures and task modalities further validate the method’s practicality.", "theoretical_claims": "The paper provides heuristic derivations rather than fully formal proofs. Notably:\n\n- The use of a first-order Taylor expansion to derive a data-free objective is a reasonable approximation given the unavailability of task data.\n- Approximating the gradient of the task loss at the pre-trained model using the task vector (i.e., −τ_j) is intuitively justified by interpreting the task vector as an accumulation of gradients.\n- The decomposition of the gradient into components within and orthogonal to the shared subspace is well-motivated, though the justification remains somewhat heuristic.\n\nWhile these theoretical insights are plausible and backed by empirical evidence, a more rigorous treatment or further formal analysis would help strengthen the theoretical claims.", "experimental_designs_or_analyses": "The experimental setup is comprehensive:\n\n- The authors evaluate on eight-task vision benchmarks using CLIP-based ViT-B/32 and ViT-L/14 models, as well as on eight-task language benchmarks using LoRA fine-tuned Flan-T5 models.\n- Multiple baselines, including both data-free and test-time adaptation methods, are used for comparison.\n- Detailed ablations assess the contribution of each module (∆ optimization, shared subspace projection, and adaptive λ), lending credibility to the claims about each component’s effectiveness.\n- Additional experiments on unseen tasks and corrupted test sets reinforce the method’s robustness.\n\nOne minor suggestion is to include more explicit details on computational overhead (I found some recent work also update ∆ by gradient descent but not sure how expensive this procedure is) and convergence behavior across varying numbers of tasks.", "relation_to_broader_scientific_literature": "The paper is well-situated within the broader context of multi-task learning and model merging:\n\n- It builds upon previous work in data-free model merging (e.g., Task Arithmetic, Ties-Merging) and test-time adaptation (e.g., AdaMerging).\n- It draws connections to multi-task learning strategies that emphasize gradient alignment and modular architectures.", "essential_references_not_discussed": "Some relevant work that tackle model merging on subspace need to be discussed:\n- Gargiulo, A. A., Crisostomi, D., Bucarelli, M. S., Scardapane, S., Silvestri, F., and Rodola, E. Task singular ` vectors: Reducing task interference in model merging. arXiv preprint arXiv: 2412.00081, 2024.\n- Stoica, G., Ramesh, P., Ecsedi, B., Choshen, L., and Hoffman, J. Model merging with svd to tie the knots. arXiv preprint arXiv: 2410.19735, 2024.", "other_strengths_and_weaknesses": "Strengths:\n\n- Comprehensive Evaluation: The extensive experimental validation across both vision and language domains, along with detailed ablation studies, convincingly demonstrates the method’s effectiveness.\n- Practical Relevance: The approach is designed to work in data-free scenarios, which is particularly appealing in settings where access to original training data is restricted due to privacy or logistical concerns.\n\nWeaknesses:\n\n- Theoretical Rigor: Some derivations, particularly the gradient approximations and the rationale behind using −τ_j as a proxy for the gradient, could benefit from a more rigorous treatment.\n- Hyperparameter Sensitivity: The method involves several hyperparameters (e.g., the subspace basis size and global scaling factor η) whose selection may critically affect performance. More discussion on sensitivity analysis would be helpful.\n- Computational Overhead: A deeper analysis of the additional computational costs (e.g., due to SVD and projection operations) would enhance understanding of the method’s scalability.", "other_comments_or_suggestions": "- Clarity in Derivations: Some steps in the derivation of the data-free objective could be elaborated further. A step-by-step explanation with more intuition would improve readability. For example, ∥θ∗ − θi∥_Gap apprears in Eq 2 without introduction.\n- Limitations and Future Work: It would be beneficial for the authors to include a discussion on potential limitations (e.g., cases where tasks are highly heterogeneous) and directions for future research.", "questions_for_authors": "- How sensitive is the overall performance to the choice of the subspace basis size and the global scaling factor η?\n\n- Could you provide further empirical or theoretical justification for approximating ∇θLj(θ0) by −τ_j? Under what conditions might this approximation break down? Should I expect gradient-based MTL methods (https://github.com/thuml/awesome-multi-task-learning) can outperform task arithmetic (e.g. MGDA, CAGRAD, PCGRAD, IMTL...)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0kaWteE80", "reviewer": "Reviewer_97J9", "summary": "This paper addresses the challenge of merging multiple task-specific models into a unified model without accessing their original training data. The authors identify critical limitations in existing methods, such as discarding task-specific information during conflict resolution and over-enforcing orthogonality, which erodes shared knowledge. They propose DOGE, a constrained optimization framework that minimizes performance gaps via data-free gradient descent, projects updates orthogonally to a shared subspace (preserving common representations), and employs task-aware merging coefficients derived from task vector norms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Taylor expansion may need to point this out and justify.\n\nA1: During fine-tuning, parameter evolution in pre-trained models is frequently minimal, indicating that training remains within the tangent space where Taylor expansion closely approximates network behavior. This aligns with MAP, which examines task vector magnitudes and employs 2nd Taylor expansion to approximate metrics. It provides a formal proof regarding the negligibility of the remainder in Taylor series, and interestingly proposes using linear regression to estimate Hessian. Thanks for suggesting this related work! It strengthens our theoretical foundation, and we include this reference to further substantiate the rationale behind our approach.\n\nWe examined the difference between the 1st order Taylor expansion and the original loss, finding them to be within the same order of magnitude, confirming the accuracy of the estimation. Since calculating the gradient $\\nabla_{\\theta}\\mathcal{L}_j(\\theta_0)$ requires specific data $\\mathcal{D}_j$, we used task vector $\\tau_j$ as an approximation. Interestingly, when we attempted to optimize using actual gradients computed from specific data, we observed performance degradation. We attribute this to highly unstable gradients at the initialization, which complicated the optimization process. Thus, approximating the original loss using task vectors appears to be the superior way.\n___\n> Q2: Whether $\\Delta$ is input or how it is initialized.\n\nA2: $\\Delta$ is initialized as a zero tensor with the same shape as the task vector.\n___\n> Q3: Strong baseline methods such as EMR merging, Twin merging.\n\nA3: As discussed in Twin merging [2], they all belong to dynamic merging, which requires **additional storage for task-specific modules**. Such methods face **parallelization** challenges during inference, necessitating either dynamic I/O loading of task-specific modules or storing all modules in GPU memory. EMR merging requires priors during inference to load corresponding modules, while Twin merging trains a router using validation datasets to select modules.\n\nBoth EMR merging and Twin merging can be viewed as lightweight WEMoE [3], yet they still impose storage demands (**2.25× our approach**). For instance, EMR merging's proposed mask implementation still uses 8-bit Bool types, and Twin merging's module reconstruction $U\\Sigma V$ requires matrix operations that may not reduce peak GPU consumption. Notably, these approaches avoid direct comparison with WEMoE, which is unsurprising. According to the no free lunch theorem, performance increases with the number of retained parameters, with complete task-specific models representing the upper performance bound.\n\nOur approach, by contrast, is a static merging plug-and-play method (like TA and Ties merging) that maintains standard model size and enables parallelized inference. We compare our method with SOTA static merging approaches such as AdaMerging and PCB-Merging. **We believe methods should first be classified before conducting fair comparisons within each category. Otherwise, MoE methods will always outperform others simply due to larger parameter count.**\n___\n> Q4: Present the time/space that DOGE requires.\n\nA4: We have reported training time and memory usage in Table 10 of the Appendix, demonstrating remarkably efficient performance with only 121 seconds total training time and a memory usage of 729MB. We will relocate this information to the main text.\n___\n> Q5: Generalization performance on SUN397, DTD, and Cars.\n\nA5: Based on your request, we conducted experiments evaluating generalization on three unseen tasks when merging five other tasks. The results reveal that SUN397, DTD, and Cars datasets pose challenges for ViT models, while MNIST/EuroSAT show limited generalization to these complex tasks. Despite this, our method consistently outperformed other model merging approaches by a significant margin.\n|Method|Seen||||||Unseen||||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||RESISC45|SVHN|GTSRB|MNIST|EuroSAT|Avg.|SUN397|Cars|DTD|Avg.|\n|Pre-trained|60.6|23.5|30.4|47.6|45.6|41.5|63.2|59.9|43.9|55.6|\n|Task Arithmetic|52.8|83.9|71.1|97.7|61.9|73.5|27.9|25.0|26.4|26.4|\n|Ties-Merging|74.6|89.1|81.8|97.7|73.7|83.4|57.5|51.9|38.7|49.4|\n|AdaMerging|73.5|76.0|81.5|97.4|69.4|79.6|42.3|37.8|32.0|37.4|\n|DOGE TA|82.6|89.4|89.0|98.6|92.3|**90.4**|58.7|54.3|41.4|**51.5**|\n___\n> Q6: Is $\\Delta$ a task-specific vector?\n\nA6: Thanks for the suggestion. $\\Delta$ is a universal modification vector to each task vector. In our experiments, using a universal modification vector yields performance nearly identical to that of task-specific modification vectors, as they are mathematically equivalent when optimizing Eq.(5).\n___\n[1] EMR-Merging: Tuning-Free High-Performance Model Merging. NeurIPS 2024.  \n[2] Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging. NeurIPS 2024.  \n[3] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. ICML 2024.", "claims_and_evidence": "- Correct me if wrong, to use the Taylor expansion, the expanded point and the pretrained model should be very close. May need to point this out and justify. \n- In addition, did the author evaluate the performance of using this 1st order Taylor expansion to approximate the loss to validate this choice?", "methods_and_evaluation_criteria": "Methods: \n- In algorihtm 1, the authors should define $\\Delta$, whether it is input or how it is initialized. \n\nEvaluation: \n- The benchmark datasets are commonly used in task vector based model merging.\n- However, I would like to see the comparison between DOGE with other strong baseline methods such as EMR merging, Twin merging.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes, I checked the experiments. As I mentioned before, it would be great to add comparisons between DOGE with other strong baseline methods such as EMR merging and Twin merging.", "relation_to_broader_scientific_literature": "This paper is related to prior ideas including twin-merging (modulating shared and exclusive knowledge) and representation surgery (trying to make the representation of the merged model close to each individual task). It uses Taylor expansion to approximate the loss without using any data (similar to the idea in MAP (using Taylor expansion to approximate loss function)).", "essential_references_not_discussed": "For the Taylor expansion part, it would be helpful to cite a related work (MAP: https://arxiv.org/pdf/2406.07529) which also uses Taylor expansion to approximate the loss function / evaluation metric.", "other_strengths_and_weaknesses": "Strengths: \n- Empirical results (performance gains, robustness to task scaling, cross-domain generalization) convincingly demonstrate DOGE’s practical efficacy.\n\n- The plug-and-play design and compatibility with architectures like ViT/LoRA are validated experimentally.\n\nWeaknesses:\n- Since the method requires additional optimization and additional modification vectors for each task, I would like the authors to present the additional time/space that DOGE requires.", "other_comments_or_suggestions": "- In methodology section, $\\lVert\\cdot \\lVert_{Gap}$ and $\\lVert\\cdot \\lVert_{Sshare}$ make it seems like you are defining some new norms. I would avoid using them as subscripts of the norm symbol. \n- Table 5 and 6 are not numbered in the order they appeared in the paper. \n- Table 5: it is interesting the tasks selected MNIST and EuroSAT are the relatively easier tasks to the ViT models. It would be interesting to see the generalization performance on SUN397, DTD, and Cars.", "questions_for_authors": "- Is $\\Delta$ a task-specific vector? Since you mentioned $\\Delta$ is a modification vector to each task vector, and it is not indexed by the task, it was a bit confusing to me at the beginning. Maybe rephrase it to \"a universal modification vector to each task vector\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XULavflUQ2", "reviewer": "Reviewer_QBR6", "summary": "The authors introduced an approach to merging tasks for a multi-task learning purpose while maintaining performance comparable to task-specific models. They formulated the problem as a constrained optimization task, solved using adaptive projected gradient descent. To facilitate task merging, they introduced a modification vector for each task, acting as a correction mechanism. To achieve this, they constructed a shared subspace using SVD to capture common features, optimizing within this space to minimize task conflicts. The gradient of the modification vector is decomposed into two components: one projected onto the shared subspace and the other orthogonal to it. Additionally, they introduced merging coefficients based on the norm of task vectors to mitigate the dominance of any single task’s gradient influence.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The omission appears intentional, since the prior work demonstrates superior performance.\n\nA1: Our approach differs fundamentally from [1,2] in both setting and methodology. Our objective is to close the performance gap between model merging and multi-task learning **without introducing additional computation and memory requirements—a core and previously unresolved challenge in model merging research**.\n\n- ### Parameter\n  MoE architecture preserves MLP layers from each fine-tuned task-specific model and the pre-trained model, while additionally training a router module. In contrast, our merged model maintains a standard model size. The parameter comparison for ViT-B/32 (8 tasks) is as follows:\n||Total Parameters|\n|-|:-:|\n|Individual|113.45M|\n|Ours|113.45M|\n|WEMoE|573.96M|\n\n  The primary motivation for model merging is parameter reduction. If performance were the sole consideration, retaining each task-specific model would be the trivial solution. **Our method aims to compress multiple models (whether 8 or even 20) into a single standard-sized model, which aligns with the typical settings in model merging and multi-task learning**. As MoE methods (89.4%) exceed the performance upper bound of multi-task learning (88.9%), comparing our approach directly with MoE would be inappropriate.\n\n- ### Data Requirements\n  MoE approaches employ unlabelled test datasets to train the router module, whereas our optimization of task vectors is data-free. The performance benefits from test-time adaptation are self-evident. **Merging based solely on model parameters is more practical and represents the focus of most model merging methods**.\n\n- ### Computational Overhead\n  Static merging maintains **inference costs** equivalent to standard models, while MoE dynamic merging consumes more memory and computational resources (router + activated experts $k$). The inference phase memory usage comparison is as follows:\n||ViT-B/32 (8 tasks)|ViT-B/32 (20 tasks)|ViT-L/14 (8 tasks)|\n|:-:|:-:|:-:|:-:|\n|Ours|963.42MB|963.42MB|3772.63MB|\n|WEMoE|2750.65MB|5346.00MB|10063.64MB|\n\n  Similarly, test-time adaptation incurs additional **training costs**, while our method requires only lightweight training overhead (as shown in Table 10 of our paper):\n  ||ViT-B/32 (8 tasks)|ViT-L/14 (8 tasks)|ViT-B/32 (8 tasks)|ViT-L/14 (8 tasks)|\n  |:-:|:-:|:-:|:-:|:-:|\n  |Ours|729MB|2448MB|2.02min|5.18min|\n  |WEMoE|3744.19MB|24535.53MB|7.07min|56.84min|\n\n  Notably, our method can be trained layer by layer, enabling model merging for large models with minimal memory requirements.\n\n- ### Regarding Figure 3\n  Figure 3 visualizes task vector magnitudes, highlighting a phenomenon inherently observable across domain benchmarks. E-WEMoE and DOGE propose different approaches to address this phenomenon. Figure 3 was drawn with assistance from E-WEMoE authors to create a new version. **Associating performance gaps with non-citation introduces a conceptual misunderstanding, as fair comparison is impossible due to differing settings.** Meanwhile, we compare our approach with state-of-the-art methods in both data-free and test-time adaptation scenarios (described in lines 314-328). We appreciate your feedback and will introduce MoE-like methods and the clear differences.\n___\n> Q2: Each dataset should have a description. Most of the included datasets focus on classification. Can this approach be applied to heterogeneous MTL?\n\nA2: We will add descriptions for each dataset. Model merging in CV indeed focus primarily on classification tasks, following common experimental settings (as acknowledged by Reviewer `97J9`). Research on heterogeneous model merging remains limited, with existing work mainly centered on VGG and ResNet architectures using CIFAR datasets. We would welcome suggestions for appropriate benchmarks to explore this direction.\n___\n> Q3: SVD is computationally expensive. Can this approach be applied to Llama 2 or Llama 3?\n\nA3: SVD computation only needs to be performed once at the beginning. As shown in Table 10, which details the computation overhead, **our approach requires minimal memory and time**. We conducted experiments following standard LLM settings, completing the merging in 58 min on a single A100 GPU. We report normalized scores on merging WizardLM-13B (Instruction-Following), WizardMath-13B (Math), and llama-2-13b-code-alpaca (Code). Our method achieves optimal average performance across tasks.\n||AlpacaEval|GSM8K|MATH|HumanEval|MBPP|Avg.|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Individual|100.0|100.0|100.0|100.0|100.0|100.0|\n|TA|102.7|91.0|70.5|50.0|87.7|80.4|\n|TIES|98.1|97.4|68.1|60.0|89.4|82.6|\n|TA + DARE|103.1|88.0|72.5|63.3|92.9|84.0|\n|TIES + DARE|107.9|90.3|65.6|80.0|92.4|87.2|\n|Ours|107.5|105.0|94.4|56.7|86.5|**90.0**|\n___\n\n[1] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. ICML 2024.  \n[2] Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging. ArXiv 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "No", "relation_to_broader_scientific_literature": "The authors are trying to tackle the issue with merging parameters of the model achieving good performance comparable to task-specific methods by reducing conflict of tasks. They mentioned most of the other literature work that addressed this issue.", "essential_references_not_discussed": "Yes, the Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging was introduced in the paper \"Merging Multi-Task Models via Weight-Ensembling Mixture of Experts\", published at ICML 2024. Additionally, an extended arxiv version, \"Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging\" (E-WEMoE), further refines this approach. Both papers should be included in the related work section for a comprehensive discussion.", "other_strengths_and_weaknesses": "Strengths:\nThey applied their approach on Vision and NLP tasks.\nWeakness:\nEach dataset should have a brief description. \nMost of the included datasets focus on a single task, primarily classification. Can this approach be applied to heterogeneous MTL?\nSVD is computationally expensive. Can this approach be applied to Llama 2 or Llama 3?\nTraditional MTL needs to be clarified more. For instance, what is its architecture?\nThe results were not compared against the WEMoE published in this paper “Merging Multi-Task Models via Weight-Ensembling Mixture of Experts” and  E-WEMoE frameworks presented in the paper \"Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging\". Additionally, Figure 3 is similar to one in the paper \"Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging”.\nFor vision tasks, your results fall short compared to those reported in the paper “Merging Multi-Task Models via Weight-Ensembling Mixture of Experts” and the paper \"Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging\".", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUku0glI3t", "reviewer": "Reviewer_tyLA", "summary": "This paper views model merging from a multi-task learning angle. It designs an adaptive projective gradient descent method that tries to minimize the gap between the merged model and individual models, subject to the constraint of retaining shared knowledge. Specifically, the method only uses gradients in the orthogonal direction of the shared space of task vectors. Experiments show performance improvement compared to baseline methods.\n\n## update after rebuttal\nAs discussed during the rebuttal, I generally support the acceptance of this paper despite its performance gap issue. The proposal should be helpful for some other researchers working in this field.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and detailed comments. We hope the following discussion can address your concerns!\n___\n> Q1: Based on Tables 1, 2, and 3, the model merging proposal does not maintain the same level of performance as the individually trained ones, and is even worse than multi-task learning in many cases.\n\nA1: Transfer learning has driven the proliferation of fine-tuned models, **but deploying separate models for each task creates significant storage and computational burdens.** While multi-task learning could address this, it involves costly training and simultaneous access to all tasks. Additionally, determining the optimal data mixture for effective multi-task training can be complex and resource-intensive. Model merging addresses these challenges by **compressing** task-specific models without requiring access to training data (privacy or copyright). The performance gap between merged model and individual models or multi-task learning is an inherent constraint, as merging multiple trained models into a single model occurs without the benefit of costly computations.\n\nWhile previous methods focus on alleviating conflicts between tasks, our approach takes a more direct path by establishing the minimization of the gap between the merged model and individual models as our explicit optimization objective (Line 24). By formulating and effectively solving this as a data-free constrained optimization problem, we achieve significant performance improvements. On ViT-L/14, our method reaches 92.6% performance, approaching the 93.5% achieved by multi-task learning—a substantial narrowing of the gap. We have revised the description of model merging requirements in Line 18, and greatly appreciate your suggestion.\n___\n> Q2: Theoretically, it is also unclear why \"only take gradient steps in the direction orthogonal to the shared space\" (Line 68) can help us achieve this ambitious goal. The argument is not convincing.\n\nA2: The optimization objective in Eq. (5) promotes orthogonality between task vectors to mitigate conflicts, **while multi-task learning similarly emphasizes shared representations**. Parameters between similar tasks can be shared (e.g., applying the MNIST task vector improves accuracy on SVHN). Therefore, we propose constructing a shared subspace $S_{share}$ to preserve common representations. By constraining task vector optimization to reduce updates along $S_{share}$, we maintain shared knowledge while minimizing the gap for each task as defined in Eq. (5).\n\nAblation studies demonstrate a 3.5% improvement with $S_{share}$. Table 7 presents a comparison of different gradient directions, revealing dataset-specific performance variations. Our method achieves significant improvements on the DTD dataset while showing decreased performance on SVHN. This pattern stems from DTD's reliance on rich textural features that are preserved in $S_{share}$. In contrast, SVHN's visual representations differ substantially from other tasks, making the primary components in $S_{share}$ less suitable. This observation is further validated by examining the performance gap between pre-trained and fine-tuned models: SVHN exhibits the lowest pre-trained performance (31.4%) but achieves remarkable results after fine-tuning (97.5%), indicating its strong dependence on task-specific features. In summary, our approach effectively preserves shared knowledge across tasks while achieving optimal overall performance.\n___\n> Q3: The connection between the motivation (keeping task-specific information) and the method is not very clear.\n\nA3: To isolate task-specific information, the task vector is defined as $\\tau_i = \\theta_i - \\theta_0$ to capture unique characteristics for each task. While preserving task-specific information through simple vector addition is straightforward, the challenge in model merging lies in managing conflicts between multiple tasks. This challenge becomes evident in Figure 1, which demonstrates how performance consistently declines across all merging methods as the number of tasks increases, directly reflecting increased task conflicts.\n\nAs shown in Eq. (3), we measure the gap between the merged model and individual models **in terms of task-specific losses**. To alleviate conflicts, we introduce a modification vector $\\Delta$ for each task vector. This leads to our optimization objective in Eq. (4), which aims to achieve optimal cross-task performance by optimizing $\\Delta$. Through this optimization process, the merged model approximates the behavior of task-specific models while effectively resolving conflicts. In short, by minimizing our proposed loss function, we ensure the merged model preserves essential task-specific information.", "claims_and_evidence": "The main goal of the paper, which is ambitious, is \"ensuring the merged model performs comparably to task-specific models on respective tasks\". This is not well supported by the experimental results. Based on Tables 1, 2, and 3, the model merging proposal does not maintain the same level of performance as the individually trained ones, and is even worse than multi-task learning in many cases. \n\nTheoretically, it is also unclear why \"only take gradient steps in the direction orthogonal to the shared space\" (Line 68) can help us achieve this ambitious goal. The argument is not convincing.", "methods_and_evaluation_criteria": "The proposed method needs more justification to help me understand why it can help keep task-specific information while model merging.\n\nThe experimental evaluations are diverse and the proposal shows better performance compared to previous model merging methods.", "theoretical_claims": "The paper does not provide much theoretical evidence.", "experimental_designs_or_analyses": "The experimental designs are satisfactory, comparing the proposal with SOTA methods and analyzing different modules of the method.", "relation_to_broader_scientific_literature": "This paper is related to many works on model merging and multi-task learning.", "essential_references_not_discussed": "I am not aware of closely related works that were not discussed in the paper.", "other_strengths_and_weaknesses": "**Other strengths:**\n\n- This paper is interesting and shows promising performance compared to previous methods based on the experimental results. \n- The manuscript is well-written, with a proper discussion of previous works.\n\n**Other weakness:**\n\n- The connection between the motivation (keeping task-specific information) and the method is not very clear.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper initially received borderline scores. The main concerns raised by the reviewers included: (1) lack of discussion on related work, such as subspace-based methods for model merging; (2) insufficient justification for the theoretical analysis, particularly the linearity assumption of the loss landscape near fine-tuned parameters; (3) limited analysis of computational cost; and (4) missing comparisons with MoE-based methods, such as approaches using routers.

During the rebuttal, the authors addressed most of these concerns effectively, leading to a consensus among reviewers in favor of acceptance. In particular, the AC believes that comparing MoE-based and static model merging approaches purely in terms of accuracy is not very meaningful and that computational cost should be considered as part of the evaluation. For example, MoE approaches with a zero-compression rate are effectively equivalent to storing all fine-tuned parameters separately, diminishing their practical advantage.
Decision: Accept (poster)