Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning
Authors: Yeoreum Lee
Jinwook Jung
Sungyong Baik
Venue: ICLR 2025 Poster (2025)
Keywords: Model merging
Multi-task learning
Sharpness-aware minimization
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We connect sharpness-aware minimization (SAM) to model merging by focusing on its ability to find wider minima, and show that SAM can enhance the multi-task performance of the merged model.

Abstract: Large-scale deep learning models with a pretraining-finetuning paradigm have led to a surge of numerous task-specific models fine-tuned from a common pre-trained model.
Recently, several research efforts have been made on merging these large models into a single multi-task model, particularly with simple arithmetic on parameters.
Such merging methodology faces a central challenge: interference between model parameters fine-tuned on different tasks.
Few recent works have focused on designing a new fine-tuning scheme that can lead to small parameter interference, however at the cost of the performance of each task-specific fine-tuned model and thereby limiting that of a merged model.
To improve the performance of a merged model, we note that a fine-tuning scheme should aim for (1) smaller parameter interference and (2) better performance of each fine-tuned model on the corresponding task.
In this work, we aim to design a new fine-tuning objective function to work towards these two goals.
In the course of this process, we find such objective function to be strikingly similar to sharpness-aware minimization (SAM) objective function, which aims to achieve generalization by finding flat minima.
Drawing upon our observation, we propose to fine-tune pre-trained models via sharpness-aware minimization.
The experimental and theoretical results showcase the effectiveness and orthogonality of our proposed approach, improving performance upon various merging and fine-tuning methods.
Our code is available at https://github.com/baiklab/SAFT-Merge.

Published as a conference paper at ICLR 2025
MITIGATING PARAMETER INTERFERENCE IN MODEL MERGING VIA SARPPEESSAAARE FINE-TUNING
Yeoreum Lee', Jinwooo Jung' & Sungyong Baik 1 Dept. of Artificial Intelligence, 2 Dept. of Data Science Hanyang University [1eeyeoreum01, 1iw 1jw970517, jeanaake .c kr
ABSTRACT
Large-scale deep learning models with a pretraining-finetumne paradigm have led to a surge of numerous task-specific models fine-tuned from a common pre-trained model. Recently, several research efforts have been made on merging these large models into a single multi-task model, particularly with simple arithmetic on pa- rameters. Such merging methodology faces central challenge: interference be- tween model parameters fine-tuned on different tasks. Few recent works have fo- cused on designing a new fine-tuning scheme that can lead to small parameter in- terference, however at the cost of the performance of each task-specific fine-tuned model and thereby limiting that of a merged model. To improve the performance of merged model, we note that fine-tuning scheme should aim for (1) smaller parameter interference and (2) better performance of each fine-tuned model on the corresponding task. In this work, aim design new fine-tuning objec- tive function to work towards these two goals. the course of this process, we find such objective function be strikingly similar sharpness-awar minimiza- tion (SAM) objective function, which aims to achieve generalization by finding flat minima. Drawing upon our observation, we propose to fine-tune pre-trained models via sharpness-awan minimization. The experimental and theoretical re- sults showcase the effectiveness and orthogonality of our proposed approach, im- proving performance upon various merging and fine-tuning methods. Our code is available https: /ttp://///tttt/b/// com/ ba i aaa/ aaalaaa/aaaaa/a.a (AFTeMerge)
INTRODUCTION
Foundation model, large deep learning model pre-trained on large-scale datasets, has shown great advancement across a wide range of downstream tasks, after fine-tuning on each task (Achiam et al.) 2023 Saab et al. 2024 Ding et al. 2023), Recent successes of the have given rise to a burst of task specific open-source models in communities, such as Hugging Face. Diversity yet ready availability of large task-specific models have naturally elicited a question from researchers: Can we combine these large models into one, while retaining the performance on each task?
paradigm
Traditionally, a single multi-task model is obtained by jointly training on data across all tasks (Caru- ana 1997: Crawshaw 2020 Vandenhende al., (022). However, given the size of foundation models and the number of tasks, joint training on all tasks incurs significant computational costs.
Motivated by the accessibility, variety, abundance, and common origin of task-specific models, sev- eral research efforts have focused on merging multiple fine-tuned models into a single model via simple arithmetic on parameters of these models. thereby removing the need for joint training (II-)
harco et al.| 2023, Yadav et al. 2023 Yang et al., 2024b Matena & Raffel 2022; Jin et al.) 2023] 2023 Yang et al.
Daheim et al.) 2024.] Li et al.
2024a). However, a central challenge remains:
parameters of different task-specific models interfere or conffict with each other, leading to the per-
formance degradation of a merged multi-task model on each task.
To bridge such performance gap, several works have tried to reduce the parameter interference
during the process merging (Yadav et al. 2023 Jin et al. 2023 Yang et al. 2024b Wang et al.
Corresponding author.

---

Published as a conference paper at ICLR 2025
2024
2024). Another line of works focuses finding new fine-tuning scheme that results in task-specific models whose parameters have lower parameter interference (also often referred to as
Yu et al.
better weight disentanglement with respect to model outputs) (Ortiz-Jimeen
et al.. 2023] Tang et al. 2025) and thus less performance degradation after merging. Few studies (Wortsman) Jin et al.
2024
2022a| Ihaacco et al. 2023] Wortsman et al.) 2022b) suggest that the effectiveness of linear arithmetic on parameters in the process of merging may be owed to the linearity of fine-tuning process. Conversely, Ortiz-Jimenez et al. (023) have refuted such hypothesis by showing that there is huge performance drop from the approximation of fine-tuned models with linearized pre- trained model. Another observation they make is that such post-hoe linearized models led to less parameter interference. Based on this observation, few recent works Ortiz-Jimenez et al. (2023); Tang et al. (2024); Jin et al. (2025) have tried to explicitly linearize fine-tuning processes in order to induce weight disentanglement.
al.
In this work, we note that we need simultaneously work towards two goals for effective model merging: (I) reducing parameter interference between fine-tuned models while (2) maintaining the performance of task- specific fine-tuned models on respective datasets. Therefore, during fine-tuning process, aim to directly optimize for both performance with respect to performance. In the course of designing a fine-tuning objective function that aligns find striking resemblances between our goals and sharpness-aware minimization (SAM) (Foret et al., 2021), which aims for better generalization by finding flat minima via mini- mization of both loss values and loss sharpness. In particular, we find the similarities between the minimization of both loss values and loss sharpness in SAM and joint optimization for performance and weight disentanglement of fine-tuned models in our goal.
each task and weight disentanglement
with our goals,
Drawing upon our observations, we propose to obtain task-specific models from pre-trained mod- els via sharpness-ave awrpe fine-tuning (SAFT), in order to achieve better performance on each task, lower parameter interference, and thus better overall performance of a merged multi-task model. Our extensive experimental results demonstrate that our proposal greatly improves the overall per- formance of merged model. The effectiveness of our proposed method is owed to achieving better performance of each task-specific model and less performance gap between task-specific models and a merged model. We further highlight the generalizability and orthogonality of our approach by demonstrating performance improvements when applied together with various merging methods and fine-tuning methods for model merging.
RELATED WORKS
Model merging. The recent emergence of large foundation models and pretraining-iinttee paradigm has motivated researchers to explore ways of merging multiple task-specific models into a single multi-task model without re-training. Model merging, the merging of models with sim- ple arithmetic on parameters, has garnered significant amount of attention for its flexibility and simplicity. However, parameters of diffcrent task-specific models may interfere with each other dur- ing merging process, resulting in performance degradation on each task, compared to task-specific models. To address the parameter interference issue, researchers focus on either designing a merg-
process to mitigate the parameter interference. Initiated with simple averaging (Utans) 1996 Shoemake)
ing process (Utans 1996 Ilbarco et al. 2023 Yadav et al. 2023) designing
1985), research works on merging process focus on representing task-specific models task vec-
tors for easier manipulation of knowledge (Ilarcco et al. 2023), or weighting parameters (Matena & Raffel 2022 Jin et al.) 2023 Yang al.. (2044b or selecting parameters (Yadav et al. 2023) Wang et al. 2024 Yu et al. 2024) according to the estimated importance of each parameter with respect to given tasks. In parallel, task- sask-specific model parameters have less interference with each other to begin with, the effectiveness of model merging can be amplified. As such, few recent works have focused on designing a fine-tuning process such that resulting fine-tuned model parameters will have less interference and result in less performance gap between a merged model and task-specific mod- els. Ortiz-Jimaa Prtiz-Jimmnzzeee et al. (2023) show that linearized fine-tuning (fine-tuning in the space tangent to pre-trained initialization) leads to less interference (specifically better weight disentanglement with respect to model outputs), aspring other linearized fine-tuning methods (Jin et al. 2025 Tang et al. 2024).
2

---

Published as a conference paper at ICLR 2025
Sharpness-aware minimization (SAM). Foret et al. (2021) introduce optimization objec- tive function that minimizes both loss and loss sharpness to seek flat loss minima that may lead to better generalization performance. SAM defines loss sharpness the maximum loss difference measured at current parameters and nearby parameters (obtained by perturbing current parameters). Several follow-up works have strived improve SAM via improving perturbation methods (Mi) et al. 2022] Kwon et al. (021), improving gradient (Wang et al. 2023; Zhao et al. (022), or combining with other flatness-aware optimizers (Cha et al. [021 Kaddour al.) 2022 for better generalization. While previous studies have primarily focused on single-task learning, Phan et al. (2022) incorporate SAM into joint multi-tast training as regularization technique for multi- task learning. We note that our method and Phan et al. (2022) target two different scenarios. |Phan et al. (2022) target a traditional multi-task learning scenario, where the training performed on tasks jointly. By contrast, our work tackles multi-task model merging, where the goal is to merge different task-specific models, each of which is independently fine-tuned from a common pre-trained model without the knowledge of other tasks. This approach eliminates the need for joint training all tasks at the same time and avoids retraining from scratch when new tasks are introduced. In multi- task model merging, the lack of knowledge of other tasks also presents several challenges, such as parameter interference between different task-specific models that cause degradation of single-task performance after merging.
In this work, we introduce a new objective function for single-task fine-tuning aimed for model merging, from which we present a new insight that draws connections between the objective of multi-task model merging and that of sharpness-aware minimization (SAM). Furthermore, we the- oretically (in Appendix D and empirically show that, sharpness-aware fine-tuning can reduce pa- rameter interference, even without the knowledge of other tasks during fine-tuning.
3 BACKGROUND
Sharpness-aware minimization (SAM). To achieve better generalization, SAM (Foret et al. 2021 seeks for wider minima by minimizing both loss value and loss sharpness during optimization, where the loss sharpness is formulated as difference between a loss at the current parameters and the maximum loss value at nearby parameter values:
min
L(O
()
eill
loss
loss sharpnoss
where e is perturbation vector which is bounded above by a predefined p that controls the radius of the neighborhood; and are network parameters to be optimized for a given loss function over dataset DD For efficiency, |Foret et al. (2021) approximate the inner maximization via Taylor approximation. Then, along with canceling identical terms L(0; D) with opposite signs, the original optimization is reduced to
min L(e + E) where é VoL(O;D) Vo D)I
(2)
The perturbations within the same neighborhood radius for all parameters may impact each param- eter differently, especially if their scales differ by several factors. To take such varying scales of parameters into account, Adaptive SAM (ASAM) (Kwon al.) 2021 proposes scale the pertur-
bation vector e according to the scale of each parameter as ĆASAM scale of perturbations according that of parameters can be even effective in large foundation
Adjusting the
PiivaC(eDD]l
models with the pretraining- geetraining-ffnetunnng paradigm, since large pre-trained models likely have large number of parameters of different scales after training large-scale datascts.
Problem setting. the pretraining- netraining-finetuinng paradigm, there exists a large pre-trained model f X x θ - V. parameterized by trained parameters Og O that is in turn fine-tuned to T downstream
tasks. Each downstream task, indexed by t, is accompanied with a dataset (()) = {xx) (1) (0) N
where x! E ( C X' is input with corresponding label y E ( C y. Employing standard loss function (e.g., cross-entropy loss for classification) and an optimizer (e.g.. SGD), fine-
tuning pre-trained model fo, each downstream task will lead to task-specific model fo, with
3

---

Published as a conference paper at ICLR 2025
its parameters d::
0. arg min C(0; D())).
Task arithmetic. To merge models into a single model by performing an arithmetic on model parameters, Ilharco et al. (2023) have introduced the concept of task vector, which is essentially vector pointing to task-specific parameters 0, from pre-trained model parameters dg. obtained by taking difference between them: T 0₂- Ou- Iiharco et al. (2023) note that task vector тi can be considered as the representation of the knowledge learned for a task t. As such, they claim that the knowledge of each task can be manipulated by simple arithmetic on pre-trained model parameters: 0, + or where 04 > 0 will add the knowledge of task / while or < O will result in forgetting the knowledge of task t, while |ay| controls the extent of leamiin/forgettnng Using these task vectors Tr with corresponding task coefficients at, task-specific models can be merged into a merged multi-task model, parameterized by meerge as follows:
i meerge 0g + Σ
T.TT.
(4)
4 MITIGATING PARAMETER INTERFERENCE VIA SHARPNESS-AWARE FINE-TUNING
Since merged model is formed by simply performing linear arithmetic on task vectors, there is high chance for interference among tasks (Ilharco et al. 2023). Such interference leads to the perfor- mance degradation on downstream tasks after merging. Some works focus on reducing interference during merging process, which is a challenging task as fine-tuned model parameters are fixed. On the other hand, few recent works propose to modify fine-tuning process that results in task-specific models whose parameters have less interference with each other. In particular, they show that fine- tuning (partially) linearized model or only its linear layers results in less interference. However, such linearization of fine-tuning process results in the performance degradation of each task-specific model, limiting the overall performance of a merged model.
In this work, we claim that we need to achieve both (1) less performance gap between a merged model and each fine-tuned model (i.e., less parameter interference) and (2) generalization per- formance of each fine-tuned model on each respective dataset. As such, we aim to design a new objective function for fine-tuning to achieve these two objectives:
0, arg min
D(1) L(0; D(O) + C(O; D(
(5)
0
Objective (1)
Objective (2)
where merge (0) is to demonstrate that mergg changes as 0 optimized, while considering parame- ters for other tasks to be fixed. While this objective function already looks similar to the SAM objec- tive function in Equation|] [ after some simplifications (deriviations are delincated in Appendix get the final objective function follows:
0, arg min L(O + Σa,T, (a, 1)T; D(!)),
(6)
0
#f1
where Eoft T,T, (at- 1)T represents the parameter offsets a model merging process would introduce to the parameters of task-specific model for task Hence. Exjt Q,TT + (a,, l)T can be considered as perturbations that would cause parameter interference during model merging, from the perspective of each task-specific model. However, we do not assume access to other tasks while fine-tuning on each task, as each task-specific model independently trained. Since other tasks are unknown (and thus Eayee ATT, is unknown), we consider Eayie O&TT + (a, - 1)7 to be random perturbations. Furthermore, because the perturbation Expt CTTT + (ax 1)T depends on 0 6u and is thus varying during training, we use ASAM that models as parameter-dependee perturbation. In other words, we use ČASAM as a surrogate Σ.y; T,T, + (at (0t-11) approximati
our objective function for fine-tuning aimed for model merging (Equation 5 as
O'VoL(0, min L(O + E)) where ē VoL(O:
4

---

Published as a conference paper at ICLR 2025
From perspective described above, consider parameter interference be caused by pa- rameter perturbations Eef Ts + (a, I) that would be introduced during model merging, the information of which is however not available during fine-tuning for each task. The perturbations will bring model to new location in the loss landscape, away from the found local minimum. If the region around the local minimum is not flat enough, the new location (i.e., merged model parameters) brought by perturbations will most likely have higher loss, resulting in a large per- formance gap between a merged model and a task- specific model. In other words, to minimize the interference caused by parameter perturbations, it is essential identify flat minima. Flat minima can effectively prevent the loss from increasing after parameter perturbations (e.g., model merging). Thus, we argue that finding flat minima (or equivalently, minimizing sharpness) via sharpness-awe aware fine-tuning (SAFT) can greatly reduce parameter interference.
EMPIRICAL AND THEORETICAL ANALYSIS
1.5
DuroSAT, ty SUNSST bejo ASAN0
2: EUSSAT,
SURG97 (W ASAM)
2:: DTD, BureSAT (WN ASOM)
DTD,
EuroAT (N) ASAM)
1,0
1.0
1.059
1.0
1.0
1.193
0.794
0.294
0.5
0.5
0.596
0.0
0.255
0.0
0.298
0.5
0.5
0.000
0.5
0.5
0.000
0.0
-0.
0.0
0.5
1.5
1.5
1.244
1.5
1.5
1.466
1.0
1.0
1.13
0.746
0.879
0 0.5
95
6 9.5
0.5
0.488
0.586
0.0
0.0
0.248
0.0
0.0
0.293
0.5
0.000
0.000
-05
2.0
0.5
3.0
1.5
-0.5
0.0
0.5
1.0
1.5
-0.
0.0
0.5
1.0
1.5
-0.5
0.0
0.5
1.0
1.5
Figure 1: Weight disentanglement visualization of two task-specific models across two tasks. Each pixel in the heatmap corresponds to the weight disentanglement error {(01,02) between a two-task-merged model, parameterized by mergg = 00 + 72, and two task- models, parameterized by 0, + a T and 0g + T2, evaluated on task 1 and task 2. We use CLIP ViT-B/32 on EuroAAT-SUN397, DTD-EurOSAT, GTSRB-SVHN, and DTD-MNIST task pairs plot these visualizations. The red box highlights the search space used to find the optimal task coefficients {a1, ) for task arithmetic.
In this section, we experimentally validate argument by showing that our proposal, SAFT, leads
to better weight disentanglement (Figure T and Figure 2). better cross-task linearity (Figure 3)
and better joint-task loss linearity (Figure 4 and Figure 5). which are the signs of less parameter
interference. Better performance by our proposed method, compared to standard SGD and other fine-tuning schemes specifically designed for model merging, further underlines the effectiveness of SAFT in reducing parameter interference. Then, we also theoretically show that the capability of SAFT reduce the dominant Hessian eigenvalues induces joint-task loss linearity (the linearity of loss on all joint tasks).
Weight disentanglement. Ortiz-Jimenez et al. (203)) argue that for model merging via task arith- metic to be effective, weight disentanglement (a task vector for one task not affecting the outputs of task-specific model on other tasks) is necessary condition. In this work, we show that SAFT indeed achieves better weight disentanglement, in comparison to a standard objective function. The weight disentanglement is achieved when the parameter updates with the task vector for the task t (i.e., TT) impart a localized influence on the output of a model only when processing an input from the task without impacting the output of the model when inputs from other tasks are processed. Ortiz-Jimenez et al. (2023) formally express the localized influence of task vectors on the input space
T
f (xx; O meerge) = 1:00 00 + Σ a,T
1=1
=z Po
aTT))
when x E (1).
(8)
5

---

Published as a conference paper at ICLR 2025
TV EuroSAT, rs: SUNSST ASAO, 2: FunSAT, Ty SUNO97 (N) ASAMO
DTD, BurSAT oeto ASSM)
tol DTD, to: EuroSAT (ay 4SAM)
1.690
1.5
1.336
1.0
1444
1.0
1.326
1218
1.119
8 0.5
0.5
8 0.5
0.5
0:746
0.204
00.5
0.5
0.500
-0.5
-0.5
0.497
G75R8, ASAM)
GT5R8, ASAMI
DTD, MMST
DTD.
(w ASAMI
1.5
1.5
1.550
1.5
1.5
1,487
1.0
1.0
1.0
1.0
1.011
1.021
8
0.5
8
0.5
0.767
0.791
0.0
0.0
0.304
0.0
0.0
0.599
-0.
2.0
0.5
1.0
1.5
0.5
0.0
0.5
1.0
1.5
-0.5
0.0
0.5
100
1.5
00.5
0.0
0.5
1.0
1.5
Figure 2: Weight disentanglement visualization of tastttttts----eged models across two tasks. Each pixel in the heatmap corresponds to the disentanglement error ((01,02) between an eight- task-merged model, parameterized by Omerge = Og + 01 + 0272 (1 Ts and each task-specific model, evaluated task 1 and 2. To visualize the landscape of the merged multi- task model on 2D heatmap, we adjust only two task coefficients corresponding to the evaluation tasks. The models and evaluation task pairs used for the visualization are the same those used in Figure ! The meaning of the red box is the same as in Figure
To evaluate how well weight disentanglement is satisfied, Ortiz-Jimenez al. 2023 quantify dis- entanglement error as the discrepancy between the output of a merged model and t-th task specific model on input data of (-th task. Lower disentanglement errors imply that each task contributes ap- propriately without adversely affecting others. Following the experimental settings by Orttiz-Jimenez al. (2023), we first evaluate weight disentanglement during the merging of two tasks:
2
E(ai, (()))02)=
Σ Ex [dist (f(x; acTt), f(x; 0, + +TTTI 0222)))
where E(a1, a2) the disentanglement error with respect to two given tasks and visualized in Fig- ure [ We further stress-test and evaluate the disentanglement error while considering merging of all task-specific models (T in this work), thereby evaluating how well an actual merged model achieves weight disentanglement. However, it is difficuul to visualize if all T' task coefficients adjusted. In this work, for case of visualization, we adjust casf-coefficients of two tasks while fixing other task coefficients, while still considering merged model with all task vectors:
Eaul (01, dist f(x: Ou +
2 Σ B 1=1
fa;)) + aI 0222 +
Σ *1(1
EanT) aoTT.)
(10)
where Eall (a1, 02)) represents the total disentanglement error across all tasks (visualized in Figure 2 and dist(,,)) -) is a distance metric measuring the divergence between the outputs of the task-specilic model and the merged model. Small 5((11, 02) or (an( (x2) implies that the merged model pa- rameter mmerge better reflects the individual contribution of each task, signifying reduced parameter interference. Indeed, the visualizations of weight disentanglement when considering two tasks in Figure I and tasks (T 8) in Figure demonstrate the effectiveness our method in achiev- ing better weight disentanglement. In particular, note that the weight disentanglement error of the model merging with standard fine-tuning optimization increases significantly when considering all tasks in model merging. compared to considering two tasks. On the other hand, proposal, SAFT, reduces the weight disentanglement even when considering all tasks, further highlighting the effectiveness of our method in model merging.
Cross-task linearity. Cross- Task Linearity (CTL) (Zhou et al. 2024) is a property that ensures the linear separability of task influences the layer outputs across all layers of the network. To satisfy CTI, for every layer E, the layer output of a merged model should be approximately equal the combination of the layer outputs of individual task-specific models scaled by their respective coefficients. Formally, Zhou et al. (2024) define CTL condition as:
f(() (x: X0, (1 x)0,) () (z;; 0x) + (1 ()
(x;; (x:;,)), 0t),
(11)

---

Published as a conference paper at ICLR 2025
TV DTD, by MNIST
Ti EuroSAT, Ty SUN397
0.3
0.3
0.0
0.0
10
Block
We ASAM
Vc ASAM
Figure 3: Verification of CTL between the merged model and task-specific models. We compare Epeuupte [1 - cos 2Xr, 2XT,)] between sharpness-aware fine-tuning and SGD. The values for the last six blocks are evaluated on the two task pairs DTD-MNIST and EuroSAT-SUN397. We set the scaling factor À to 0.3.
where R scaling factor; æ (x)) U ( is an input from either task s or 0x O, are
parameters of task-specific models fine-tuned on task s and respectively; and f(() (æ: 0) represents the response (or feature) of (-th layer of a network f for the given input æ. The cross-task linearity each layer implies that the influence of one task on another minimal, thereby facilitating effec- tive weight disentanglement. Zhou et al. (2024) demonstrate that satisfying CTL condition leads to reducing the disentanglement error {(a).. J. To evaluate whether CTL is satisfied, the following cosine similarity metric is used:
cos (*) (x; 2AT 2AT.)
1 ((0 A(T + Ti)), ( 2ATs) + g (x; ((0
+ 2ATI)
(12)
CoS
The metric measures the cosine similarity between the layer output of merged model and the averaged layer outputs of the task-specific models. Following the settings in (Zhou et al. (0224),
use the metric [D (os (x:; 2AT,, 2AT:)] ) to evaluate how well CTL is satisfied, where smaller values of [E (f 2AT,, 2AT:)1 indicate stronger CTL. Since satisfying CTL leads to better weight disentanglement, smaller values of Ep cos (æ;; 2λT,, 2AT,) should result in lower disentanglement error E(a1, (22). as noted by Zhou et al. (2024). Figure shows that our method has lower Ep[[ co (x; 2XT,, 2) in comparison to SGD, demonstrating that our method results in not just better weight disentanglement, but also better cross-task linearity.
Joint-task loss landscape. ference by finding flatter minima across the joint tasks. shows the joint-task loss landscape visualizations for two task-specific models trained on corresponding two tasks. We observe that our method allows models to reach flatter minima across the joint tasks compared to SGD, particularly around the boundaries of the task coefficients search space task arithmetic. Our method increases the likelihood of finding merged model connected to each task-specific model along low-loss path, which indicates a smaller performance gap between the merged model and the task-specific models. Consequently, our proposal makes easier to find merged model with reduced parameter interference, compared to SGD.
empirically demonstrate that our method reduces parameter inter-
Yet, the tendencies observed with two tasks may not generalize in the case of more tasks, as the chance and extent of interference increases with the number of tasks (Yadav et al.) (023). There- fore, we also visualize the loss landscape of the multi-task model built by merging all eight tasks, parameterized by O = 0g + E. OuT To visualize the loss landscape of an eight-task-merged model on a 2D heatmap, we vary only the coefficients of two randomly chosen tasks.
Figure showws shows the loss landscape of multi-task model obtained by merging the parameters of eight task-specific models. Compared to Figure 4 the minima in the landscape shrink in every case as the number task-specific models to be merged increases. However, while the minima found by SGD shrink significantly, the minima found by method remain to be flat and wide in all cases. This suggests that our method maintains the ability to reduce parameter interference and preserve the performance of the merged model, even when more task-specifi models merged.
Theoretical results. Here, we further validate our proposal by theoretically demonstrating that SAFT leads to the loss of merged model and task-specific models being connected along linear
7

---

Published as a conference paper at ICLR 2025
TV DuroSAT, ts: SUNSST ASAMO 21 FurSAT, Ty SUNOST (N ASAMO
DTD, BurSAT oets ASSM)
To DTD, to: EuroSAT (ay 4SAM)
9.336
1.5
7.000
1.0
6.157
1.0
5,10
4.613
8 0.5
0.5
8 0.5
0.5
2821
2219
0.5
0.5
1.042
-0.5
-0.5
1.021
67588, ASAM)
GTSRB, ASAMI
DTD,
1.5
1.5
0.262
1.5
1.5
6.654
1.0
1.0
1.0
1.0
3874
4,302
8
0.5
8
0.5
2,400
3.246
0.0
0.0
1.486
0.0
0.0
2.110
05
2.0
0.5
1.0
1.5
0.5
0.0
0.5
1.0
1.5
-0.
0.0
0.5
100
1.5
0.5
0.0
0.5
1.0
1.5
Figure 4: Joint-task loss landscape visualization of two task-specific models across two tasks. Each pixel in the heatmap corresponds the loss values [(O [(0msrge D(1) + [(O D(2)) ) of the pro-task-merger model, parameterized by m = 0g + 01 Ti + Q2 evaluated on task and task 2. The setting of the model, task pairs, and red box is the same in Figure [ We use CLIP ViT-B/32 on the box highlights the search space used find the optimal task coefficients {a1, of task arithmetic.
and DTD-MNIST task pairs to plot these visualizations. The red
1.3
DuroSAT, Tp. SUNSST bejo ASAMO 2: EUrSAA, SURORT (N) ASAM)
12.54
CTD,
DureSAT oet ASOM)
DTD, DurSAT (N) ASAM)
7.541
1.0
1.0
10.50
1.0
1.0
6.717
8.45
5.593
0.5
0.5
0.0
0.0
3.345
0.5
0.5
2:33
0.5
0.5
2.221
-0.5
0.0
0.5
-0.
0.0
1.0
1.5
1.5
6:595
1,5
1.5
6.467
100
5
1.0
5,515
4.563
8 0.5
0.5
8 0.5
0.5
3232
3.11
0.0
2.111
0.
2,659
0.900
-0.5
1.707
0.0
0.5
1.0
-0.
0.0
0.5
1.0
Figure 5: Joint-task loss landscape visualization of tight-task-merged models across two tasks. Each pixel in the heatmap corresponds the loss values L [(0merge D(1)) + C(Omerge D(2)) of the hight-task-mergee model, parameterized by m 0 + Σ. ar Tr, evaluated on tasks and 2. We adjust only the two task coefficients corresponding to the evaluation tasks to visualize the weight disentangiement on a 2D map, as in Figure 2 The setting of the model, task pairs, and red box are the same as in Figure We use the same models and task pairs as illustrated in Figure
path on the loss landscape over all tasks. Formally, we define the loss over all tasks as joint-task loss and models being connected along linear path on the joint-task loss landscape as joint-task loss linearity as follows:
Definition (Joint-task loss). Given a joint-task dataset = DDD and model with parameters O, we define Joint-Task Loss, denoted as L (0: D),
CSTL D) L(0; D.) L(O; D:).
(13)
where L(0;D) Da) and L(0; D.) denote losses incurred by model, parameterized by on datasets D. and D., respectively.
Definition 2 (Joint-task loss linearity). Joint-task loss linearity (JTL linearity) describes the lin- ear relationship between the joint-task loss of an interpolated model and the weighted sum of the individual josn losses of task-specifer models. Specifically, given models, parameterized by O, and O, fine-tuned on respective tasks with datasets D, and Dr, we say JTL. linearity holds over joint-task dataset D = D. D.
Larz(af, + (1 a);;; aLmLL(0; (1 (1-1))/raaiil (),,),(
(14)
where E [0, 1] is scalar.
8

---

Published as a conference paper at ICLR 2025
Theorem 1 (SAFT induces joint-task loss linearity). If models, parameterized by 0, and t2, are obtained by fine-tuning from a common pre- pre-trained model via SAFT on their respective datasets, the models better satisfy the joint-task loss linearity. A proof is relegated to Appendix D due space constraints.
Joint-task loss linearity induced by SAFT implies that SAFT leads to merged model with less performance degradation in comparison to task-specific models on the joint task, thus implying that models fine-tuned by our method experience less parameter interference during the process of model merging.
EXPERIMENTS
In this section, following the settings from Ortiz-Jimenez al. (2033), conduct experiments diverse tasks to demonstrate the effectiveness of our proposal, sharpness-aware fine-tuning (SAFT), in improving the overall performance of merged model. We compare against three fine-tuning ap- proaches: SGD, linearized fine-tuning in the tangent space (FTTS) (Ortiz-limenez et al. (023), and fine-tuning linear layers only (FTLO) (Jin al.) 2025). We also validate the effectiveness, applica- bility, and generalizability of SAFT by assessing its performance in combination with three different model merging methods: weight averaging, task arithmetic (Ihharco et al. 2023), and TIES (Yadav
et al. 2023
6.1 TRAINING SETUP
Following the same training protocol outlined ITharco et al. (2022), we fine-tune two CLIP (Rad-
ford et al.) 2021) models: (a) ViT-B/32 and (b) ViT-B/16. Our experiments conducted across
eight diverse datasets: (1) Cars (Krause et al. 2013), (2) DTD (Cimpoi et al.) (014)), (3) Eu-
roSAT (Helber et al. 2019), (4) GTSRB (Stallkamp et al. 2011). (5) MNIST (Deng) 2012). (6)
RESISC45 (Cheng et al.] (017), (7) SUN397 (Xiao et al. 2016). (8) SVHN (Netzer et al.] 2011).
All fine-tuning processes begin from the same CLIP pre-trained checkpoint obtained from the open_clip (Radford et al. 202I) repository. We fine-tune each model for 8000 iterations with batch size of 128 and learning rate of 10-5 for all backbones and all fine-tuning methods. The learning rate schedule follows a cosine annealing approach with 500 warm-up steps, and optimiza- tion performed using the AdamW (Loshchilov & Hutter 2019). Consistent with I]harco et al. (022), we freeze the weights of the classification layer derived from encoding a standard set of zero-shot template prompts for each dataset. This strategy ensures that no additional learnable pa- rameters introduced during fine-tuning and does not compromise model accuracy. For more experimental details, please refer to Appendix
6.2 MAIN RESULTS
We evaluate the effectiveness of SAFT in closing the performance gap between merged model and each task-specific models, in comparison to other three fine-tuning approaches. Table shows that SAFT achieves the higher absolute and normalized accuracies in every case, compared to other fine-tuning methods. Normalized accuracy is defined as the absolute accuracy divided by the cor- responding accuracy of the fine-tuned task-specific model, evaluating the performance gap between a merged model and task-specific models. These results suggest that SAFT not only improves per- formance in downstream tasks but also narrows the performance gap between the merged model and fine-tuned models, improving the overall performance. Moreover, SAFT can be used together with other fine-tuning methods (FTTS (Ortiz-Jimenez enhancing the performance in multi-task settings during model merging, demonstrating its gener- alizability and applicability. In particular, FTTS (Ortiz-Jimenez al. 2023) and FTLO (Jin et al.) 2025) demonstrate better multi-task performance compared to SGD, these fine-tuning methods reduce interference between tasks by encouraging weight disentanglement (Malladi et al., 2023] Ortiz-Jimenez et al. 2023). Thus, the performance improvement brought by SAFT top of these fine-tuning methods demonstrates the orthogonality of our proposal.
al. 2023) and FTLO (Jin al., 2025).
In T our method is shown bring consistent performance improvement across diverse model merging methods and image encoder model backbones. Notably, our method brings performance improvement when used together with both weight averaging and task arithmetic. Task arithmetic

---

Published as a conference paper at ICLR 2025
Table I: Multi-task performance across different fine-tuning methods. We report the average absolute and normalized accuracies for three fine-tuning baselines: SGD, FTTS, and FTLO. Results are shown for three fine-tuning methods, grouped by whether SAFT-ASAM is applied. ViT-B/32 used the image encoder of CLIP, with task arithmetic as the model merging method in every case across eight tasks.
SGD
FTTS
FTLO
Fine-tuning method (-)
Abs.
Norm.
Abs.
Norm.
Abs.
Norm.
w/o SAFT-ASAM
68.23
75.47
78.35
86.83
75.93
85.74
SAFT-ASAM (Ours)
69.45
76.32
79.38
87.72
77.49
88.77
Table Multi-task performance across different model merging methods and image encoder models. We report the average absolute and normalized accuracies for three model merging meth- ods: weight averaging, task arithmetic, and TIES merging. We also compare the performance of two different models used as the image encoder of CLIP: ViT-B/32 and ViT-B/16. All cases are fine-tuned using SGD and evaluated across eight tasks.
Weight averaging Norm.
Task arithmetic Abs.
TIES merging
Merging method (-+)
Abs.
Norm.
Abs.
Norm.
ViT-B/32
SGD
65.72
72.91
68.23
75.47
74.57
82.29
SAFT-ASAM (Ours)
66.76
73.62
69.45
76.32
75.45
82.86
VIT-B/1G
SGD SAFT-ASAM (Ours)
71.58
77.37
73.40
79.31
77.94
84.04
71.84
77.53
76.77
82.50
80.14
86.23
searches for the optimal task coefficient within a given search space, while weight averaging is specific case of task arithmetic, where a; J. This suggests that our method finds flatter minima that covers the task coefficients search space in task arithmetic compared to SGD. Moreover, SAFT also performs better the case of TIES merging, indicating that interference mitigation by our method complements the parameter interference mitigation achieved by TIES merging.
7 CONCLUSION
In this work, draw connections between two research fields of machine learning: sharpness- aware minimization and multi-task model merging. Particularly, the connections are drawn from the formulation of two objectives of model merging: (1) reducing parameter interference between task-specific models and (2) achieving better generalization each task-specific model. Building objective function for fine-tuning, from
upon the objectives of model merging, we derive which we find similarities with sharpness-aware minimization. Upon observation, we propose to approximate newly derived fine-tuning objective function with sharpness-aware minimization: sharpness-awarr aware fine-tuning (SAFT). Experimental and theoretical results demonstrate that SAFT indeed results in less performance interference and better performance of merged model, even when applied together with other merging and fine-tuning methods designed for model merging. Motivated by the effectiveness and applicability of our proposal, we hope that this work encourages further research on investigating the relationship between SAFT and model merging, opening new research avenue.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education (N0.RS-2024-0033333555 and In- stitute of Information & communications Technology Planning & Evaluation (ITP) grant funded Artificial Intelligence Graduate School
by the Korea government (MSIT) Program (Hanyang University)).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXivv2303.0887/, 2023.
Atish Agarwala and Yann Dauphin. Sam operates far from home: eigenvalue regularization dynamical phenomenon. In ICML, 2023.
Rich Caruana. Multitask learning. Machine Learning. 1997.
Junbum Cha, Sanghyuk Chun. Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In NeurIPS, 2021.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the Proceedings of the IEEE, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796. 2020.
Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty- gradient matching. ICLR, 2024.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2012.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023.
Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In NeurlPS, 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awarr minimiza- tion for efficiently improving generalization. ICLR, 2021.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song. Hannanch Hajishirzi, Si- mon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulay models by interpo- lating weights. In NerulPS, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon W. Aver- aging weights leads to wider optima and better generalization. In UAI, 2018.
Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie Su, and Li Shen. Fine-tuning attention modules only: Enhancing weight disentanglement in task arithmetic. In ICLR, 2025.

---

Published as a conference paper at ICLR 2025
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro. and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. ICLR, 2023.
Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J. Kusner. When do flat minima optimizers work? NeurIPS, 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- tne-grained categorization. In ICCV Workshop, 2013.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for inveriinva learning of deep neural networks. In ICLR, 2021.
Tao Li, Weihao Yan, Zehao Lei, Yingwen Wu, Kun Fang, Ming Yang. and Xiaolin Huang. Efficient generalization improvement guided by random weight perturbation. arXiv preprint arXiv:2211.11489, 2022.
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimization. In CVPR, 2024.
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arxiv preprint arXiv:2309.15696 2023.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-awan aware minimization. In CVPR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 2019.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In ICML, 2023.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In NeurIPS, 2022.
Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness-aware minimization stronger: A sparsified perturbation approach. In NeurlPS, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco. Baolin Wu, Andrew Y. Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In NeurlPS, 2023.
Hoang Phan, Lam Tran, Quyen Tran, Ngoc N. Tran, Tuan Truong, Nhat Ho, Dinh Phung, and Trung Le. Improving multi-task learning via seeking task-based flat regions. arXiv preprint 2rXiv:2211.13722 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. ICMI., 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10..0/883......... 2019.
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother. Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models medicine. arXiv preprint 4404 18416, 2024.
Ken Shoemake. Animating rotation with quaternion curves. In SIGGRAPH, 1985.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In IJCNN, 2011.
12

---

Published as a conference paper at ICLR 2025
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial linearization. In ICLR, 2024.
Joachim Utans. Weight averaging for neural networks and local resampling schemes. In AAAI Workshop, 1996.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Denxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NerulPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In ICML, 2024.
Pengfei Wang. Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-awant gradient matching for domain generalization. In CVPR, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblihh, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. /CML, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In CVPR, 2022b.
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection scene categories. International Journal of Computer Vision, 2016.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. In NeurlPS, 2023.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.0744 07666, 2024a.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang. and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024b.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving gen- eralization in deep learning. In /CML, 2022.
Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence cross- task linearity in the pretraining-finetuiigg finetuming paradigm. In ICML, 2024.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
A.1 FINE-TUNING BASELINES
We compare the following three fine-tuning baselines with and without SAFT:
(1) SGD: This refers to standard fine-tuning that uses only an optimizer such as Adam W (Loshchlloy
& Hutter 2019).
(2) Fine- TuninT in the Tangent Space (FTTS) (Ortz-Jimenez et al. 2023): This fine-tunes the model in the tangent space its pre-trained initialization. achieves this by linearizing the model using a first-order Taylor expansion Stin (0; D) ʃ(0;; D) + (0 0o) V ((0;; D), where Og repre- sents the parameters of the pre-trained model and D is the training dataset. The method freezes dg and updates only 0.
(3) Fine-Tuning Linear Layers Only (FTLO) (Jin et al., 2025); This exclusively fine-tunes the linear layers within the attention module. Therefore, this method can only be applied to model architectures that include attention modules such as Transformer (Vaswani et al. 2017).
We utilize ASAM (Konn et al. 2021 as a default SAFT method in every experiments, since finds minima adaptively by considering correlation between generalization gap and sharpness. We set the value of ASAM to 0.5, following the default setup outlined in ASAM, along with all other ASAM hyperparameters.
A.2 MERGING METHODS
We merge the models that achieve the best performance for each corresponding task. These best models are selected based their performance validation set split, which is split from the training set at 0.1 ratio, as specified IIharco al. (2023).
We use the following model merging methods as baselines:
(1) Weight averaging: This merges fine-tuned models by averaging their parameters element-ve wise, denoted as merge T Σi= 0,, where 0, represents the fine-tuned parameters for each correspond- ing downstream task, T is the number of downstream tasks being merged.
(2) Task arithmetic (Ilharco et al., 2023): This method calculates task vectors = 0, 0 for each downstream task f, where 0, represents the fine-tuned parameters for task and O, represents the pre-trained parameters. A linear combination of these task vectors is then added to the pre- trained parameters, denoted as m 0, + Σ a,T,, where is a task coefficient that scales the corresponding task vector. This method generalizes the weight averaging when = for t = 1,,,,...,
Since the search space for a, becomes too large as the number of tasks increases, we set the task coefficients to be the same for all tasks and search for the optimal coefficient within the range [0.1, 0.3, 0.5, 0.7, 0.9, 1.0] using the validation set of each task.
(3) TIES merging (Yadav et al. 2023): This method mitigates parameter interference before merg- ing models. First, it trims parameters changed that change minimally during fine-tuning, these small changes in each model can become more pronounced after element-wise parameter merging. Second, it resolves parameter interference due sign conflicts by determining the sign of each parameter through a majority election before merging the models.
We apply TIES merging to task arithmetic. To find the optimal merged model, search for the task coefficients in task arithmetic within the range (0.1,0.3, 0.5, 0.7, 0.9, 1.0) and the percentile of task vectors to be pruned
within (0.7, 0.8, 0.9), using the validation set for each task.
A.3 VISUALIZATION SETUP
We produce the joint-task loss landscape and disentanglement under two distinct settings: (1) merging two task-specific models across two tasks, and (2) merging all eight task-specific models across two tasks.
14

---

Published as a conference paper at ICLR 2025
1. Two task-specific models across two tasks: In the first setting, we parameterize the merged model as:
merge 0 + 7 + Q272
where Ti and T2 represent the parameters of models fine-tuned on tasks 1 and 2, respec- tively.
2. The hight-task-mereer model across two Tasks: In the second setting, parameterize the merged model as:
i merge
0g + 0171 + 0227 +
Σ
ef
with 0.3, where т, denotes the parameters of the additional six tasks.
For both settings, use (a1, a2) pairs spanning from -0.5 1.5 with 21 evenly spaced points along each axis, resulting in 21 x 21 grid. For each (0) task coefficient pair on the grid, we compute the disentanglement eтror {(01. (2) and visualize the error values using contour plots to identify regions where the weight disentanglement is stronger. Since task coefficients are real numbers, we utilize contour plots effectively visualize the variations in loss landscape and disen- tanglement error across the continuous (01, (12) parameter space.
B DERIVATION OF EQUATION 6
We start with simplifying Equation 5 which is the objective function that incorporates the goals of model merging:
0. arg min C(O merge (0); D(1)) L(1; D( L(1; D()))
arg min mergg (0); ()))).
Here, consider task coefficients {a,} and other task vectors {T.)ayt to be fixed. Since then, instead of merge 0 + Σx=1 T, in Equation we express meerg as Oo- +Eoft T, acT, where T = 0 0g since 0, has not been found yet during the process of optimizing 0 for task We now have
0. arg min L(O, + Σ COST + OCT; D(9)
sylt!
arg min L(Op + T T +
Σ
T.T. + act; D(!)
##1
arg min C(O + Σ
asTa a:T; ((t)) : = 0 +
0
arg min L(1 +
Σ acTT (ar 1)T: Dit)).
sfft
C ADDITIONAL RESULTS
C. EFFECT OF THE NUMBER STEPS DURING FINE-TUNING
SAFT enables continued accuracy gains through longer training without the risk of overfitting. Therefore, we conduct an ablation study to evaluate whether SAFT can enhance the performance of single downstream task by doubling the number of training steps. In this study, employ ASAM. As shown in Table CT appears that SGD converges, as performance plateaus after 2000 steps. In contrast, SAFT continues to consistently improve performance up 8000 steps.
C.2 FINE-TUNING PERFORMANCE OF SAFT VARIANTS
To empirically justify our choice of SAFT variants for our main experiments, we evaluate various the SAFT variants the same datasets (i.e., eight vision tasks) our main experiments. In particular,
15

---

Published as a conference paper at ICLR 2025
Table Cl: Average accuracies of fine-tuned ViT-B/32 over steps across the eight tasks.
Fine-tuning steps
2000
4000
8000
SGD
90.37
90.21
90.48
SAFT-ASAM
90.50
90.84
91.03
we investigate how the performance of merged model changes when applying SAM (Foret et al., 202T ASAM wwon et al.) 202T), Friendly SAM (Li et. 2024), WA-SAM (Kaddour al. 2022),
SAGM Wang et al. 2023). PGN (Zhao et al. 2022), SSAM-F (Mi et al. 2022h, and SSAM-D (Mi
ct al. 2022), as shown in Table C2 The results demonstrate that ASAM brings better performance
improvement, compared to SAM and other SAFT variants. As result, ASAM shows the best single-task performance among other variants. Therefore, we use ASAM as a default SAFT variant in all experiments.
Table C2: Average accuracy of fine-tuned ViT-B/32 over steps across various SAM variants and fine-tuning methods.
SAFT variants
Accuracy
SGD
90.45
SAFT-SAM
90.16
SAFT-ASAM
91.29
SAFT-Friendly SAM
90.29
SAFT-WA-SAM
91.06
SAFT-SAGM
90.96
SAFT-PGN
90.90
SAFT-SSAM-F
90.80
SAFT-SSAM-D
90.60
C.3 CROSS-TASE LINEARITY (CTL)
We provide additional results that demonstrate that SAFT satisfies cross-task linearity on other pairs of datasets in Figure CI
We utilize VIT-B/32 the image encoder visualize this figure, just the same as Figure 3 The results show that our method achieves lower CTL scores across all layers for various task combinations. This suggests that our approach better satisfies CTL for a broader range of data, implying improved weight disentanglement and task arithmetic properties. Consequently, can be concluded that our method reduces parameter interference and minimizes the performance gap.
C.4 LOSS BETWEEN A MERGED MODEL AND -PECCICCIIIC MODELS
To demonstrate that SAFT indeed reduces the loss sharpness and the performance gap between a merged model and task-specific models, visualize loss changes between a merged model and a task-specific model on given task in Figure C2 Sharpness-awart fine-tuning indeed results in reduced loss barrier, leading to less performance gap exhibited in less weight disentanglement error, better cross-task linearity, and better overall performance in our main paper.
traverse along linear path
C.5 ADDITIONAL RESULTS OF FINE-TUNING BASELINES AND MODEL MERGING METHODS
Following Section 6. conduct experiments on all combinations of fine-tuning baselines (SGD, FTTS, FTLO) and model merging methods (weight averaging, task arithmetic, TIES), as summa- method leads to performance improvements
rized in Table C3 In the of weight averaging, in most cases, and for task arithmetic, it achieves performance improvements in all cases. For weight averaging, method improves performance in most cases, while task arithmetic consistently yields
16

---

Published as a conference paper at ICLR 2025
T:: Cars, T;
0.5
WD ASAM
ASAM
0.2
Stcck
Block
Bluck
Block
elack 4
Blck
$0,3 6
Blcck7
Blckk
Block
Block
Block11
Ti Cars, Ty: SUN397
ASAM
ASAM
0.3
0.0
DTD, EuroSAT
0.5
0.2
Slock
Black
Sloce
Booc
Bloce 4
Reckk
Bosk
Block
Bedk
Dicc
Sleck
Bioce
T: DTD, Ty MNIST
we ASAM
ASAM
0.3
0.0
Buck
Block
Bluck
GTSRB, SVHN
0.4
0.1
Back
Bisce
Ty
0.5
wi ASAM
ASAM
0.2
Slock
Slece
Bick:33
Beck
Diock
Meck
Diace
Modk
Disce11
T: EuroSAT, SUN397
web ASAM
0.4
A565
0.0
0.5
0.2
Beckk
Black
Sloce
Book
Oloce 4
Beckk
Bosk
Block
Stck
Dicca
Bock
Bioce 11
Figure Cl: Verification all blocks CTL between the merged model and task-specific models. We compare Eposupin (o (x;; 2XT,, 227:)). We set the scaling factor À 0.3.
17

---

Published as a conference paper at ICLR 2025
Task: Cars
Task: DTD
2.5
2.0
2.00 +1
1.5
1.5
1.0
Interpoiate corpppattion df
m/o ASAN
Interpalation of
Whb ASAM
Less Barnir Iwib ASAM:: MI: 0.75
ASAM
Loss Barner (elt ASAMI:: 0.00
ASAM
Task: SUN397
Task: SVHN
2.5
0.8
2.0
1.5
Interpolate atppoaatiinn I
M/o ASAM
Interpolation o Loss Barrier leih ASAMI:: 0.42
Less Barrier (eub ASAM:: Mt: o.
w/ ASAM
Figure C2: Test loss barrier between the merged model and each task-specific model.
performance improvements across all cases. contrast, TIES shows performance improvements in only half of the cases. Upon closer examination, when linear fine-tuning methods such FTTS and FTLO which regularize the model output to satisfy linearity - are used without SAFT-ASAM, TIES generally outperforms task arithmetic. However, with SAFT-ASAM, TIES consistently per- forms worse than task arithmetic.
This seems that since the combination of linear fine-tuning and SAFT-ASAM has already enhanced weight disentanglement and reduced parameter interference, parameter trimming via TIES may rather remove critical parameters ically, the combination of TIES and SAFT-ASAM delivers the best performance. Conversely, with FTTS and FTLO, task arithmetic paired with SAFT-ASAM achieves superior results. In some stances, TIES combined with SAFT-ASAM performs similarly to weight averaging. Thus, for linear fine-tuning methods like FTTS and FTLO, combining SAFT-ASAM with TIES can negatively im- pact performance. Additional analysis of this behavior is reserved for future work.
noisy parameters, leading to performance degradation. Specif-
C.6 MERGING WITH FIXED Ct.
Previous research (Ilharco al. 2023 Jin al. 2023 Matena & Raffel 2022 Yadav et 2023)
on model merging has focused on finding better merged models through hyperparameter tuning. However, such methods become increasingly costly as the number of hyperparameters grows, and they need to be re-applied whenever tasks are added or changed. Therefore, it is essential to create a robust merged model that performs well regardless of the selected hyperparameters.
Our method achieves robustness by identifying flatter minima for joint loss and weight disentangle- ment compared to SGD, enabling the discovery of optimal hyperparameters across a wider range of conditions. To support this claim, we evaluate task arithmetic by fixing the task coefficients to 0.4 for merging tasks, following the recommendation of Illarco et al. (2023). As shown in Table C4 our method outperforms other fine-tuning baselines in all cases, achieving improvements of up to 30% in both absolute accuracy and normalized accuracy. Additionally, there are several nearly identical to those of hyperparameter tuning. Therefore,
cases where the performances method ensures that a merged model with reliable performance can be obtained, even when arbitrary hyperparameters chosen.
18

---

Published as a conference paper at ICLR 2025
Table C3: Multi-task performance when merging CLIP image encoder on eight tasks. We report the average absolute and normalized accuracies for different five model merging, methods. Results are shown for the six fine-tuning methods, categorized by whether SAFT-ASAM applied.
Merging methods (-)) Fine-tuning baselines (L)
Weight averaging Abs.
Task arithmetic Abs.
TIES merging
Norm.
Norm.
Abs.
Norm.
ViT-B/32
SGD
65.72
72.91
68.23
75.47
74.57
82.29
SAFT-ASAM (Ours)
66.76
73.62
69.45
76.32
75.45
82.86
FTTS
72.47
82.04
78.35
86.83
76.89
86.84
FTTS w/ SAFT-ASAM (Ours)
75.10
86.10
79.38
87.72
73.77
84.46
FTLO
65.96
73.83
75.93
85.74
77.39
85.89
FTL.O w/ SAFT-ASAM (Ours)
65.34
72.78
77.49
88.77
76.30
84,62
VIT-B/16
SGD
71.58
77.37
73.40
79.31
77.94
84.04
SAFT-ASAM (Ours)
71.84
77.53
76.77
82,50
80.14
86.23
FTTS
77.20
84.87
79.37
87.33
81.09
89.05
FTTS w/ SAFT-ASAM (Ours)
78.09
86.45
79.78
88.26
78.41
86.72
FTLO
70.97
77.11
80.00
86.55
78.25
84.91
FTLO w/ SAFT-ASAM (Ours)
71.03
76.78
82.59
89.11
79.49
85.92
Table C4: Multi-task performance of task arithmetic with and without hyperparameter tun- ing. We compare the average absolute and normalized accuracies of task arithmetic whether the hyperparameter is tuned. The fixed hyperparameter is 0.4 for all eight vision tasks.
w/o tuning
w/ tuning
w/o tuning
w/ tuning
Fine- tine-tuning basclines (L)
Abs.
Norm.
Abs.
Norm.
Abs. Norm.
Abs.
Norm.
V/T-B/32
ViT-B/16
SGD
46.34
48.72
68.23
75.47
46.91
49.98
73.40
79.31
SAFT-ASAM (Ours)
57.27
62.06
69,45
76.32
71.37
76.34
76.77
82.50
FTTS
72.89
81.97
78.35
86.83
76.69
84.05
79.37
87.33
FTTS w/ SAFT-ASAM (Ours)
75.21
85.78
79.38
87.72
78.87
87.17
79.78
88.26
FTLO
48,20
55.29
75.93
85.74
77.36
83.60
80.00
86.55
FTLO w/ SAFT-ASAM (Ours)
79.68
87.92
77.49
88.77
82.50
88.95
82.59
89.11
C.7 MULTI-TASK PERFORMANCE. OF OTHER FLAT-MINIMA TECHNIQUES
We also evaluate the performance of other flat-minima techniques, in addition to SAFT variants like ASAM. Flat-minima techniques, including SWA (Izmailov al., 2018). RWP (L et al. 2022), and SAGM (Wang et al. (023). are fine-tuning methods designed to minimize the loss while finding flat- minima during model training. As shown in Table both our method and SWA improve multi-task performance compared to SGD. However, RWP and SAGM show worse performance than SGD. In addition, as discussed in Appendix C.6 we also present the results of performance evaluation without hyperparameter tuning in Table C Our method not only achieves the best performance when the hyperparameters are tuned but also outperforms in cases without tuning. Furthermore, the performance gap between method and SGD, well other flat-minima techniques, widens significantly in these scenarios. This demonstrates that our method is superior to other flat-minima techniques and exhibits less performance degradation due variations in parameter settings.
19

---

Published as a conference paper at ICLR 2025
Table C5: Multi-task performance across difffrent flat-minima techniques. We compare the average absolute and normalized accuracies of ViT-B/32 for five fine-tuning methods including three flat-minima techniques: SWA, RWP, and SAGM. We also compare the performance of merged model with and without hyperparameter tuning. All cases are merged with eight vision tasks by task arithmetic.
w/o tuning
w/ tuning Norm.
Fine-tuning basclines (L)
Abs.
Norm.
Abs.
SGD
46.34
48.72
68.23
75.47
SWA
48.94
52.46
68.58
76.11
RWP
34.97
36.58
62.48
73.02
SAGM
40.18
42.93
64.36
71.16
SAFT-ASAM (Ours)
57.27
62.06
69.45
76.32
The primary difference among flat-minima techniques lies in the strategies used to derive pertur- bations. This seems to influence how effectively these techniques can reduce the performance gap between each task-specific model and a merged model. Our method introduces perturbations by minimizing the loss difference between the current point in the parameter space and the point with the highest loss in its neighborhood during fine-tuning, This approach closely aligns with the ob- jective of model merging, which aims minimize the loss difference between the merged model and the individual task-specific models. Therefore, perturbation strategies derived from fine-tuning objectives similar to the model merging objective could result in greater performance improvements compared to other flat-minima techniques.
C.8 RESULTS ON NATURAL LANGUAGE PROCESSING TASKS
Table C6: Multi-task performance of the merged model across four NLP tasks. We report the average absolute and normalized accuracies four GLUE benchmark tasks: CoLA, MPRC, RTE, and SST-2. We fine-tune specific models.
using either SGD or SAFT-ASAM and merge the four task-
Fine-tuning baselines (4)
CoLA
MRPC
RTE
SST-2
Average Abs.
Abs.
Norm.
Abs.
Norm.
Abs.
Norm.
Abs.
Norm.
Norm.
SGD
58.77
75.58
25.74
29.75
37.55
43.52
64.11
68.68
46.54
54.38
FTTS
66.06
92.61
28.19
34.96
1.81
2.35
87.39
94.90
45.86
56.21
FTLO
66.83
96.67
67.40
79.71
13.30
14,50
36.88
47.72
SAFT-ASAM (Ours)
68.65
99.31
42.16
52.92
49.82
60.00
45.30
49.38
51.48
75.47
To demonstrate the effectiveness of our method in other domains, we evaluate our method on NLP lasks. Following the evaluation settings of Ilhacoo et al. (2023), we fine-tune the Flan-T5-base (Raf-)
Tel et al. 2019; Wei al. 2022) on four natural language processing (NLP) tasks: CoLA, MRPC, RTE, and SST-2 in GLUE benchmark (Wang et al. 2019). All fine-tuning processes start from the Flan-T5-base pre-trained checkpoint available on HuggingFace. We fine-tune each model for 8000 iterations with batch size of 16 and learning rate of 10-5, AdamW is used as the optimizer, and a linear annealing approach without warmup applied as the learning rate scheduler. For efficient fine-tuning, we convert all downstream NLP tasks into a text-to-text format, following the approach in Jin et al. (2025). We measure the multi-task performance the multi-task model merged by all four tasks using task arithmetic.
Table C6 indicates that FTTS and FTLO exhibit a tendency for certain finetuned parameters to exent undue influence after the merging process. This dominance leads to significant performance degra- dation on specific tasks, notably RTE, where both FTTS and FTLO experience extreme performance drops. These results suggest that FTTS and FTLO are still susceptible to parameter interference.
20

---

Published as a conference paper at ICLR 2025
Although they achieve commendable performance some tasks, their performance is inconsistent across the entire task set. In contrast, our proposed method effectively mitigates parameter inter- ference, resulting in a more balanced performance profile across all tasks. This balanced approach leads to a superior average performance, as demonstrated by the highest absolute and normalized accuracy, with the normalized accuracy being approximately 20% higher than the second-best per- forming method, FTTS.
The parameter dominance observed in FTTS and FTLO further substantiated by an analysis of the predicted text from the merged model. For clarity, the label texts for CoLA are "unac- ceptable" or "acceptable", for MRPC "not.equivalen"" or "equivalent". for RTE "entailment" or "not entailment", and for SST-2 "negative" or "positive". This label text configuration aligns with the settings established Jin et al. (2025). Our analysis reveals that texts predicted by FTLO frequently contain or resemble "acceptable", "unacceptable", "equivalent", or "not_equie "ho__equivleet", ir- respective of the input text data. While these outputs may be appropriate for CoLA and MRPC datasets, they are irrelevant for RTE and SST-2, resulting in the observed performance decline these tasks. This analysis highlights that after merging, FTTS is significantly influenced by the parameters finetuned for specific tasks like SST-2 and CoLA. FTLO exhibits similar pattern of influence. Conversely, although our predictions of our method may not always be correct, consis- tently predicts of the correct text labels for the corresponding task of the input data, across all four tasks. This consistency demonstrates an improved ability to mitigate parameter interference. Notably, while FTLO and FTTS outperform our method on only one task each, our method achieves superior performance on the remaining three tasks. Specifically, our method outperforms FTLO on CoLA, RTE, and SST-2, while also outperforming FTTS on CoLA, MRPC, and RTE. This result demonstrates an improvement in overall performance and suggests greater resilience to parameter interference.
C.9 TRAINING COSTS OF FINE-TUNING
Table C7:
Training cost of SAFT.
ViT-B/32
ViT-B/16
Finc-tuning baselines (4)
Time (it/s)
VRAM (GB)
Time (it/s)
VRAM (GB)
SGD
2.53
7.3
0.68
21.5
FTTS
1.30
12.6
0.37
37.2
FTLO
3.11
5,8
0.84
19,0
SAFT-ASAM (Ours)
1.34
7.8
0.34
21.6
Table C7
presents comparison of training costs between SGD and our method across various models and fine-tuning methods.
We use AdamW an optimizer, setting the batch size 128. Additionally, all training is conducted using NVIDIA Quadro RTX 8000 GPUs.
The use of SAFT-ASAM leads a nearly twofold increase in training time, whereas VRAM usage experiences only minor increment.
Recently, there has been active research aimed at reducing the computational cost of SAM (Du et al. 2022 Liu et al. (002)). Model merging is an approach designed to efficiently build multi-task models, and since our work seeks to establish a connection between model merging and SAFT, we believe our research can significantly contribute to works focused on improving the efficiency of SAFT.
D PROOF FOR THEOREM
In this section, we provide a proof for Theorem I which is restated below for the convenience:
Theorem 1 (SAFT induces joint-task loss linearity). If models, parameterized by 6, and 0, are obtained by fine-tuning from common pre-trained model via SAFT their respective datasets
21

---

Published as a conference paper at ICLR 2025
Wi ASAM
ASAM
3-
Amall m)())
0.0
Figure DI: Comparison of dominant Hessian eigenvalue for parameters on the line segment between the pre-trained parameters and fine-tuned parameters. We compare the dominant Hessian eigenvalue Amax 0 D(O) of parameter A along the line segment 0s0r, where O = Oo + (0, 0o) for y E [0, IJ. ( denotes the dataset for task k. The line represents the mean of the dominant Hessian eigenvalues for all tasks.
D3 and Dr, the models are linearly connected the loss landscape over the joint-task datasets (D UDD.
Proof. Let 8 represent the difference between the linear interpolation of Voint-Task, Losses incurred by task-specific models parameterized by O, and O, and Joint-Task Loss incurred by interpolated model parameterized by a0, + +1 a)O,:
= f.rT + -1 a)0,; D) - aLrrI -
(1- -1- (1-
(15)
D) alrr -10-
[f(ad, + - a)),)), aL(O,; - -1 (0,))
+ +L[(a0, + a1- D.) aL(O,; D,) (1 (e))
5. 6r,
(16)
where
5 L(af, + (1 - a.);;;): aL(O,; D.) - (1 ()(, D.).
8, L(a0, + - a)),) D.) aL(O,; D.) - -(1 (),((),) D.).
Performing third-order Taylor expansion of C(a0, + (1- a); D.) around 0,: L(a0, + (1 a)), = ( a1- (.) (0: - 0.)
+ 1 ク - a0) T (0, 0.) R.s.
(17)
where H, V2C((,:D.) and Rg is the remainder term. Similarly, expand L(0,; around 0,:
=() = C(0,; VoC(0, -0,, 1 (0, 0.) TH, 0.) 0.) + (18)
Multiply both sides by (1 - a):
(1-a) CL-a) = (1- (CCCC(0)))000---a0000000--------
+ (1 (), 0.) TH, 0.) + R)
(19)
22

---

Published as a conference paper at ICLR 2025
Compute 8g:
6_ = -1000-100000)))0001222-0---1111)
0.) + 2 a) 0,) H,, 0,) Rg
-al(Os; - [(1-a) VoVo D.) 0.)
+ (1 [)]- 0.)) (, 0.) + (1-00) a
(20)
Simplify the expression:
6, ((),; -
(1 - Vlloa)V D.) (0, -.) -1 - (1oo))o(((. (0: -0))
- ()) - 0.) H, - -0) 2 - a) - 0.) H. - -0,)
+ R. (1 - a)R² R'
- a1 () TH, (0, 0.) (R. -(1 -1--a)
(21)
Similarly, compute 5, by expanding around A;:
6. 4--50(1 - (0, - 0.) H, - 0.) + (Re - alt),
where H. V2C((,; D.).
(22)
Combining 6, and d,:
1 2a - (O) H.) 0.)
8 +
- +
where E c (1 (1-a) R).), + (Re aRt)
(23)
Since the dominant Hessian eigenvalue Amax (0: D) is the largest eigenvalue of H(0; D), have
(0: 0,) ". (0: 0.) S A, 1. 1111
(0: - 0,) H (0: 0.) < 0e 1.11 where >, Ammax (0,; de = Amax (0; D.). Then, |8| bounded as:
(24)
|8| < a11 a) + A,) 0,1l +
(25)
where 6 - (R, o.) R2, RR a R2) is the remainder term. Agarwala Dauphin (2023)
have demonstrated that SAM reduces the dominant Hessian eigenvalue throughout the learning tra- jectory. According Equation 25 the reduction of the dominant Hessian eigenvalue by SAFT leads
to smaller 88 and thus better satisfaction of joint-task loss linearity.
23


Reviews:
[{"review_id": "Hos3djAw7S", "reviewer": "Reviewer_ENRa", "summary": "This paper presents a model merging method by fine-tuning each pre-trained model using SAM, aiming to reduce parameter interference and improve task-specific performance. A comprehensive empirical analysis in weight disentanglement, along with experiments, demonstrates the effectiveness of the proposed methods.", "questions": "1. Could you interpret the meaning of the perturbations in Eq.(7)? This could provide readers with more intuitive explanations.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and well-organized. I enjoyed reading it.\n2. The analysis in Section 5 is comprehensive, although primarily from an empirical perspective.\n3. The results are promising for some merging methods.\n4. The method is versatile can be applied to existing merging approaches.", "weaknesses": "1. Confused motivation: For Eq.(6), the proposed method still needs to use the dataset of each task and fine-tune $\\theta_t$ of each task. Therefore, if we have all task-specific datasets, why not directly perform joint training?\n\n2. Using \"sharpness-aware\" is misleading for the goal of addressing parameter interference. Although Eq.(7) takes a similar form to the SAM function Eq.(2), the perturbations in Eq.(7) are not derived from the goal of \"sharpness-awareness.\" In my opinion, there is no link between \"sharpness-aware\" and \"interference reduction.\"\n\n3. When optimizing Eq.(6), we optimize $\\theta_t$ while keeping other models frozen. Afterwards, when optimizing $\\theta_s$, which version of $\\theta_t$ should we use? If we use the latest version, which task-specific model should we optimize first? Should we consider the forgetting issue for this sequential optimization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dEQ9tzGIe8", "reviewer": "Reviewer_AjRq", "summary": "This paper propose utilizing Sharpness-Aware Minimization (SAM) during the finetuning of pre-trained models to achieve better generalization and weight disentanglement for model merging. The central hypothesis is that SAM can lead to flatter minima, which in turn reduces interference between task-specific models and enhances the performance of the merged model. The authors demonstrate the effectiveness through several experiments.", "questions": "* The resource cost of this method should be reported. Does this method increases the training cost than the baseline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The description of the background is very clear, making the motivation of this work reasonable.\n* The paper shows consistent performance improvements in merged models.\n* They demonstrate the effectiveness of this method through weight disentanglement visualization and CTL.", "weaknesses": "* The contributions seem to be not enough.\n* Limited experimental results in this paper. They focus on the vision tasks and the results on other domains are unclear.\n* The performance gain of several results are relatively small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FyWx5ehAlX", "reviewer": "Reviewer_22A4", "summary": "The paper presents a method that aims to reduce the interference during the merging of multiple models. This is achieved by optimizing the performance gap between the merged model and each individually finetuned model, as well as optimizing per-task losses. To minimize these objectives, the authors incorporate Sharpness-Aware Minimization (SAM) during the finetuning process. This approach not only helps reduce parameter interference but also enhances the generalization of finetuned models. Empirical results suggest that SAM facilitates weight disentanglement and improves cross-task linearity. Additionally, the performance of the final model is enhanced on different merging methods, which demonstrates the orthogonality of the proposed method.", "questions": "- Could you clarify how the parameters $\\alpha_t$ with $t>2$ are defined in Equations (10) and (12)?\n- I am wondering how other flat minima techniques can help bridge the performance gap between merged and individually finetuned models (e.g. SWA [1], RWP [2], ...).\n- It would be beneficial to discuss a related work [3], which aims to find the common low and flat loss region of per-task objectives.\n\n[1] Izmailov, Pavel, et al. \"Averaging weights leads to wider optima and better generalization.\" arXiv preprint arXiv:1803.05407 (2018).\n\n[2] Li, Tao, et al. \"Efficient generalization improvement guided by random weight perturbation.\" arXiv preprint arXiv:2211.11489 (2022).\n\n[3] Phan, Hoang, et al. \"Improving multi-task learning via seeking task-based flat regions.\" arXiv preprint arXiv:2211.13723 (2022).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-structured and articulate, making it easy to follow. It presents an intriguing finding that employing SAM can significantly narrow the performance gap between a merged model and task-specific models. The experiments conducted across various merging methods effectively illustrate the effectiveness of the proposed approach.", "weaknesses": "The connection between the objective of SAM and Equation (7) is relatively loose and it is uclear how SAM can help minimize the objective in Equation (7). Thus, further ablation studies are needed to demonstrate the motivation of SAM. Otherwise, it seems that any optimization technique that targets flat minima could potentially enhance the performance of the merged model by steering the parameters towards regions where interpolation between different parameters does not increase the loss values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6TUuDT8m7d", "reviewer": "Reviewer_cmQk", "summary": "This paper proposes leveraging Sharpness-Aware Minimization (SAM) during fine-tuning to enhance the performance of model merging. The authors find that SAM effectively reduces parameter interference, addressing a core challenge in model merging. With its inherent capacity to improve generalization, this paper shows that SAM can be a viable and effective optimization method for model merging.", "questions": "Please refer to the weaknesses highlighted above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents an interesting perspective on improving model merging by using SAM as the optimizer. The authors show convincing evidence that SAM can reduce parameter interference, which has not been deeply explored. The experiment results also show improvement of SAM in many scenarios.", "weaknesses": "1. The experiments in Table 2 rely on SGD as the base optimizer, whereas Table 1 indicates that FTTS and FTLO significantly outperform SGD. It would be beneficial to show results combining FTTS, FTLO, and TIES merging to better understand the method's upper performance bounds.\n2. The experimental scope is limited to vision tasks, and expanding the evaluation to include NLP tasks would strengthen the demonstration of the method’s applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)