Efficient Model Editing with Task-Localized Sparse Fine-tuning
Authors: Leonardo Iurada
Marco Ciccone
Tatiana Tommasi
Venue: ICLR 2025 Poster (2025)
Keywords: task arithmetic
parameter-efficient fine-tuning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Task arithmetic has emerged as a promising approach for editing models by representing task-specific knowledge as composable task vectors. However, existing methods rely on network linearization to derive task vectors, leading to computational bottlenecks during training and inference. Moreover, linearization alone does not ensure weight disentanglement, the key property that enables conflict-free composition of task vectors. To address this, we propose TaLoS which allows to build sparse task vectors with minimal interference without requiring explicit linearization and sharing information across tasks. We find that pre-trained models contain a subset of parameters with consistently low gradient sensitivity across tasks, and that sparsely updating only these parameters allows for promoting weight disentanglement during fine-tuning. Our experiments prove that TaLoS improves training and inference efficiency while outperforming current methods in task addition and negation. By enabling modular parameter editing, our approach fosters practical deployment of adaptable foundation models in real-world applications.

Published as a conference paper at ICLR 2025
EFFICIENT MODEL EDITING WITH TASK- SPARSE FINE- TUNING
Leonardo Iurada' Marco Ciccone", Tatiana Tommasi!
Politecnico di Torino, Italy Vector Institute, Toronto, Ontario, Canada "Correspondance to: lecnardo. iurada@polito..tt
ABSTRACT
Task arithmetic has emerged as promising approach for editing models by rep- resenting task-specific knowledge as composable task vectors. However, existing methods rely on network linearization to derive task vectors, leading to compu- tational bottlenecks during training and inference. Moreover, linearization alone does not ensure weight disentanglement, the key property that enables conffict-free composition of task vectors. To address this, we propose TaLos which allows to build sparse task vectors with minimal interference without requiring explicit linearization and sharing information across tasks. We find that pre-trained models contain subset of parameters with consistently low gradient sensitivity across tasks, and that sparsely updating only these parameters allows for promoting weight disentanglement during fine-tuning. Our experiments prove that TaLos improves training and inference efficiency while outperforming current methods in task addi- tion and negation. By enabling modular parameter editing, our approach fosters practical deployment of adaptable foundation models in real-world applications
INTRODUCTION
Large pre-trained models (Radford et al. 2021 Raffel et al. 2020 Brown et al., 2020) have be-
come the cornerstone of modern machine learning, showcasing impressive capabilities across broad spectrum of tasks. Currently, their development is confined to a few computationally and financially well-resourced research groups, but once publicly released they provide a wealth of reusable knowledge that greatly benefits downstream applications. Indeed, fine-tuning large models to achieve optimal performance on specialized tasks or to align with user preferences becoming an increasingly democratized practice, thanks to efficient methods enabling model customization on affordable consumer GPUs crrameter-efficeen fine-tuning (PEFT) (Hu et al. 2022 Liu et al.) 2022 2024), sparsity (Ansell et al. 2022 (024), and quantization (Dettmers et al. 2023) are some of the techniques that fueled the growth of a rich ecosystem of task-specific models. They are, in tur,, readily shared open platforms (Pfeiffer al. 2020 Poth et al. 2023) fostering collaborative knowledge building by enabling users to adapt and integrate specialized modules (Raffel 2023).
In this context, task arithmetic (Ilharco et al., (023) has emerged as promising framework for scalable and cost-effective model editing. It encodes task-specific knowledge using task vectors, derived by fine-tuning a pre-trained model and subtracting its original weights from the fine-tuned ones. Task vectors can be combined through addition and subtraction to enhance specific tasks, suppress undesired behaviors, or merge functionalities. However, when task vectors are independently fine-tuned in decentralized collaborative settings, task interference becomes a significant concern (Yadav et al. 2023 Wang et al. 2024), as adding or removing a functionality disrupts previously acquired knowledge. Task interference occurs when fine-tuning modifies parameters that are critical to other tasks, resulting in unintended behavioral shifts. To prevent this, data from disjoint regions in the input space (representing different tasks) should affect only their corresponding regions in the activation space. Ortiz-Jimenee et al. 2023) formalized this concept as weight disentanglement. Their research showed that this property is emergent feature of pre- rre-training, which makes foundation models inherently suited for task arithmetic. The key question therefore becomes: how can fine-tuning preserve weight disentanglemen?
Explicitly linearizing the model during fine-tuning has been a promising direction to maintain weight disentanglement, albeit with increased computational overhead (Ortiz-Jimenez et al. 2023). In this
Code available at: http:////att..o...o////aaaaaaaaa

---

Published as a conference paper at ICLR 2025
work, we first show that model linearization alone is not sufficient, as its task functions can still activate for arbitrary inputs. Instead, we propose a set of function localization constraints to exactly implement the weight disentanglement property on linearized networks. Then, introduce novel sparse fine-tuning approach that implements such constraints while avoiding the need for explicit model linearization. The proposed method strategically updates subset of model parameters, simul- tancously promoting linearized behavior and enforcing function localization. Extensive empirical analyses and theoretical justifications demonstrate that our approach effectively promotes weight dis- entanglement, ensuring compatibility between task vectors without the need for sharing information between users and tasks. This enables efficient and robust model editing through the simple addition and subtraction of sparse task vectors, facilitating decentralized collaborative strategies.
We can summarize our main contributions as follows.
We advance the field of task arithmetic by deriving a novel set of function localization constraints that provide exact guarantees of weight disentanglement on linearized networks.
We empirically observed that the least sensitive parameters in transformer-basee architectures pre-trained on large-scale datasets can be consistently identified regardless of the task. We exploit this regularity to satisfy the localization constraints under strict individual training assumptions.
We introduce Locsllzodl Sparse Fine-Tuming (TaLoS) that enables task arithmetic by jointly implementing the localization constraints and inducing a linear regime during fine-tuning, without incurring in the overheads of explicit network linearization.
Overall, our work addresses critical gap in task arithmetic, providing a more complete and practical framework for parameter-space model editing, targeting real-world applications.
2 RELATED WORKS
Tone-Tuning. Sparsity has emerged as a fundamental concept in efficient deep learning, manifesting in both training and adaptation methodologies. Sparse fine-tuning strategies (Guo et al. 2021 Xu et al. (02I) improve training efficiency by selectively updating subsets of model parameters. These approaches often leverage the Fisher information matrix (Fisher
Sparsity & Parametee-Fffficien
1922 Amari 1996) to identify important weights for updating (Sung et al. 202T: Ben Zaken et al.
2022) or, conversely, focus on fine-tuning only the least important parameters to minimize disruption
of the original model's knowledge (Liao et al. 2023 Ansell et al. (022).. Sparse masking techniques
(Wortsman et al. 2020: Mallya et al.) 2018 Mallya & Lazebnik, 2018 Havasi et al. 2020) further
exploit this principle by employing subnetworks for continual and multi-task learning. Parameter-
efficient fine-tuning (PEFT) represents another approach to adaptation with minimal parameter
updates. Popular PEFT methods include adapter layers (Houlsby et al. 2019), prefix tuning (Li &
Liang 2021), and low-rank adaptation (LoRA, (Hu al. 2022) ). LoRA in particular approximates model updates through rank decomposition matrices while keeping pre-trained weights frozen. In complementary direction, direction,/Ansell /Anelll al. (2022): [Panda et al. (2024) investigate sparse weight addition as flexible approach to model composition. These sparse adaptation techniques connect to the broader field of model pruning, which has traditionally been applied post-training for efficient storage and inference (Blalock et al., 2020). The Lottery Tickets Hypothesis (Frankle & Carbin 2019) expanded this idea by demonstrating that sparse subnetworks identified at initialization can, when trained, match the performance of the original dense model while significantly reducing computational costs.
Model Merging. The goal of model merging is to combine multiple task-specific models into a single multi-task model without performing additional training. This requires merging techniques that prevent negative interferences among separately learned parameters. While simple parameter averaging can be effective, particularly when fine-tuned models share the same initialization (Worts) man et al.] [022] Ramé et al. (2023). it does not always yield optimal results. As result, existing approaches explored tailored re- weighting schemes, though these often come with high computational costs. RegMean (Jin et al. (202)) solves local linear regression problem for each individual linear layer in the model that requires transmitting extra data statistics of the same size as the model and additional inference steps. Fisher Merging (Matena & Raffel 2022) exploits the Fisher Information Matrix. This method, however, requires computing gradients, resulting high memory costs. A recent approach exploits extra unlabeled data to learn the model merging weights (Yang et al. 2024).
Task Arithmetic. Task arithmetic (Ihaaco et al. 2023) was introduced as a paradigm for editing models based arithmetic operations over task vectors obtained by fine-tuning a base pre-train pre-trained model and then subtracting the pre-trained weights from the fine-tuned ones. This concept has also been used in model merging, with methods that prepare task vectors before adding them together
2

---

Published as a conference paper at ICLR 2025
to produce a single multi-task model. Recent examples of this strategy are TIES-Merging Merging Yadav ct al. 2023) which resolves parameter overlap and sign conflicts after merging using heuristics, and TALL Masks / Consensus (Wang et al. 2024) that deactivates irrelevant parameters through binary masking. Other approaches sparsify task vectors by randomly dropping and rescaling parameters (Yu al. (2024) or masking weight outliers (Davari & Belilovsky 2024). However, task arithmetic goes beyond model merging as it aims at adding to or deleting knowledge and capabilities from a model in modular and efficient manner. Its effectiveness relies on weight disentanglement, property emerging during pre-training, as shown by OOtiz- Jimenez et al. (003). They proposed to preserve weight disentanglement by fine-tuning in the tangent space via full model linearization with high computational costs. To improve efficiency. [Tng et al.] (2024) proposed to use linearized low-rank adapters in the attention modules during fine-tuning. Still, linearization alone does not guarantee task localization, potentially letting weight disentanglement decrease during fine-tuning.
Our work fits within task arithmetic as a PEFT approach to construct sparse task vectors. By leveraging strategies from pruning and sparse fine-tuning, we introduce parameter update criterion that induces a lincarized regime without explicit linearization and ensures functional task localization.
BACKGROUND
Consider a neural network f with parameters 0 E R'", pre-trained on a mixture tasks P to obtain parameters 0g. We interested in fine-turr fine-tuning the pre-trained model f(- 0b) on set of T distinct classification tasks, with associated non-intersecting task data support D [ D.} S Dp (i.e. Vt, if * t' then D, n Drr = Ø).
In this setting, the core idea behind task arithmetic, introduced in Ilharco et al. (2023) to represent the knowledge acquired for each task the initial parameters from the fine-tuned parameters. Intuitively, this vector captures the direction and magnitude of change in the model's 's weight space induced by learning task f. By manipulating tasks via task arithmetic operations we can effectively add, combine, or remove knowledge in the pre-trained model producing actual functional behaviors directly in the parameters space.
task vector T; 0, Oo. obtained by subtracting
As formalized by property around Og if holds
et al. (2023). a network f is said to satisfy the task arithmetic
f {(
T Σ
[( Ou + aTT)) [(x Po)
x E Dr x & U D.
ar,00
Oo
mm1
with scaling factors (a1:..,, combination of task vectors to the initial parameters Oo is equivalent to selectively applying each task-specific modification to the model. In other words, the performance of the pre-trained model on different tasks can be modified independently if the task vector т₃ does not modify the output of the model outside D.
E A C RT. This equation essentially states that adding linear
To fulfill the task arithmetic property, Ortiz-Jimenez et al. (003)) states that the model f must exhibit
a form of weight disentanglement with respect the set of fine-tuning tasks, Le., f should behave as a composition of spatially localized components corresponding to functions that vanish outside the caa be re- re-written as
task's 's data support. In practice, Equation I
T
x. 0
Σ T:Ti
f(x 0o)1
U t=1
f(e, 0. + aa))) D.) Σ t=1
(2)
1=1
= + Σ (
where gr( T:):) = 0 for æ & D, and f t= 1,..., T, and (x()) = 0 for æ E U D., capturing the base behavior of the pre-trained model on inputs outside any of the task support.
Previous works (Tang et al. 2024 Ortiz-Jimenee et al. (023) have sought to achieve task arithmetic by focusing on linearized neural networks (Ortiz-Jiménez al. (202I). as they explicitly constrain if to be represented as linear combination of functions. Specifically, the linearization of f can be achieved by its first-order Taylor expansion centered around 0,:
(xx Oo + a.TT)) (ar, On a.T.) = fax, 0o) + OCT, Vol(a, 0o)
3

---

Published as a conference paper at ICLR 2025
The model fin 0u rı) represents linearized neural network. For this type of networks, when combining together multiple task vectors, it holds
T
T acT (of
fúi
x. 00
+
TCTT
f(æ 0o) +
Σ 1=1
(5)
While Equation 5 appears to closely resemble the weight disentanglement condition presented in Equation this similarity is superficial unless each term a.T, (xr 0o) corresponds function that vanishes outside its task data support (i.e. is localized within Dz). In the following, we will
demonstrate how efficiently impose a condition of function localization.
TASK-LOCALIZED SPARSE
To formalize the condition of function localization for task arithmetic, we begin by revisiting the linear approximation of used in linearized fine-tuning. For Equation [ to satisfy the weight disentanglement conditions in Equation 3 we must ensure that each t-th task-specific function T Vef(x, 0o) is active (non-zero) only for inputs within its corresponding task support, i.e., Dr. This requirement can be expressed as a set of constraints:
Vx E Dest. T, Vef 0o) 0.
(6)
Satisfying these conditions ensures that updating the model's weights by training on task t does not affect how the model processes data from other tasks, preventing interference between task vectors.
Directly implementing Equation poses a significant practical challenge. Enforcing the constraint Væ Dr requires simultaneous access data from all other tasks (t' A t) during fine-tuning on task t. However, this is an impractical requirement in realistic settings where contributors optimize their model asynchronously on private, task-specific data. To address this, we assume that during pre-training the model is exposed to a vast mixture of tasks, including some that are similar to the T fine-tuning tasks under consideration. Consequently, we expect the gradients Vof(- 0n) exhibit shared structure across tasks, thereby bypassing the need for accessing all task data during fine-tuning.
4.1 FUNCTION LOCALIZATION UNDER INDIVIDUAL TRAINING CONSTRAINTS
As the gradient Vo (x, 0o) quantifies the influence of each parameter on the model's 's output for given input x, it serves as a direct measure of parameter sensitivity, describing how small variations in each parameter affect the model's input-output behavior.
Consequently, satisfy the function localization constraints in Equation 6 our goal is to identify those parameters that have minimal impact on the model. In particular, by denoting the j-th element
of 0 E Rm as Ozj] we define the leust-sensitive parameters as the ones for which Vay, / (ax, 0n) Rs O. We hypothesize that such parameters remain least sensitive across all tasks (i.e. Vz D) and can thus be determined independently of the specific task, without having to access all task data.
To test our hypothesis, we conduct a sensitivity analysis following Chaudhry et al. (2018); Pascanu Bengio (2013); Matena & Raffel (2022). We define fax 0o) -br log pa, (y/ az), where pe, (y x) denotes the probability of assigning class y æ. To quantify how changes in the parameters influence
the model's 's output, we rely on the Fisher Information matrix (FIM) (Fisher 1922
Amari 1996),
positive semi-definite symmetric matrix given by:
F(Ou, D:) BAND, (Eyro (x)2² Vo log Po, (y ( Vo log Po, (u|z) J].
For a parameter with index j E 1 1,..., m. the corresponding value on the diagonal of the FIM represents its sensitivity,
Fui D.)
1 N
Σ
E By-ro, (1/=;) Vo log pe, (y)2
where Ti...., EN D, are i..dd examples, while the expectation the output be computed via sampling from the distribution of pe, (y)2z)). The lower (6. Dt), the less the model will be affected by the j-th parameter changes.
Least sensitive parameters are shared across tasks. To study the role of the least sensitive parameters across tasks, we performed a pruning experiment, illustrated in Figure T We first.
4

---

Published as a conference paper at ICLR 2025
VIT-B-32
VIT-L114
T5-Small
T5-Large
Mask Calibration Detasel
Cery
Idb
1::0
DT
100 K01
Fousdat
100
1:00
PAWS
RMS
S045
Stry Skon
Seyy Cloce
SUN007
SU/000
Weog
140
Wreglerde cee
D70
070
Test Dataset
Test Dataset
Test Dataset
Test Dataset
Figure 1: Relative performance when pruning parameters with low sensitivity. The heatmaps illus- trate the effect of pruning the parameters with the lowest sensitivity (measured by [Fis.]] (0., D,) D:) on different tasks across various pre-trained models using data from different tasks. Each grid com- pares the accuracy ratios for models after pruning. where the rows represent the task dataset Dj used to identify the parameters with the lowest sensitivity, and the columns show the model's zero-shot performance on each task after pruning those parameters. The accuracy ratios are normalized by the model's 's performance before pruning. The sparsity ratio (10%) was found as the maximal sparsity that minimally influenced the model's output on the mask calibration dataset.
identified the parameters with the lowest (o. D,) by using only data from task t. We then pruned these parameters from the network and evaluated its performance on and other tasks '' * t. The results show that the pruned model retains its zero-shot performance over all tasks. We conclude that the least sensitive parameters can be effectively identified independently of the specific task, empirically supporting our hypothesis (further validation of this phenomenon and discussion in Appendix A.7pp
Consequently, function funcciioo as for such updates the resulting dot product in Equation 6 is expected to be minimal across all tasks (we will expand on this in Section 4.3). Thus, we propose learning task vectors via selective Sparse Fine-Tuning (TaLoS), wherein only the parameters with the lowest sensitivity are sparsely updated during fine-tuning.
localization can be achieved by updating only the least sensitive parameters,
4.2 TALGS IMPLEMENTATION
Sparse fine-tuning consists in introducing binary mask c E {0, 1 1 are updated during gradient descent. Specifically, at each i-th iteration, the update rule becomes:
Th to control which parameters
0 = (i 7/c @ VoL(f(æ, o ((i-1)),)) v)]
(8)
where y is the learning rate, L is the loss function, and @ represents the element-wise product.
To achieve function localization we selectively update only the parameters with minimal impact on the model's output. Based on what was discussed earlier, we score each parameter using the diagonal elements of the FIM 8 = [Fiss] (0. p 1 E R and sort them to identify the index j of the k-th lowest element in This value is adopted as threshold and we set Clyl 0 if sty] 815-1 effectively freezing these parameters. Otherwise, cti] = 1, allowing these parameters to be updated during fine-tuning. Note that the estimation of may be susceptible to gradient noise (Tanaka et al.) 2020). Thus, we follow standard Pruning-at at- at-Initialization practices (Tanaka et al., 2020) and iteratively refine in multiple rounds (we provide full details of TaLos, alongside its pseudocode in Appendix A.2
4.3 INSIGHTS ON SPARSITY AND LINEAR BEHAVIOR
TaLoS promotes linear behavior. Parameters with the smallest (ideally near-zero) F(.J] (0g D.) are associated with flatter regions in the loss landscape, as for Ou the FIM equals the Gauss-Newton approximation of the Hessian (Pennington & Worah 2018 Kunstner et al. 2019). Updating param- eters in a flat subspace allows the gradient to be approximately constant throughout fine-tuning, a necessary condition for operating in the linearized regime (Malladi et al.) 2023b). This means that fine-tuning the least sensitive parameters inherently promotes a linear behavior without requiring explicitly lincarizing the network. We follow Ortiz-Jimenez al. (2023) to confirm this claim in Appendix A.4
"Sensitivity scoring be implemented through different approaches, as long as they preserve the same ranking as the FIM. For instance, given scalar output and f f(x 0o) A log po, (v(æ). / (z, 0o)]] yields the same ranking as the diagonal of the FIM F. [[ log Po, (1) [)), as the absolute value function
(h(x)) = |=)) and the squaring function (h(z) x²) are both monotonically increasing the interval J0, +ool
5

---

Published as a conference paper at ICLR 2025
Function localization in TaLoS. Given the least sensitive parameters are shared across tasks, the function localization constraints of Equation 6 for TaLoS can be rewritten and upper bounded as
Vo E Drs |c & (T, Vof 0o)l - ||e Tell max =ED;
||e@ O Vof(æ, 0o)l - k2 p.
Here, 11 maxe V 0o)l is the magnitude of the k-th largest gradient element, capturing the maximum sensitivity of the fine-tuned parameters to input data. μ == max, |Ctj @ Teu represents the maximum change in any of the updated parameters during fine-tuning. Inequality provides an upper bound on the degree of function localization of Te obtained via TaLos. Having this quantity equal zero ensures no task interference, as the overall output falls back to f()00) which by definition weight disentangled. Yet, this means that no learning has occurred. Instances of this are when no parameter is updated (k- 0) or when only parameters with exactly zero influence (1) = 0) are fine- tuned. Apart from these cases, fine-tuning the least sensitive parameters allows for minimal increase of this bound while still allowing to learn the task, as even parameters with marginal influence can collectively contribute to task performance (Ben Zaken et al. 2022 Xu et al., 2020. Liao et al. (023) (in Appendix [A66 we show that LaS enables learning on par with other PEFT baselines). Indeed, as the model is robust to changes within the flat subspace defined by its least sensitive parameters, learning Tr in this subspace ensures minimal impact the model's 's output for other tasks as well (we empirically validate this in Figure 3). As detailed in Appendix A.I k is a hyperparameter controlling the sparsity ratio of thus, indirectly controlling the degree of function localization. We tuned it at the task level, resulting in optimal sparsity ratios between 90% and 99% (ablation in Appendix A.5)
9
5 EXPERIMENTS
Our experimental evaluation focuses on the established Task Arithmetic framework outlined by Ilharco et al. (202 2023), specifically targeting Task Addition and Task Negation, encompassing both language and vision domains. In the following we describe the baselines we compared our TaLos against. Further details regarding the experimental setups, the relevant metrics. the implementation of the experiments, as well as the data and architectures used, are deferred to Appendix A. Additionally, in Appendix [A.8 we test different model merging schemes on task vectors obtained with TaLos.
Baselines. We consider three families of methods as references. (i) Full fine-tuning methods aim to produce task vectors Ti by fine-tuning all the parameters of the network. Specifically, Non-linear fine-tuning (FT) ([haacoo et al. 2022 2023) minimizes a standard cross-entropy loss, while Linearized et al. (2023). (ii) Post-hoc
FT fine-tunes the linearized counterpart of the network, as in methods refine т, after it has been obtained via fine-tuning (as prescribed by the respective methods, we apply these post-hoc approaches on non-linear FT checkpoints). TIES- WWE--Megging (Yadav et al., 2023) reduces redundancy in Ti by magnitude pruning, keeping only the top-k highest magnitude parameters, and addressing sign conflicts when merging task vectors. TALL Mask / Consensus (Wang et al.) 2024) identifies task-specific parameters in 7i by comparing them the sum of task vectors. It then merges multiple task vectors by using an clement-wise OR operation between masks to further identify and remove conflicting parameters. DARE (Yu al. (024)) randomly sparsifies T, to eliminate redundancy and upweights the remaining parameters based on the percentage that was removed. Breadcrumbs (Davari & Belilovsky 2024) reduces redundancy using magnitude pruning and eliminates weight outliers within the retained top-k parameters. Although these methods have been presented for task addition, we also test their ability of handling task negation. (iii) Parameter-efficient fine-tuning (PEFT) methods aim to obtain task vectors by efficiently fine- tuning the network, using far fewer resources compared to full fine-tuning. We compare against L-LoRA (Tang et al.] 2024). which applies linearized low-rank adapters to the Q and V projections in self-attention layers. This approach was specifically designed for Task Arithmetic and offers superior performance over standard LoRA. For sparse fine-tuning, use LoTA (Panda et al.) 20244, method that leverages the Lottery Ticket hypothesis (Frankle & Carbin 2019) to select the top-k parameters when sparsely fine-tuning the network, making it suitable for model merging.
5.1 TASK ARITHMETIC RESULTS
We thoroughly evaluate TaLoS on its ability to derive task vectors that enable model editing through simple arithmetic operations on model parameters.
Task Addition. In this benchmark, the sum of the task vectors E, QUT, is added to pre-traue traitrained checkpoint to produce a multi-task model f(., 0n + E. T.). The success is measured in terms of
the maximum average accuracy over the different tasks. As done by Ortiz-limenez et al. (2023) Tang

---

Published as a conference paper at ICLR 2025
Ab.
Num 11x
Als (1)
Num. (D)
Ab..t)
Nem. (7)
Neer (1)
(T)
Norry
Ab
Nemr.
colaed Clercsbao
.0.22
55.33
55.70
53.51
$1.11
Fell Fanturieg Methach
Naviica 77 Hlane 2000
71.25
12.08
71.11
86.00
90.04
650M
9136
74.30
50400
13.37
$3:23
Jano Honeraa
26.0
85.96
3001
$3.20
$5.28
$3.24
61.13
71.00
60.12
64.14
78455
Pot-fre Metbels
TIES Mazia
14.99
12.8-
82.12
88.16
$2.95
02.53
96.83
70.66
02.37
14.00
$5.30
TALL Nar
0224
74.3
1427
7402
80.80
20.11
00.11
15.34
91.00
17.31
$7.54
Dean
7131
76.99
85.90
03.51
60.00
80100
76.00
$5.51
Treaderahh Donnn Tellonds p12]
09.99
79.51
7190
73.94
$4.79
$2.97
01.10
90.35
02.70
73.01
$7.00
Pialott
2xted
65/04
Table 1: Task Addition results. Average absolute accuracies (%) and normalized accuracies (%) of different CLIP ViTs and T5 pre-trained models edited by adding task vectors on each of the downstream tasks. We normalize performance of each method by their single-task accuracy. Bold indicates the best results. Underline the second best.
Makal
VIT-8016
VITL/14
75-Seall
Lage
Cott
Owt
Oer
Own (t)
2464
30.3
70.79
Lixutted ET Ooulltncae
11:20
66774
00.92
10.95
72.43
46.47
4494
40.27
4137
45.30
Por Mitheds
bu Xrd
21.44
61.99
19.22
55.99
24.90
7541
3501
43.90
4000
45.15
46.19
43.9
TALL Mark
Gquaes Werg
2004
1131
00.56
3071
05.17
22.35 15
75.30
43.43
4541
40.14
40.20
41.26
45.99
N4
3581
60/0
30333
60.88
7140
70:66
40.53
65%
40.44
65
01.0
65.0
Inseereel Mion Tellindy 2204
3431
60.6
31.08
65.35
31.09
72.45
5145
/519
10/6
14.11
01:00
65.1
Phrass (refeees Langinnng Netheds
19.29
69775
39.33
35.99
19.39
75.14
55.30
45.24
51.35
45.10
48:37
45.51
1oth
25.21
$6.10
65
11:33
25.77
Table 2: Task Negation results. Average minimal accuracy (%) of different CLIP ViTs and T5 pre-trained models edited by subtracting a task vector from a target task while retaining at least 95% of their performance on the control task. We average the minimal accuracy over each of the downstream tasks. Bold indicates the best results. Underline the second best.
et al. (004)), we also report the average normalized accuracy over the tasks. The normalization is performed with respect to the single-task accuracies achieved by the model fine-tuned on each task (see Appendix I. The results in Table demonstrate the effectiveness of our proposed method across various model scales and modalities. TaLeS consistently outperforms existing approaches, with evident improvements in normalized accuracy of 1.88% to 4.65% over the second best method across all model variants. Such metric provides insights into the outstanding ability of TaLoS to maximize the benefits of model combination while mitigating interference.
For vision models, TaLos exhibits strong performance across all scales, with absolute accuracy gains of up to 2.61% % over the closest competitor. In NLP, TaLoS maintains its leading position, although the gains are less striking than in vision experiments. Nevertheless, the improvements are particularly pronounced in larger models, suggesting that TaLoS scales well with model size. Notably, Tao'ss s performance surpasses both full fine-tuning and post-hoc methods across the board. This suggests that our parameter-efficient approach can achieve superior results while potentially reducing computational costs, crucial factor when working with large-scale models.
Task Negation. In this benchmark a task vector т. is subtracted from the pre-trained checkpoint to reduce the performance on task f, producing the model f(. 0. - T.).). By following et al. (2023). the success is measured in terms of the maximum drop in accuracy on the forgetting task that retains at least 95% of the accuracy on the control task. Results are averaged over tasks and presented in Table 2 For vision models, TaLoS achieves the lowest target task accuracies while maintaining high control task performance, indicating superior ability to selectively remove targeted task information. For T5 models, all methods, including TaLos, face significant challenges in Task Negation. The results show a much tighter clustering of performance across different approaches. This suggests that negating specific language tasks without substantially impacting the control task accuracy is inherently more difficult than in vision models. Despite this challenge, TaLoS still manages to achieve the best balance between target and control task performance.
5.2
WEIGHT DISENTANGLEMENT AND LOCALIZATION
The improved localization provided by TaLoS seems to play a crucial role in driving effective task arithmetic. Here delve deeper into this aspect with tailored analyses. First, we assess how well the weight disentanglement property holds. Then, for each training recipe, we evaluate the degree of task component localization on each task.
7

---

Published as a conference paper at ICLR 2025
Non-lin. FT
Lin. FT
L-LORA
LoTA
TaLoS (Ours)
((miing))
Cars RESISC45
-30
100%
-1.0
3.0
EuroSAT SVHN
-1.
1.5
3.5
Non-lin.
Lin.
L-LoRA
LoTA
TaLoS (Ours)
-30
100%
QASC Story
-1.0
1.0
3.0
PAWS Winogrande
3.0
-1.
3.9
3.0
-1.0
1.0
3.0.5.0
-1.0
3.0.3.0
-1.0
1.0
3.0
1.0
1.0
3.0
8.0
3.0
[r)
41
Figure 2: Visualizing weight disentanglement error. The heatmaps illustrate the disentanglement сπror {(a1, a2) of each fine-tuning strategy on both a CLIP ViT-B/32 model (top) and T5-Small model (bottom) across two task pairs. Lighter areas highlight regions of the weight space where disentanglement is more pronounced. The red box indicates the search space within which the optimal values were searched (refer to Appendix[A.1] We chose the task pairs to visualize by following Ortiz-Jimenez et al. (2023) for vision and criterion akin to the one used in Tang et al. 2024 for language.
Weight disentanglement error visualization. Ortiz-Jimenez al. (2023) Tang et al. 2024
proposed to evaluate the disentanglement error defieed as
2 E(a1, a.)) - ExED, (dist(f(x,,6) +aTt),, an + o2T2))
(10)
where the prediction error dist(y), y2) = 1y) * y2) is taken as the distance metric. Generally, given pair (01, a:2), the smaller the value of {(a), 022) the more weight disentangled model is. Maintaining a low disentanglement error as oa and 02 increase provides an even stronger evidence of the weight disentanglement property.
In Figure 2 we report E(01, 02) across different fine-tuning strategies for both the CLIP ViT-B/32 and T5-Small models on task pairs. Overall there clear difference in disentanglement patterns between vision and language models. For the latter, the patterns are more consistent across strategies, which may explain why the differences in task arithmetic performance are notable in vision experiments and less pronunced in language experiments (ref. to Tables 2
By focusing on vision models we observe that Linearized FT, L-LoRA, and our approach demonstrate improved disentanglement (indicated by lighter regions) than non-linear fine-tuning, with our method performing the best overall. We remind that L-LoRA approximate behavior Linearized FT adapters but still lacks to optimize the task localization property. Interestingly. LoTA shows much lower degree of disentanglement. We remark that this approach selects and updates task-specific parameters while TaLoS focuses task-generic ones and this difference accounts for the observed behavior.
For language, Linearized FT and L-LoRA yield mixed results depending the pairs of considered tasks. LoTA seems able to improve over non- linearize FT but with different extents across tasks and it is consistently outperformed by TaLoS.
8

---

Published as a conference paper at ICLR 2025
Non-inear FT
Linearized FT
L-oRA
LoTA
Tale8 (Cung)
Fine-tunng Detozet
ont
Cardee
takon
Dalasot
Test Datasel
Dataset
Dataset
Test Dalaser
Engor
Linecried
Tales (Curs)
QACC
1M
Cuentry
Fire Freeuring Laring
Dalecet
Dateset
Test Daliset
Test Datasel
Test Datiset
Figure 3: Function localization. The heatmaps present the accuracy ratios for fine-tuned models across tasks for CLIP VIT-B/32 (top) and T5-Small (bottom) models. Each row indicates a model fine-tuned on specific task, with columns representing its performance on different test datasets. Accuracy ratios are normalized by the pre-trained model's 's performance. Lighter colors indicate better performance, suggesting minimal interference between the fine-tuned model and other tasks' input spaces. The red diagonal highlights each model's ' test performance on its specific fine-tuning task.
Tal.88 (Qurs) VVT-8/32
LoTA- VIT-B/32
TaloS (Curs) T5-Small
LoTA- 15-Small
a 27
vocideed Grothedd dock aegie
0.55
340
desc
excddrr M.GeMaantoo iegge
0.24
125 4.90
0.75
0.35
Weggtt Remaining Ratio
Weight Ramaaning
Weight Romaning Ratio
Weight Ramaning
Figure 4: Visualization of mask calibration. Percentage of parameters selected for sparse fine- tuning in transformer block of a ViT-B/32 (left) and T5-Small (right) models, after our method's mask calibration vs. LoTA's mask calibration, at 90% sparsity. On ViT-B/32, we calibrate the masks
on the Cars dataset (Krause et al., 2013), while on T5-Small we use QASC (Khot et al. 2020p. Full visualizations of all masked layers are reported in Appendix A.3
Function localization. We experimentally assess the function localization property of TaLoS by comparing it with other fine-tuning methods. From the definition in Equation [) we know that when this property holds, each task activates only for its specific data support. Thus, we should observe an advantage in the prediction output when testing that task, and the same performance of the pre-trained model for all the others tasks. Figure confirms the expected behavior for TaLoS in vision, while the competitors display more interference between tasks, as indicated by darker hues off the diagonal. Interestingly, for NLP tasks all methods exhibit natural function localization, as reflected by the lighter regions in the figure. This provides us the opportunity to remark the importance of extensive model analysis as conclusions drawn from a single domain where linearization is sufficient might be misleading.
5.3 WEIGHT SPARSITY STRUCTURE AND EFFICIENCY
Visualizing task vector masks. To understand the nature of our sparse fine-tuning approach, we analyze the structure of the masks calibrated using TaLoS and compare it with the ones produced by LoTA.
a visualization of the layer-wise percentage of parameters selected for sparse fine-tuning in transformer block of a ViT-B/32 and T5-Small models. The results reveal distinct patterns in parameter selection between TaLoS and LoTA across both models. TaLoS exhibits a highly structured selection, predominantly preserving parameters in the multihead self-attention layer, particularly in the Q and K projections. contrast, LoTA's selection appears more distributed across different layers of the transformer block. Interestingly, our analysis reveals some notable contrasts

---

Published as a conference paper at ICLR 2025
Midiel
Ellesing Casl
Fincitazng
Teak Addiio
Negaka
Frenar Hautboad Pus Time to)
Ontin Sup Drae
The becatoo Tire
Park Mernor Stage (Cit)
Abe. it
Norr. (t)
They (P)
Coer.
Now-teaa P1
Laaantta 2002
0.3658 a 2.00
0.0114 2000
0,3122 9000
6.
71.25
26.54
2404
00.25
Lieartoed
3175
(16815
0.000 4.915
0.666
10.2
N6.10
$5.96
09.71
Tex
(13222 +: 1anns
0.6066 : 1967
0.316 8980
52
25.00
20.18
17.29
00.35
1n1k
01:785 - 20001
0 11575 + 9995
0,155 0006
6.1
61:11
N6.17
71/00
61.81
Tatod (Dan)
0,1256 2.44
0.0858 1 1944
0.1644
47
79.67
90.33
11.03
60.59
VIM/
0.0156
Liscaiosi
3123
1,8200
0.0052
18462
21.3
88.12
23.01
10.6
L-aRA Timu
(15133 J.37
0.662 2.015
0.5235
62
87:30
91.37
10.3
73.14
Latt parta M050
08133 -1a0
0:1499-:000 0919----011
1. 1.987
154
68
10.11
Now-lnea FT
2003
0.9047 2axs
0.084412000
0.9941 8.77s
300
15.31
83.25
41.54
43.49
(itie Tranno 43
3175
1
0 2o00
1 - 8100
15.1
09.34
76.95
11.72
45.70
192
Lotk
Pizta
0.3325 2.943
0.3942 2.000
32.1
23.84
83.14
44.33
JatoS (0ans
0.458 2amms
0,000 : 2944
0.486 1998
1211
24.7
90.68
55,20
45.78
Table 3: Computational cost and memory footprint of fine-tuning, Average iteration time (in seconds) and peak memory usage (in Gibibytes) of different fine-tuning approaches CLIP ViT- B/32, ViT-L/14 and T5-Large models, alongside their performance on the task arithmetic benchmark. To improve granularity, we report also the average forward-backward time of a single iteration and the average step time of the optimizer. We separate full fine-tuning methods from parameter-effiiennt fine-tuning methods. Further details on the resource monitoring process can be found Appendix A.1 Bold indicates the best results. Underline the second best.
with L-LoRA (Tang et al. 2024). method specifically designed for task arithmetic. While L-LoRA arbitrarily fine-tunes the Q and V projections, our findings suggest that, generally, Q and K play a more significant role in task arithmetic than V in the multihead self-attention layers. Additionally, for CLIP ViT-B/32 biases also seemingly play crucial role for function localization. This structured sparsity not only provides insights into our method's 's mask calibration mechanism but also hints at potential efficiency gains, which we explore further in the following.
Computational cost and memory footprint. The observed structured sparsity pattern of TaLos suggests that it also provides highly efficient task arithmetic fine-tuning strategy. To verify it we performed a comparative analysis of the computational cost and memory footprint of TaLoS against several fine-tuning methods.
In Table 3 we present the collected time and memory costs with detailed average time (in seconds) for single training iteration's forward and backward pass. This is separated because approaches like Linearized FT and L-LoRA involve specialized forward passes that require Jacobian-vector products with respect to LoTA and TaLos, which operate similarly to non-linear FT. We also report the time (in seconds) spent by the optimizer updating parameters, as LoTA and TaLoS require an additional mask-based element-wise multiplication prevent updates to certain parameters by masking gradients. Additionally, we provide the total time (sum of these two values) and the peak memory usage (in Gibibytes) recorded during fine-tuning. Overall, the ability to freeze large number of parameters, thanks to well-structured mask sparsity of our approach improves the total iteration time. Although our method has a slower optimizer step compared to other approaches, the faster forward-back ward pass compensates, making TaLoS the leading method. In terms of memory usage, the benefits are especially notable for large models, where only a small subset of parameters requires fine-tuning thus, yielding pronounced savings.
6 CONCLUSION
In this work we have proposed LaL.S, an efficient and effective strategy to edit pre-trained models in the framework of task arithmetic. We started from the observation that the parameters showing the least variation in the fine-tuning process of single task are also those minimally relevant for other tasks. Thus, we have leveraged them through sparse learning process that promotes task localization and avoids task interference. A thorough experimental analysis across vision and language domains confirmed that TaLoS yields state-of-th -ate-offtheeart results in task addition and negation, showing significant efficiency advantage over competitors. Moreover, with a tailored set of evaluations we assessed model linearization and function localization properties, providing insights on the inner functioning of our approach.
Overall, we have discussed how preserving the regularities provided by a large scale pre-trained model sufficient maintain weight disentanglement and observe beneficial effects in task arithmetic. Future work may investigate whether explicitly enforcing localization constraints during fine-tuning could enhance performance and further advance model editing capabilities.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have made significant efforts ensure the reproducibility of our results. Full implementation details are provided in Appendix A.1
Pseudocode for our algorithm is included in Appendix [A.2 to clarify key steps, well as practical design choices to address potential challenges in implementing our experiments. Additionally, we publicly released our code to further facilitate reproducibility at https://gi hwtp:/ttttoooooooooooooooee.eee http://uaraaaaaa ttalos task arithmetic
ACKNOWLEDGEMENT
The authors thank the reviewers and area chair for their valuable comments. M.C. also thanks Derek Tam and Colin Raffel for their fruitful discussions and feedback on the early state of this work. L.I. acknowledges the grant received from the European Union GextraeneeaiionUUU (Piano Nazionale di Ripresa E Resilienza (PNRR)) DM 351 on Trustworthy AL. T.T. acknowledges the EU project ELSA European Lighthouse on Secure and Safe AI. This study was carried out within the FAIR Future Artificial Intelligence Research and received funding from the European Union Next- SextteeneratioEEU (PIANO NAZIONALE RIPRESA RESILIENZA (PNRR) MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 D.D. 1555 11/10/2022, PF00000013), This manuscript reflects only the authors' views and opinions, neither the European Union nor the European Commission can be considered responsible for them. We acknowledge the CINECA award under the ISCRA initiative for the availability of high-pee pelfoomerrmmcc computing resources and support.
REFERENCES
Shun-ichi Amari. Neural learning in structured parameter spaces-natural riemannian gra- dient. In Advances in Neural Information Processing Systems (NeurIPS), 1996. URL
wwur.gg...e..rr.p cc http://rrrrs...caeeaeeaaeeee///a
3 39e4 0oeef777ba3222220000070005553333
pdf
Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), 2022. URL https::/ http://s///laaahoooo..coooo org.loogoyoo//2222 acl-long: 125
Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M. Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401. 2024. URL https
6405
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), 2022. URL https:
//aclantholoogyorrg//D00
short.1
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Proceedings of Machine Learning and Systems (MLSys), 2020. URL
https://arxix: wtw//rggg////oooooooooo 03033
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voos Voss, Gretchen Krueger, Tom Henighan, Rewon Child. Aditya Ramesh, Daniel Ziegler, Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot leamner. Advances Neural Information Pro- cessing Systems (NeurIPS), 2020. URL URU/ntpps//prrceeenss..eee cwtpiippppppppppp.eeeepeppeppppe pdf
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European conference on computer vision (ECCV), 2018. URL https:t//////.......ttt..tl //tp:////xiv..v/.a
httttta////xx...o///////a org/abs/
1801. 10112
11

---

Published as a conference paper at ICLR 2025
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 2016. URL httpa:://axxii.oss///da
1604.06174
Gong Cheng, Junwei Han, and Xianqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865-1883, 2017. URL https://
Leeexplore ieee orge www....ege...ueeeeeeeeeeee//eeee www.d/de/ueeteeeeeeeee /791144
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and An- drea Vedaldi. Describing textures in the wild.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. URL
httips:/openaccess.o thecvf..ev..cc. www.o/o//cooooooooooooccoooooooo cotnterrt_oopr22204//ppaa CCimpoi_ Describing Textures in 2014 CVPR pwwpr.ppppppppp
Ido Dagan, Oren Glickman, and Bernardo Magnini.
The pascal recognising tex- tual entailment challenge. In Machine learning challenges workshop, 2005. URL
ist. edu ptt.tppp....oobommetttteeeeeeeee htt.://uumnotttt....e.eooeeeeee. @60F22ccleeecc000cccelee22222283
Mohammad Reza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision (ECCV), 2024. URL
pettps://arxig httt:s:/aaxxx..orrr/////3/33331 wttp/b///gg///////
06795
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In IEEE conference on computer vision and pattern recognition (CVPR), 2009. URL http:::tppsspooottte proceedi hgs CVPR 2009/000 /attt////aaaaaa patpttaappprss/111103 pdf
cnrs. fr/ tr/imaginn/put
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Advances Neural Information Processing Systems (NeurIPS), 2023. URL
https://arxiv org/ abs////////ao
/t.b///233 /20.. 14314
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. URL URLLattpss//arxi..ooo//aas////
Ronald Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers mathemiicca or physica character, 222(594-6044)339 1922. URL httpp::/ http:p///oopalloaeeyyaaaaaassssa
10. .itetta...99222. 1922. 0009
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019. URL
https::/arxiv. www//bsg////ooooooooo./ooee.
abs/ /et/s//133.00 03665
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL httpt:ttp............. / raaaaiiv....mm.m www./orgggooo.. ats///////333 wrtp////////aa...o.oa/ 14108
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference Natural Language Processing (Volume I: Long Papers), 2021. URL
ott.aaacboooooy...../22......... 378
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction, arXiv preprint arXiv:2010.06610, 2020. URL :/appiiv/////....///////a/ crq/abs/2210 06610
12

---

Published as a conference paper at ICLR 2025
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019. URL https:
//ieeexpl ore. leee. org/ http:///cmmeeeet/////00 87366788
Geoffrey Hinton. The forward- algorithm: Some preliminary investigations. arXiv preprint 1rX1v:2212.13345, 2022. |ttpstss::///xx.vvooo
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), 2019. URL https
//arxiv. //tpxvv.rcg/aa//////// att.i/.o..//aab//12...000
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL htLtpt/s///axa
org/ als///a////// waw./2a. httgg//bb/2106.. 2206. 09685
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In Advances Neural Information Processing Systems (NeurIPS), 2022. URL https
7/arxiv. abtg/aba/2aa 2208 05592
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. International Conference on Learning Representations (ICLR), 2023. URL https: http///wwwxww.../a//a///o//////a
21100.08 08207
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations (ICLR), 2023. URL
2212 09849
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020. URL httts::////attt
org/ abs/ 1910. 11473
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, 2013. https: /ww. cV cv- www.-atioiiio..oommmmoo org/ openaccess/conten
iccv www.wwows_ooo11111
2013 http://////pppppeaaeeeee/ otte
201 3 1ea13_IcV_papeee pdf.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approxima- tion for natural gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL/attps://arat
www. 1atee//19...1.333
Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruming framework for transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL httpps://proceeding:s neurips 98
neuri wwtpppprppppcppcppp
Yann LeCun. The mnist database of handwritten digits, 1998. URL httt::::////tn.eeee com/ exdb/mnist/
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In
Thirteenth international conference on the principles of knowledge representation and reasoning
(KR), 2012. M https::////.. // cdd. iaatn.aaai. http://ww////a///a/aaa//////////
1PB.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuus promptss for generation. arXiv preprint arXiv:
2021. URL. Jtl.https:://rrxiv..rr////s
00190
13

---

Published as a conference paper at ICLR 2025
Baohao Liao, Yan Meng, and Christof Monz. Parameter-efficien fine-tuning without introducing new latency. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023. URL lttttaa://allannooooy.ooooo
ac .oo......0 233
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-effiiiet fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https
//arxiv. org/ abs 20.D0/2200 2205. 05638
Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang. Kwang-Ting Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML), 2024. URL https: //axxiv///vvvv..//a /ttts:///////
owg/abs////ooooo ence on Learning Representations (ICLR), 2019. URL http:s//aaiii..rr/rabbb//////// 2402
09353
lyya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- net.//ww..e.eeee.eeeeeeeeeeeeee forum?
id=Bkg6RiCqY.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems (NeurIPS), 2023a. URL
http:///rrceeeinos..ei..o..o...o a62 602780000ee11113222888882----220
Sadhika Malladi, Alexander Wettig. Dingli Yu, Danqi Chen, and Sanjeev Arora. kernel- hernel-based view of language model fine-tuning. In International Conference on Machine Learning (ICML). 2023b. URL htttpss:/ppocceeeciigggr /tt.://ceeea.aatl...ppppes/////a
ml pwwpsse2ee//aaaaaaaspappp aad mall adii3dii2a 23a fdd.)
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition (CVPR), 2018. URL http::ttp:://t..t..
/tttpp::/arxivee /ttpiaiivvv.. www.aiiwwww.aaooooaaooooooooooot abt/aaas///1/// http:///aww.....//////..1....../ 05769
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European conference on computer vision (ECCV), 2018.
/aattps://arxiie org/ abs /1///111111 06519
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. htttps://gite thub
com/ Coon/hagginnfacce/peet 2022.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems (NeurIPS), 2022. htttttttts/:/trtttt org abs/
2111. 09832
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In Ad- vances in Neural Information Processing Systems (NeurIPS) Workshops, 2011. URL htttss://staticc.ooollesssss
com.:/eenttcccmmmeeeeeeeeeeeee httddddareeee.hheeeeeeeeae it httppppb///arcci////////333 pdf
com/
Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL http:ttsss/:/proceeeeeeee
neur i ps. cc. pctpp/pp//////1///////// pdf
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent
space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
net www.neettp..... /pf p 11d--1--2222 j2DGWW
Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sammi Koyejo, Tsachy Weissman, and Prateek Mittal. Lot- tery ticket adaptation: Mitigating destructive interference in llms. arXiv preprint arXiv:2406. 16797, 2024. URL http:::/t/saatt www.:iv..orrrggreee wwt.:oggg//aaaooo /2406 16797
14

---

Published as a conference paper at ICLR 2025
R Pascanu and Y Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013. htttttt://aaiiiy.e http//ab//1111111 3584
Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of single-hidden-layer neural network. In Advances in Neural Information Processing Sys- tems (NeurPPS), 2018. htLthttps////ppptee
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP): Systems Demonstrations, 2020. URL
ac wwwwwb..........c wwwpwwewwwr.e.oeeeeooooooooooooe anthology/
1020.emmlp--emor
Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulié, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: unified library for parameter-efficient and modular transfer learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2023.
URL https:
org. 202g/2222333 emnlp-demo, 13.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning URL aattps://aaaat wwg.:/awwiii.....oooooooooo abs / 22032113....0 00020
(ICML), 2021.
Colin Raffel Building machine learning models like open souree software. Communications the ACM, 66(2):3 40, 2023. URL htttps://dd acm. org/
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
orrppp///nn..oorr//pprrr /vtpers////////// 21111/20000...1... 074.html
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang. Matthieu Cord, Léon Bottou, and David Lopez- Paz. Model ratatouille: Recycling diverse models for oul-of-distribution generalization. In International Conference on Machine Learning (ICML), 2023. URL httttp:://axxv..rggg
2212
10445
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
URL https: /dl. acm. org/ doii dtt/paa//1..115553553333
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (ACL), 2018. URL htttpps//acllamttooooy.ooooo
Iddf
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: multi-class classification competition. In International Joint Conference on Neural Networks (ICNN), 2011. URL httl:ttttt:::eeppppllll:
feee. org/ .eee..og//oouueettt document/
6033395
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL httpps///axxii
arq/abs/211 abs./ill..llaaaappbb.abbbal..lll 09839
Yi-Lin Sung, Jaehong Yoon, and Mohit Bansal. Ecoflap: Efficient coarse-to-fine layer-wise pruning for vasion-language models. In International Conference Learning Representations (ICLR),
2024. /ttp::///aaa
ULLtttess//aaxv...ooro/////.... 02998
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. Quartz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 conference empirical methods in natural language processing (EMNLP), 2019. URL https::/ :/tpss://arxv.... o haa/bb//1900
031553
15

---

Published as a conference paper at ICLR 2025
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems (NeurIPS), 2020. httpl::///aaaatt
atgrg//bb/2005 05467
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial linearization. In International Conference on Learning Representations (ICLR), 2024. URL httttht http:/://axxvv..rro org/ als/ /atos/2310.0m 04742
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In International Conference on Muchine Learning (ICML), 2024. URL http:: htttt::tt/xxxxii/e
07813
Yite Wang, Dawei Li, and Ruoyu Sun. Ntk-sap: Improving neural network pruning by aligning training dynamics. In International Conference on Learning Representations (ICLR), 2023, URL
https://arxiv:: wwwp//////////oooo/ooe 02840
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-opess, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022. URL https //arxiv. org/ abs/ 221.//223....5
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016. URL htttps://lin
s1 111663004--7700
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise child in large language model: Towards effective and generalizable fine-tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, URL /ttp///allaatthtt waaeehhhoooy....oooo . orgg 2021 emnlp-maa main.. 749
Shichao Xu, Yixuan Wang, Yanzhi Wang, Zheng O'Neill, and Qi Zhu. One for many: Transfer learning for building hvac control. In International Conference on Systems for Energy-Efficient Built Environments (BuildSys). 2020. URL fotttttss/:/xxiiy org/ abs/ 2008 03625
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL hctps: /ttpss://rxiv.. orq/ abs/ / 220/ 01708
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao.
Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations (ICLR), 2024. URL https: / org/ abs/ 2310. 02575
Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing (EMNLP), 2015. UtLhtts:://acll
orgg/15- pdf
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning (ICML), 2024. URL http::tttt///attt org/ abs /2///211..000 .00.1...30999
Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. URL htl.tttss: : //ta//aii/// org abs 904 01130
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Computational resources. We execute all the vision experiments using ViT-B/32, VIT-B/16, and ViT-L/14 on a machine equipped with two NVIDIA GeForce RTX 2080 Ti (11 GB VRAM), an Intel Core 17-9800X CPU @ 3.80GHZ and 64 GB of RAM. For all the language experiments using T5-Small, T5-Base, and TS-Large we employ a machine equipped with a a single NVIDIA A100 SXM (64 GB VRAM), an Intel Xeon Platinum 8358 CPU 2.60GHz and 64 GB of RAM.
Starter code. We developed our codebase starting from the repositories provided by Ortiz-limenez al. 2023 I (based on the code by IIharco al. (2022) 2023 and Yadav et al. 2223 m which allow
to reproduce the full fine- tieeetuning results (Non- linear FT and Linearized FT). TIES-Mergihn Merging (Yaday)
et al. Davari & Belilovsky 2024 and LoTA (Panda et al. 2024 9
f TALL Mask Consensus (Wang et al. 2024 DARE YYu et al. 2024 Breadcrumbs
provide official implementations
of their methods from which we carefully adapted their code to work within the Task Arithmetic
but the guidelines in the paper are sufficient to reproduce their results. To this end, we used the peft framework. L-LoRA (Tang et al. 2024) unfortunately doesn't 't provide any official implementation.
library (Mangrulkar et al. 2022 for implementing the LoRA modules.
Hyperparameter selection. As highlighted by Ortiz-Jimenez et al. (2023), task vectors that per- form well in Task Negation tend to exhibit higher degrees of weight disentanglement in Task Addition. This relationship informed our hyperparameter selection strategy. For each method, we cross-validate its hyperparameters on each individual task by leveraging Task Negation per- formance on a small held-out portion of the training set, as implemented by IIharco et al. (023);; Ortiz-Jimenez et al. (2023). It's important note that hyperparameter selection shall not be per- formed separately for addition and negation, as each choice of hyperparameters yields a unique task vector. Hyperparameter search of each method is carried out according to the guidelines pre- sented in each paper. Specifically, for post-hoc methods, the sparsity ratio is searched in the set E {0.1, 0.2, ..., 0.9, 0.95, J. Furthermore, for TALL Mask Consensus (Wang et al. 2024) we also tune the consensus threshold in the set {0, ..., ... where T is the number of tasks. For Breadcrumbs (Davari & Belilovsky 2024) we also tune the percentage of top-k parameters con- sidered outliers, using values from the set {0.8, 0.9, 0.95, 0.99, 0.992, 0.994, ..... 0.999). Regarding pfrfceter--ffficientt fine-tuning methods, when using L-LoRA (Tang et al. (024) we progressively reduce its rank r E {512, 256, 128, 64, 32, 16, 8). While, for LOTA (Panda al. 2024) and our method we tune sparsity at the task level using values in the set 10.1, 0,.,,.2,..... 0.9, 0.95, 0.99). Re- garding the amount of data used perform mask calibration on each task, we align with Panda et al.) (2024) by using the validation split as it accounts for the 10% of the total training data. For LoTA, we set the number of iterations for mask calibration to match the number of mask calibration rounds used by our method (further details at Section A.2). This ensures that the drop in performance is negligible with respect to using the full training split while significantly reducing the computational overhead.
Datasets & Tasks. In line with what introduced in []barco et al. (2022) 2023); p; Ortiz-Jimenee et al. (2023), our vision experiments consider image classification across various domains. We adhere to the proposed experimental setup by utilizing eight datasets: Cars (Krause et al. 2013). DTD (Cimpoi
et al. 2014). EuroSAT (Helber et al. 2019), GTSRB (Stallkamp al. (011)) MNIST (LeCun 1998).
RESISC45 (Cheng et al. 2017). SUN397 (Xiao et 2016)) and SVHN (Netzer et al. 2011
follow the methodology outlined in Yaday et al. (2023). incorporating seven prescribed datasets: three regarding question answering
For the natural language processing (NLP) experiments,
(QASC (Khot al. 2020), WikiQA (Yang et al. (0055) and QuaRTz (Tafjord al. 2019 ) one for
paraphrase identification (PAWS (Zhang et 2019)), one focusing on sentence completion (Story Cloze (Sharma et ) and two for coreference resolution (Winogrande (Sakaguchi et al. 202
htttorr/ggthhu.ccc/////ttt////aa task arii
httear//daaaaaccccoo///fmmoaeeaa vectors
htte:://iktuub.oomnnkk-a///a111 masks
17

---

Published as a conference paper at ICLR 2025
and WSC (Levesque et al. 2012)). Concerning Task Negation, we align with Jitiz-Jimenez et al. (2023) and consider ImageNet (Deng et al. 2009) as the control dataset for vision experiments, while for NLP, we utilize RTE (Dagan et al. 2005). as it provides distinct task (i.e. natural language inference) with respect to the others considered for the NL. experiments.
Architectures & Pre-trained models. By following Iharco et al. (2023); Ortiz-Jimenez al. (2023); Yadav et al. (202)), on vision experiments, we use three variants of CLIP (Radford et al. 2021) with V/T- ViT-B/16, and ViT-L/14 models (Dosovitskiy et al. 2021). Regarding the NLP experiments, we employ T5-Small, T5-Base, and T5-Large models (Rafiel et al. 2020).
Fine-tuning details. All fine-tuning experiments on vision adhere the training protocol outlined by Ilharco et al.] (2022) ): OOtiz-Jimenez et al. (2023), with minor modifications made to the training code to accommodate the additional baselines and our method. Specifically, we fine-tune all datasets starting from the same CLIP pre-trained checkpoint, which is obtained from the open_ci repository (Gadre et al. 2024). Each model fine-tuned for 2,000 iterations with a batch size of 128, learning rate of 10 and a cosine annealing learning rate schedule with 200 warm-up steps. We use the AdamW optimizer (Loshchilov & Hutter 2019). Following [harco et al. (2022). the weights of the classification layer, which are derived from encoding standard set of zero-shot template prompts for each dataset, are frozen during fine-tuning. Freezing this layer ensures no additional learnable parameters are introduced and does not negatively affect accuracy (Ilharco al. 2022). Regarding the language experiments, we aligned with Yadav et al. (2023); Ihharco et al. (2023) and utilized three variants of the T5 model (Raffel et al. 2020), namely T5-Small T5-Base, and Ta-Large, with training conducted for a maximum of 75,000 steps. We employed an effective training batch size of 1024, with a learning rate of 10 4, To prevent overfitting, we implemented an early stopping mechanism with a patience threshold of 5. During training, we used io ffloati taatt 6 and the maximum sequence length was set to 128. Evaluation carried out by performing rank classification, where the model's 's log probabilities for all possible label strings are ranked. The prediction is considered correct the highest-ranked label corresponds to the correct answer.
Disentanglement error heatmaps. As prescribed by Ortiz-Jimenez et al. (2023). we produce the weight disentanglement visualizations of Figure 2 by computing the value of the disentanglement error {(a1, 0:2) 20 x 20 grid of equispaced values in [-3,3] x [-3, 3]. Estimations are carried out on a random subset of 2,048 test points for each dataset.
Tuning of in Task Arithmetic experiments. As outlined in Ilharco et al. (2023) Ortiz-Jimener et al. (2023). we employ a single coefficient, denoted as a, to adjust the size of the task vectors used modify the pre-trained models (i.e. an 02 ....r). For both the task addition and task negation benchmarks, following fine-tuning, we evaluate different scaling coefficients from the set E {0.0,00 0.05, 0.1, ..., 1.0} and select the value that achieves the highest target metric on small held-out portion of the training set, as specified in Ilharco et al. (2023), y; Jimenzz et al. (2023). To account for the lower norm of task vectors obtained via sparse fine-tuning (LoTA and TaLos) we extend this range by x1 x1/(1- - k) where l is the sparsity ratio of the task vector. Specifically, we aim to maximize the normalized average accuracy for Task Addition and ensure the minimum target accuracy for Task Negation while maintaining at least 95% of the original accuracy of the pre-trained model on the control task. The tuning of a is performed independently for each method.
Measuring computational costs and memory footprint. The timings in Table 3 are obtained using the perf_ccoutee clock from Python's 's time module. We monitored memory footprint using the NVIDIA nvml library All measurements are obtained during tineituning, with the very same setup explained in the fine-tuning details. Then, for each method, the mean and standard deviation of the timings computed over all iterations of all tasks. Peak memory usage, instead, is taken the maximum over all tasks. Memory usage is recorded at regular intervals of second, starting from the first forward pass and ending when the training loop breaks.
Normalized accuracy calculation in Task Addition. Normalized accuracy is computed by taking the average of the normalized individual accuracies over the T tasks. Given a task t, the normalized individual accuracy for is computed by taking the accuracy of the multi-task fused model on and dividing by the single-task accuracy that the fine-tuned checkpoint obtained on before being fused. Formally.
Normalized Accuracy T t=1
1
Σ
Accuracy| (P,,, 0n- + Σt (eTT)) Accuracy[f(D. O, a.T)
(11)
htttss:///ccs nwidi ctt.aaa.ccmcpcppceee htceddppppyyyylee cy/nvm1 1
18

---

Published as a conference paper at ICLR 2025
Algorithm I:
TaLoS to obtain task vectors
Input :Pre-trained model 0, E _e", neural network f(x 0) log pe (v(æx, task dataset D,, final sparsity k, number of rounds R, number of epochs F., learning rate " loss function C
Output: T'ask vector T. Rm for performing task arithmetic
// Calibrate sparse fine-tuning mask
2 c 2-=3 1
b Initialize weight mask to all Is
for
r- 1,2..... R do
p - k(//R)
> Compute the current sparsity at round r
8 1-00 0
> Initialize paramcter-wisc scores to all Os
// Compute diagonal FIM score according to Equation 7 for D, do
y ~ P(e oo) (y|z) end
Sample from oulput distribution of the model > Update scores on current example & sampled y
+ Ve log Pie e (v)[))
10
11
// Update retain only the bottom-k parcmeeeess
å sort descending (s)
Sorted scores in descending order I compute bottom-p threshold index
13
p <- [p.m]
14
for j = 1, 2, m do
15
if [8) 81p) > 0 then
Cly + 0
Set the mask of the j-th parameter to zero
17
end
18
end
19 end
20 // Sparse fine-tuning, starting from from 0u and obtaining 0:
21 for epoch = 1, 2, ...; E do
22
for (x, y) E D. do
23
0 0 7 l (o
Update rule, mask gradients with c
24
end
25 end
Ti 0 0o
Compute final task vector for task t
27
return TI
A.2 DETAILS ON MASK CALIBRATION & COMPUTATIONAL OVERHEAD
Sparse fine-tuning prescribes to mask gradients when updating the model parameters. Thus, is foundational that the mask is correctly calibrated before training. We mask only Linear, Attention, LayerNorm, and Convolutional layers (Kwon et al. 2022). Embedding layers and final projection layers are kept frozen. Furthermore, following standard procedures in Pruning-an at- attinniializttionn (Pal) (Tanaka et al.] 2020 Wang et al., (023), we iteratively refine the mask in multiple rounds to obtain better estimates from the mask calibration procedures. In detail, at each round, we select the bottom-p parameters (according to our score, detailed in Section 4 and we exponentially increase the current sparsity p. We repeat this process until reach the target sparsity k. For the sake of major clarity, we report in Algorithm [ the pseudocode for our procedure, encompassing both mask calibration and sparse fine-tuning. We remark that the choice of the bottom-k: values may lead layer collapse (Tanaka et al. 2020), namely, removing all parameters in a layer, disrupting the information flow in the network. To face this problem, we set e to some positive value close zero (e.g. 0.01) and we don't 'tn't include in the ranking those entries that are already soft-masked. This ensures that we are not changing the nature of our estimation, while countering the possibility of disrupting gradient flow in the network, during calibration.
Unfortunately, mask calibration introduces some amount of overhead before training. It is of paramount importance that such overhead doesn't 't hinder the computational gains obtained during fine-tuning.
Time overhead. The time spent for single iteration of mask calibration is comparable to that of single forward-backward iteration of non-linear fine-tuning (refer to Table Our mask calibration process typically employs an average of 10 iterations per round, with satisfactory results already observed at just 4 rounds (i.e., approximately 40 iterations total, we use the same batch size for mask calibration for fine-tuning! Given that fine-tuning generally requires around 2,000 iterations for vision experiments and substantially more for language tasks, we argue that the time overhead introduced by our mask calibration is negligible.
19

---

Published as a conference paper at ICLR 2025
Avgenge Execution Time
Menory Usage (GiR)
Negation
Train
Targ. (1)
Non-lincad Cene--icarrrre
18.6
56.09
Linearized FT Omi-Jimenez 2023
3311.77
3311.77
21.3
58.29
10.86
72.43
I-LoRA Tane ot al
9024
1053,07
1053.07
9.7
9.7
57.77
91.87
19.39
73.14
LoTA Panda et al.
2024
$1.84
2592.40
2644.24
12.9
15.4
15.4
$7.60
91.89
22.02
23.22
TatoS (Our)
63.04
656.23
719.27
7.8
7.8
7,8
88.40
95.19
10.63
73.55
Table 4: Computational cost and memory footprint of mask calibration and tine-tuning, Av- erage time (in seconds) and peak memory usage (in Gibibytes) of mask calibration and fine-tuning approaches CLIP ViT-L/14, alongside their performance both LoTA and TaLoS, we used batch size 128 for 40 iterations (in detail, 10 iterations per round for TaLoS, with 4 rounds total). We employ gradient checkpointing during mask calibration. Further details on the resource monitoring process can be found in Appendix A.1 Bold indicates the best results. Underline the second best.
the task arithmetic benchmark. For
LOTA VIT-B/22 (Cars)
Weight Rumainng Weight Ramaiing
TaLoS (Ours) VIT-B/32 (Cars)
(GASC)
Weggt Ramaninn Remaning
TaLoS (Qurs) T6 Smal (OASC)
Figure 5: Visualization of mask calibration. Percentage of parameters selected for sparse fine- tuning in ViT-B/32 (top) and a T5-Small (bottom) models, after method's mask calibration vs. LoTA's mask calibration, at 90% sparsity. On ViT-B/32, we calibrate the masks on the Cars dataset (Krause al. 2013). while T5-Small we QASC (Khot et al. 2020).
Memory overhead. The memory cost of each mask calibration iteration is equivalent that of each training iteration in non- non-linear fine-tuning. While we have not implemented any specific mechanism to reduce the memory footprint for calculating gradients (used scores) during mask calibration, there are several approaches available to achieve this. Most of these methods involve estimating gradients using zeroth-order information (Hinton. 2022
Malladi al. 2023a Sung et al. 2024), which allows
20

---

Published as a conference paper at ICLR 2025
to trade off speed for reduced memory usage by approximating gradients through multiple forward passes, eliminating the need to store computational graphs for automatic differentiation. Alternatively, gradient checkpointing (Chen et al. 2016) is another practical solution.
To further clarify the overall computational cost TaLos, encompassing both mask calibration and sparse fine-tuning,
provide a comparison in of the timings in seconds (averaged over the vision tasks) and the peak memory usage in Gibibytes of mask calibration and fine-tuning on a CLIP ViT-L/14. The results show that mask calibration time is approximately the same for TaLoS and LOTA, however, the costs in terms of memory are very different (LoTA requires storing optimizer states). Regarding total time, we recover what was presented in Table highlighting the beneficial effect of the highly structured sparsity of TaLoS on fine-tuning. The task arithmetic results are in line with Tables | with no detrimental effect given by the usage of gradient checkpointing.
A.3 FULL MASK CALIBRATION VISUALIZATIONS
For the sake of completeness, we provide a full visualization in Figure of the masks obtained after calibration with Tal.oS and LoTA. As shown, a repeating sparsity pattern emerges for our method across each transformer block. Notably, TaLoS consistently identifies only the and parameters for fine-tuning, demonstrating more structured behavior. In contrast, the mask generated by LoTA appears far more unstructured, with no clear pattern across the blocks.
A.4
ANALYZING THE FINE-TUNING BEHAVIOR
VIT-8/52
VIT-8/16
VIT-L/14
Fieekting
Work we
35
Latts
30
Fire tuning Accuraty
Delivet
RundaT
OTSRR
MN/ST
46
0L/037
Post-hoc Linearization Accuracy
Post-hoc Linearization Accuracy
Post-hoc Linearization Accuracy
Testing linearized behavior. Single-task accuracies of different fine-tuning strategies, each used to obtain their corresponding task vectors Tr, and the accuracy of their post-hoc linearization fim(-, Og T.). Different colors represent distinct fine-tuning strategies, while different markers indicate different tasks. Points that lie on the bisector (black dashed line) indicate that the fine-tuning process exhibited linearized behavior.
Figure 6:
Non Incar
Linearized
LoTA
ToLo8 (Ours)
oig Tameslop
Traninn
Temolp
DTD
SRUET
RES
SUROS7
SVIN
Figure 7: Change in parameter sensitivity throughout fine-tuning, We visualize the average relative change the output derivative of the parameters CLIP ViT-B/32 model when fine-tuned using different approaches. The starting point is the same for all methods.
We provide empirical validation the linear fine-tuning regime of our TaLoS (i.e. the change in the network output can be well-appor cell-approximated by its first-order Taylor expansion around 0,). As discussed by Orttiz-Jimnnz et al. (202)) cheap test consists of performing post-hoc linearization of
21

---

Published as a conference paper at ICLR 2025
the fine-tuned model around Og and checking whether the performance produced by such linearized model matches that of the original fine-tuned model. We use this approach and report the results in Figure 6 The scatter plots compare the fine-tuning accuracy against the post-hoc linearization accuracy for various tasks and fine-tuning strategies across different ViT architectures. Our method, TaLos, consistently demonstrates linearized behavior during fine-tuning for most tasks, evidenced by its proximity to the bisector line. This supports our claim that sparse fine-tuning, which both TaLos and LoTA employ, inherently promotes the emergence of linearized behavior during fine- tuning. Interestingly. while TaLoS exhibits this property across wide range of tasks, LoTA does not consistently demonstrate the same level of linearized behavior. This discrepancy can be attributed to differences in parameter selection, as discussed in the next paragraph, closely matching what happens during linearized fine-tuning. It's worth noting that linearized behavior may arise for various fine-tuning strategies, but its occurrence depends on the interaction between the task and pre-training (Malladi et al. 2023b). For instance, tasks such as GTSRB (Stalkkamp et al. [01I]. MNIST (LeCun 1998), and SVHN (Netzer et al. (001T) do not exhibit fine-tuning in the linear regime, hinting at potential mismatch with the pre-training, as evidence suggests (Radford et al. (021).
To further test the fine-tuning regime, we examine the evolution of parameter sensitivity during fine-tuning across different methods, as depicted in Figure 7 Inspired by Malladi et al. (2023b), we measure the average change in sensitivity as E,. [llVos ( 0())- - Vel (x. 0o) at each i-th training step, with æ from small subset of 2,048 examples from D.. Notably, for TaLos, the gradient fa 0) remains almost unchanged throughout training. closely mirroring the behavior of linearized fine-tuning. In contrast, LoTA diverges from this pattern, behaving more in line with non-linear fine-tuning. This phenomenon reinforces our claim that our method fine-tunes in the linearized regime, maintaining constant / (z., 0) during fine-tuning is critical for operating in the linearized regime (Malladi et al. 2023b).
A.5 ABLATIONS ON MASK SPARSITY RATIO
viTaiie Task Acdiion
VITL/14 Task Addtion
T5-5mall Task
Larpe Task Additon
95
90
95
85
00-
75
85
83
60
75
Botoom: Sparsity (%)
Batomm sparsity 170)
Bodom Sparshy (%)
Bettom sparsty (S)
Avg. Single-Task Acc.
Aug. Absolute Acc.
Aug. Normalized
VIT-R2 Negation
Task Negstion
T8-Small- Negetion
arge Teak Nagaton
42
30
10
30
38
50
10
75
Botom/
sparsity
Bostomi
sparsity
Aug Target Task Apc.
Avg. Control Task Acc.
Figure 8: Effect of the choice of in TaLos. Results of hyperparameter tuning of ls in TaLos for task addition and negation on both vision and language. Note that we tune k indirectly by controlling its value via the sparsity ratio. For task addition (top) we report the average single-task accuracy (before addition), absolute and normalized accuracies (after addition). For task negation (bottom) we report average target and control accuracies (after negation).
For clear understanding of the effect of sparsity on TaLoS, we report in Figure 3 the task arithmetic performance achieved by TaLos, while varying the sparsity level. At 0% sparsity, we recover full mon-linear)) fine-tuning results. Increasing the sparsity improves the task arithmetic performance, while slightly decreasing the average single-task accuracy, as fewer parameters are updated during fine-tuning. Optimal values for absolute accuracy (in task addition) and target accuracy (in task negation) are observed for sparsity level of 90% across a variety of models. After 90% sparsity there is a slight drop in both task arithmetic and single-task performance, making such sparsity levels not ideal. Intuitively, if the fine-tuning involves too little weight the resulting entries in the task vector will be mostly zero, reducing the ability to perform task arithmetic effectively. We can conclude that, like other parameter-efficient fine-tuning methods, our approach trades some single-task performance
22

---

Published as a conference paper at ICLR 2025
VIT-B/32
VIT-B/16
VIT-L/14
MHIST
CarcMNiST
CarsMNIST
Care
T5-Small
T5-Base
TS-Large
WIKIDA
WIKIOA
WIKIAA
PAWS
QASC
QASC
Storycloze
Storycloze
StoryGoze
Winogrande
Winegrande
Winogrando
Pre: trained (Zero-shot)
Non-linear FT
Linearized FT
Linearized LoRA
LoTA
Talas (Ours)
Figure 9: Task performance after fine-tuning, Single-task accuracies obtained by different fine- tuning approaches across vision and language experiments. Results are displayed for three model sizes of CLIP ViT (B/32, B/16, L/14) and T5 (Small, Base, Large), with outer edges representing higher accuracy. The dashed line represents the accuracies before fine-tuning.
for parameter efficiency. But this trade-off allows also for superior task arithmetic capabilities for TaLoS (Tables while maintaining competitive single-task accuracy, especially for larger models where the performance drop becomes negligible (Figure D).
A.6 SINGLE-TASS PERFORMANCE OF FINE-TUNING METHODS
In this analysis we focus discussing the single-task performance of TaLoS before task addition. To this goal, we compare in Figure the the accuracies obtained by TaLoS (at 90% sparsity) vs. the other fine-tuning strategies. In almost all cases TaLoS achieves approximately the same performance of full fine-tuning methods (Non-linear FT and Linearized FT), occasionally improving over Linearized FT (ViT-B/32 SVHN), which is remarkable, as TaLos updates only very small subset of parameters, while full fine-tuning (both linearized and non-linear) updates the whole set of model parameters. Furthermore, compared with parameter-efficient fine-tuning methods, which allows for a truly fair comparison (the parameter count is the same across methods), almost always TaLoS improves with respect to Linearized LoRA and matches the performance of LoTA. However, we remark that the task arithmetic performance of TaLoS is much higher than the latter (see Tables 2
A.7 ADDITIONAL EVIDENCE ON THE PARAMETER-SHARIE PHENOMENON
In this section, provide additional validation of the phenomenon observed in our motivating example, namely that insensitive parameters are consistently shared across tasks. First, we revisit the relationship between parameter sensitivity and the Fisher Information matrix (FIM) Fisher (1922). highlighting why the FIM serves as suitable tool for conducting sensitivity analysis. Next, we present further experimental evidence to support the findings of Section 4.1 Specifically, instead of pruning the least sensitive parameters, we analyze the effect of perturbing them and subsequently examine whether masks calibrated on different tasks exhibit significant similarity.
Parameter sensitivity analysis and connection to Fisher Information. Applying a perturbation 0,- - Og + 80, to a subset of the pre-trained weights O, and observing no change the output fa 0²) ~ fxx 0o) intuitively means that those weights have low sensitivity to the task. So, pruning or randomizing them would not affect input- behavior. However, there may be problem in assessing sensitivity via extreme aundomizations/petuuratttt very high magnitude perturbations (perhaps, additive), then such perturbations will not be suitable assess the sensitivity of the parameters, this could potentially move the current solution
if "extreme" randomization refers to
23

---

Published as a conference paper at ICLR 2025
ViT-B/32 [1
0.0061
(1,099)
T5-Smal
10045)
Mask Calibration Datasel
Cars
1:00
C:0
RSF
CASC
WARQA
EVriBAT 0U
QuaRTa
VR8ST
RES/8045
Bary
Wingrande
SYHN
0.0
s07
3M0
WBC
0TO
MPNOT
MAuGh
Grantr
Gtory Cles
Test Dataset
Test Dataset
Figure 10: Perturbing parameters with low sensitivity. The heatmaps illustrate the effect of perturbing the parameters with the lowest sensitivity (measured by (F) D.)l on different tasks across various pre-trained models. Each grid compares the accuracy ratios for models after pruning, with the rows representing the task Dd used to identify the parameters with the lowest sensitivity and the columns showing the model's performance on each task after pruning those parameters. The accuracy ratios are normalized by the model's performance before perturbation. The average magnitude μu and standard deviation across perturbed parameters, prior applying noise are also reported. The ratio of perturbed parameters (10%) is chosen based on the experiment of Figure [
(parametrized by 0, R" away from the current local optimum, distinct region of the loss landscape. Indeed, sensitivity analysis generally refers to "robustness to small perturbation". This concept, alongside how perform proper sensitivity analysis on the parameters of neural network, has been formalized by a rich literature dedicated to applications of information geometry (Amari) 1996. Chaudhry al., 2018; Pascanu & Bengio 2013). Specifically, as shown by Chaudhry al. (2018); Pascanu & Bengio (201)), to assess the influence of each weight on the output of a network, we can use the Kullback-Leblee (KL) divergence between the output distribution induced by the original network (po,) vs. the one induced by the perturbed network (P0++50). Mathematically, assuming 80 (a small perturbation).
DEL (Po, (0o)
2 50 F(0,)60 (115011))
The KL divergence is zero if the perturbation doesn't affect the output, revealing that the modified weights are not influential for the output. It larger than zero otherwise. Here F(0o) E R= is the Fisher Information matrix (FIM) (Fisher 1922 matrix defined as,
Amari 1996). It is a positive semi-definite symmetric
F F(Oo) = (yll (vlam) Vo log pe, (y|æ) Ve log pe, (3/=²) []]
It can be used to relate the changes in the parameters to the changes in the outputs, effectively implementing a proper sensitivity analysis of the parameters of a neural network by studying the magnitude of its diagonal elements, they represent the sensitivity of each parameter (Chaudhry) let al. 2018 Pascanu & Bengio [013) Matena & Raffel] 2022). Formally, for each parameter j E ..... m. its corresponding entry on the diagonal of the FIM has value
Fis.i (o)) Ex [ey-o (ylm)) Ve log pe, (1]=))]]]
The higher this value, the more the model will be affected by the j-th parameter changes.
Perturbing the least sensitive parameters. We repeat in Figure 10 the the experiment of Figure I but by adding noise distributed as (0, 20/) to the bottom-10% of parameters, instead of pruning them. is the standard deviation of the parameters, previous to perturbation. The results align with the analysis reported Figure T highlighting the stability of these parameters across tasks.
Measuring masks intersections across tasks. Additionally, in Figure I1 we provide further evidence about the overlap of low- sewsiiiiiivity parameters across tasks. For each parameter, we compute the mean Intersection over Union (mloU) of masks, between each task pair: starting from pre-traine trairre parameters tn. we predict the mask on task t and then check its intersection over union against the mask predicted task t' (which acts ground truth). A mIoU of 1 signals perfect mask overlap
24

---

Published as a conference paper at ICLR 2025
VIT-BV32
VIF-L/14
TS-Small
TS-Large
Mask Celibration Dalasel
DecG
Wi-O
WiLDA Cat
C2T
19079
HWYS en
Siery Sian
Sery
SUNDR0
Wenger
Whoggand
2/-8
SWaN
#803
380
WeG
6.00
Quenta
Clock
Stay
Ground Truth Dalasel
Ground Truth Dalaset
Ground Truth Dataset
Ground Truth Dataset
Figure 11: Masks intersections of low sensitivity parameters. The heatmaps illustrate the mean Intersection over Union (mloU) between masks pairs of the lowest sensitivity parameters (measured by [F]r)] (0. D,)ly= on all tasks across different pre-trained models. For each mask, the amount of selected parameters (10%) is chosen based on the experiment of Figure T
Mathod
VIT-R/32
TS-Small
Abs. (1)
Norm.
Ahs. (t)
Norm.
Pre-trained (Zero-shot)
47.72
55.70
Non-linear FT IThrrrn al
2023
71.25
76.94
65.04
87,98
TIES-Merging et
2023
62.53
Task-wise AdaMerging Yang et al
2024
73.39
79.02
66.19
89.86
Layer-wise AdaMerging Yang et al 2024
77.06
82.98
66.61
89.86
(Ours)
65.04
97.22
TaLoS TeES-Merging
78.15
89.10
54.54
85.42
TaLoS Task-wise AdaMerging
79.73
90.84
66.47
99.21
TaLos Liyer-wise AdaMerging
80,25
91.40
66.76
99.63
Table 5: TaLoS on different model merging schemes. Average absolute accuracies (%) and normalized accuracies (%) of CLIP ViT-B/32 and T5-Small pre-trained models edited by adding task vectors on each of the downstream tasks. We normalize performance of each method by their single-task accuracy. Bold indicates the best results. Underline the second best.
between tasks. The number of parameters selected by each mask is 10%, in line with the experiment of Figure [] Smaller vision models (ViT-B/32) exhibit high parameter sharing (> 0.7 mloU) of low-sensitivity parameters, while smaller language models (T5- share fewer (0.3-0.5 mloU). However, with a fixed 10% mask sparsity, larger models in both vision and language domains share more low-sensitivity parameters across tasks.
A.8 COMBINING TALoS WITH OTHER MODEL MERGING SCHEMES
We extend Table i Table 5 by testing our TaLoS in combination with other merging schemes (TIES-Merging Yadav et (2023) and AdaMerging Yang al.) (2024)). Specifically, for TIES- Merging we skip the sparsification part, as the task vectors obtained by TaLoS are already sparse. Regarding AdaMerging, we test both Task-wise AdaMerging and Layer-wise AdaMerging. As we can see, in both vision and language experiments, applying T'EE-Merging to our TaLoS is harmful. Seemingly, the signs of task vectors obtained via La play an important role and disrupting them according to some heuristics causes drop in performance. Regarding AdaMerging, TaLoS has full compatibility with existing methods for automating the selection of optimal merging coefficients, highlighting its versatility. However, by itself TaLoS is already robust enough that doesn't 't benefit this much from neither task tunings nor wise tunings.
can see that
25


Reviews:
[{"review_id": "7kzDO7B8En", "reviewer": "Reviewer_qd2R", "summary": "The paper proposes a novel fine-tuning strategy based on sparsity that is designed explicitly to facilitate merging a posteriori. The proposed method first identifies a per-task parameter mask that actively promotes weight disentanglement and is used during fine-tuning to constraint the parameter updates only to those in the mask. The method includes vision and NLP benchmarks for task addition and negation as well as weight disentanglement plots which are common in the literature", "questions": "1. Can you explain the discrepancy in the normalized results in t5-small? TA is better at absolute compared to tall masks but much lower in normalized.\n2. Table 3: why is non-linear ft forward-backward pass time so much more than the proposed method? Given that it does not have the masking of equation 7, shouldn't it be a lower bound to TaLoS? Also, where does the difference in peak memory usage come from? this would be the case if entire layers are excluded from the mask, saving space in optimizer states, but this is not explicitly mentioned.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The motivation of the paper is clear and the writing is easy to follow\n2. The method is simple and intuitive.\n3. The performance is superior compared to the closest baselines in both task addition and negation, highlighting the importance of sparsity for task localization and the effectiveness of the method.\n4. The experiments on mask calibration are interesting, e.g. Figure 4. and showcase that the method clearly identifies a pattern in the weights of the model.", "weaknesses": "The paper has several flaws that, if addressed, can improve the paper significantly. Specifically:\n\n\n1. The applicability of the method is undermined by the fact that the user needs to redo the fine-tuning from scratch.\n2. Mask construction part of the algorithm is not clear: L236 refers to $\\mathbf{x}$ but it is not clear if the process involves a single batch or not. The reader needs to go to Algorithm 1 of the appendix to find that multiple “rounds” are used. Both the algorithm and the term “rounds” are never mentioned in the main text. Thus, the reader cannot understand the computational overhead of the method and the assumptions on task availability.\n3. The level of mask sparsity is not mentioned on the paper, while it is a major part of the algorithm, apart from the caption of figure 4. It is not clear if 90% is used throughout. Constraining the fine-tuning process too much will result in worse single-task performance (before merging), limiting the method’s applicability. This is an important hyperparameter of the method and it is not discussed/ablated. Moreover, the absolute fine-tuned results should also be given. This is especially important given that fine-tuning in the tangent space “may cause single task performance drop” (L118). The reader can only imply the difference in performance from Figure 3, where the proposed method achieves lower performance compared to some tasks such as SVHN, Cars and DTD.\n4. Logical jump in motivation. The paper includes a nice experiment for Figure 1, discussed in lines 208-226. However, this experiment focuses on the *zeroshot* and there is a logical jump on how and why this translates to the fine-tuned checkpoints. The paper would benefit from better explanation\n5. Presentation of results is sometimes sloppy. For instance, T5-large Absolute: 77.31 > 76.20, making the +2.87 in the last row wrong. \n6. Unfair comparison for task negation: all posthoc methods are designed for task addition, making the comparison unfair imo. this should be stated clearly.\n7. Need rephrasing:\n\t1. “This inherent property of sparse finetuning increases the likelihood that the linearization condition will hold, effectively rendering explicit network linearization unnecessary.” This is not shown and should be framed as intuition rather than fact\n\t2. Contribution 1 needs to be rewritten and be more specific: the current version can be said for any model merging method\n8. Related work: apart from Fisher merging, the methods outlined in the model merging paragraph are proposed primarily for task addition and, therefore, should be merged with the task arithmetic paragraph. The authors should include a paragraph specific to fine-tuning specific for merging, as these are their baselines and most related works.\n\n“Existing task arithmetic solutions are strongly tied to model linearization which leads to computational bottlenecks” → does this refer to tangent? task arithmetic/ties etc have no bottleneck\n\nMinor comments:\n\n\n1. L47: interference missing refs from ties and tall masks\n2. L52: “is crucial for preventing interference” this does not seem like a correct characterization\n3. L137-138: this is taken from [1], reference is missing. Moreover, equations 2 and 3 seem redundant\n4. L194: fix notation of the subscript\n\n[1] Ortiz-Jimenez, G., Favero, A. and Frossard, P., 2024. Task arithmetic in the tangent space: Improved editing of pre-trained models. *Advances in Neural Information Processing Systems*, *36*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5eVAH4Xawv", "reviewer": "Reviewer_S7QQ", "summary": "This paper presents a new approach for efficient model editing through Task Arithmetic, specifically using a method called Task-Localized Sparse Fine-Tuning (TaLoS). TaLoS differs from traditional linearization-based approaches by selectively updating only parameters with low gradients, reducing computational load while minimizing task interference. The proposed method enables precise task addition and removal across vision and NLP tasks and demonstrates high performance in experimental results. Additionally, TaLoS achieves a reduced memory footprint and runtime by avoiding explicit linearization, verified by extensive evaluations on computational cost.", "questions": "- Uniformity of Parameter Sensitivity\nRegarding Figure 4, if the gradient norms are not uniform across layers, key parameters might differ across tasks. In this scenario, could the TaLoS approach risk inadvertently pruning important parameters? Could the authors provide further insights on this?\n\n- Conditions for Task-Specific Parameter Pruning\nLine 217 mentions that removing parameters deemed unimportant for a specific task does not degrade performance on other tasks. Could the authors clarify the specific conditions or limitations under which this holds? Insights into the effect of model size or architecture variations would also be beneficial.\n\n- Pruning Criteria Unclarity\nIn Figure 1, the specific pruning criteria (e.g., values for bottom-k) were not readily apparent. Additional detail on this would enhance reproducibility.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novelty and Effectiveness: This work introduces a novel technique that improves computational efficiency by updating only parameters with low sensitivity while maintaining task separability. This approach significantly reduces the computational burden associated with linearization-based methods and is a valuable contribution in terms of efficiency.\n- Empirical Performance: Experimental results demonstrate that TaLoS consistently outperforms baseline methods, including LoTA, across a range of tasks. As shown in Figure 4, TaLoS maintains high accuracy when adding tasks and minimizes the impact on target tasks when removing tasks, showcasing its robustness in task arithmetic.", "weaknesses": "- Explanation of Task Interference: Regarding the sensitivity analysis in Figure 1, I understand that it evaluates the performance on other tasks when parameters with low gradients in a given task are pruned. However, could you provide a more detailed explanation on whether altering these selected parameters, rather than simply pruning them, would also have no impact on inference? For example, could you test whether applying extreme random changes to these parameters would still leave other tasks unaffected? This additional analysis would clarify the stability of these parameters across tasks.\n- Key Concern: While intuitively, changing parameters with low sensitivity should not significantly affect the function for data drawn from task t , it is less clear why this would not affect other tasks post-finetuning. A detailed explanation of this phenomenon such as measuring the impact of these parameter changes on other tasks' performance or analyzing the overlap of low-sensitivity parameters across different tasks would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rBurMuGLti", "reviewer": "Reviewer_51Ub", "summary": "This paper proposes TaLoS, a novel sparse fine-tuning approach for efficient model editing in the task arithmetic framework. The key idea is to prevent interference between tasks by constraining updates to parameters that have low sensitivity across multiple tasks. By selectively fine-tuning only these parameters, the method aims to adapt models to new tasks while minimizing interference when performing task arithmetic operations.", "questions": "- Can the authors provide a clearer, more concise explanation of the core algorithm, perhaps with pseudocode?\n\n- Is it possible to derive theoretical guarantees or convergence properties for the proposed method?\n\n- How sensitive is the method to the choice of k, and are there principled ways to select its optimal value?\n\n- Can the authors elaborate on the direct relationship between gradient magnitudes and task interference, ideally with theoretical justification?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novel approach combining sparse fine-tuning with task localization for improved task arithmetic\n\n- Comprehensive empirical evaluation across vision and language tasks showing state-of-the-art performance\n\n- Efficiency gains in both computation and memory usage compared to existing methods\n\n- Detailed analysis of weight disentanglement and function localization properties", "weaknesses": "1. The caption for Figure 1 lacks clarity, particularly in explaining that \"sensitivity\" refers to gradient values. This makes the figure difficult to interpret without careful reading of the surrounding text.\n\n2. The key algorithm and mechanism are not presented clearly. For example, there is a grammatical error in the sentence \"To prevent interference between tasks and enable task arithmetic, prevents significant changes to the parameters that are highly sensitive to multiple tasks.\" This verbose phrasing obscures the core mechanism.\n\n3. The paper lacks theoretical analysis or proof of convergence for the proposed method. While empirical results are strong, a theoretical foundation would strengthen the work.\n\n4. Equation 8 introduces parameter k as a significant factor in the upper bound, but there is insufficient discussion of its impact and optimal selection. A more comprehensive analysis of this parameter would be valuable.\n\n5. The direct relationship between gradient values and task interference is not thoroughly explored or justified theoretically. While intuitively reasonable, a more rigorous examination of this connection would bolster the method's foundation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DdwzSpuGuu", "reviewer": "Reviewer_YVQa", "summary": "This paper introduces Task-Localized Sparse Fine-Tuning (TaLoS), a novel approach to efficient model editing through sparse fine-tuning, aimed at enhancing task-specific performance in pre-trained models without explicit linearization. The method selectively updates parameters with minimal cross-task impact, achieving weight disentanglement and enabling efficient task arithmetic. Extensive experiments across vision and language domains demonstrate TaLoS's superiority over baseline methods in task addition/negation performance, computational cost, and memory efficiency.", "questions": "see weekness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper is well-written, with clear explanations of the motivation.\n+ The empirical analyses offer valuable insights into the model’s behavior, which is interesting and well-reasoned.\n+ The experiments are comprehensive.", "weaknesses": "- Could the authors clarify further how TaLoS achieves linear behavior? The paper mentions that parameters unimportant for a given task are also generally unimportant for other tasks, which might imply a shared subset of parameters across tasks. It would be helpful if the authors provided experimental results showing parameter overlap across task vectors.\n- It is unclear how the mask sparsity ratio is determined in practice. A sensitivity analysis of this hyperparameter might be valuable.\n- I am curious about how this method might perform in combination with other model merging techniques, such as Ties-Merging or AdaMerging. Although such an investigation would require additional work, discussing this potential integration would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)