Efficient Model Editing with Task-Localized Sparse Fine-tuning

Authors: Leonardo Iurada, Marco Ciccone, Tatiana Tommasi
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: task arithmetic, parameter-efficient fine-tuning

Abstract:
Task arithmetic has emerged as a promising approach for editing models by representing task-specific knowledge as composable task vectors. However, existing methods rely on network linearization to derive task vectors, leading to computational bottlenecks during training and inference. Moreover, linearization alone does not ensure weight disentanglement, the key property that enables conflict-free composition of task vectors. To address this, we propose TaLoS which allows to build sparse task vectors with minimal interference without requiring explicit linearization and sharing information across tasks. We find that pre-trained models contain a subset of parameters with consistently low gradient sensitivity across tasks, and that sparsely updating only these parameters allows for promoting weight disentanglement during fine-tuning. Our experiments prove that TaLoS improves training and inference efficiency while outperforming current methods in task addition and negation. By enabling modular parameter editing, our approach fosters practical deployment of adaptable foundation models in real-world applications.

Decision: accept (poster)
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730188271818

thread_type:
official_review

<thread_object>
creator: reviewer_yvqa

<content>
summary:
This paper introduces Task-Localized Sparse Fine-Tuning (TaLoS), a novel approach to efficient model editing through sparse fine-tuning, aimed at enhancing task-specific performance in pre-trained models without explicit linearization. The method selectively updates parameters with minimal cross-task impact, achieving weight disentanglement and enabling efficient task arithmetic. Extensive experiments across vision and language domains demonstrate TaLoS's superiority over baseline methods in task addition/negation performance, computational cost, and memory efficiency.

strengths:
+ The paper is well-written, with clear explanations of the motivation.
+ The empirical analyses offer valuable insights into the model’s behavior, which is interesting and well-reasoned.
+ The experiments are comprehensive.

weaknesses:
- Could the authors clarify further how TaLoS achieves linear behavior? The paper mentions that parameters unimportant for a given task are also generally unimportant for other tasks, which might imply a shared subset of parameters across tasks. It would be helpful if the authors provided experimental results showing parameter overlap across task vectors.
- It is unclear how the mask sparsity ratio is determined in practice. A sensitivity analysis of this hyperparameter might be valuable.
- I am curious about how this method might perform in combination with other model merging techniques, such as Ties-Merging or AdaMerging. Although such an investigation would require additional work, discussing this potential integration would be interesting.

questions:
see weekness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730380667601

thread_type:
official_review

<thread_object>
creator: reviewer_51ub

<content>
summary:
This paper proposes TaLoS, a novel sparse fine-tuning approach for efficient model editing in the task arithmetic framework. The key idea is to prevent interference between tasks by constraining updates to parameters that have low sensitivity across multiple tasks. By selectively fine-tuning only these parameters, the method aims to adapt models to new tasks while minimizing interference when performing task arithmetic operations.

strengths:
- Novel approach combining sparse fine-tuning with task localization for improved task arithmetic

- Comprehensive empirical evaluation across vision and language tasks showing state-of-the-art performance

- Efficiency gains in both computation and memory usage compared to existing methods

- Detailed analysis of weight disentanglement and function localization properties

weaknesses:
1. The caption for Figure 1 lacks clarity, particularly in explaining that "sensitivity" refers to gradient values. This makes the figure difficult to interpret without careful reading of the surrounding text.

2. The key algorithm and mechanism are not presented clearly. For example, there is a grammatical error in the sentence "To prevent interference between tasks and enable task arithmetic, prevents significant changes to the parameters that are highly sensitive to multiple tasks." This verbose phrasing obscures the core mechanism.

3. The paper lacks theoretical analysis or proof of convergence for the proposed method. While empirical results are strong, a theoretical foundation would strengthen the work.

4. Equation 8 introduces parameter k as a significant factor in the upper bound, but there is insufficient discussion of its impact and optimal selection. A more comprehensive analysis of this parameter would be valuable.

5. The direct relationship between gradient values and task interference is not thoroughly explored or justified theoretically. While intuitively reasonable, a more rigorous examination of this connection would bolster the method's foundation.

questions:
- Can the authors provide a clearer, more concise explanation of the core algorithm, perhaps with pseudocode?

- Is it possible to derive theoretical guarantees or convergence properties for the proposed method?

- How sensitive is the method to the choice of k, and are there principled ways to select its optimal value?

- Can the authors elaborate on the direct relationship between gradient magnitudes and task interference, ideally with theoretical justification?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582777597

thread_type:
official_review

<thread_object>
creator: reviewer_s7qq

<content>
summary:
This paper presents a new approach for efficient model editing through Task Arithmetic, specifically using a method called Task-Localized Sparse Fine-Tuning (TaLoS). TaLoS differs from traditional linearization-based approaches by selectively updating only parameters with low gradients, reducing computational load while minimizing task interference. The proposed method enables precise task addition and removal across vision and NLP tasks and demonstrates high performance in experimental results. Additionally, TaLoS achieves a reduced memory footprint and runtime by avoiding explicit linearization, verified by extensive evaluations on computational cost.

strengths:
- Novelty and Effectiveness: This work introduces a novel technique that improves computational efficiency by updating only parameters with low sensitivity while maintaining task separability. This approach significantly reduces the computational burden associated with linearization-based methods and is a valuable contribution in terms of efficiency.
- Empirical Performance: Experimental results demonstrate that TaLoS consistently outperforms baseline methods, including LoTA, across a range of tasks. As shown in Figure 4, TaLoS maintains high accuracy when adding tasks and minimizes the impact on target tasks when removing tasks, showcasing its robustness in task arithmetic.

weaknesses:
- Explanation of Task Interference: Regarding the sensitivity analysis in Figure 1, I understand that it evaluates the performance on other tasks when parameters with low gradients in a given task are pruned. However, could you provide a more detailed explanation on whether altering these selected parameters, rather than simply pruning them, would also have no impact on inference? For example, could you test whether applying extreme random changes to these parameters would still leave other tasks unaffected? This additional analysis would clarify the stability of these parameters across tasks.
- Key Concern: While intuitively, changing parameters with low sensitivity should not significantly affect the function for data drawn from task t , it is less clear why this would not affect other tasks post-finetuning. A detailed explanation of this phenomenon such as measuring the impact of these parameter changes on other tasks' performance or analyzing the overlap of low-sensitivity parameters across different tasks would strengthen the paper.

questions:
- Uniformity of Parameter Sensitivity
Regarding Figure 4, if the gradient norms are not uniform across layers, key parameters might differ across tasks. In this scenario, could the TaLoS approach risk inadvertently pruning important parameters? Could the authors provide further insights on this?

- Conditions for Task-Specific Parameter Pruning
Line 217 mentions that removing parameters deemed unimportant for a specific task does not degrade performance on other tasks. Could the authors clarify the specific conditions or limitations under which this holds? Insights into the effect of model size or architecture variations would also be beneficial.

- Pruning Criteria Unclarity
In Figure 1, the specific pruning criteria (e.g., values for bottom-k) were not readily apparent. Additional detail on this would enhance reproducibility.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674922062

thread_type:
official_review

<thread_object>
creator: reviewer_qd2r

<content>
summary:
The paper proposes a novel fine-tuning strategy based on sparsity that is designed explicitly to facilitate merging a posteriori. The proposed method first identifies a per-task parameter mask that actively promotes weight disentanglement and is used during fine-tuning to constraint the parameter updates only to those in the mask. The method includes vision and NLP benchmarks for task addition and negation as well as weight disentanglement plots which are common in the literature

strengths:
1. The motivation of the paper is clear and the writing is easy to follow
2. The method is simple and intuitive.
3. The performance is superior compared to the closest baselines in both task addition and negation, highlighting the importance of sparsity for task localization and the effectiveness of the method.
4. The experiments on mask calibration are interesting, e.g. Figure 4. and showcase that the method clearly identifies a pattern in the weights of the model.

weaknesses:
The paper has several flaws that, if addressed, can improve the paper significantly. Specifically:


1. The applicability of the method is undermined by the fact that the user needs to redo the fine-tuning from scratch.
2. Mask construction part of the algorithm is not clear: L236 refers to $\mathbf{x}$ but it is not clear if the process involves a single batch or not. The reader needs to go to Algorithm 1 of the appendix to find that multiple “rounds” are used. Both the algorithm and the term “rounds” are never mentioned in the main text. Thus, the reader cannot understand the computational overhead of the method and the assumptions on task availability.
3. The level of mask sparsity is not mentioned on the paper, while it is a major part of the algorithm, apart from the caption of figure 4. It is not clear if 90% is used throughout. Constraining the fine-tuning process too much will result in worse single-task performance (before merging), limiting the method’s applicability. This is an important hyperparameter of the method and it is not discussed/ablated. Moreover, the absolute fine-tuned results should also be given. This is especially important given that fine-tuning in the tangent space “may cause single task performance drop” (L118). The reader can only imply the difference in performance from Figure 3, where the proposed method achieves lower performance compared to some tasks such as SVHN, Cars and DTD.
4. Logical jump in motivation. The paper includes a nice experiment for Figure 1, discussed in lines 208-226. However, this experiment focuses on the *zeroshot* and there is a logical jump on how and why this translates to the fine-tuned checkpoints. The paper would benefit from better explanation
5. Presentation of results is sometimes sloppy. For instance, T5-large Absolute: 77.31 > 76.20, making the +2.87 in the last row wrong. 
6. Unfair comparison for task negation: all posthoc methods are designed for task addition, making the comparison unfair imo. this should be stated clearly.
7. Need rephrasing:
	1. “This inherent property of sparse finetuning increases the likelihood that the linearization condition will hold, effectively rendering explicit network linearization unnecessary.” This is not shown and should be framed as intuition rather than fact
	2. Contribution 1 needs to be rewritten and be more specific: the current version can be said for any model merging method
8. Related work: apart from Fisher merging, the methods outlined in the model merging paragraph are proposed primarily for task addition and, therefore, should be merged with the task arithmetic paragraph. The authors should include a paragraph specific to fine-tuning specific for merging, as these are their baselines and most related works.

“Existing task arithmetic solutions are strongly tied to model linearization which leads to computational bottlenecks” → does this refer to tangent? task arithmetic/ties etc have no bottleneck

Minor comments:


1. L47: interference missing refs from ties and tall masks
2. L52: “is crucial for preventing interference” this does not seem like a correct characterization
3. L137-138: this is taken from [1], reference is missing. Moreover, equations 2 and 3 seem redundant
4. L194: fix notation of the subscript

[1] Ortiz-Jimenez, G., Favero, A. and Frossard, P., 2024. Task arithmetic in the tangent space: Improved editing of pre-trained models. *Advances in Neural Information Processing Systems*, *36*.

questions:
1. Can you explain the discrepancy in the normalized results in t5-small? TA is better at absolute compared to tall masks but much lower in normalized.
2. Table 3: why is non-linear ft forward-backward pass time so much more than the proposed method? Given that it does not have the masking of equation 7, shouldn't it be a lower bound to TaLoS? Also, where does the difference in peak memory usage come from? this would be the case if entire layers are excluded from the mask, saving space in optimizer states, but this is not explicitly mentioned.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733307372211

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Note - Outcome of the discussion period

comment:
We thank all Reviewers again for their time and effort in providing their valuable feedback on our work. 

We clarify general doubts about the placement in literature of our submission:

- Our paper places itself in the **Task Arithmetic literature**, a framework for scalable and cost-effective model editing. Unlike conventional model merging which aims to only create multi-task models, Task Arithmetic focuses on **deriving task vectors that can edit model behavior**, enhancing capabilities via task addition or suppressing unwanted behaviors through task negation. However, interference between tasks remains a significant barrier, as combining independently fine-tuned models often disrupts performance on previously learned tasks.

- The authors of [1] highlighted the importance of **weight disentanglement**, a condition ensuring tasks modify only their respective activation regions. They demonstrate that pre-training inherently promotes some level of disentanglement but fine-tuning may disrupt this property. Prior solutions relying on explicit model linearization have proven computationally expensive and insufficient, as they fail to ensure task-specific activation.

Thus, in our work we propose a principled way to perform fine-tuning and maintain weight disentanglement for task arithmetic:

- We derive a novel set of **function localization constraints** that provide exact guarantees of weight disentanglement on linearized networks, ensuring task-specific activation.

- We empirically observe that least sensitive parameters in architectures pre-trained on large-scale datasets can be consistently identified across tasks. This insight enables satisfying localization constraints under strict individual training assumptions, **without cross-task communication**.

- We introduce Task-Localized Sparse Fine-Tuning (TaLoS), **a mask selection strategy for sparse fine-tuning** that produces task vectors enabled for Task Arithmetic. TaLoS jointly fulfills the localization constraints and induces a linear regime during fine-tuning, all without the computational overhead of explicit network linearization.

Extensive theoretical and empirical analyses confirm that our proposed approach maintains weight disentanglement, prevents task interference, and ensures compatibility between task vectors. This enables robust and efficient task arithmetic, establishing TaLoS as a practical solution for scalable and targeted model behavior editing.

In response to individual reviewers’ requests, we updated our manuscript as follows:

- **Clarification on mask construction & its computational cost [qd2R, 51Ub, YVQa].**
We added more precise information at LL245-248 and L260 about TaLoS (including the usage of multiple rounds and a pointer to the mask calibration details and pseudocode). We now state explicitly at LL259-260 that the best results are observed in sparsity ratios between 90% and 99%. To provide a more detailed overview of the effect of sparsity on TaLoS we introduced section A.5 in the Appendix and presented the results in Figure 8. We extended Table 3, providing a detailed comparison of the memory and time costs of mask calibration in Appendix A.2 (Table 4 and discussion at LL1117-1123).

- **Single-task fine-tuning results [qd2R]**. 
We added section A.6 in the Appendix to show and discuss the single-task performance of TaLoS before task addition: Figure 9 presents the accuracies obtained by TaLoS (at 90% sparsity).

- **Rephrasing some parts & reorganization of the related works**.
We updated the claim in LL273-277 referencing earlier the experimental results of Appendix A.4. We addressed the Reviewer’s suggestion about the contributions and rephrased more clearly all of them. According to the Reviewer’s suggestions, we improved how we present and refer to previous works, moving TIES-Merging and TALL Masks from the “Model Merging” subsection to the paragraph dedicated to post-hoc methods in the “Task Arithmetic” subsection. Additionally, we reworked the paragraph dedicated to fine-tuning strategies for task arithmetic (LL118-121) **[qd2R]**. We improved the caption of Figure 1 to be more clear and specific about the chosen pruning criterion and sparsity ratio **[qd2R, S7QQ, 51Ub]**.

- **Sensitivity analysis & overlap of low-gradient parameters [S7QQ, YVQa]**.
We extended the experiment of Figure 1 by providing further evidence of the stability and overlap of low-gradient parameters across tasks in Appendix A.7.

- **Combining TaLoS with other merging schemes [YVQa].**
We dedicated Appendix A.8 to the results when using other schemes for selecting optimal merging coefficients. We choose to test both TIES-Merging and AdaMerging, as suggested by the Reviewer.

For easier reference, we highlighted new or strongly altered sections in a different color in the revised version of our manuscript.

**References**

[1] Ortiz-Jimenez et al "Task arithmetic in the tangent space: Improved editing of pre-trained models" NeurIPS 2023
</content>
</thread_object>

</thread>
<thread>
creation date:
1739176315692

thread_type:
public_comment

<thread_object>
creator: ~leonardo_iurada1

<content>
title:
Acknowledgements

comment:
We would like to extend our sincere thanks to the Program Committee, the Reviewers, and the Area Chair for their careful evaluation and constructive feedback throughout the review process. We deeply appreciate the time and effort invested in helping us improve our work.

We also wish to address the additional comment provided by the Area Chair regarding the tone of some of our responses. We apologize if any of our replies were perceived as aggressive. We have streamlined our communications to convey our points as clearly and efficiently as possible but our intention was never to be confrontational. We fully recognize the importance of maintaining a respectful and collegial tone and will certainly take this feedback to heart in all our future interactions.

Thank you once again for your invaluable contributions, which have significantly enriched our work.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents  a method for improving task arithmetic by selectively updating a sparse subset of parameters during fine-tuning, thereby achieving weight disentanglement and minimizing task interference. The approach falls in the class of methods that requires a modification of the fine-tuning procedure. The results show a promising comparisons to some post-hoc and full fine-tuning methods obtaining strong performance across a variety of benchmarks. The paper had some issues with clarity and inclusion of important details, most of them addressed in the rebuttal. The authors emphasized that their work focused on task arithmetic thus comparisons to model merging methods were not appropriate, some reviewers disagreed. Indeed these two areas have a blurry border, the authors have now included experiments comparing to AdaMerging and Ties Merging combined with their method and in isolation. A lingering concern was the tradeoff in single task performance. Overall the strengths of the paper exceed the weakness thus acceptance is recommended.