Mastering Task Arithmetic: $\tau$Jp as a Key Indicator for Weight Disentanglement
Authors: Kotaro Yoshida
Yuji Naraki
Takafumi Horie
Ryosuke Yamaki
Ryotaro Shimizu
Yuki Saito
Julian McAuley
Hiroki Naganuma
Venue: ICLR 2025 Poster (2025)
Keywords: task arithmetic
model editing
task vector
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Our proposed $\tau$Jp regularizer improve the performance of task arithmetic and lead to its practical applications.

Abstract: Model-editing techniques using task arithmetic have rapidly gained attention.
Through task arithmetic, simply through arithmetic operations on the weights of pre-trained and fine-tuned models create desired models, such as multi-task models, models in which specific tasks are unsolvable, or domain-transferred models.
However, task arithmetic faces challenges, such as poor reproducibility and the high cost associated with adjusting coefficients in the arithmetic operations on model parameters, which have limited its practical success. In this paper, we present three key contributions in the context of task addition and task negation within task arithmetic. First, we propose a new metric called $\tau$Jp which is based on the product of the task vector ($\tau$) and the Jacobian of the pre-trained model with respect to its weights. We show that $\tau$Jp has a causal relationship with the interference that occurs from arithmetic operations. Second, we show that introducing regularization to minimize $\tau$Jp significantly mitigates interference between task inference, which leads to the elimination of coefficient tuning and improved accuracy on each task.
Third, in the context of incremental learning, we demonstrate that our $\tau$Jp regularization achieves more robust performance in environments where access to future tasks is unavailable, thus validating the scalability of the approach.
Finally, we demonstrate that the $\tau$Jp regularizer further reinforces the performance of task arithmetic by leveraging publicly available fine-tuned models, offering practical benefits for real-world applications.
Our code is available at https://github.com/katoro8989/tau-Jp_Task_Arithmetic

Published as a conference paper at ICLR 2025
MASTERING TASK ARITHMETIC: TJP AS A KEY INDICATOR FOR WEIGHT DISENTANGLEMENT
Kotaro Yoshida 1e.1, Yuji Naraki Takafumi Horie Ryosuke Yamaki 3,4
Ryotaro Shimizu Yuki Saito Julian McAuley", Hiroki Naganuma 7,8
Institute of Science Tokyo, Independent Researcher, Ritsumeikan University, ProPlace, ZOZO Research, University of California San Diego, Université de Montréal, Mila
ABSTRACT
Model-editing techniques using task arithmetic have rapidly gained attention. Through task arithmetic, simply through arithmetic operations on the weights of pre-trained and fine-tuned models create desired models, such as multi-task mod- els, models in which specific tasks are unsolvable, or transferrdd models. However, task arithmetic faces challenges, such poor reproducibility and the high cost associated with adjusting coefficients in the arithmetic operations on model parameters, which have limited its practical success. In this paper, we present three key contributions in the context of task addition and task negation within task arithmetic. First, we propose new metric called rJp which is based on the product of the task vector (T) and the Jacobian of the pre-trained model with respect its weights. We show that rJp has causal relationship with the inter- ference that occurs from arithmetic operations. Second, we show that introducing regularization minimize 7Jp significantly mitigates interference between task inference, which leads to the climination of coefficient tuning and improved accu- racy on each task. Third, in the context of incremental learning, we demonstrate that our rJp regularization achieves more robust performance in environments where access future tasks is unavailable, thus validating the scalability of the approach. Finally, we demonstrate that the rJp regularizer further reinforces the performance of task arithmetic by leveraging publicly available fine-tuned mod- els, offering practical benefits for real-world applications. Our code is available at /ettss//qqite wwt.://www.c.....ooooeeeeeo/e/ee
Taskk Arithmetin
INTRODUCTION
While there is growing demand for foundational models recent machine learning trends, the high computational costs associated with their training (Zhou al.) 2023 Kaplan et al. 2020 Villalobos al. 2022) remain a significant barrier to broader practical use. To address this, model-editing tech-
niques using task arithmetic (Ilharco al. 2023) have rapidly gained attention in the fields of deep
learning (Yadav et al.) 2023 Davari & Belilovsky 2023] Yu et al. 2024| Tang et al.) 2023b Ortiz-
Timenez al.] 2023). Task arithmetic offers a significant advantage
traditional approaches
by enabling the efficient creation of edited models without the need for additional training, simply through arithmetic operations on the weights of pre-trained and fine-tuned models. Specifically, task arithmetic enables three operations: the creation of single model capable of handling multiple tasks (task addition), model that selectively reduces the performance for a specific task (task negation), and a model capable of handling tasks not explicitly included in the training data (task analogies). These are realized by basic operations such as scalar multiplication, addition, and subtraction.
However, task arithmetic faces challenges, such as low reproducibility and the high cost associated with adjusting coefficients the arithmetic operations on model parameters, which have limited its practical success (see Table and Table In addition, there is still limited theoretical understand- ing of why and how these techniques work (Ortiz-Jimenes et al. 2023). Ortiz-Jimenez et al. (2023)
Work was performed when K. Yoshida and T.Horie were ProPlace interns Cooresponding Authors: yoshi da. 0253@m. sct ac. jp, naganuma hi roki @mi queeeec

---

Published as a conference paper at ICLR 2025
demonstrated in their experimental setup for task addition and task negation that the degree of inter- ference between task inference can be quantified using a metric called the weight disentanglement error. They also observed that linearizing the model by the neural tangent kernel (NTK) (Jacot et al.) 2018) approxiatto reduced the weight disentanglement error. However, while their study provides important insights into the conditions for successful task arithmetic, its scope is limited to indirect explanations and approaches to improvement.
Ensuring high reproducibility and minimizing computational costs while avoiding task interference is essential for the practical application of task arithmetic. To address these challenges, we shed light on the product of task vectors and the Jacobian matrix of the model function with respect to its parameters. In particular, we investigate the relationship between this product and weight disentanglement, drawing insights from the NTK regime and model linearization (Jacot al. 2018 Ortiz-Jimenez et al. (.03). We introduce a novel metric, rJp, and theoretically demonstrate that it has causal link to weight disentanglement. Based on this insight, we introduce the regularization to minimize rJp and acquire task vectors with small interference between tasks. Moreover, we demonstrate the effectiveness of the rJp regularizer in scenarios where future tasks to be learned remain unknown inaccessible. This is critical requirement for scaling task arithmetic to more complex and realistic environments. We further explore improving task arithmetic performance by applying rJp regularization the continual training of existing fine-tuned models. Our results show that this approach is effective even for publicly available fine-tuned models, providing practical advantages for real-world applications.
In this paper, we present three key contributions in the context of task addition and task negation within task arithmetic.
We propose a new metric, rJp (r-Jacobian product), which can be shown to have a causal relationship with weight disentanglement. We show that rJp tends to be inversely corre- lated with normalized accuracy, i.c., the metric of performance variation from accuracy before task arithmetic (Section
By introducing regularization during fine-tuning to minimize rJp, we significantly reduce the interference between task predictions, thereby greatly reducing the need for coefficient adjustments (Section 4.] and Section 4.2)
We demonstrate that the regularization of TJp is effective in two practical scenarios: i) when future tasks to be learned are unknown, or ii) when using publicly available fine-tuned mod- els. Our regularization method demonstrates both scalability and practical applicability. (Section 4.3)
We believe that these contributions will facilitate the practical application of model-editing tech- niques using task arithmetic.
2 BACKGROUND
Notation. Let E IR' represent the weights of a neural network f ;
X - y, where X C I2d and y C Re are the input and output spaces with dimensionalities d and respectively. The parameter 0 has dimensionality p. representing the total number of model parameters. Additionally, let dg represent the pre-trained weights and 8* represent the fine-tuned weights. Let T denote the set of all possible task indices. Define T C T as the subset of task indices used. For each task t E T, the
corresponding dataset D; { Vt, (..)),,, [Pel
|it is defined, where 2t, E X and yt, E Y. For task t, IPPd L(f(x,, 0), v..). starting from D..]
fine-tuning is conducted by minimizing the loss function 1 0o. and yielding the fine-tuned weights 07.
2.1 MODEL EDITING VIA TASK ARITIMETIC
Task arithmetic (Ilharco et al. 2023) represents the difference between the weights of a fine-tuned model and those of pre-trained model specifically, 7 8* fly ing arithmetic operations, such as the addition or subtraction of multiple task vectors, and adding the result to the pre- pre-trained weights 8, the model can be effectively edited. Two key methods lever-
task vector. By perform-
By task arithmetic,, we refer to the definition provided in Equation (1) of Oriz--Jnnnez et al. 2023). This definition does not account for task analogies, which were proposed as the third approach in Iharco al. 2023). Therefore, we excluded task analogies in this work.
2

---

Published as a conference paper at ICLR 2025
aging task vectors for model editing have gained recognition in the field. Task addition creates multi-task model by summing task vectors obtained from various tasks and then adding this sum to the weights of pre-trained model. Task negation suppresses erases the abilities and properties only learned from a specific task by subtracting the corresponding task vector from the pre-trained model's ss weights. For instance, it can be used to forget toxic behaviors biases learned during training.
2.2 WEIGHT DISENTANGLEMENT
Ortiz-limenez et 2023) introduced the concept weight disentanglement measure the de- gree of interference between task vectors in task arithmetic. Weight disentanglement is satisfied if the following condition holds:
x; 00
Σ CiTT 1ET
E1;; = - E D.) + (o) x
UDD IET
(I)
RET
The above equation implies that when performing task arithmetic using all task vectors within T the model will produce the same output as when using only the task vector тı for given task t, and for tasks outside of I' the model will produce the same output as the pre-trained model. To assess weight disentanglement between two tasks, weight disentanglement error was proposed.
& (2)
Σ Bz-m (dist (f (x; aaT)), (z; 0 a1 (272))]
where dist(:;.)) measures the distance between two models' vector outputs. For classification tasks, checks whether the predicted labels from the two models, ú1 and f2, match, i.e., dist (2) 1 (ύ) + (02). This етror captures the difference in output distributions when task vectors are ap- plied individually or jointly to a pre-trained model, reflecting interference between task vectors in function space. Ideally, in task arithmetic, each task vector would independently influence the model's 's output, resulting in the error being small.
2.3 NEURAL TANGENT KERNEL
The Neural Tangent Kernel (NTK) (Jacot al. 2018) is kernel that linearizes the learning dynam- ics of infinite-width neural networks. In infinite- width networks, parameter updates during training become infinitesimally small, which allows the following first-order Taylor approximation hold:
f(z: 0) f fx; Bu) + (0 0u) Vo 0p).
This approximation valid in a regime commonly referred to as the NTK regime, or tangent space, where the relationship between the parameter space and function space becomes linearized. Recent studies have observed that fine-tuning large pre-trained neural networks often operates within the NTK regime, the parameter changes during fine-tuning remain sufficiently small (Malladi al., 2023 Ren et al.) (023). In contrast, it has also been reported that in practice, the fine-tuning of finite-width models does always result in perfectly linear behavior, and fine-tuning can exhibit non-linear characteristics (Ortiz-Jimenee et al. 2023).
2.4 TASK ARITHMETIC IN THE NTK REGIME
In task arithmetic, linear operations in the weight space of neural networks translate directly to changes in the function space, which can be explained by the following NTK approximation:
f(x; 0o + aaTT f(2;00)
) (Vf((z;)))
For all E T, denotes the task vector for task defined as 0 0. and a, E R. In simple terms, in the NTK regime, the linearity of operations on task vectors is preserved in the model's output, leading to linear effects performance.
In practice, it has been reported that explicitly enforcing fine-tuning within the NTK regime im- proves task arithmetic (Ortiz-Jimenez al. 2023 Tang al. 2023b) Ortiz-Jimenez, et al. (023))

---

Published as a conference paper at ICLR 2025
Tang et al. (2023b) demonstrated that fine-tuning within the NTK regime lowers weight disentangle- ment error and improves the performance of task addition and negation. One linearization method proposed by Ortiz-Jimenee et al. (2023) is to fine-tune lincarized models fin (in(2, f) within their NTK regime when creating task vectors, which is formulated as follows:
fin 0) f(z 0o) T Vef(x,00)
(5)
However, it remains unclear why linearizing the model suppresses weight disentanglement error and how this enhancement, in turn, improves task arithmetic. These questions have not yet been fully addressed from a theoretical standpoint. We focus on the term T (af 8b) in the NTK approximation and aim to provide theoretical explanation. Building this theoretical foundation, we propose a novel method to enhance task arithmetic.
3 CAUSAL IMPACT OF THE T-JACOBIAN PRODUCT ON WEIGHT DISENTANGLEMENT
We theoretically explain weight disentanglement in the NTK regime and propose the 7-Jacobian product as the underlying mechanism that drives weight disentanglement. We also experimentally validate the relationship between the r-Jacobian product and model interference.
3.1 WEIGHT DISENTANGLEMENT IN THE NTK REGIME
In this section, we provide a theoretical explanation of the relationship between weight disentan- glement and the task vector Jacobian product in the NTK regime. For simplicity, consider task arithmetic involving two tasks, A and B. the NTK regime, the model's output can be approximated follows:
fax 0o *AAAA + (BTB) R f(x, + OAT Vof(x, + HBTT Vof Bo)
(6)
with CA E R. In this case, for inputs IA and FB from tasks and B, achiev- ing weight disentanglement error of 0 in Eq. 2 is equivalent
satisfying the condi- tions f (εA;00 OATA + (BTR) = I [x 0o + QATA) and / (I + QATA + + = (xB; Ou (BTB) for any d and og, which leads to Eq. 77 below.
00 + + R f(zA, 0o) + a Vel 0o) +0 f(zA, aATA),
(7)
f(z,, 0g + aATA + QBTB) f(EB, 0o) 0 agTB Vo F (Z 0o) ~ f(I,, 0o + aBTB).
Eq. 7 implies that the weight disentanglement error is when the task vectors satisfy the following conditions:
T Vof((E 0o) 0,
TB (o = 0.
These conditions imply that the task vector for given task is orthogonal to the Jacobian of the pre- trained model, with respect to its parameters oy. on the other task. In other words, linearizing the model alone does not guarantee weight disentanglement; it is also necessary satisfy the conditions in Eq. (8) as demonstrated theoretically.
We propose the following r-Jacobian product (rJp) as measure of how well the condition in Eq. (8) satisfied between two tasks:
Jp Vol 00)1 + Ve/
The TJp the mean squared norm the product between task vector and the gradient of the pre-trained model with respect to its weights on the other dataset, averaged across both datasets. According to the condition Eq. smaller rJp desirable.
3.2 RELATIONSHIP BETWEEN 7 JACOBIAN PRODUCT AND INTERFERENCE
As demonstrated in Section 3.1 smaller TJp improves weight disentanglement and reduces inter- ference between task vectors. In this section, we experimentally show that minimizing rJp effec- tively mitigates task vector interference.
4

---

Published as a conference paper at ICLR 2025
Cars-MNIST
DTD-GTSRB
EuroSAT-RESISC45
[dlp:
Weight Disee Pileernaang Maannaanngggn Nement Error
Linear PT
1.4-
10.
10-
07
-10
9
3.0
-3.0
-1.0
0.0
1.0
3.0
-3.0
-3.0
-1.0
0.5
1.0
3.0
-3.0
-3.0
-1.
0.5
1.6
300
0.0
o1
q1
q1
Itlp: 0.7)
[t)p: 1.3)
(tjp: 1.71
3.0
3.0-
3.0
(Weigh maanglement Error
Ours
3.0
0.0
0.0
-10
3..0
-3.0 0.0 1.0
3.0 -3.0
Figure 1: Visualization of weight disentanglement ViT-B-32 with respect to rJp. The upper row illustrates the linearized model without regularization, while the lower presents the model with our proposed regularization. Overall, it is observed that when rJp is large, weight disentanglement becomes sensitive to the coefficients. As 7Jp increases, weight disentanglement shows greater bustness to variations in the coefficients. Furthermore, our proposed regularization enhances this robustness with respect the coefficients. The red cross the center represents the pre-trained model, and the red box indicates the typical coefficient search range in task arithmetic.
In the experiments, linearized fine-tuning (FT) Ortiz-Jimenez al. 2023) of different pre- traitrained Vision Transformers (ViT's (Dosovitskiy et al. 2021) under the same conditions as in [lharco et al.
2022 Ortiz-limenez et al. (023) was conducted with CLIP (Radford et al.] 2021) eight image
lasks. Specifically, the eight tasks are Cars (Krause et al.) (2013). DTD (Cimpoi et al. 2014). Eu-
roSAT (Helber et al., 2019), GTSRB (Stallkamp et (201)), MNIST (LeCun 1998). RESISC45
Cheng et al. 2017). SUN397 (Xiao et 2016). and S SVHN (Netzer et al. 2011)
First, we investigated the relationship between rJp and weight disentaagleement Figure visualizes weight disentanglement alongside rJp. In the top row showing Linear FT, we can see that when rJp large, the blue area becomes more prominent, indicating that the weight disentanglement error is more sensitive to each coefficient and interference is not being prevented. As rJp decreases, the error tends become more robust to changes in the coefficients.
Next, we focus on the actual performance of task arithmetic. We analyzed the correlation between normalized accuracy and rJp. presenting the resulting scatter plot in Figure 2 Each data point rep- resents a model trained by performing task addition accuracy is defined as the accuracy of each task after applying task arithmetic relative to its accuracy before task arithmetic, which is set to 1.0. Across all model scales, observed consistent trend where task pairs with smaller rJp values tend to exhibit higher normalized accuracy.
two out of the eight image tasks. Normalized
4 ENHANCING TASK ARITHMETIC BY MITIGATING INTERFERENCE BETWEEN TASKS
4.1 7-JACOBIAN PRODUCT FOR REGULARIZATION
As demonstrated in Section to prevent interference between task vectors in task arithmetic and to improve performance, necessary not only to linearize the model but also keep rJp small simultaneously. Building on these theoretical and empirical insights, we propose a novel method to enhance task arithmetic. Specifically, introduce regularization during fine-tuning that encourages
5

---

Published as a conference paper at ICLR 2025
VIT-B-32
VIT8--16
VIT-L-14
Nommlized Accuracy
Normalized Accuracy
1.000
Normalized minalized.o Accuracy
1.00
1.00
0.99
0.99
0.98
0.98
0.96
0.97
0.98
0.96
0.97
0.94
2.5
7.5
10.0
0.95
2.5
5.0
7.5
10.0
10
15
20
T-Jacobian product
T-jacobian product
T-Jacoblan product
Correlation Coefficient: -0.800
Correlation Coefficient: -0.841
Correlation Coefficient: -0,606
p-value: 0,000
p-value: 0.000
p-value: 0,000
Ours
Linear FT
Figure 2: Visualization of the relationship between rJp and normalized accuracy. Each point rep- resents a pair of tasks from the set of eight tasks, yiclding (:) combinations, i.c., 28 in total. We observed correlation, where smaller 7Jp values associated with higher normalized accuracy. The blue dots represent the results from traditional linearized task addition, while the orange stars denote the results using task vectors obtained through our proposed regularization. A significant difference in rJp values between the two approaches is evident, indicating that our proposed regu- larization reduces 7Jp and improves task addition performance.
JJp to be small - that is, we promote learning to occur in a subspace where is orthogonal to the Jacobian of the pre-trained model, with respect to fgg for different tasks.
Inspired by this requirement, we propose the following rJp-based regularized loss function:
Ixsp([in(1;;0)))) 0), T.fSim (x: 0),
Σ 11(0 - 0o) 0o) V.VVolin (Zt, 0o)11
(10)
where Toch denotes the set containing indices of other tasks for which we aim to suppress interfer- ence. Our objective to ensure that A* Og is orthogonal is, we add the 1.2 norm of their product as regularization term. The hyperparameter À adjusts the strength of the regularization. It is important to note that, for the computation the regularization terms, only the input data for each sample in Torth is required, and labels are not necessary.
Vof(x,, fo) for Tonh : that
However, in practical applications, when there are numerous tasks Tonh where interference needs to be reduced, calculating penalties for all tasks each iteration results in significant memory and computational overhead. To address this, we propose the following more efficient implementation:
L(fin (z; 0). v) Ali(O To
Vooimmaa ( mod (Tml)) 00)
(11)
where i denotes the iteration number, and at each iteration, the task for which the penalty is cal- culated is rotated within Tom (specifically, (i mod [Toma] )). With this approach, it is sufficient to calculate the penalty for one task per iteration, ensuring scalability with respect to the size of Tonh-
In Appendix C we conducted comparison between the loss functions in Eq. (TO) and Eq. (11) using ViT-B-32. Although the latter exhibited slightly lower capacity to reduce interference be- tween task vectors, it significantly improved computational efficiency. Moreover, the performance difference was not statistically significant. Therefore, for the remainder of the experiments, we will adopt LTp in Eq.
4.2 ENHANCEMENT THROUGH - 7-JACOBIAN PRODUCT REGULARIZATION
Settings. We conducted experiments to compare linearized fine-tuning with the regularization in Lysp- standard fine-tuning (Non-lin. FT), and fine-tuning with only lincarization (Linear FT), as well recent task arithmetic methods such as Ties-Merging (Yadav et al. 2023) and AdaMerging (Yang) ct al. 2024). in both task addition and negation scenarios.
For vision tasks, the experimental setup for task addition followed the methodology described in Section 3.2 In task negation, we introduced a control task, ImageNet (Deng et al. 2009). to maintain performance during negation.
For NLP tasks, we followed experimental configurations consistent with Ilharco et al. 2023 Task addition experiments used four selected tasks (MRPC, RTE, CoLA, SST-2) from the GLUE

---

Published as a conference paper at ICLR 2025
Table 1: Results of task addition using the eight tasks presented in Section 3.2 In the "Task vector coef." column, the method of determining the task vector coefficients is presented. "1.0" indicates that all coefficients were fixed at 1.0, without any coefficient adjustment.
Our method demonstrates significant performance improvements, particularly in reducing tuning costs by eliminating the need for extensive coefficient adjustments.
Method
Task vector coef.
ViT-B-32
ViT-B-16
WT---14 14
Abs. (T)
Norm. (1)
Abs. (1)
Norn. (1)
Abs. (1)
Norm, (1)
Pre-trained
47.3
54.5
65.1
Indivisual
89.9
92.2
93.7
MTI.
87.8
90.8
92.6
Non-lin. FT
Ilharco et al.
2023
19.9
20.5
19.1
19.7
37.6
39.0
Grid-searched
70.4
78.0
75.5
81.5
84.0
89.3
Linear FT Oriz Jimenez et al.
2023
1.0 Grid-searched
55.4
61.7
58.2
63.6
80.5
86.7
74.3
85.0
78.7
87.6
85.8
92.8
1.0
74.2
84.8
78.6
87.6
85.0
91.9
Ties Merging
Yadav al
2023
searched
74.2
78.6
85.0
AdaMerging
Yang et al.
2024
Traine
80.1
88.5
84,9
92.1
90.8
96.4
1.0
84.2
97.2
87.5
98.4
90.8
99.0
Ours
Grid-scarched
84.5
87.6
90.8
We report the minimum accuracy on the target tasks while maintaining 95% of the pretrained model's accuracy on control tasks (note: results in (-) are reference values where control task performance did not exceed 95% the pretrained model's accuracy). The results show that our method achieves better forgetting of target tasks while preserving higher performance on control tasks compared to existing methods.
Table 2: Results of task negation using the eight tasks presented in Section 3.2
Method
Task vector coef.
VIT-B-32
VIT-B-16
VIT-L-14
Targ.
Cont. (?)
Targ.
Cont. (1)
Targ.
Cont. (f)
Pre-truined
47.3
66.7
54.5
69.3
65.1
77.3
1.0
(10.9)
(44.7)
(10.8)
(51.6)
(15.2)
(68.6)
Non-lin. [Tharco al. 2023
Grid-scarched
24.0
60.7
20.3
64.7
18.4
72.4
Linear
Ortiz Ortzz-imenez
2023
1.0 Grid-searched
(6.3)
(57.2)
(5.4)
(62.2)
(3.0)
(67.9)
11.8
60.6
8.8
65.0
8,3
72.2
Ties-Merging Yadav et al. 2023
1.0 Grid-searched
21.8
61.6
24.3
67.0
26.6
74.4
21.8
61.7
24.3
67.0
26.6
74.4
1.0
11.8
62.5
11.8
67.8
15.1
75.1
Ours
Grid Grid-searched
6.7
73.0
benchmark (Wang al. 2019), while task negation focused on mitigating model toxicity in text generation. Specifically. we extracted instances with toxicity scores above 0.8 from Civil Com- ments (Borkan al.] 2019), performed causal language modeling on this data obtain task vectors, and subtracted these vectors from the pre-trained model. Text toxicity was measured using Detox- ify (Hanu & Unitary team 2020), with perplexity on WikiText- 103 (Merity et al., (016) used as control metric. The models used were T5-small (Raffel et al. 2023 for task addition and GPT-2 small (Radford et al. (219)) for task negation.
Further details the fine-tuning settings can be found in Appendix
Results on Vision Tasks. Table [ shows that our method consistently outperforms existing ap- proaches and achieves notable improvements in both average absolute (Abs.) and normalized (Norm.) performance, regardless whether task vector coefficients are grid-searched. Even with- out coefficient adjustment, approach performs better than prior methods while reducing the cost of tuning the inference-tine time hyperparameter. Notably, for ViT-L-144, our method yields the same results with and without coefficient adjustment, indicating that a; 1.0 is optimal, and achieve normalized accuracy of 99%. This shows that performance is barely degraded by the addition of task vectors.
Next, examining the task negation results presented in Table we observe that although our method without task vector coefficient adjustment does achieve sufficient forgetting the target task
In hardware environment, the memory capacity insufficient, report the results presented by Yang (2024). The experiments conducted under the same experimental settings, except for the hardware conditions.
'AdaMerging trains the coefficients of task vectors trainable parameters using general optimizer (e.g..
Adam).

---

Published as a conference paper at ICLR 2025
Table The results of task addition using T5-small on four GLUE tasks (MRPC, RTE. CoLA, SST- 2) are shown, with task vector coefficients grid-searched for all methods. Our proposed approach consistently outperforms existing methods across all tasks.
Method
SST-2
Avg.
Abs.
Norm. Abs.
Norm.
Abs. (t) Norm. Abs.
Norm.
Abs.
Norm.
Pre-trained
31,8
5.0
7.
32.3
19.1
Indivisual
93.5
93.7
76.8
94.6
89.7
Non-lin. FT [lbrrco et al. 2023
76.7
82.0
78.1
83.3
75.8
98.7
66.0
09.8
74.2
83.5
Jruzz
89.5
89.0
73.0
Ties-Merging
Voday
95.4
60.1
70.5
80.6
Ours
79.1
87.5
82.8
90.6
76.5
99.6
92.5
98.5
82.7
94.0
Table 4: The results task negation for mitigating toxicity in text generation using GPT-2 are pre- sented. Task vector coefficients were grid-searched, and the largest coefficient that kept perplexity within 0.5 of the pre-trained model's value on WikiText- 103 was selected. Our method successfully reduces toxicity, measured by two toxicity metrics, while preserving the general linguistic capa- bilities of the pre-trained model.
Method
Toxic generation (L)
Average toxic score
WikiText- perplexity (1)
Pre-trained
1.3
0.03
29.4
Non-lin, FT (lharco el ul.
2023
1.1
0.02
29.7
Lineur FT OrtzzJimenez
2023
0.9
0.02
29.6
Ties-Merging Yadav et al.
2023
2023
1.0
0.02
29.6
Ours
0.4
0.01
29.9
(Targ.), it significantly outperforms existing methods in preserving the performance on the control tasks (Cont.). Conversely, with coefficient adjustment, our method greatly enhances the forgetting of the target task, while still surpassing existing methods in preserving control task performance across all cases.
We clarify why the method without coefficient adjustment (with a 1.0) was effective for task addition but not for task negation. As detailed in Appendix [A] in the ideal case where rJp is zero (no interference), the optimal coefficient ay for task addition is 1.0. Conversely, for task negation, the optimal a should be infinitely large in this ideal scenario. However, in realistic situations where rJp not and interference exists, there is no well-defined theoretical optimal coefficient for task negation. This makes the fixed coefficient method with a; = 1.0 insufficient to induce adequate forgetting, necessitating coefficient adjustment.
Finally, to verify whether our regularization effectively improves weight disentanglement, we present the lower row of Figure [ Compared to the upper row, which shows the linearized model without regularization, it is evident that weight disentanglement is significantly enhanced, indicating that sensitivity to coefficients has been mitigated.
Results on NLP Tasks. The results of task addition on the GLUE benchmark are shown in Table where task vector coefficients were grid-searched. Our proposed method consistently outperforms other approaches across all tasks, with superior average performance. Notably, for the SST-2 task, performance degrades significantly (normalized accuracy: 61.3) without regularization, likely due to interference from the CoLA task vector, as both are single-sentence tasks. Applying our proposed regularization substantially mitigates this issue, achieving a normalized accuracy 98.5.
The results of task negation to mitigate toxicity in text generation are presented in Table with task vector coefficients also grar-scar Our method achieves the greatest reduction in toxicity while maintaining perplexity within 0.5 points of the pre-trained model.
Results using fixed coefficients of 1.0 without adjustment are presented in Appendix E.6
4.3 SCALABLE REGULARIZATION IN PRACTICAL APPLICATIONS
First, in situations where tasks are introduced incrementally, similar to incremental learning, we demonstrate that comparable performance can be achieved by applying regularization exclusively to previously learned tasks(Section 4.3.1. Then, we demonstrate that simply adding few additional steps of regularization-basel training to existing linearized task vectors yields significant improve- ments (Section 4.3.2).
8

---

Published as a conference paper at ICLR 2025
4.3.1 INCREMENTAL ADDITION
Table 5: Comparison of original regularization and the incremental regularization in task addition on ViT-B-32
In practical applications, scalability to new tasks is critical. Here, we consider scenario of incremental task addition within the previously discussed eight-task task addition framework. Specifically, when training on a task / E T future tasks are not taken into account, and reg- ularization is applied only with respect to past tasks, i..,, aTrth (1,2..... -t))...,.111)
Method
Abs. (1)
Norm. (f)
No reg. (Linear FT) Incremental reg. (Ours)
74.3
85.0
83.6
96.5
Full reg. (Ours)
84.5
97.6
Table 5 presents a comparison task addition on ViT-B-32 using three approaches: applying regu- larization all tasks (Full reg.), applying regularization incrementally (Incremental reg.), and Linear FT (No reg.). The results show that applying regularization to all tasks leads to the highest perfor- mance and helps to prevent task interference, consistent with theoretical expectations. However, the incremental regularization approach also demonstrates substantial improvement over the existing unregularized method, indicating that our approach is highly scalable new tasks.
4.3.2 PENALIZATION ON EXISTING TASK VECTOR
We also examine the effect of applying our regularization-baser learning in addition to the task vectors already created by other users, as shown in Figure The horizontal axis rep- resents the number of steps in the additional training, starting from the initial point, which is
Normalized Accuracy
99.5
99.0
the task vector obtained via Linear FT. The left vertical axis (blue) shows the normalized accu- racy during task addition, while the right verti- cal axis (red) represents rJp. It can be observed that both metrics improve sharply within the first 100 steps, with normalized accuracy ex- ceeding 99%. Subsequently, the improvement more gradual. This indicates that even when linearized task vector already exists, small amount of additional training with our regular- ization can significantly enhance performance.
96.5
98.0
97.5
200
900
Steps from Linear FT
Figure 3: Regularization-Bee based additional training for task addition between EuroSAT and SVHN, using ViT-B-32, where interference was particu- larly severe.
RELATED WORK
The attempt to merge and average the parameters of multiple neural networks originates from the work of Utans (1996). In recent years, various methods have been proposed for large-scale neural networks with numerous parameters, aimed at manipulating their properties or enhancing perfor- mance through addition and subtraction in the parameter space. For example, by merging language model specialized in medical knowledge with one specialized legal knowledge, would pos- sible to develop model capable of solving tasks related medical litigation. Among the various methods for realizing the integration of models and their knowledge, many are related to task arith- metic, including task analogies, as well model merging. One of the simplest approaches model merging involves taking the parameters of multiple models fine-tuned from the same pre-trained model and computing their simple average (Wortsman et al. 2022a Choshen al.] (022). ). Building on this, various extensions have been proposed. For instance, Don- Yehiya et al. (2023) presents
framework for the distributed fine-tuning and fusion of multiple models. [Ramé et al. (2023) adopts a strategy where the same pre-trained model is fune-t using diverse auxiliary tasks, and the param- eters of these fine-tuned models are fused. This approach aims to maximize the diversity of model parameters and thereby improve generalization performance. Volicoeur-Martines poses a method in which model merging is performed periodically during the tine-tuning process to ensure that the parameters of individual models do not deviate too far from the population mean. Muqeeth et al. (2023) introduces a technique in the context of Mixture-of -f-eerert- (MoE), where a merged expert is created by computing the weighted average of parameters across multiple expert
et al. (2024) pro-

---

Published as a conference paper at ICLR 2025
networks. Other approaches such as linearly interpolating between the pre- pre-trained model and the fine-tuned model, rather than merging parameters of fine-tuned models, have also been explored (II-
harco al. 2022 Wortsman al. 2022b).
On the other hand, in the integration of models via model merging or task arithmetic, interference among the parameters of multiple models task vectors can arise, and various methods have been proposed to mitigate such conflicts. For instance, several methodologies utilize masking operations
on task vectors (Tang al. 2023a Wang et al. 20224] Huang al.] (022), while others involve
trimming or scaling techniques (Ydday et al. 2023 Davari & Belilovsky 2023 Yu et al. 2024),
or leverage model linearization (Tang et al. 2023b Ortiz-Jimenez et al., 2023) Additionally, in
incremental learning (Wang et al.] (2024b). Huang et al. (2021) and Waan et al. (2023) introduced
regularization techniques aimed at minimizing task interference during the training multiple tasks
on the same neural network. These methods ensure that the subspaces in the parameter space asso-
ciated with each task remain orthogonal and disentangled.
Theoretical and analytical studies on the effectiveness of model merging and task arithmetic include research based on analyses of the loss landscape (Entezari et al. [2022] Qin et al. [022 Gueta et al.) 2023), as exemplified by linear mode connectivity (Frankle et al. 2020),, as well as approaches that leverage model lincarization within the Neural Tangent Kernel (NTK) regime (Jacot et al.) 2018). These studies have demonstrated that during the integration of multiple neural networks via model merging, techniques such as parameter permutation to align different models within the same basin in the loss landscape (Ainsworth et al. 2022) or inducing weight disentanglement between task vectors through lincarization (Ortiz-imenezz et al. 2023) can be effective.
Our proposed method addresses key limitations in existing model integration and task arithmetic techniques, specifically task interference and the high cost of coefficient tuning. We introduce the rJp metric (the r-Jacobian product), which quantifies weight disentanglement, showing an in- verse correlation with task interference. This metric provides a novel approach to reducing inter- ference, distinct from conventional masking or trimming techniques. Additionally, by minimizing rJp through regularization during fine-tuning, we significantly reduce the need for costly coeffi cient adjustments. Our method effective even in practical scenarios, such as when future tasks are unknown or when using publicly available fine-tuned models, thereby enhancing scalability and broadening real-world applicability.
6 LIMITATIONS
Our experiments are based on the linear approximation, assuming learning occurs in the NTK regime. As noted by (Ortiz-Jimenez al. (2023), this linear approximation increases the computa- tional time for forward calculations by two to three times compared to that of a mon-linearizer model. The regularization proposed this study is based on such linearized models, and this aspect has not been improved. However, linearization methods leveraging parameter-efficient approaches, such as LoRA (Hu al. 2022), have also been proposed (Tang et al., (023b). Combining these methods with our regularization has the potential to reduce computational costs while enabling more effi- cient and effective task arithmetic. Our contribution lies in elucidating the internal structure of task arithmetic using 7Jp and confirming the sufficient effectiveness of our regularization under precise lincarization. Validation on larger models (e.g.. LLMs) using approximate linearization methods, such those mentioned above, is left as future work.
CONCLUSION
In this paper, we proposed a novel metric, rJp, to better understand weight disentanglement in task arithmetic and demonstrated its inverse correlation with normalized accuracy. By incorporating regularization to minimize rJp during fine-tuning, we significantly reduced task interference, mini- mizing the need for coefficient adjustments in task addition and negation. In incremental learning, we found that our 7Jp regularization method shows strong performance in situations where future tasks to be learned are unknown accessible, confirming the scalability of the approach. Further- more, the rJp regularizer improves the performance of task arithmetic by utilizing publicly avail- able fine-tuned models, which makes it beneficial for practical in real-world scenarios. These findings contribute to advancing the practical application of model- -editigg techniques through task arithmetic.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Our deepest gratitude goes out to the anonymous reviewers whose invaluable insights substantially enhanced the quality of this manuscript. We sincerely thank Yoshikazu Ikeda (ProPlace Inc, Osaka University) for his invaluable assistance in setting up the infrastructure, which greatly contributed to the success of this research. This research was supported by the GCP Startups Booster Program and Microsoft for Startups. Hiroki Naganuma also sakknwwledges funding support from the ANRI Fellowship for this work. Ryosuke Yamaki also acknowledges funding support from JST SPRING (Grant Number JPMJSP2101) and JST ACT-X (Grant Number JPMJAX24CS).
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprini arXiv:2209. 04836, 2022.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 105(10):1865-1883, 2017.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, arXiv preprint arXiv:2204.03000 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3606-3613, 2014.
MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Sealing multi-task model merging with sparse masks. arXiv preprint arXiv:2312.06695, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-244225 248-255. Ieee, 2009.
Shachar Don Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, and Leshem Choshen. ColD fusion: Collaborative descent for distributed multitask finetuning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 788-806, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: /oopeneeveww wwtp::nettwttneeee FNNNTT
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur: The role of permutation in- variance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022. URL https: hwtpp//pppe.eeeeeeeeeppeee net htte///oummmiiiooe -it.diiytdddddddllllll
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In Proceedings of the International Conference Machine Learning, pp. 2269, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation, 2015. URL http:://aasii..og////aaaa//
7495
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowledge is region in weight space for fine-tuned language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023. URL https:
//openreview.rr net. www.:etrtt//eettee wtwprummmmmiiqqiee d
11

---

Published as a conference paper at ICLR 2025
Laura Hanu and Unitary team. Detoxify. Github. https///gthbb.ccmmmmiiaaaaaaaadd
2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217 000:2221222222 2226, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https::/opperevv//
net forum?
id=nzevKeeFYf9
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning--ree pirh-peeforrance model merging. arXiv preprint arXiv:2405. 17461, 2024.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cot- terell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 2736-2746, Online, June 2021. Association for Computational Linguis- tics. doi: 10. 10.18653/v1/2022 mawcl-main.n11111 218. URL
naacl-ma in.ain.. 218
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzbak Gadre, Shuran Song. Hannanch Hajishirzi, Si- mon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpo- lating weights. Advances in Neural Information Processing Systems, 35:29262-293 222277, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Alexia Colicoeur-Martineal Emy Gervais, Kilian FATRAS, Yan Zhang, and Simon Lacoste-Julien. Population parameter averaging (PAPA). Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL http::ppp///peeeeeee....... net fttt://oormmmdddd////////oo
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL
arxiv. hbtgg//b//200000 08361
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13); 3521-3526, March 2017. ISSN 1091-6490. doi: 10.1073/pnas. 161 1835114. URL http:
7/dx. doi. org.orr//1e 11.0g//10...00 1073 pnas. 1611835114
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554 561, 2013.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In Proceedings of the International Conference on Machine Learning, pp. 23610-23641, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 1609.08843 2016.
Mohammed Muqeeth, Haokum Liu, and Colin Raffel. Soft merging of experts with adaptive routing. arXiv preprint arXiv:2306.03745 2023.
12

---

Published as a conference paper at ICLR 2025
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre- pra-tee models. Advances in Neural Information Processing Sys- tems, 2023. URL https:
htttlttts///ppeeeeee///e https////peeneeeeeeeooom net. Eorrrorunnnn forrmm i 0A9E2 J2DGW
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23), UIST '23, New York, NY, USA, 2023. Association for Computing Machinery.
Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Exploring mode connectivity for pre-trained language models. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 6726- 6746, 2022. doi: 10.18653/v1/2022.ooo eea...emmlllmaaa 4511 URL https///aaaannooooy.ooooo 2022. ewn pammain. 451
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL
10683
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In Proceed- ings of the International Conference on Machine Learning, pp. 28656-28679, 2023.
Yi Ren, Shangmin Guo, Wonho Bae, and Danica Sutherland. How to prepare your task head for finetuning. arXiv preprint arXiv:2302.05779 2023.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In The 2011 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, and Dacheng Tao. Concrete subspace learning based interference elimination for multi-task model fusion. arXiv preprint 1rXi0:2312.06173, 2023a.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial lincarization. arXiv preprint arXiv:2310.04442, 2023b.
Joachim Utans. Weight averaging for neural networks and local resampling schemes. Proceedings of the AAAI Workshop on Integrating Multiple Learned Models, pp. 133-138, 1996.
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho, and Marius Hobbhahn. Machine learning model sizes and the parameter gap. 2022. URL ptttps://arxiv...
abs/2207 02852
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi- task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL http:::pps hattps://openzeviee
het//orrum:
13

---

Published as a conference paper at ICLR 2025
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. Proceedings of the International Conference Machine Learning, 2024a.
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. JEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-533 5383, 2024b. doi: 10.1 11000TTAAM22222.333722222
Xiao Wang. Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310. 14152, 2023.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the International Conference Machine Learning. volume 162, pp.
PMLR, 2022a. URL https: cettps://proceediinn
mlr. wwwpppeessssvee. wwort sman22a html
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, pp. 7959-7971 2022b.
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119: 3-22, 2016
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES Her-mng Re- solving interference when merging models. Advances in Neural Information Processing Systems, 2023. URL /URp::ttts:///peeeeeee
net/ Xtt
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang. and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024. httptttpss///ppeeeeeeeee
net/ forum?
id=nžP DNND3QX
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Proceedings of the International Conference on Machine Learning, 2024.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang. Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. A comprehensive survey on pretrained foundation models: A history from bert chatgpt, 2023. URLbttps://aat http::::////iiii//// wwtp:/wwwwrrraaggooooooo wwwpg///a/aa/aeeee 2b020000.2230291919
14

---

Published as a conference paper at ICLR 2025
A SCALING COEFFICIENTS FOR TASK VECTORS IN THE NTK REGIME
We provide theoretical insights into the coefficients applied to task vectors in task arithmetic, em- ploying the NTK regime.
A.I THEORETICAL INSIGHTS
First, we present the following two important theorems preliminary step.
Theorem 1. In the weight space R2, let 8, E IR denote the initial point and 0 E IP the fine-tuned point. For any scalar IR, define point the straight line passing through Og and 0
0(a) = (1 11l-a)0 a300 + at"
Under the NTK regime, the model's output can be approximated as:
f(x; (())) = (1 - (2; 0o) + (x;
In other words, linear interpolation in the weight space corresponds to linear interpolation of the outputs in the function space.
Proof. Noting Eq. 3 and that f(x;0") x f(z;00) + f2; based on it, we obtain the following:
f(;; ((a))(a))) = [(x; a)00 a00)
f (2; a 0o))
[(x;00 + aT)
[(=00)) + +ar Vol(2;00))
f(;;0)) ((:))))))((
- ax; (a(((0))
Theorem 2. Consider convex loss function (x;f():)) with respect to the model output fxr 0). Then, in the NTK regime, the loss function ((f (a;)) is convex with respect to
Proof. According to Theorem 1, in the NTK regime, f(x;0(a)) is a linear interpolation between fa; Bo) and fa; 8²), such that
((a)) = (1-a))(z;;0) 0n) + (=;
Since the loss function L convex with respect to the model output, the composite function L[f(x; (()))) is convex with respect sition of convex function with a linear function is convex.
R. This follows from the property that the compo-
Specifically, because L is convex and f(;; ((a)) is linear function of a, the function L(f(x; is convex with respect to a.
Finally, based on Theorem 2, we derive the following explanations for the coefficients in task addi- tion and negation of the linearized model.
Task addition. If the 4* obtained through fine-tuning for each task optimal (i.e., minimizes the loss), then, according to Theorem 2, the loss is minimized at 1.0. Furthermore, if all rJp are zero, setting the task-specific coefficients on = 02
or 1.0 enables complete task addition without any performance degradation for each task.
Task negation. the 0* obtained through fine-tuning for a particular task is optimal (i.e., mini- mizes the loss), then the loss decreases monotonically in the direction from 0g towards 0* along T. Conversely, moving in the direction of - leads to an increase loss (i.e., forgetting occurs). This is because the loss is convex with respect a. Therefore, in this case, optimal coefficients cannot theoretically be obtained, and as long as the NTK regime holds, increasing a indefinitely in the negative direction results in greater forgetting.
15

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
All our experiments using CLIP were conducted on four NVIDIA V100 GPUs, each with 16GB of memory.
B.I FINE-TUNING DETAILS
Vision Tasks. The fine-tuning process for each task was primarily based the implementations of Ilbarco et al.] (2022); Ortiz-Jimenez et al. (2023). Specifically, for all tasks, we set the number of steps to 2000, the batch size to 128 (with gradient accumulation for the ViT-L--14 model), and used the AdamW W optimizer with a learning rate of le-5, weight decay of 0.1, and a learning rate schedule based on cosine annealing, incorporating 200 warm-up steps. As noted by [harco et al. (0tz))(Ortzzllimeeee et al. (2023), freezing the text encoder during the fine-tuning of CLIP does not significantly impact final performance, so we adopted a fixed classification head by using the output of the pre-trained text encoder on class-specific text prompts (e.g.. "a photo of {classname] "), while fine-tuning only the image encoder. For the fine-tuming of the linearized model, we followed the exact implementation outlined in OrtiizJimenez et al. (202)). In our proposed method, during training for each task within the eight tasks, 7 orth consisted all tasks except the target task, as well as ImageNet. The same task vectors were used for evaluation in both task addition and negation.
NLP Tasks. For task addition in the GLUE benchmark, we adopted a configuration similar to that used for vision tasks, with the key difference being the use of T5-small as the model. For task negation, we followed the setup described in Ilharco et al.) (2023), using GPT-2 small Fine-tuning on Civil Comments involved causal language modeling with learning rate of 1 1e-5, batch size of 16, and training for epochs.
The computation of the r-Jacobian product for the proposed regularization efficiently performed using Jacobian-vector products, as in Ortiz-Jimenez et al. (2023). To reduce computational costs, we used a batch size that was 5 of the batch size used for computing the loss on the target task for vision tasks, and 4 for NLP tasks. The strength of the regularization term, represented by the hyperparameter X, was tuned using grid search over the range [le-3, le-2, le-1]. Validation accuracy was used as the evaluation metric. Due to limited computational resources, the value obtained from specific task (Image: Cars, NLP: CoLA) was reused for other tasks. Despite this simplification, the proposed regularization consistently achieved strong performance across various tasks. This suggests that empirically, the proposed regularization is not highly sensitive to the choice of X.
B.2 TASK VECTOR COEFFICIENTS
In all experiments where task vector coefficients were determined via grid search, the coefficients aT. For task
were unified across all task vectors. Specifically, in Eq. (4) we set a11 a2 addition, the grid search range was set to a E {0.0, 0.05, 1.0], and for task negation, the range was a E f0.0, ...0...........
As demonstrated in Appendix A under the NTK regime, theoretically, if there is no interference be- tween task vectors. an optimal coefficient of 1.0 should be achieved for task addition. However, for task negation, the NTK approximation theoretically allows a to grow arbitrarily large within the valid approximation range. Therefore, we adopted broader search range for task negation com- pared to previous approaches.
For coefficient selection, in task addition for both vision and NLP tasks, we chose the coefficient that yielded the highest normalized accuracy the validation split. For vision task negation, we selected the coefficient that achieved the lowest accuracy on the target task while maintaining at least 95% of the pre-trained model's accuracy the control task (ImageNet) validation split. In NLP task negation, we selected the largest coefficient that kept the perplexity on WikiText- 103 within 0.5 of the pre-trained model's perplexity.
16

---

Published as a conference paper at ICLR 2025
B.3 EVALUATION DETAILS
In the NLP task negation experiments, toxicity was measured using Detoxify (Hanu & Unitary team 2020 Following the methodology llharco al. (2023), 1000 text samples were generated with the prefix "I don't care if this is controversial," and Detoxify was used to compute the toxicity scores for these samples. The average of these scores reported as the "Average toxicity score" Additionally, the proportion of samples with a toxicity score of 0.5 or higher was calculated and reported as the "Toxic generation rate"
C COMPARISON OF STRICT REGULARIZATION AND CYCLICAL REGULARIZATION
Applying our proposed regularization strictly penalize every iteration, as in (TO), is computa- tionally and memory expensive. Therefore, as shown in (II), we propose more effcient approach by penalizing each task cyclically. Here, we compare the performance and computational cost of this efficient regularization with the original strict regularization, demonstrating that its practical use justified.
Tablee[ the results of a comparison between the two approaches and Linear FT (No reg.) in the context of task addition using ViT-B-32. The evaluation metrics include absolute accuracy, nor- malized accuracy, and the actual time taken per iteration. From the perspective of accuracy, the strict regularization (Strict reg.) slightly outperforms the efficient implementation (Cyclical reg.). indicat- ing that the strict implementation of proposed regularization can nearly eliminate interference. On the other hand, while the efficient implementation performs slightly worse in terms of accuracy, the difference is not significant. Notably, in terms of actual computation time, achieves around 80% reduction. Furthermore, despite having a runtime comparable to Linear FT, it demonstrates a significant improvement performance.
Based on the above observations, the approximate regularization in provides faster and suffi- ciently effective regularization.
Table Comparison of the strict regularization and the efficient regularization task addition on ViT-B-32
Method
Abs. (t)
Norm. (1)
Sec. / Ittr.())
No reg, (Linear FT)
74.3
85.0
0.361
Cyclical reg. 11]
84.5
97.6
0.374
Strict reg. 110
86.4
99.3
2.027
D TASK ARITHMETIC AND MULTI TASK LEARNING
Multi-Task Learning (MTL) Caruana 1997) involves training single model simultancously us- ing data from multiple tasks. When sufficient input data and labels are available for each target task, leveraging them concurrently enables the construction of unified model capable of handling multiple tasks effectively.
However, if even one of the tasks has limited access to sufficient data or lacks labels, achieving this single training process becomes challenging. Additionally, adding new capabilities pre-trained model while maintaining its performance on other tasks (Kirkpatrick et al.) 2017). or forgetting toxic abilities, is not straightforward task.
In contrast, task arithmetic offers high practicality, flexibility, and scalability. Firstly, in practical applications, task arithmetic does not require complete access to all task data simultaneously during training. Instead, it allows for learning in environments where only partial access data is available, and the weights can be integrated afterward to create a multi-task model. In addition, since each task has its independent weight (task vector), task arithmetic offers flexibility represent wide variety of models. For example, with task vectors for N tasks, it is possible to represent 2 different models through task vector addition negation. In the context of recent advancements such as
17

---

Published as a conference paper at ICLR 2025
Large Language Model (LLM))based) based chatbots and multi-agent systems (Park et al. 2023) where diverse models are needed to adapt to various situations, the flexibility of task arithmetic is highly significant. Furthermore, discussed in Section 4.3] our method can be easily extended in the context of continual learning while maintaining performance on previous tasks.
Furthermore, Table 7 compares the performance, data requirements, and flexibility of Non-lin. FT, Linear FT, our proposed method, and MTL. Notably, MTL requires access to inputs and labels for all tasks, whereas task arithmetic operates under more restricted assumptions. Unlike conventional task arithmetic approaches (Non-lin. FT and Linear FT), our method requires access to inputs from other tasks. This extension naturally aligns with the context of Unsupervised Domain Adaptation (Ganin & (Lempiisky] 2015), where only the data distribution Fother from other domains is accessible. By leveraging access to this data distribution, our method learns task vectors in orthogonal directions, enhancing the disentanglement of model weights. In contrast to other methods, which cannot utilize the data distribution Fother in such scenarios, we propose an effective learning strategy tailored to this specific context.
Table 7: Comparison of task addition and MTL. On the right side, the table shows the types of data required for training task t T, as well as the flexibility of the model. Here, zeelf represents the input data of the current task, yself represents the labels of the current task, and Tother and Yother refer to data from tasks other than t. In MTL, training requires data that includes labels from all tasks, whereas task addition can be applied in more relaxed scenarios where MTL is not feasible. Flexibility indicates whether the model's performance can be easily modified for specific tasks after training. On the left side, the table shows the accuracy across eight tasks for each model scale, demonstrating that task addition using our regularization achieves performance comparable to MTL, even in more relaxed scenarios.
Method
ViT-B-32
ViT-B-16
ViT-L-14
Fzelf
Yself
Tother
Yother
Flexibility
Non-lin. FT
70.4
75.5
84.0
/
X
X
Linear FT
74.3
78.7
85.5
/
/
X
X
/
Ours
84.5
87.6
90.8
/
/
V
X
/
MTL
87.8
90.8
92.6
V
f
V
X
E ADDITIONAL RESULTS
Here, we present more detailed experimental results related to the analysis in the main text.
E.1 SINGLE TASK ACCURACY ON EACH TASK
Figure presents the accuracy for each task obtained with the three FT methods described in Sec- tion
along with the pre-trained model.
As noted by (Ortiz-Jiment et al. (202)), Non-linear FT outperforms the linearized FT methods (Linear FT and Ours) due the non- mon-linear advantage.
Notably, despite applying regularization in our proposed method, which constrains learning to subspace orthogonal to Vof By). E Tonh there no degradation in performance compared to the original Linear FT. This demonstrates that our method successfully prevents task interference while maintaining performance by guiding learning in a space that mitigates inter-task interference.
E.2 EFFECT TASK ADDITION EACH TASK
In Figure 5 we present the absolute and normalized accuracies for each task after task addition, com- paring different methods. The right-hand plots of normalized accuracy demonstrate that our method not only achieves the highest accuracy across most tasks but also maintains consistent performance across all tasks, indicating that task- inkiindeeendent regularization is effectively achieved. Moreover, in the left-hand plots of absolute accuracy, method outperforms existing methods all tasks cept for AU (Helber et al. 2019). These results suggest that our method successfully prevents interference between tasks while preserving absolute performance.
18

---

Published as a conference paper at ICLR 2025
VIT-b-32 (single task)
VIT-b-16 (single task)
1.0
1.0
Absolute Accuracy
Absolute Accuracy
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
5VHN
SUN397
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SVHN
SUN397
VIT--114 (single task)
1.0
Absolute Accuracy
0.8
0.6
0.4
0.2
0.0
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SVHN
SUN397
Non-linear FT
Linear FT
Ours
Pre-trained
Figure 4: The absolute accuracy after fine-tuning for each of the eight tasks, comparing Non-linear FT (blue), Linear FT (orange), Ours (red), and the pre-trained model (green).
E.3 TJP ON EACH TASK PAIR
Figure illustrates the rJp between task pairs for both Linear FT and Ours. These results demon- strate that our proposed regularization reduces the rJp between tasks. Compared Linear FT, method shows a notably lower rJp in the off-diagonal components of the heatmap, i.e., between different datasets. These results suggest that our proposed method effectively decreases the 7Jp values.
E.4 RELATIONSHIP BETWEEN JJP AND COSINE SIMILARITY OF TASK VECTORS
Interpreting similarity interference between tasks in terms of cosine similarity of their task vectors has been a common practice (Ilharco et al. 2023 Wang et al. (023). However, explanations for the interpretations remain limited and it is unclear whether cosine similarity fully accounts for those relationships between tasks. In this section, we attempt to analyze the relationship between task interference and task vector similarity through the lens of rJp.
We consider two tasks, A and B. Assuming that the fine-tuning of task B is conducted with single update using the entire dataset, тв can be expressed as follows:
TB = Vo
aLn
Vof(IB, 0n)
(12)
0f(xB, 00)
19

---

Published as a conference paper at ICLR 2025
Table The results of task addition using T5-small on four GLUE tasks (MRPC, RTE, CoLA, SST- 2) are shown, with task vector coefficients 1.0. Our proposed approach consistently outperforms existing methods across all tasks.
Method
MRPC
SST-2
Avg.
Abs.
Norm.
Abs.
Norm.
Abs.
Norm.
Abs.
Norm.
Abs.
Pre-trained
31,8
5.0
7.3
3233
19.1
Non lin. FT
Burrco et al. 2023
70.3
91.6
72.2
77.2
90.2
96.3
57.1
60.4
72.5
61.4
Linear FT
Oriz Jimenez d 2023
74.7
97.4
79.1
89.8
81.7
89.4
56.2
59.8
72.9
$4.1
Tees-Merging
Yaday
89.7
Ours
Table 9: The results of task negation for mitigating toxicity in text generation using GPT-2 are presented. Task vector coefficients were fixed 1.0. Our method effectively reduces toxicity while maintaining the perplexity of the pre-trained model, whereas other methods result in a significant increase in perplexity.
Method
Toxie generation rate (U)
Average toxic score (L)
WikiTTex- 103 perplexity (1)
Pre-trained
1.3
0.03
29.4
Non-lin, [Ilharco
2023
0.01
95.7
Linear FT Ortiz-Jimenzz al 2023
0.0
0.00
66.7
Ties-Merging Yadav et al. 2023
0.6
0.02
87.7
Ours
0.5
0.01
30.7
The first equation above derived from the fact that the loss function becomes convex with respect to the weights in the NTK regime (Theorem 2 in Appendix A From this expression, the cosine similarity can be rewritten as:
T TB
cos COS,
[TAl ITT
θL.
|Ta| Vef 0o)
(13)
TAl
Of 0n)
In the above, T (or( 0p) component of rJp, which, as we have demonstrated, explicitly affects task interference (or weight disentanglement) in the model. Although π (z (, 0n) is included in the cosine similarity, based the equation, the presence of other components also affects their relationship, making it difficult to claim a theoretical correlation between them.
Figure 7 shows the cosine similarity between task pairs for both Linear FT and Ours. In Linear FT, the cosine similarity between MNIST (LeCun 1998) and SVHN (Netzer et al. 201T) ) is particularly high, whereas in Ours, the values are much smaller and comparable to those of other task pairs. On the other hand, the cosine similarities between Cars and SVHN in Linear FT is higher than the ones in Ours. Therefore, no consistent trend was observed between cosine similarity and rJp values.
In Figure g we present a scatter plot with rJp on the horizontal axis and the cosine similarity between the two task vectors the vertical axis. Weak positive correlations were observed between these values in the three model sizes. In particular, since cosine similarity tends to be small value when the number of dimension large, the correlation considered weak in the setting of ViT-L-14.
Based on this analysis, cosine similarity appears to be less effective in representing weight disentan- glement compared to rJp. This implies that rJp regularization performs better than cosine similarity for reducing task interference.
E.5 WEIGHT DISENTANGLEMENT AND GENERALIZARIILTY ON UNSEEN TASKS
In the context of MTL, generalizability to unseen tasks is often topic of significant discussion. In this study, examine how improved weight disentanglement impacts the generalization perfor- mance on unseen tasks.
We first conduct theoretical analysis. Based the weight disentanglement defined in Eq. we argue that for unseen tasks those not corresponding to the task vectors used during training -the task vectors should not influence the model's outputs. As a result, the pre-trained model's
20

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of task addition on unseen tasks using ViT-B-32.
Method
Training Tasks Avg. (f)
SVHN (1)
RESISC45 (1)
Pre-trained
48.8
31.6
60.2
Non-lin. FT
73.4
50.2
52.2
Linear FT
77.4
38.7
46.6
AdaMerging
80.3
60.9
50.2
MTL
86.3
60.8
42.9
Ours
85.4
42.4
54.3
performance should be preserved, but no further performance improvement is expected, indicating that weight disentanglement does not contribute directly to generalization performance such cases.
To validate this hypothesis, conducted task addition experiments summarized in Table T0] The experimental setup follows the protocol outlined in Yang et al. (2024), where the training tasks consist of six datasets: [Cars, GTSRB, DTD, EuroSAT, MNIST, SUN397], while the unseen tasks are RESISC45 and SVHN.
For the unseen tasks:
SVHN is 10-class digit classification task similar to MNIST. Here, knowledge learned from MNIST in the training phase could be effectively transferred improve performance on SVHN. Therefore, weight disentanglement is not expected to generalize well in this case.
RESISC45 is classification task involving aerial imagery, akin to EuroSAT, However, RE- SISC45 includes approximately 35 additional classes not present in EuroSAT 10-class setup. a result, the knowledge gained from EuroSAT alone insufficient to classify most samples in RESISC45. In this scenario, preserving the knowledge from the pre- trained model via weight disentanglement is expected to yield better performance.
The results confirm our thcoretical insights. As shown in Table I0 MTL and AdaMerging, which does not prioritize weight disentanglement and is optimized under the MTL framework, achieves strong generalization performance on SVHN. However, its generalization performance on RE- SISC45 is relatively poor due to its adverse impact on the pre-trained model's knowledge retention. Conversely, our method, which emphasizes weight disentanglement, demonstrates lower general- ization performance on SVHN but successfully maintains the pre-trained model's performance on RESISC45.
In summary, the reduced generalization performance SVHN is an expected and intentional out- come of weight disentanglement, reflecting the method's design focus. Conversely, the superior performance RESISC45 highlights the effectiveness of our approach in retaining pre-trained knowledge for tasks that require it.
E.6 ADDITIONAL RESULTS ON NLP TASKS
Table addition) and Table 9 (task negation) present the results of the NLP experiments where task vector coefficients were not adjusted.
In task addition, our method consistently outperforms other approaches, achieving a normalized accuracy of 94.0% even without coefficient adjustment, effectively preventing interference between task vectors during addition.
In task negation, while Linear FT achieves the greatest reduction in toxicity, it significantly impacts the pre-trained model's perplexity, increasing it by +37.3 points, thereby affecting the original lan- guage capabilities. In contrast, our method limits the perplexity increase to just +1.3 points while sufficiently mitigating toxicity.
21

---

Published as a conference paper at ICLR 2025
ViT-b-32 (addition)
ViT-b-32 (addition)
1.0
1.0
Normalized Accuracy
Absolute Accuracy
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SVHN
SUN397
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SVHN
SUN397
ViT-b-16 (addition)
ViT-b-16 (addition)
1.0
1.0
Normalized Accuracy
Absolute Accuracy
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Cars
DTD
EurosAT
GTSRB
MNIST
RESISC45
SVHN
SUN397
Cars
DTD
EuroSAT
GT5RB
MNIST
RESISC45
SVHN
SUN397
ViT-I-14 (addition)
ViT--114 (addition)
1.0
1.0
Normalized Accuracy
Absolute Accuracy
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SVHN
SUN397
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SVHN
SUN397
Non-linear FT
Linear
Ours
Figure 5: The absolute accuracy (left column) and normalized accuracy (right column) for each of the eight tasks after task addition, comparing Non-linear FT (blue), Linear FT (orange), and Ours (red).
22

---

Published as a conference paper at ICLR 2025
Linear FT (VIT-B-32)
Ours (VIT-B-32)
Cars 9.036
3:420
5.221
5.148 4.134 4.990
4.896
5.716
Cars
7.578 1.46
0.909
0.948
0,721
0.933
0.887
1.096
-30
-0
DTD 5.420
10.128
7.767
5.644 6.366 7.092 6.691
7.064
DTD
1.046
5.406
1.377
1.79
1,006
1.44
1.267
1.356
GTSRB
5.148
5.297
20
GTSRB
20
MNIST 4.134
6.366
7.726
8.396
12.973
5.488
9.692
5.131
MNIST
0.721 1.006
0.869
1.329
12.497
0.833
1.685
0.771
RESISC4S 4.900 7.002
10.056
6.297
5.488
11.385
6.225
6.693
RESISCAS
0.97 1.424 1.694
1.104
0.833
7.734
1.058
1237
9.692
SUN397
Care-
DTD
RUROSAT
GTSRO
MMST-
RESIS44S
SWHN
SUN399
Can--
DTD
RUNIAAT
GTSRD
MNST-
RESISCAS
SWH
SUN22
Linear (VIT-8-16)
Ours (VIT-B-16)
35
35
Cars. 5,760
5.464
Cars
EuroSAT
5.6621
8.155
16.072
10.012
7.642
10.105
9.802
7.294
25
EuroSAT- 0.835 1311
8.616
1491
0.827
1607
1.325
0.970
25
GTSRB 5,225 7.000
10.012
13.790
8.511
6.894
9.564
6.403
GTSRB 0.858
1226
1.491
9.88
1.187
1.85
1.993
0.871
MNIST
4.422
6.105
5.405
-15
MNIST 0.615 0.902
1.427
13
RESISC4S
RESISCAS
-10
10
SWHN 5.324
7.826
9.564
8.980
7.057
SVHN
0,793
1.205 1.325
0.995
11.718
0.874
SUN397 6.084
74488
7,291
5.103
5,281
7.067
6.497
8.639
SUN397
0.999
1270
0.97
2471
0.670
1201
0.874
4,817
Cars-
DTD
EuroSAT
GTSRB
MNIST
RESISCAS
SVHN
SUN297
Cars-
DTD
FurSSAT.
MNSST
RESISCAS
SUHN
Linear FT (ViT-L-14)
Ours (VIT-L-14)
9.715
9.801
6.185
8.459
9.005
9.012
13.007
Cars
30.897
1573
1.596 1.211 1.254 1855 1.451 1.769
30
DTD
9.715
17:11
13.53
10.961
11,079
11.950
11.438
12.149
DTD
1.57 6.633
1.83
1.779
1.430 1.77 3.764
1.779
EuroSAT
9.601
13.5
53,275
1991
15.49
17.907
EuroSAT
1.56
12.235
1873
4.601
2.884
1.600
-20
MNIST
8.499
33.689
10:365
-15
MNIST
1.430
1.85
1.585
1141
15
RESISC45
9.403
11,950
18.494
10,910
10.385
22:311
11.477
11.653
RESISCAS
1.653
1.78
4.601 1.745
1.285
11:408
1.545
1.744
SVHN 9.012
11.438
17.907
17.204
19.543
11.477
26.420
9.741
SVHN
1.431
1764
2.684
4.431
1.585
1545
36.765
1336
SUN397
11.651
cars-
DTD
GTSRD
MNIST-
RESISCES
SWHN
SUN397
care-
DTD
EuroSAT-
G15RD
MNIST-
RESISCES
SWHN-
SUN391
Figure 6: Heatmaps visualizing TJp on each task pair. The darker the color of the cell, the higher the value it represents. The values within cells indicate 7Jp. The figures in the left column show the model with our proposed regularization, while the figures in the right columns show the existing linearized model without regularization. Our proposed regularization results lower rJp between different tasks.
23

---

Published as a conference paper at ICLR 2025
Linear FT (VT-8-32)
Ours (VIT-8-32)
1.0
1.000
0.032
DTD 0.311
1,300
0.243
D.258
0.044
0.044
5.349
0.052
2.8
DTD 0.42
1.000
0.007
D.005
0.011
0,029
0.015
0.025
0.8
EuroSAT- 6,200
0,083
1,000
0.038
0.023
0.945
0.025
6.027
EuroSAT 0.19
0.007
1.009
9.032
0.020
0,006
0.011
0.013
-.66
0.6
1.000
8.033
MNIST- 0.044
0.044
0.059
1,000
0.026
0.040
0.011
1,000
0.033
-.4
0.4
RESISC45 6.224
0,064
0.245
D.030
0.027
1.000
0,208
0.035
RESISC45
0.046
0.029
0.006
0.024
6.032
1,000
0.070
0.040
SWIN 0.222
0.069
0.035
0.070
0.122
0.028
1,000
0.025
SWNN
0.031
0.015
0,011
0.009
0.063
0.030
1,000
0.030
Cars
EuroSAT
GTSRB
MNIST:
RESSC45
SVHN
SUN397
Cars
DTD.
G7SRB
MNIST:
RESISC45
SVHN
SUN397
Linear
(VT-B-16)
Ours
(VIT--16)
1.0
1.0
0.028
0.037
0.030
0.088
1,000
0.065
0.050
9.054
0.0G4
-.8
0.048
1,000
0.021
0.026
3.8
EuroSAT 6.244
0.288
1.000
0.004
0.033
0.041
0.016
0.032
EuroSAT
0.14
0.004
1.000
0.011
0.029
0,041
0.000
0.016
2.6
GTSRB 0.84
1.000
0.080
0.043
GTSRD
0.017
0.011
0.019
0.026
0.025
RESISC4S 6,22T 0.267
0.243
D.035
0.028
1,000
0.230
0.041
RESISC4S
0.003
0,008
0.111
0.038
1,000
0.001
0.002
SVIN 6.27 0.064
0.026
0.082
0.103
6.030
1,000
0.034
SVHN
0.065
0.045
0,010
0.031
0.055
0.001
1.000
0.045
0.034
1.000
Cars
DTD-
EUroSAT:
GTSRB
MNIST:
RESISCAS.
SVHN
SUN397
Cans
DTD
GTSRD
MNIST:
SVHN
SUN397
-0.0
Linear FT (VIT-L-14)
Ours (VIT-L-14)
1.0
1.0
Cars 1.000
0.022
0.188
0.016
0.028
Cars
0.029
0.033
0.029
0.029
0.028
5.8
EuroSAT 6018
0,002
1,000
D.277
0,022
5,011
0.028
EuroSAT
0.33
0.005
1.000
0.033
0,032
0.006
0.030
GTSRB 6,22 0.044
0.227
1.000
0.035
6.023
3.528
0.052
GTSRB
0.299 0.019
0.236
1.000
0.027
0.051
0,028
0.027
MNIST- 0018
1000
0.015
0.021
1000
RESISCAS- 6.020
0.023
0.015
6.033
RESISCAS
0.029
0.022
0.040
1.000
0.020
SVHN 0016
0.027
0.221
0.028
0.045
0.017
1,000
0.021
9.2
SVHN
0.031
0.022
0.026
0.028
0.006
0.036
1.000
0.019
SUN397 0.288 0.048
0.026
0.022
0.024
6.033
0.021
1.000
SUN397
0.026
0.016
0.030
0.027
0.024
0.020
0.019
1.000
Cars
615RB
MNIST
RESISC4S-
5VAN
SUN397
-0.0
Carz
DTD
EuroSAT
GTSRB
MNIST-
SVAN
SUN397
-0.0
Figure 7: Heatmaps visualizing cosine similarity of task vectors each task pair. The darker the
color of the cell, the higher the value it represents. The values within cells indicate cosine similarity. The figures in the left columns show the model with proposed regularization, while the figures the right columns show the existing linearized model without regularization.
24

---

Published as a conference paper at ICLR 2025
VIT-8-32
VIT-8-16
VIT-L-14
0.125
Cosine Similarity
Cosine Similarity
0.100
Cosine Similarity
0.100
0.04
0.075
0.075
0.050
0.03
0.025
0.02
0.025
0.000
0.01
0.000
T-Jacobian product
T-Jacobian product
T-Jacobian product Correlation Coefficienti 0.246 0.067
Correlation Coefficient: 0.502
Correlation Coefficient: 0.555
0.000
p-value: 0.000
Ours
Linear FT
Figure 8: Visualization of the relationship between rJp and cosine similarity. Each point represents a pair tasks from the set of eight tasks, yielding (e) combinations, i.c., 28 in total. The blue dots represent the results from traditional linearized task addition, while the orange stars denote the results using task vectors obtained through our proposed regularization.
25


Reviews:
[{"review_id": "R9h4B2JLpU", "reviewer": "Reviewer_hopm", "summary": "The paper tackles the task of task arithmetics, i.e. how to combine *task vectors/parameters* to form a multi-task model. A key issue is to determine the best combination weights as to minimise interference between tasks, and maximise sharing of information / positive transfer.\nMore specifically, the authors make use of two previously introduced notions: **(i)** the notion of **weight disentanglement** which was  proposed as a measure of task interference in task arithmetic. And **(ii)** the Neural Tangent Kernel (NTK) which designates a training regime where parameter updates can be expressed with a linearised approximation.\nPrevious works have suggested that performing task arithmetics under the NTK regime can lead to better MTL performance. the authors investigate this behaviour in more depth. Based on this analysis, they also propose a regularisation technique to further reduce task interference when performing task arithmetic, which involves slightly fine-tuning the task vectors themselves.", "questions": "* I find the notion of \"One-shot\" and \"fine-tuned\" experimental setting could be improved; First because the notion of fine-tuning can become confusing between the coefficients $\\alpha$ vs the model parameters $\\theta$ fine-tuning. Second, because it is not clear if it is referring to a specific method/objective for fine-tuning the task coefficients (e.g. AdaMerging or others) or simply hyperparameter search.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well motivated and grounded in previous related work\n* The proposed method is simple and could adapt to different task arithmetic variants\n* Interesting insights on the link between the proposed regularisation and weight disentanglement\n* A more efficient implementation of the method is proposed for handling larger number of tasks (Equation 11)", "weaknesses": "* Missing baselines on more recent task arithmetic work: The main tables should include some recent task arithmetic results (e.g. TIES-Merging and AdaMerging) as well as standard single task and MTL baselines (although it is in the appendix), if only to better understand the existing gap in performance.\n* Missing discussion about the extra cost: The paper briefly mentions efficiency of the method (e.g. equation 11 or line 364), however I think this could be discussed in more depth: On the one hand, the proposed method seems more robust to task coefficients $\\alpha$, which could save on hyper parameter tuning; On the other hand, it involves a fine-tuning procedure which requires knowledge/access to all tasks simultaneously (Equation 10) as opposed to directly combining task vectors obtained independently from one another.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HyIMhBTXVO", "reviewer": "Reviewer_onTx", "summary": "This paper proposes a new metric called $\\tau$Jp ($\\tau$-Jacobian product) for measuring weight disentanglement in task arithmetic operations on neural networks. The authors theoretically analyze the relationship between $\\tau$Jp and interference between tasks, and introduce a regularization method based on minimizing $\\tau$Jp during fine-tuning. Experiments on image classification tasks demonstrate improved performance and reduced need for hyperparameter tuning compared to existing methods.", "questions": "- How does the computational cost of the proposed method compare to existing approaches?\n- Can the method be adapted to work with limited or no access to data from other tasks?\n- How well does the approach generalize to other domains beyond image classification?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a comprehensive theoretical and empirical study of the relationship between the proposed $\\tau$Jp metric and task interference in neural networks.\n\n2. The introduction of $\\tau$Jp as a new metric for weight disentanglement is novel and well-motivated.\n\n3. The proposed regularization method eliminates the need for tuning inference-time hyperparameters ($\\alpha$), which is a practical advantage.", "weaknesses": "1. The method requires access to data from all other tasks during training, which is often unavailable in realistic task arithmetic scenarios. This limits the practical applicability of the approach.\n\n2. The computational cost of calculating τJp is likely very high, as it involves multiple Jacobian-vector products. The paper does not report runtime or resource requirements, making it difficult to assess scalability.\n\n3. Experiments are limited to image classification tasks. Evaluation on other domains like language tasks would strengthen the claims of generality.\n\n4. The derivation of Equation 7 from the weight disentanglement definition is non-trivial and should be explained more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fc0LwO1Qz5", "reviewer": "Reviewer_rGLy", "summary": "The paper introduces a novel metric, $\\tau \\text{Jp}$ ($\\tau$-Jacobian product), to improve understanding of weight disentanglement in task arithmetic. It demonstrates that τJp inversely correlates with normalized accuracy, suggesting it as an indicator for weight disentanglement. A regularization technique is proposed to minimize τJp during fine-tuning, effectively reducing the need for coefficient adjustments in task addition and negation. It also proves valuable in incremental learning scenarios where future tasks are unknown.", "questions": "Suggestions：\n\n1.The related work section could be improved by explicitly connecting prior studies to this paper's contributions, emphasizing how the proposed method addresses existing limitations.               \n2.Consider moving the related work section after the methods section, especially since the current structure delays the introduction of the proposed method until page 5. This change would allow readers to quickly understand the proposed approach before diving into comparisons, enhancing readability and engagement.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper addresses an important and timely topic: in an era where foundation models are prevalent, better understanding weight disentanglement is particularly valuable for enhancing the practical applicability of these models.\n\n2.The proposed metric offers a deeper understanding of weight disentanglement, and the regularization method effectively reduces task interference, minimizing the need for coefficient adjustments.\n\n3.The success of the proposed method in incremental learning scenarios aligns well with real-world applications, demonstrating its scalability and practical relevance when future tasks are unknown.", "weaknesses": "1.While the paper introduces the $\\tau \\text{Jp}$ metric and explains its relationship with weight disentanglement, the theoretical justification for why $\\tau \\text{Jp}$ regularization effectively reduces task interference could be further elaborated.\n\n2.The proposed regularization method lacks a comparison with other existing regularization techniques, which makes it difficult to fully assess its relative strengths and weaknesses.         \n\n3.The paper mentions task addition, task negation, and task analogies in the introduction and background sections as key operations in task arithmetic, but there are no experiments evaluating task analogies. This inconsistency weakens the completeness of the experimental validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0SuYXopDIx", "reviewer": "Reviewer_n6ZH", "summary": "The paper presents a novel approach to task arithmetic in neural networks, which leverages a novel metric that quantifies the relationship between task vectors and the Jacobian of pre-trained models. The authors claim that by minimizing this metric through regularization, they can significantly reduce interference between task predictions and enhance the accuracy of task arithmetic operations. The experimental results demonstrate substantial improvements in performance for both task addition and task negation.", "questions": "Could the proposed regularization affect the model's plasticity? Specifically, how might the addition of this regularization impact the fine-tuning performance, potentially influenced by the strength of the regularization?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ This paper is well-written and easy to follow.\n+ The experiments are extensive, and the results sound good.\n+ The design of metric τJp is reasonable and interesting.", "weaknesses": "- The novelty of this paper may be limited. My consideration is that this paper seems to fundamentally align with the approach proposed by Ortiz-Jimenez et al. (2023) [1], which also emphasizes fine-tuning models in the tangent space. Although using the specific regularization term, this paper does not sufficiently differentiate itself from this existing work.\n- While the empirical results are compelling, the paper lacks a thorough theoretical explanation for why the proposed regularization leads to better performance compared to other methods, such as those discussed in Ortiz-Jimenez et al. (2023). I am confused about why a simple and soft regularization results in such improvement compared to [1]. A deeper theoretical analysis could strengthen the paper's contributions.\n- The authors briefly mention tuning the regularization strength but do not provide sufficient details on how this hyperparameter was selected. The sensitive analysis of this hyperparameter is also necessary for the paper.\n\n[1] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)