ICLR 2025

Published as a conference paper at ICLR 2025
MASTERING TASK ARITHMETIC: τJP AS
A KEY INDICATOR FOR WEIGHT DISENTANGLEMENT
Kotaro Yoshida1,∗,†, Yuji Naraki2, Takafumi Horie3,∗, Ryosuke Yamaki 3,4,
Ryotaro Shimizu5,6, Yuki Saito5, Julian McAuley6, Hiroki Naganuma 7,8,†
1 Institute of Science Tokyo, 2 Independent Researcher, 3 Ritsumeikan University, 4 ProPlace,
5 ZOZO Research, 6 University of California San Diego, 7 Universit´e de Montr´eal, 8 Mila
ABSTRACT
Model-editing techniques using task arithmetic have rapidly gained attention.
Through task arithmetic, simply through arithmetic operations on the weights of
pre-trained and fine-tuned models create desired models, such as multi-task mod-
els, models in which specific tasks are unsolvable, or domain-transferred models.
However, task arithmetic faces challenges, such as poor reproducibility and the
high cost associated with adjusting coefficients in the arithmetic operations on
model parameters, which have limited its practical success. In this paper, we
present three key contributions in the context of task addition and task negation
within task arithmetic. First, we propose a new metric called τJp which is based
on the product of the task vector (τ) and the Jacobian of the pre-trained model with
respect to its weights. We show that τJp has a causal relationship with the inter-
ference that occurs from arithmetic operations. Second, we show that introducing
regularization to minimize τJp significantly mitigates interference between task
inference, which leads to the elimination of coefficient tuning and improved accu-
racy on each task. Third, in the context of incremental learning, we demonstrate
that our τJp regularization achieves more robust performance in environments
where access to future tasks is unavailable, thus validating the scalability of the
approach. Finally, we demonstrate that the τJp regularizer further reinforces the
performance of task arithmetic by leveraging publicly available fine-tuned mod-...