Title: Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: task arithmetic, model editing, task vector

Abstract: Task Arithmetic (TA) provides a modular and scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors, but existing approaches typically require external task data, which conflicts with TA’s modularity and availability constraints like privacy concerns. We propose a data-free approach by framing representation drift regularization as a curvature matrix approximation problem. This allows us leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature (KFAC) to obtain practical regularizers. Our method is data-free, has constant complexity with respect to the number of tasks, and improves performance on TA benchmarks.

TL;DR: We tackle cross-task interference in Task Arithmetic with a data-free approach. By viewing drift as curvature approximation and applying K-FAC, we achieve scalable task disentanglement and improved benchmark performance.