Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
Authors: Angelo Porrello
Pietro Buzzega
Felix Dangel
Thomas Sommariva
Riccardo Salami
Lorenzo Bonicelli
Simone Calderara
Venue: ICLR 2026 Poster (2026)
Keywords: task arithmetic
model editing
model merging
model reuse
task vector
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We tackle cross-task interference in Task Arithmetic with a dateless approach. By viewing drift as curvature approximation and applying K-FAC, we achieve scalable task disentanglement and improved benchmark performance.

Abstract: Task Arithmetic yields a modular,  scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors; however, existing approaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, eliminating the need for held-out tuning.

Published as a conference paper at ICLR 2026
DATALESS WEIGHT DISENTANGLEMENT IN TASK ARITHMETIC VIA KRONECKER-FACTORED APPROXIMATE CURVATURE
Angelo Porrello
Pietro Buzzega'
Felix Dangel
Thomas Sommariva' Riccardo Salami
Lorenzo Bonicelli'
Simone Calderara'
University of Modena and Reggio Emilia, Italy name surname? unimore..
Vector Institute, Toronto
f feangellvectorisstiitue
ABSTRACT
Task Arithmetic yields modular, scalable way to adapt foundation models. Com- bining multiple task vectors, however, can lead cross-task interference, causing representation drift and degraded performance. Representation drift regulariza- tion provides a natural remedy to disentangle task vectors; however, existing ap- proaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless ap- proach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage estabistall techniques; in particular, adopt Kronecker- -acctored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and nega- tion. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, climinating the need for held-out tuning.
INTRODUCTION
Task arithmetic (TA, Iharco et
promises a scalable approach for adapting foundation mod- els. Indeed, fine-tuning produces spsk-ip parameter updates called task vectors that can be added or subtracted edit model behavior. This enables reuse of task-specific knowledge across do- mains and even backbones (Rinaldi et al. 2025) without retraining. In practice, composing multiple task vectors degrades performance due to cross-task interference: when a new task vector added, modifies shared representations, disrupting those used by other tasks. To prevent such interference, task-specific components must be decoupled to preserve other tasks' representations. This property, whereby distinct directions in parameter space lead to changes confined to con-ovellapping regions of the input space, is called weight disentanglement (Ortiz-Jimenez et al. (023).
2022)
Encouraging weight disentanglement. To promote this property. might regularize the fine- tuning procedure to explicitly preserve other tasks' representations (Yoshida et al. 2025) or, in other words, prevent representation drift i.e., change in a task's activations when new task vectors added. Nonetheless, such regularizers often require access to other tasks' training data, which is impractical under privacy or regulatory constraints and contradicts modularity and reusability.
Therefore, our goal is to design a computationally efficient regularizer for weight disentanglement that be used without requiring access to the training data.
This task relates to approximating neural network function space distances (Dhawan et al. 2023 which measure how much model's behavior changes without requiring access to the original data. Building on this perspective, we incorporate an additional insight specific to TA: fine-tuning the first- order Taylor approximation of the model around its pre-trained parameters empirically enhances weight disentanglement sentation drift simplifies into quadratic form of the network Jacobian's 's Gramian, which can be pre-computed on, and shared instead of, the data to enhance weight disentanglement (Fig. [ How- ever, the Gramian is intractably large, its size grows quadratically with the number of parameters.
et al., 2023). We show that, under linearization, the repre-

---

Published as a conference paper at ICLR 2026
Link to curvature approximation. The Jacobian Gram matrix is instance of the generalized Gauss-Newton (GGN) matrix (Schraudolph] 2003), an extensively studied object in the context of second-order optimization (Martens] (0000/2020) This link allows us to leverage prior research on efficient curvature approximations. Specifi- cally, we adopt fronecker-factored approx-
Cams166 RESISCAS
Cars196 RESISCAS
100
imate curvature (KFAC, |Matenns & Grosse 2015), block-diagonal approximation of the GGN, where blocks correspond to layers and each block is Kronecker product of two small matrices. KFAC drastically reduces storage and computation while still capturing most intra-layer correlations, bridging the gap be- tween oversimplified diagonal approximations and the intractable full GGN of interest.
-0
-1
Figure 1: Weight disentanglement (left) without and (right) with Jacobian Gram regularization.
Adapting KFAC for TA. KFAC-based regularization faces a key limitation when applied to multi- task arithmetic: its associated regularizer cannot be accumulated exactly across tasks. The per-task regularizers induce memory and computational costs that grow linearly in the number of tasks. Going beyond the existing approximation, we propose an aggregation scheme that merges per-task curvature factors into single surrogate, yielding constant complexity in the number of tasks.
We show that linking the weight disentanglement objective eurvature- aware optimization yields state-of-the-are performance in task addition and negation (Ilharco et al. 2022). Furthermore, our method exhibits desirable properties, such task localization i.e., distinct task vectors govern separate, localized regions in function space associated with different tasks and robustness to task vector rescaling, which renders performance insensitive to scaling coefficients and thus eliminates the need for held-out tuning. In summary, our contributions are the following:
We derive regularizer for task arithmetic called TAK (Task Arithmetic with KFAC regular- ization)- - that improves weight disentanglement without using external data.
We scale representation drift regularization by aggregating per-task regularizers into a single surrogate, ensuring constant complexity and storage regardless of the number of tasks.
2 BACKGROUND: TASK ARITHMETIC AND LINEARIZED FINE-TUNING
Setup. Let f : x RP + CC denote a neural network that processes a datum æ E RDD via parameters O E IR. into prediction f(x, 0) RC During training. these predictions are compared to a target y E R via a criterion function c RC x RY + with the goal to minimize the empirical risk over a training data set D (( yn)) mm We start from model pre-trained large source dataset Do. yielding pre-trained weights Og. Our goal is to fine-tune this model on a specific downstream task t with data set De, to obtain the task-specific fine-tuned weights 07.
Task Arithmetic. The above fine-tuning procedure is typically repeated for multiple (I') tasks, yielding task vectors {T. := 0j 0o) Such vectors form the core of TA, which posits that simple linear operations in weight space can induce targeted transformations in function space. This enables combining the capabilities of multiple task vectors build multi-task model without addi- tional training, through simple linear combination (task addition): given the individual task vectors (T.T- the composed model has parameters On + Dt OrT,, with at R (in the simplest case, 0g = 1). TA also addresses the removal of task-specific knowledge (lask negation) by subtracting, rather than adding, a task vector. However, naive linear composition is prone to interference, as overlapping task-vector updates often conflict and degrade the composed model's performance.
Linearized fine-tuning, Ortiz-Jimenez et al. (2023) empirically show that TA benefits from model linearization, particularly when applied during both training and inference. This approach replaces the network with its linear approximation around the pre-trained weights, (f. 0v) <> fun as
(zn 0) = f(x 0b) + Jof (e) 0u),
(1)
with (x, 9b) E CCP the Jacobian of the model's prediction on datum x with respect to its parameters, evaluated at 0g. This encourages weight disentanglement in TA, property whereby task vectors influence the model only on their own tasks, leaving its behavior unchanged elsewhere. Our goal is to construct regularizer encourage this property during linearized fine-tuning.
2

---

Published as a conference paper at ICLR 2026
Algorithm Idealized and practical representa- Algorithm 2 Linearized FT on task vector Tr tion drift regularizer for task t'
Require: Initial weights do. dataset De' task
Require:
Network 00), tasks {D=)== 1=1
vector Ty merged curvature matrix G-r'
I: Linearize the net: (f, 00) + fin( Tp - Oo)
1: Compute per-task GGNs [
(Eq.
(approximate via KFAC, Sec.
3.3
2: while not converged do
3: Draw a mini-batch B ~ De
2: Merge over tasks: G-v =
12 X.
4: Minimize objective Eq. 7 on B w.r.t. TM
(optional: merge KFACs, Eq.
5: end while
3: return Quadratic form: T G_
return Task vector Tr
MAKING REPRESENTATION DRIFT REGULARIZATION DATA-FREE
Simplified setup with two tasks. Model linearization simplifies the learning dynamics, allowing us to analyze how editing affects the model. We conduct this analysis in feature space through the lens of representation drifi, the change in the last-layer activations of task when adding a new task t':
Pre-edit (reresseutation)
Z₁ (xx)
Jiin (ar Oo + axTTT) - (x)) = frin (2x, Go + CuTi + Tx)T)
edit
Post-edit (represcntation
(eoprosentation)
(x)) = (2)) 2xx21)1
(2)
If the drift (x+ vanishes for all x E Dr. the newly added task vector Tr will not interfere as it does not change the model's behavior for inputs from task t. Interference between the two tasks can be reduced by penalizing representation drift (Yoshida et al. (025) via the neural network (Tr)
function space distance (Dhawan et al., 2023) (T+)) == I/P,1 LægD, Ai-r, (az). However, the regularizer for Ty requires accessing data of the external task t. This may violate segregation policies, impose significant storage demands, and prevent independent training, ultimately reducing flexibility for decentralized training. These issues make direct optimization of this objective imprac- tical in many real-wo wallworld settings, such as decentralized (McMahan et al. 2017 Kairouz et al. 2021) or privacy-preservinn preserving learning scenarios (Abadi et al. 2016 Bonawitz et al. 2017).
3.1 CONNECTING REPRESENTATION DRIFT REGULARIZATION TO CURVATURE MATRICES
Now, we reformulate the regularization objective to eliminate its dependence on external task data. Thanks the linearization, the representation drift from Eq. ( simplifies into A
JJJe (asT: (aiTe + arTe))l? a [| fin (xx 0o) Tell2. 11 The associated regularizer
Arritt Li-t (Tr) = T, G. (O)) with (. = DD. ELED, Jos 0o) Jo Jo 0o)
1
(3)
Note that the network Jacobian's Gramian (.,) E PxP after initial pre-computation does not require further data access. This idealized training loop shown in Alg. (black font).
In exchange for eliminating the data dependency, however, we now face the challenge of computing the x P Gramian. This is infeasible even for small neural networks. Thankfully, we can interpret G. as curvature matrix that is well-known in the optimization literature: the generalized Gauss- Newton (GGN) matrix (Schraudolph 2003 Martens 2020). This connection allows us to build on well-established approaches from the optimization literature to efficiently compute structural parametric approximations of G,, ultimately allowing us to make Alg. practical (red font).
3.2 THE GENERALIZED GAUSS-NEWTON (GGN) MATRIX
The GGN is a curvature matrix related the Hessian and arises from partial linearization: The Hessian of a function composition = o f is 226= v2 f), while the GGN is V² fin)- The standard setting the second-order optimization literature sets f to be the neural network, and the criterion function used for training. We now introduce the GGN in this context, showing that the Jacobian Gram matrix from Eq. (3) is an instance of the GGN that results from replacing the training criterion with the squared loss. We can then easily transfer existing GGN approximations.
In the following, suppress in since the Jacohians of and fin coincide 0g.
3

---

Published as a conference paper at ICLR 2026
GGN in the training setting. Consider the neural network f with criterion function (e.g. cross- entropy) and training data D from Sec. 2 For sample n, define fn := f( In)) and On == c( yn)) The example- wise loss is then given by L, Cn o fn. and training minimizes the empirical risk
( = Doj Σ,, (z (T.. yn)) )) = [D) Σn (0) == po) En (.)
(4)
For brevity, we use Cp. to denote the value c. (f.(())))))) and [+]: for slicing (c.g. [a]; is the 1th entry of a). Differentiating the empirical risk twice and applying the chain rule yields the Hessian and its Gauss decomposition (Schraudolph) 2003 Martens 2020), containing the GGN G(Ø):
V2 = G()) R(0)= কoa (Jof.) (Jofk)) + T E. VV (f (5)
For models that are linear in the parameters, the residual R(0) vanishes, it depends on second derivatives, (zero in the linear case). The GGN then coincides with the Hessian of the risk under linearization and, for likelihood-based losses, with the Fisher information matrix (Amari 2000).
The Jacobian's ' Gram matrix as GGN. The GGN in Eq. (5) generalizes the Jacobian Gram matrix from Eq. (3). used for representation drift regularization, by additionally weighting the Jacobians with the criterion function's Hessian & we choose squared error en (f) = 1/2|1 yn 1l' rather than the training criterion, the GGN becomes the Jacobian Gram matrix exactly, since V Ic.
Hence, the matrix G. of the quadratic form in Eq. 3 corresponds to curvature matrix: the GGN of the loss ((0) (Eq. 4 when the training criterion is the squared loss.
While the GGN is impractically large to compute or store for neural networks, the literature has developed scalable structured approximations for it. In the following, we build on these approxima- tions (specifically, KFAC) and study how to adapt and extend them in the context of task arithmetic.
3.3 KRROECKKR-FACTOORK
APPROXIMATION OF THE GENERALIZED
We rely on structured GGN approximation called Kronecker-Factored Approximate Curvature (KFAC) introduced by Martens & Grosse (2015) for culneccon,, then generalized to convo- lutional (Grosse Martens 2016), recurrent (Martens et al. 2018), and transformer architectures (Eschenhagen et al.] 2023). KFAC has been successfully applied to optimization (Osawa et al., 2019), pruning <Wang et al. 2019), Laplace approximations (Daxberger et al., R
Ritter et al,
2018 and influence functions (Grosse et al. 2023). For an in-depth tutorial, see Dangel et al. (2025)
KFAC approximates the GGN as block-diagonal. Each block corresponds to layer, G0) = blockdiag (G( (())) and is further approximated as a Kronecker product, G(O)) s B' @ A'. To evaluate the approxima- tion's 's quadratic form for representation drift regularization, {(B), h from task f, then evaluate (without instantiating the Kronecker product Loan 2000))
Parametric form. For a net with L. layers and parameters 0- fl,....,0
,..,,,,,,,
simply store the Kronecker factors
t (Te) = T, (.o)))T:
KFAC &
af Dt. of (B) A,T
(6)
with r' denoting the part of T corresponding the parameters in layer 7.
KFAC for a single layer. To illustrate the approximation, consider a single fully-connected layer in neural network, with associated weights W Pix>> (we omit biases for simplicity). The layer processes an intermediate input representation a E D2 for datum v. into an intermediate output representation 2/. Wa E R.D. Further, let 0² = vec W RD.D= denote the row-flattened weights. The layer's ss GGN block is G(vec into a sum of Kronecker products by using the chain rule J. Jvec wi fn (J2z, (J) wiz!.) where Juee wiz = ID. @ a (e.g. Dangel et al. 2020) to obtain
1/12) Σn, (Jor f..)TT C2 and simplifies
G(vec W 1 En (J fx) (Jen( @ a' = E [B', & A.
For the last equality, we use E,,["] = 1/10 Σn, on for averaging over the data set. KFAC assumes E, n & +n En nnl & E, [+.], yielding single Kronecker product involving the small factors
A E D2 B E R D to approximate the intractable block G((ee W E R D.xD, D2.
G(vec W
KFAC R
D (JzN Vec @ ( En a. := B' A'.
4

---

Published as a conference paper at ICLR 2026
SVHI
SVHP
RESISCAS
TSRB
RESISCM
CTSRB
Pre-trained
Ding. Linear FT (FAC.
Pre-trained
Only
Linear FT
Non-linear
Only FT KFAC
Figure 2: Impact of regularization on "8 Vision" CLIP ViT-B/16 (abs. accuracy). Left: linearized fine-tuning regime. Right: non-linear regime. See the Appendix for CLIP ViT-B/32 and -L/14.
Variations. KFAC computes two covariances per layer: (i) the imput covariance A' = En Enla, a 9n.
and (ii) the output gradient covariance B' En,m I Ii of (Jug, fn) 8n,m obtained by backpropagating vectors $n,m E IR related to the Hessian 22c, There exist different variations compute B' and since it is priori unclear which approach works best in the context of 'AA we consider two variants that differ in cost (details in (Dangel et al. 2025 (i) Exact (Botev al., 2017) uses C' backpropagations per datum and exactly computes B': (ii) Monte-Carle (MC, Martens & Grosse (013)) randomizes the exact approach and computes an un- biased MC estimate of B' using M bbackppoopggttioss per datum (typically, M 1).
3.4 MULTI-TASK TRAINING PROCEDURE & REGULARIZATION MERGING
Naïve multi-task regularization. While we focused on two tasks, extending multiple tasks introduces new challenges. To promote disentanglement when training the task vector Tv, we pe- nalize representation drift with respect to other tasks o t'. Starting with the standard training loss 0o), y). the overall fine-tuning objective becomes
LDD (Te) = 1/1p, E (m. c o(fim
CD, (T,) Eeger Cdrii (Te) LD, (Te) 3 El Tf
ET (B) & A1) (7)
where 3 and X control the overall and task-specific regularization strengths, respectively. We weight tasks by data set size, A |Pi1/E.yy [Dr). Given pre-computed KFAC of each task * t', this formulation enables regularization without requiring direct access to data sets external tasks.
Accumulated regularizer. A key limitation of the objective in Eq. ? is that we must store the Kronecker factors individually for each task, incurring ( memory and run time cost. To ad- dress this, we build upon the accumulated regularizer (O)) Eeger Ge (0)) for layer and approximate it with a single Kronecker product that captures the contribution of all other tasks:
KFAC 2
Enpe N,
A
merge
(Erge
Bi
& CEegedee
(8)
Empirically, this heuristic (Eq.
8 matches the un-merged formulation's performance (Eq.
4 EXPERIMENTS
Task addition. We evaluate performance on the 8 Vision benchmark (Ilharco et al. 2022 which covers eight classification datasets. Using CLIP (Radford et al.) 2021) as the foundational vision backbone, collect eight checkpoints during training for each method and subsequently merge them into a single unified model. Additional details on training and datasets are provided App. Following the original setup (Ortiz-Jimenez Jimenez et al., 2023). report both absolute and normalized accuracy. We further analyze the role the rescaling coefficient a: (i) setting a, = a for all tasks, corresponding to plain task-vector addition, and (ii) tuning cross-task validation set.
5

---

Published as a conference paper at ICLR 2026
Table I: Task addition results 8 Vision. The "a" column specifies how task vector coefficients are chosen. "1.0" denotes that all coefficients are fixed to 1.0, with no tuning. Numbers marked with for TaLoS [Turada et al. 2025) are taken from the original paper. See Fig. f task-wise plot.
ViT-B/32
ViT-B/16
VIT-I/14
Method
Dataless
Abs.
Norm.
Abs.
Norm.
Abs.
Norm.
Pre-trained
48.4
55.4
65.0
Individual
90.9
92.4
93.8
Linear Fine-Tuning
1.0
76.7
87.2
80.2
88.9
88.0
94.8
Linear FT
Best
78.8
89.9
82.0
90.9
88.0
94.8
1.0
85.0
97.4
88.2
98.3
90.9
98.3
TJp Yoshida et al.
2025
Best
85.6
98.2
88.6
98.7
91.1
98.5
1.0
80.1
92.3
82.9
93.2
87.9
96.3
Diag. GGN (Porrello et al.
2025
V
Best
80.2
92.5
83.0
93.3
88.0
96.4
1.0
85.8
97.6
88.3
97.9
91.6
99.3
TAK, Ours
V
Best
86.0
97.8
88.3
98.1
91.6
99.3
Non-Linear Fine- Fine-Tuning
1.0
32.0
32.9
27.4
28.2
45.3
47.5
Non-linear FT
Best
73.5
80.4
77.0
82.9
84.5
89.7
1.0
22.5
23.3
22.8
23.4
66.2
69.7
Attn. Only FT (Jin
2025
Best
78.2
86.3
80.4
87.1
88.2
93.8
TaLos' (lurada al.
2025
V
Best
79.7
90.8
82.6
92.4
88.3
95.2
Attn. Only FT
1.0
60.3
64.5
59.0
62.3
82.1
87.2
v
TAK, Ours
Best
83.1
91.3
84.3
91.0
89.9
95.9
Comparison with related works. We present a comparative analysis of our regularizer TAK in two distinct regimes. On one hand, we evaluate it in the linearized regime, for which it was originally designed: on the other, we examine whether its benefits also extend to the non-linear regime. If so, this would broaden the applicability of our approach to most state-of of-the-ar learning frameworks.
L Linearized fine-tuning regime. We refer to Fig. 2 (left) for depiction of the per-task absolute ac- curacy of the merged model the linearized regime, while Tab. [ reports the quantitative results on the 8 Vision benchmark. The results indicate that our KFAC-regularized approach yields substantial improvements against the baseline, achieving performance on par with rJp (Yoshida et al. 2025) while avoiding any reliance external data from other tasks. This makes method not only more flexible but also inherently privacy preeoyryrreeeiinn... without sacrificing accuracy. Furthermore, whereas competing methods often require coefficient grid search, TAK proves highly robust: addition of task vectors (a 1) performs competitively, suggesting that post-hoc tuning can be safely omitted. As a side note, the evidence ViT-B/32 suggests that the smaller the model scale, the more crucial curvature regularization becomes for achieving strong final performance.
simple
In this setup, we also compare against an approach inspired by Porrello ture regularization using coarse diagonal approximation of the GGN, While both methods exploit curvature information from the pre-trained model, ours relies on KFAC, providing more accurate estimate that captures intra-layer dependencies. Results show that improved curvature approxima- tions yield larger gains in Task Arithmetic; notably, even diagonal regularization outperforms naive linear fine-tuning, underscoring the role of regularization in enabling weight disentanglement.
(0255) and apply curva-
Non-linear fine-tuning regime. We now consider the non-linear fine-tuning regime (Tab. D and Fig. [ right). In this setting, alternative approaches attempt to approximate linear behavior without fully linearizing the model. For example, TaLoS (Turada et al., (2025) follows a different route and identifies a subset of parameters that consistently exhibit low gradient sensitivity across tasks and updates only these sparse components. This promotes weight disentanglement during fine-tuning while avoiding the computational bottlenecks of full linearization, enabling efficient task addition and negation. Instead, the authors Attention- Only Fine-Tuning Tuning (Jin et al.) (025) fine-tune only the attention layers of Transformers, showing that this strategy implicitly induces kernel- like behavior.

---

Published as a conference paper at ICLR 2026
Table 2: Task negation on 8 Vision. We report the minimum accuracy on target tasks while pre- serving at least 95% of the pretrained model's accuracy on control tasks.
ViT-B/32
VIT-B/16
VIT-L/14
Method
Dataless
Targ.
Cont. f
Targ. +
Cont. f
Targ. +
Cont. 1
Pre-trained
48.4
63.3
55.4
68.3
65.0
75.5
Non-linear FT
20.4
60.5
20.4
65.3
18.1
72.4
Linear FT
9.3
60.5
8.3
65.5
7.5
72.1
TaLos" (lurada et al. 2025
11.0
60.7
10.6
66.1
10.7
73.6
TJp Yoshida et al. 2025
X
6.7
60.8
4.7
66.0
3.7
73.0
TAK. Ours
3.4
62.4
3.4
66.4
3.5
72.6
T5-base (Lincar FT)
TS-base (Non-Linear
Method
Dataless Abs. Norm.
SNLI
ONLI
RTE
SICK
Individual
85.9
MTL.
83.6
Non-lin. FT
75.7
87.7
Sci.
MNLI
Sch.
Linear FT
76.9
92.8
Attn. Only FT
72.9
85.2
TaLos
76.3
93.4
QNLI
SNLI
rJp
X
81.3
100
SICK
TAK, Ours
78.7
98.9
La.. FT w. Day. GN
Nan-in FT
Neu-iia FT (Ati.) w. KFAC
(a) Task addition results for T5-base. All reported scores correspond to the best- performing values; the results obtained with a are provided the appendix.
(b) Impact of training and regularization choices on language (abs. accuracy). Left: linearized regime with no regularization and with the diagonal approximation. Right non-linear regime, with attention-only fine-tuning with and without regularization.
Figure 3: Results for language tasks. Left: impact of different training strategies and sensitivity to hyperparameter. Right: effects of different regularizations on linear and non-linear fine-tuning.
In this regard, although our regularization is not theoretically exact in the non-linear regime, its ap- plicability can still be justified whenever linearized behavior is implicitly enforced. For this reason, in the non-linear setting we pair our regularizer with Attention-Only Fine-Tuning, which has been shown to induce approximately linear fine-tuning dynamics in Transformers, thereby providing a practical way to extend our method beyond the strictly linearized regime. The results in Fig, 2 (right) show that this is the case: when fine-tuning only attention layers, our approach proves bene- ficial even in the non-linear regime. Moreover, in this setting, the choice of the coefficient has stronger impact on the final accuracy. However, TAK remains the most robust on average, a trend further confirmed by experiment reported in one of the subsequent paragraphs.
Unlearning. We herein investigate a setting where each task vector is subtracted from the pre-trained model. doing so, use ImageNet as a control task verify whether subtraction selectively removes the corresponding task without erasing general knowledge. As shown in Tab. 2 our model achieves stronger forgetting of target tasks while better preserving the control task, surpassing that of the main competitor, rJp Yoshidda et al. 2025). Notably, since our regularizer is dataless, it avoids the challenges associated with transterring and storing "large" data set such as ImageNet to perform regularization. This property is especially promising in the context of the massive data sets used today to train conversational models, where the cost of data access and management is critical.
we test across six natural language 2018), SICK (Marelli et al. 2014), (Wang al. 2018). fine-tuning the T5-base model (Raffel et al. 2020). As shown in Fig S TAK consistently outperforms the baselines, particularly under non-linea fine-tuning, thus corroborating the findings observed in vision. How-
Task addition (language tasks) Following Stoica et al. (2025) tasks: SNLI (Bowman et al. 2015), MultiNLI (Williams et al.
SciTail (Khot et al. 2018), 8), RTE (Wang et al. 2018). and QNLI
ever, leveraging data from other tasks (rJp) yields additional gains, suggesting that textual domains may still benefit from even more accurate curvature estimation.

---

Published as a conference paper at ICLR 2026
Absolute Ace. (%)
Normalized Acc. (%)
100
30
70
90.0
87.5
85.0
$60
85.8
84.2
84.7
84.3
84.2
84-2
842
184
83.8
82.0
60-
82.5
80.0
27.5
40
[a]
[a]
75.0
0.75 1.25 1.75
0:25 0.5 0.25 10
TA
TSV
IS0
TIES
TA Linei FT TA- KFAC
TSV Non-lin FT
-an--i FT
Linear PT
KFAC (Best o)
Linear FT (Des FT
Non--n
Lincar FT
(a) Model Merging (Non-linear FT) vs. TA (Linearized FT)
(b) Model Merging & Linearized FT
Figure 4: For VIT-B/32 (8 Vision), analyze the sensitivity of different merging strategies to the scaling coefficient a; a similar analysis for ViT-B/16 is reported in the Appendix. Left: a-sweep accuracy of post-hoc merging strategies in the non-linear regime, compared with our linearized and regularized models. Right: performance of merging methods on linearized checkpoints.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Linear FT
000
W. KFAC
0:0
Inlier (training distribution)
Outlier (other tasks)
Figure 5: Distribution of f 0o) 2 for inputs originating from the training distribution of task& & (inliers) versus from other tasks (outliers), under both regularized and non-regularized FT.
Comparison of model merging strategies. Fig. 4 compares existing post-hoc approaches for merg-
ing task vectors, including TIES (Yadav et al. 2023). TSV (Gargiulo al. 2025), and ISO (Marczak) et al. 2025). We remark that these methods operate after training and are therefore complementary to our approach, which instead acts during training and produces explicitly weight-diseetangled task vectors. To assess the benefits of in un-triining regularization, in Fig, 4a we perform an a-sweep over the range [0, focusing on performance stability here, a scales the merged parameters 0o + aM((T)) { where M(-) denotes the merging strategy. Under KFAC regularization (green curve), simple task-vector summation (Task Arithmetic, TA) achieves the best peak performance and exhibits strong robustness, with accuracy remaining stable over wide interval of values. This property makes our approach particularly suitable when a cannot be tuned, c.g., in the absence of validation set. In practice, this robustness removes the need to access validation data from other tasks, which may be unavailable or undesirable to share. Moreover, as our method TAK relies on simple Task Arithmetie, it avoids expensive operations such as the SVD required by ISO and TSV. As a result, it can be applied in on-the-fly and adaptive model-merging settings (Crisostomi et al., 2026), enabling efficient personalization for specific user requests.
In Fig. 4h we analyze merging techniques applied to checkpoints obtained in the linearized regime. TA and TIES benefit the most from curvature regularization, whereas ISO and TSV already perform competitively without it. Nevertheless, their performance remains consistently below that of TAK, i... Task Arithmetic with curvature regularization. Additional results are reported in App. F.
Curvature regularization enables Task Localization. We show that our approach enables clear separation between training and out-of-distriburt examples. Indeed, given an input a and task vector T₁, we measure [ar, Te which we interpret as a normaley score for task With our regularization (Eq. (3). these scores are indeed forced to remain low for examples outside the t-th training distribution. As illustrated in Fig. 5 this is exactly what we observe in practice: the distribution of [[Jof( 0, is pushed toward zero whenever the input does not belong to task t. With the naive linear fine-tuning, this behavior is instead much less clear.
8

---

Published as a conference paper at ICLR 2026
Table 3: Our Kronecker-accumulaton heuristic vs. the idealized multi-task formulation.
ViT-B/32
ViT-B/16
T5-base
Method
Complexity
Abs.
Norm.
Abs.
Norm.
Abs.
Norm.
Naive Multi- Task FT
((T)
1.0
86.5
98.4
88.0
97.5
78.5
97.0
Best
86.6
98.5
88.1
97.6
78.5
97.0
1.0
85.8
97.6
88.3
97.9
78.6
98.7
Accumulated reg. (TAK)
(1)
Best
86.0
97.8
88.3
98.1
78.7
98.9
Training times (Forward-bckkwrd)
VRAM Memory Peak
Exact
MC=1 (ours)
1350
A [s]
1.4
1.4
5.033
B [s]
91.5
0.2
LUCC
1.b
VIRAN
Total [min]
198.7
3.9
(b) Computation time for the KFAC approx- imation. Reported times for A and G corre- spond to the averuge over a batch of 8 exam- ples, while the last row shows the total time (in minutes) required to compute the KFAC approximation for all tasks of 8 Vision.
Non PT
Linear
Not
Fi
Lisese
Linear
Nen-lin
Non-lin, (ARt)
Nen-lin (Atte) w. KFAC
(a) Computational overhead: training times and GPU peak.
Figure 6: Analysis the overhead of KFAC regularization during training and pre-computation.
This indicates that, under TAK's curvature regularization, each task vector influences the network output only for inputs drawn from its own training distribution. Moreover, this property suggests natural use of our method for our-of-distribttion detection, as it provides a principled mechanism assess whether an input lies within the model training distribution. A complementary analysis in the non-linear fine-tuning regime is provided in App. F5 where we compare our method against TaLoS and attention-onle onny fine-tuning and observe that the same task-localization behavior persists.
Naïve multi-task training vs. accumulated regularizer. We herein investigate the impact of the heuristic used in our approach, which accumulates the Kronecker matrices (see Eq. (8) and thereby avoids a linear cost in the number of tasks. To this end, we run experiments using the idealized naïve multi-task training described Eq. 7 Our findings, reported in Tab. 3 show that the gap between the idealized and the actual approach marginal for medium-sized architectures such as ViT-B/16 in vision and T5-base text. For ViT-B/32, we instead observe a small but consistent gap in favor of the idealized training objective, which aligns with our experience that smaller architectures tend to be more sensitive to curvature regularization and hence to the quality of the approximation.
Training costs. Fig. 6 analyzes the overhead introduced by our approach, which is twofold: timating the KFAC matrices (before training) and computing the regularizer (during training). No overhead is introduced at inference time. With single Monte Carlo sample, estimating all KFAC matrices for the 8 Vision tasks (128 examples per task) takes only 4 minutes, a very limited amount of time compared to the exact approach from Botev al. (2017). During training, the overhead mainly depends on the chosen regime, with linearized fine-tuning having the largest computational footprint. Nonetheless, KFAC regularization requires only a negligible amount of additional re- sources, i.c., roughly one third of the training time of rJp (Yoshida et al. 2025). This efficiency arises because the TJp penalty requires a second forward-backwarr backward pass through the (slower) lin- earized model. Moreover, since TAK does not rely on data for regularization, it avoids the repeated cost of loading new batches into GPU memory, another factor that slows down r.pp
Memory footprint. Fig. 6 (right) reports the peak VRAM usage across training regimes. KFAC introduces a small increase relative to unregularized baselines: in the linearized regime, it shows a +12% overhead (11.5 + 12.9 GB) w.r.t. linear fine-tuning, while in the non-linear attention-only training shows a +22% increase (6.8 + 8.3 GB). For reference, TJp peaks at 12.3 GB (+7 (+7% linear FT), and standard non-linear fine-tuning reaches 8.5 GB. No memory overhead incurs at inference since regularization is inactive. Notably, aggregating all per-task KFAC factors into a single surrogate keeps the training footprint of our method at O(1) w.r.t. the number of tasks.

---

Published as a conference paper at ICLR 2026
#MC (decrcasing)
Varying only #examples
Varying only #MC
128
32
16
8
Full
86
MC-1
of date exocl a w par.ara
KFAC Storaee (MB)
500
Abx (9)
86
400
84-
300
Fruned 0.7
SWD Fi1S
82
82
200
Prined 9.55 Mir
100
844
126
128
81
82
83
10
57
#examples (BMC-1)
#MC (#examples=322)
#examples (increasing)
Abs. Acc. (%)
(a) Varying the number of datapoints/Monte Carlo samples for the KFAC. (b) KFAC Storage vs. Accuracy.
Figure 7: Effect of KFAC approximation efficiency on performance. Left: impact of the number of datapoints used estimate the KFAC matrices on downstream accuracy of the merged model. Right: accuracy-memory trade-off induced different KFAC matrix compression strategies.
KFAC estimation. In Fig. 7a we analyze the effect of varying the number of examples and MC samples used for curvature estimation. Our findings (Fig. 7a Left) indicate that using 128-256 examples is already sufficient to saturate performance, yielding results comparable to those obtained with 30% of each training set. Moreover, final performance generally on par with that obtained with the exact approximation of [Botev et al. (017). With respect to Monte Carlo sampling, only a few samples per example (1-2) are sufficient. Surprisingly, performance deteriorates beyond this point, with variance across seeds increasing as the number of MC samples grows. Overall, increasing the number of MC samples is less effective than using more data with fewer MC samples.
KFAC compression. Unfortunately, the memory cost of storing KFAC matrices scales quadratically with the layer width, which may become challenging for very large models. To mitigate this cost, evaluate how aggressively KFAC matrices be compressed via dynamic 8-bit quantization, structured pruning. block diaggoaaiizziiiinn,, and truncated SVD (see App. [.66) without harming ac- curacy. On VIT-B/16 (8 Vision), these techniques yield substantial memory savings with only minor performance loss (Fig- 7b). The block-based strategy provides the best trade-off, decreasing mem- ory from approximately 550 MB (full KFAC) about 70 MB 87% reduction while incurring only -p-oint drop in absolute accuracy (88.3 to 87.1).
We additionally analyze whether the KFAC matrices can be moved off-GPU during training without introducing prohibitive overhead. To do so, we evaluate a regime where the penalty loss is computed and backpropagated only once ev-
90.0
Bed
65.5
67.4
97.1
970
87.5
ery N' training steps. As illustrated in Fig. g applying the loss every 16 steps leads to
Abs. Ace.
85.0
modest degradation (~1.4 points) relative to applying it every iteration. This demon- strates that scheduling curvature updates can effectively amortize memory transfers and en- able GPU-CPU factor shuffling without com- promising the usefulness of the regularizer.
82.5
CIN
80.0
N-1
N-2
N-4
N-8
N-16
No KFAC
Loss computed every N steps
Figure 8: Applying the KFAC loss every N steps.
CONCLUSIONS
We investigate curvature-basee regularization as means to enhance weight disentanglement in Task Arithmetic and propose TAK (Task Arithmetic with KFAC regularization), a dataless, efficient, and effective approach that makes the simple summation of task vectors competitive with state-of-the-an merging strategies, without additional tuning. We demonstrate applicability in linearized and non- linear regimes, and show that it enables a clear separation between in- and out-of-distribution ex- amples. Our work calls for releasing additional assets together with the pre-trained weights without having to open-source the training data. Such information, e.g. gradient accumulators of the adap- tive optimizer used for training (Li et al. 2025), or in our case KFAC, enable further downstream applications with foundation models. Finally, further extending these ideas to models adapted either via standard full parameter-efficien fine-tuning remains important direction.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We acknowledge the CINECA award under the ISCRA initiative, for the availability of high- performance computing resources and support. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, and com- panies sponsoring the Vector Institute. Simone Calderara supported by the Horizon Europe Chips Joint Undertaking under the NexTArc project (HORIZON-JU-Chi) 200hhiss-024-2-RRAA Nex Next Generation Open Innovations in Trustworthy Embedded AI Architectures for Smart Cities, Mobility and Logistics (Grant Agreement ID: 101 194287, DOI: 10.3030/100 1/130//11110 194287). Additionally, the search activities of Angelo Porrello have been partially supported by the Department of Engineering "Enzo Ferrari" through the program FAR2025DIP (CUP E93C250003700055) We also gratefully acknowledge Symboolic s.r.l. for funding the PhD position Thomas Sommariva and for the sig-
nificant contribution of Lorenzo Bonicelli.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, the complete source code, including model implemen- tations, hyperparameters, and evaluation scripts, is integrated into the Mammoth framework. The codebase will be made publicly available at lsttps:// //g1 thub. com/ a hataaaaaaaaaaaaammaaaaaaaaaaaaae support further research and facilitate benchmarking.
DISCLOSURE ON THE USE OF LANGUAGE MODELS
Large Language Models (LLMs) were used exclusively to improve the clarity and polish of writing. All scientific ideas, methodological contributions, experimental designs, analyses, and con- clusions presented in this paper originate entirely from the authors.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016.
Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Lqf: Linear quadratic fine-tuning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2021.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 2000.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 2019.
Sanjeev Arora, Simon Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets small-data tasks. International Conference on Learning Representations, 2020.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. International Conference on Machine Learning, 2017.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
11

---

Published as a conference paper at ICLR 2026
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014.
Donato Crisostomi, Alessandro Zirilli, Antonio Andrea Gargiulo, Maria Sofia Bucarelli, Simone Scardapane. Fabrizio Silvestri, lacopo Masi, and Emanuele Rodolà. MASS: Moerging through adaptive subspace selection. In International Conference on Learning Representations, 2026.
Felix Dangel, Stefan Harmeling, and Philipp Hennig. Modular block-diagondl curvature approxi- mations for feedforward architectures. In International Conference on Artificial Intelligence and Statistics, 2020.
Felix Dangel, Runa Eschenhagen, Bálint Mucsányi, and Tobias Weber. Kfac from scratch. arXiv, 2025.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux- effortless bayesian deep learning. In Advances in Neural Infor- mation Processing Systems, 2021.
Nikita Dhawan, Sicong Huang. Juhan Bae, and Roger Baker Grosse. Efficient parametric approx- imations of neural network function space distance. In International Conference on Machine Learming. 2023.
Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, and Philipp Hennig. mronecker-factored approximate curvature for modern neural network architectures. In Advances in Neural Information Processing Systems, 2023.
Antonio Andrea Gargiulo. Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2025.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2021.
Roger Grosse and James Martens. A kronecker-factored approximate Fisher matrix for convolution layers. In International Conference Machine Learning, 2016.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large lan- guage model generalization with influence functions, 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-disttibution generalization. In IEEE International Conference on Computer Vision, 2021.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2022.
Leonardo Jurada, Marco Ciccone, and Tatiana Tommasi. Efficient model editing with task-localized sparse fine-tuning. In International Conference on Learning Representations, 2025.
12

---

Published as a conference paper at ICLR 2026
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in Neural Information Processing Systems, 2018.
Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie Su, and Li Shen. Fine-tuning attention modules only: Enhancing weight disentanglement in task arithmetic. International Conference on Learn- ing Representations, 2025.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Ad- vances and open problems in federated learning. Foundations and trends in machine learning, 2021.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Seitail: A textual entailment dataset from science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graid grained categorization. In Proceedings of the IEEE international conference on computer vision work-
shops, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report University of Toronto, Toronto, Ontario, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 2002.
Yu Xin Li, Felix Dangel, Tam Derek, and Colin Raffel. Fishers for free? approximating the fisher information matrix by recycling the squared gradient accumulator. In International Conference on Machine Learning (ICMI.), 2025.
Wu Lin, Felix Dangel, Runa Eschenhagen, Kirill Neklyudov, Agustinus Kristiadi, Richard E. Turner, and Alireza Makhzani. Structured inverse-free natural gradient descent: Memory--fficient & numerically-stable KFAC. In International Conference on Machine Learning (ICML), 2024.
Tian Yu Liu and Stefano Soatto. Tangent model composition for ensembling and continual fine-
tuning, In IEEE International Conference on Computer Vision, 2023.
Tian Yu Liu, Aditya Golatkar, and Stefano Soatto. Tangent transformers for composition, privacy and removal. In International Conference on Learning Representations, 2024.
Charles VaV Loan. The ubiquitous Kronecker product. Journal of Computational and Applied Mathematics, 2000.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, 2023.
Daniel Marczak, Simone Magistri, Sebastian Cygert, Barthomicj Twardowski. Andrew D Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task- specific subspaces. International Conference on Machine Learning, 2025.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), 2014.
James Martens. Deep learning via Hessian-free optimization. In International Conference on Ma- chine Learning, 2010.
James Martens.
New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 2020.
James Martens and Roger Grosse. Optimizing neural networks with kronecker approximate curvature. In International Conference on Machine Learning, 2015.
James Martens, Jimmy Ba, and Matt Johnson. Kronecker froneccke-facttredd curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2026
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication -fffcciinnn learning of deep networks from decentralized data. In International Conference Artificial Intelligence and Statisties, 2017.
Fangzhou Mu, Yingyu Liang, and Yin Li. Gradients as features for deep representation learning. In International Conference Learning Representations, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-traines models. Advances in Neural Information Processing Sys- tems, 2023.
Kazuki Osawa, Yohei Tsuji, Yuichiro Ueno, Akira Naruse, Rio Yokota, and Satoshi Matsuoka. Lcrge-scale distributed second-order optimization using kronecker-factee approximate curva- ture for deep convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2019.
Aniello Panariello, Daniel Marczak, Simone Magistri, Angelo Porrello, Barthomiej Twardowski, Andrew D. Bagdanov, Simone Calderara, and Joost de Weijer. Accurate and efficient low- rank model merging in core space. In Advances in Neural Information Processing Systems, 2025.
Angelo Porrello, Lorenzo Bonicelli, Pietro Buzzega, Monica Millunzi, Simone Calderara, and Rita Cucchiara. A second-order perspective on model compositionality and incremental learning. In- ternational Conference Learning Representations, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Yi Ren, Shangmin Guo, Wonho Bae, and Danica J Sutherland. How to prepare your task head for finetuning. In International Conference on Learning Representations, 2023.
Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli, Donato Crisostomi, Federico Bolelli, Elisa Ficarra, Emanuele Rodolà, Simone Calderara, and Angelo Porrello. Update your transformer to the latest release: Re-basin of task vectors. In International Conference on Machine Learning, 2025.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 2018.
Nicol N. Schraudolph. Fast curvature matrix-vecte products for second-order gradient descent. In International Conference on Artificial Intelligence and Statistics, 2003.
Hyounguk Shon, Janghyeon Lee, Seung Hwan Kim, and Jummo Kim. Dleft: Deep linear continual fine-tuning for general incremental learning. Proceedings of the European Conference on Computer Vision, 2022.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In The 2011 international joint conference on neural networks, 2011.
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. In International Conference on Learning Representatiinss 2025.
14

---

Published as a conference paper at ICLR 2026
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial linearization. In International Conference Learning Representations, 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. 2018.
Chaoqi Wang. Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. In International Conference Machine Learning, 2019.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning, 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection scene categories. International Journal of Computer Vision, 2016.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 2023.
Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryosuke Yamaki, Ryotaro Shimizu, Yuki Saito, Julian McAuley, and Hiroki Naganuma. Mastering task arithmetie: $ ttauSSp as a key indicator for weight disentanglement. In International Conference on Learning Representations, 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX / SUPPLEMENTARY MATERIAL
The appendix is organized as follows:
App. discusses the main limitations of approach, including memory requirements and curvature-estimation challenges.
provides derivation and formal bound on the approximation error introduced when merging multiple KFAC factors using the Kronecker heuristic.
App. C
App. D presents additional plots illustrating the disentanglement error.
App.
details the implementation of our methods, with separate discussions for the vision
and text domains.
App. reports additional experiments. These include:
Core analyses:
* per-task performance analysis,
* alpha-sweep robustness study (App. F2
* ablation on the regularization coefficient (App- F3
* evaluation of shared KFAC computed on reference dataset (App. F.4)
* lask-localization analysis under non-linear fine-tuning (App. E.5
extended experiments:
analysis of task localization under memory-efficien KFAC approximations, in- cluding block-based, SVD-based, pruning, and 8-bit quantized variants (App.[F6)
* additional results on more challenging vision domains using a caas-inccemmentll partitioning protocol (App. F.7.
App. G provides concise overview prior work on linearized fine-tuning and its recent developments.
B LIMITATIONS
KFAC requires storing the Kronecker matrices in GPU memory two per layer, each with quadratic complexity in the number of units. For large models this can become problematic, suggesting that alternative strategies based on matrix compression or structured Kronecker factors (Grosse et al. 2023 Lin et al. 2024) should be explored. While we combine the well-established KFAC with an accumulation strategy, designing curvature approximations that can easily be merged without sacri- ficing accuracy may be worth exploring in the future. Moreover, our experiments in the text domain indicate room for improvement, raising the question of whether more sophisticated techniques for curvature estimation could further enhance Task Arithmetic.
C APPROXIMATION ERROR OF THE MERGED KFAC FACTORS
For clarity, we focus on a single layer and assume all layers contribute equally, omitting the task weights 7 Let (A.)T= and BB BB)TT_ _ denote the KFAC factors associated with the tasks involved the merge. The heuristic used in Eq. Breellces the sum of Kronecker products with the Kronecker product between aggregated factors
T
T
Σ
B. 8 A
Σ
B
A
T
We now provide simple bound that quantifies the error introduced by this approximation. To do so, define the empirical means and the deviations from the mean
T
T
A
B
M
B
AA = Ar A,
AB, B, - B. (10)
T
T
t=1
t
16

---

Published as a conference paper at ICLR 2026
Note that, by construction, E, E, AB, 0. Substituting A, A + AA, and B, B + AB; into the left-hand side of Eq. (9 yields
T
T
Σ B
B + AB, 0 (A+ AA) Σ 1=1
(11)
B A AB, A+ AB, 00 AA,)
(12)
T
T
T
T
B A + 8 Σ
AA +
Σ
AB,
&
A + Σ
AB, @ AA.
(13)
t=1
[=1
1=1
TBA
= =T A
Σ AB. AA.
(14)
Substituting A, À + AA, and B, B + AB, into the right-hand side of Eq. 9
instead, yields
T
T
Σ
Be
Σ
A
=T
=
(15)
-=1
Hence the approximation error is
T
T
T
T Σ
E==
2 B.
T
Σ
B₁
x
Σ
A
AB @ AA.
[=1
1=1
Error bound. Using the Frobenius norm and the property
X 69 Y
F
|| ||||,, we obtain
T
T
| S
Σ
| <
AB
Σ
1A..
(16)
Defining the deviations (standard deviations in matrix space), we obtain:
T
1
σ we finally obtain the compact bound
on =
T
JBB.I
(17)
| < TOATO.
(18)
The approximation error is proportional to the product of the variations of the KFAC factors across tasks. When the task-specific factors (A,, B,) cluster tightly around their means, both δ and σg are small, yielding a small deviation between the true mixed KFAC term and its merged approximation. This situation is particularly likely to occur when the matrices estimated from a fixed pre-trained backbone such as CLIP: since the underlying feature extractor remains unchanged across tasks, the induced activation and gradient statistics tend vary only mildly. As result, the corresponding KFAC factors exhibit limited task-to-task fluctuation, further justifying the accuracy of the merged approximation.
Interpretation.
D ADDITIONAL PLOTS ON WEIGHT DISENTANGLEMENT
In Fig. we report the disentanglement emoor, metric introduced by Ortiz-Jimenez et al. (2023):
2
E(a1, = Σ Enpy, (dist (f(:; Oo + aUTT), f(;; 0, + AITI +227)))
(19)
17

---

Published as a conference paper at ICLR 2026
EureSAT SVHN
DTD SUN397
Cars196 RESISCA5
100
Diventango Cohnntanglommet Error (16)
Non Linear FT
61
44
20
0
100
Diventanglmmen Erroe (16
Linear ITT
100
Atn. Only FT
Dientanglement Ervor
Dienntanglomen Error (%N)
OURS
03
Figure 9: Visualization of weight disentanglement Ortti-Jimenez et al. 2023 in ViT-B/16. Non linear fine-tuning Ilharco al. (2022). Linear fine-tuning Ortiiz-lmmnnz et al. (2023). Attention-
Only fine-tuning Jin et al. (2025). Linear fine-tuning with KFAC regularization.
where dist(y1, 92) (1y1 # v2). When {(a), a2) 0, tasks Ti and 72 merge without interference for the corresponding values of ai and a2.
As shown in the plots. linearized fine-tuning substantially improves the disentanglement of task vectors. This property is further enhanced under our regularization regime, where only a few darker regions remain, mostly for 1, setting that is never used in practice. Notably, in our experiments the disentanglement error is consistently close to zero along the diagonals, which is the most relevant case, since in the literature the common choice on 02
On-
E IMPLEMENTATION DETAILS
The GGN information matrices were estimated using a single Monte Carlo sample and computed on 33% of the available training data. However, empirical analysis showed that sampling only 250-300 training points sufficient to obtain a reliable estimation of the curvature matrix.
KFAC factors are estimated for all layers involving linear projections in the model namely, at- tention and ferd-forward projections. In contrast, for LayerNorm parameters and the class token, whose scaling, bias, and token parameters grow linearly rather than quadratically with the embed- ding dimension, computing the full GGN matrix is tractable. For these components, we therefore the original, approximation-free GGN instead of its KFAC approximation.
18

---

Published as a conference paper at ICLR 2026
ViT-B/32 (Linear FT)
ViT-B/16 (Linear FT)
VIT-L/14 (Linear FT)
Cars
Cars
Cars
SUN39T
DTD
SUN399
DTD
SUN39
DTD
30
60
20
EAT
SVHN
RESISC
CTSRB
RESISCAE
OTSRB
RESISC
GTSRB
MNIST
MNIST
MNIST
Pre-trained
Lincar
Lincar
Diag.
Linear K-FAC
VIT B/32 (Non-linca) FT)
VIT- B/1- (Non-lineat FT)
VIT-L/14 (Non Noncain FT)
Cars
Cars
Cars
SUN397
DTD
SUN399
DTD
SUN397
DTD
70
30
30
SVHN
BUOSAT
SVHN
EOOSAT
SVHN
RESISCAS
GTSRB
RESISC
OTSRB
RESISCAS
GTSRB
MNIST
MNIST
MNIST
Pre-trinedd
Non-linear FT
Atmn. Only FT
KFAC (TAK)
Figure 10: Impact of training and regularization choices on vision tasks (absolute accuracy). Top:
linearized regime, compared against the diagonal approximation. Bottom: non-linear regime, com- pared against attention-only fine-tuning.
The KFAC regularization loss is applied to all fine-tuned layers. Empirically, we found it beneficial rescale the regularization weight of the last layer of the CLIP visual encoder by factor of 0.1.
E.1 VISION DOMAIN
We leverage the Vision protocol (Iharcoo et al. 2022 and conduct experiments on Stanford
Cars (Krause al. 2013), DTD Cimpoi al. 2014). EuroSAT (Helber al. 2019). GTSRB (Stal-
Ikamp et al. 2011). I. MNIST (LeCun et al. 2002). RESISC45 (Cheng et al.] 2017). SUN397 (Xiao 2011). For training the task vectors. we followed the setup of previous works Iharco et (202); Ortiz-Jimened et al. (2023); Yoshida et al. (025). adopting batch size of 128. We used the Adam optimizer with learning rate of 3 x of 0.1, and cosine annealing learning rate scheduler. Unlike prior approaches, we did not apply gradient clipping during training. The regularization term in the loss was weighted by 100 VIT-B/32, À 500 for ViT-B/16, and À 2000 for VIT-L/14.
al. 2016). and SVHN (Netzer
weight decay
Compared to previous work, we employed higher learning rate. Since our formulation includes an explicit regularization term in the loss, this allowed to increase the learning rate without intro- ducing interference across tasks.
E.2 TEXT DOMAIN
We follow the 6NLI benchmark (Stoica al. 2025 Panariello et al. 2025). including SNLI (Bow- man et al. 2015). MultiNLI (Williams et al. 2018). and SICK (Marelli et al. 2014) which are three-way way classification tasks where the relation between premise and hypothesis must be iden- tified as entailment, contradiction. or neutral. Additionally, SciTail (Khot et al. 2018). RTE (Wang
al. 2018), and QNLI (Wang al. 2018) binary entailment tasks, and therefore fine-tuning
19

---

Published as a conference paper at ICLR 2026
Table Comparison of different merging strategies in the linear fine-tuning regime, with and with- out KFAC regularization. Results are reported for a 1.0 and the best-performing
ViT-B/32
ViT-B/16
Method
Abs.
Norm.
Abs.
Norm.
1.0
77.1
87.8
80.1
88.7
Linear FT + TIES
Yadav et al.
22033)
Best
77.2
87.8
80.3
88.9
1.0
84.2
96.3
86.7
96.5
Linear FT + TSV
Gargiulo et al.
2025
Best
84.2
96.3
86.8
96.6
1.0
83.8
95.8
86.9
96.7
Linear FT ISO Marezak et al.
2025
Best
84.4
96.4
87.3
97.1
1.0
81.8
92.7
86.6
95.9
TAK, Ours + TIES
Yadav et al.
2023
Best
82.0
92.8
87.1
96.6
1.0
84.7
96.3
87.6
97.2
TAK, Ours TSV
Gargiulo et al.
2025
Best
84.7
96.3
87.7
97.3
1.0
84.2
95.6
87.2
96.8
TAK, Ours ISO
Marezak et al.
2025
Best
84.2
95.6
87.3
96.9
Absolute Ace. (%)
Normalized Acc. (%)
100
90.0
87.6
66.88
86.7
67.3
$1.2
81.3
869
ST.
86.6
87.5
85.0
82.0
S02
605
800
80.0
27.5
50
[x]]
75.0
TA
TSV
150
TIES
TA Linear FT TA-Lincu KFAC
TSV Non- lin FT
TIES Non lia.
Linerr PT KFAC (Best o)
Liner (Bes o)
Nun-lin
Lincar FT
(a) Model Merging (Non-linear FT) vs. TA (Lincarized FT)
Model Merging & Linearized FT
Figure 11: For ViT-B/16 (8 Vision), analyze the sensitivity of different merging strategies to the scaling coefficient a. Left: a-sweep accuracy of post-hoc merging strategies in the non-linear regime, compared with linearized and regularized models. Right: performance of merging methods on linearized checkpoints.
and evaluation are restricted to two labels. For training language task vectors, we adopted a hatch size of 128, using an AdamW optimizer with a learning rate 10-4 with an iteration-based cosine-annealing scheduler and weight decay of 0.01. Like in vision tasks, we did not apply gra- dient clipping during training. The regularization term the loss is set À 20 for the KFAC regularization and to À = 0.1 for the diagonal regularization.
F ADDITIONAL EXPERIMENTS
In this section we present the results of additional experiments on task addition conducted on the Vision benchmark, complementing those already reported in the main paper.
F.1 PERFORMANCE
Fig. T0 provides a per-task breakdown of the same experiment reported in Tab. [ Interestingly, the larger ViT-L/14 backbone exhibits smaller relative gains from regularization, particularly in the non-linear regime, where its behavior closely resembles that of its linearized counterpart. Consis- tent with prior work Ortiz-Jimenez et al. (2023), this suggests that very large models may already display an implicit form of regularization. Conversely, the ViT-B/32 benefits the most from regular- ization, showing that smaller architectures require more careful fine-tuning to enable effective task arithmetic.
20

---

Published as a conference paper at ICLR 2026
KFAC & Non-linear FT (ViT-B/32)
KFAC & Non-linear FT (ViT-B/16)
Ace. (%)
Nomaliral Acc.
Arc. (%)
Nammliized
50
60
60
10
29
[e]
[0]
24 0.75
15
15:55
00 05
00.0.500 0.75 10
05
-+- TA AM.. Only FT.. KPAC
TA Ars. Only FT
TA-Nor-a F7
- TA A... Only FT KFAC
TA Ann Only FT
TA No FT
(a) VIT-B/32, a-sweep comparison.
(b) ViT-B/16, a-sweep comparison.
Figure 12: Sensitivity the scaling coefficient a the non-linear fine-tuning regime. We report a-sweep results for ViT-B/32 (left) and ViT-B/16 (right), comparing standard non-linear fine-tuning, attention-only fine-tuning Jin et (2025). and its variant regularized with the KFAC.
Table 5: On 8 Vision, ablation of on ViT-B/32 (left) and ViT-B/16 (right). All performances are reported in terms of absolute accuracy using a
ViT-B/32
VIT-B/16
À
Seed
Seed 21
Seed 42
AVG.
À
Seed
Seed 21
Seed 42
AVG.
0
75.00
75.4
75.1
75.2 + 0.028
0
79.1
78.7
79.1
79.0 + 0.188
82.2
82.4
80.6
81.7 0.648
1
83.2
83.4
83.8
83.5 1: 0.265
10
85.2
85.1
85.1
85.1 0.002
50
86.9
86.8
87.0
86.9 + 0.059
100
86.2
85.8
86.0
86.0 0.026
500
88.0
87.9
88.2
88.0 0.114
1000
86.5
86.4
86.4
86.4 0.002
5000
88.3
88.4
88.4
88.4 0.015
10000
84.5
84.4
84.3
84.4 + 0.006
50000
86.7
86,6
86.6
86.6 + 0.002
F.2 ROBUSTNESS UNDER TASK ARITHMETIC: ALPHA-SWEEP ANALYSIS
In Fig. extend the analysis presented in the main paper to the ViT-B/16 backbone. The same trends observed for ViT-B/32 hold also in this setting, confirming the consistency of our findings across model scales. For completeness, we additionally report in Tab. the explicit performance of the different model merging strategies evaluated the linearized regime.
We then conduct similar a-sweep analysis focusing the application of method in the non- linear fine-tuning regime. As shown in Fig. 12 across both ViT-B/32 and ViT-B/16, attention- only fine-tuning Jin et al. (2025) and its KFAC-regularized variant exhibit increased robustness to variations of the scaling coefficient a compared to standard non-linear fine-tuning, with our method achieving both higher peak performance and improved robustness. However, when compared the analyses in Figs. and II which examine the linearized and KFAC-regularized model (i.e.. TAK), the non-linear regime remains significantly more sensitive to a, suggesting an intrinsic advantage of approaches that combine linearization with disentanglement-avaal regularization.
F.3 ABLATION ON THE REGULARIZZTIIN COEFFICIENT
This section presents an ablation study investigating the impact of the scaling coefficient À applied to the regularization term in the loss function. In Tab. we evaluate the performance of ViT-B/32 and ViT-B/16 using six values of the regularization coefficient, ranging over five orders of magnitude from to 10 and repeated each experiment with three random seeds. The case À = 0 serves as the baseline, corresponding to non-regularized fine-tuning. should be noted that these results differ from those reported in Tab. [ as the linear fine-tuning therein follows the hyperparameter configu- ration of [lharco et al. (2022), whereas the experiments presented here employ the hyperparameter setting described in E
The results indicate that the proposed method is robust with respect to the choice of X. Optimal performance is observed for values of À between 102 and 103, while only minor degradation occurs for À = 10 and À = 10a. This behavior confirms that successful model merging primarily depends on the presence of regularization based information from the generalized
matrix,
21

---

Published as a conference paper at ICLR 2026
Table 6: Task addition results on the eight vision datasets when using either task-specific KFAC factors or single shared KFAC computed on ImageNet-1k. Results show that a universal, task- agnostic KFAC (ImageNet- retains most of the benefits of our regularizer while requiring no access to auxiliary task-specific data.
ViT-B/32
VIT-B/16
Method
Dataless
Abs.
Norm.
Abs.
Norm.
1.0
76.7
87.2
80.2
88.9
Linear FT
Best
78.8
89.9
82.0
90.9
1.0 Best
85.8
97.6
88.3
97.9
TAK. Ours
86.0
97.8
88.3
98.1
1.0
84.7
97.0
86.0
95.4
ImageNet-TAK, Ours
Best
84.7
97.0
86.0
95.4
and that the magnitude of this term must be sufficiently emphasized. However, the results also show that no precise tuning is required to achieve strong performance.
F.4 ELIMINATING TASK DEPENDENCE WITH A UNIVERSAL. KFAC
Although our framework completely removes the need for raw auxiliary data, it still requires pre- computed input and gradient covariance factors from the tasks to be disentangled. This dependence may be limiting in scenarios where such factors cannot be shared due to practical difficulties in stor- ing or distributing task-specific curvature statistics, or simply because the set of tasks to be composed known in advance training time.
To assess whether this dependence can be relaxed, we test whether broad curvature statistics extracted from large, natural-imann distribution can serve a proxy and effectively replace the per-task KFAC factors. In details, we build a variant, denoted ImageNer-KFAC. in which every layer uses a single pair of A/ B matrices computed on ImageNet- Ik. Ideally, these factors capture universal visual covariances, and hence they can remain fixed for all downstream tasks. During fine- tuning, these shared factors can entirely substitute the task-specific ones normally employed by our regularizer.
As shown in Tab. [ despite using non-task-specinn information, this proxy KFAC recovers approx- imately 97-99% of the performance obtained with full task-specific factors on both VIT-B/16 and ViT-B/32 (8 Vision). The absolute accuracy reached by the ImageNet-KFAA variant 84.7% on ViT-B/32 and 86.0% ViT-B/16, closely matching the performance of the original approach while substantially surpassing diagonal or no-regularization baselines as well as competitive alternatives such TaLoS attention-only fine-tuning.
These results indicate that a task-agnostic curvature prior, captured by a single shared factorization, delivers most of the benefits of dataless regularizer without accessing any task-specific statis- tics. In practical scenarios, this makes the method fully data-agnostic with respect to the problem, effectively eliminating any residual coupling to external tasks.
F.5 TASK LOCALIZATION UNDER NON-LINEAR TUNE-TNNIG
In this section we extend the localizzoii analysis presented in the main paper to the non-linear fine-tuning regime. The goal is assess whether the separation between in-task and out-of-task examples, induced by our curvature regularizer under linearized training, persists when full model parameters are updated. In details, we measure the same ceiting-locaaization metric used in the main paper, namely the difference between the Jacobian- Jacobian-projected output variation [|Jef (xo 6o) Tell? for inputs belonging to task t versus those coming from other tasks.
As shown in Fig. 13 we evaluate four methods: the standard non-linear fine-tuning, TaLoS Turada et al. (2025) attention-only fine-tuning Jin et al. (2025), and our proposed KFAC-based curvature regularizer. For each approach, we fine-tune the model in the fully non-lineat setting and compute the distribution of normalcy scores for in-task and out-of-task inputs.
22

---

Published as a conference paper at ICLR 2026
Cars 196
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
TaLoS Non-linean FT
Attn-Only FT
907
w. KFAC
Inlier (training distribution)
Outlier (other tasks)
Figure 13: Task localization under non-linear fine-tuning. We report the distribution of the dacobian-projected normalcy scores [ar 0o) 2 for inputs belonging task t (in-task) ver- sus inputs from all other tasks (out-of-task).
The results show consistent pattern across all datasets. Our method maintains clear and sharp separation between in-distribution and out-of-distribution examples, closely mirroring the behavior observed under the linearized regime. TaLoS and attention-only fine-tuning preserve part of this effect but yields a weaker distinction. Overall, these findings confirm that curvature regularization continues to restrict the influence of each task vector its corresponding training distribution even when the full network is fine-tuned.
F.6 KFAC COMPRESSION STRATEGIES AND TASK LOCALIZATION
To the robustness of our curvature regularizer under memory constraints, evaluate several compression strategies applied directly to the KFAC factors. All strategies described below are applied independently to both A and B matrices for every layer.
The first strategy is a block-diagonal approximation ("Block 8"), in which each factor is parti- tioned into eight equally sized blocks along the main diagonal, with all off-diagonal blocks dis- carded. This yields substantial reduction in memory while maintaining a structured representation and preserving dominant second-order interactions.
The second strategy relies on truncated SVD. Given the factorization A = U.VT we keep only the top singular components, either by selecting fixed rank (32 in our experiments) or by retaining a percentage of the original rank (25%). The truncated reconstruction A U. V provides low-rank surrogate that preserves the principal curvature directions.
third strategy applies unstructured magnitude pruning. Each KFAC matrix is converted COO sparse format, and only the largest-magnitude entries are preserved. We consider two keep ratios, 30% and 15%, corresponding to increasingly aggressive sparsification. All remaining entries are set to zero, effectively reducing memory and bandwidth requirements.
Finally, we evaluate dynamic 8-bit quantization. Each factor is quantized on-the-fly to an 8-bit integer representation, with per-row scaling ensuring that reconstruction enrors remain controlled.
Task localization. We further investigate whether the liskloocaizattonn behavior observed in the main paper remains stable when applying memory-efficient KFAC approximations. In particular, we focus on the block-based compression strategy, where each KFAC factor is decomposed into 8 diagonal blocks, substantially reducing storage while preserving the structure of the Kronecker
23

---

Published as a conference paper at ICLR 2026
Cars 196
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
W. KFAC (8 Block) w. KFAC
Inlier (training distribation)
Outlier (other tasks)
Figure 14: Task localization under linearized fine-tuning with block-compressed KFAC. The sepa-
ration between the two distributions closely matches that of the full KFAC model, indicating that the block-based compression has negligible impact on task localization and that curvature-based task isolation remains robust even under aggressive memory reductions.
Table 7: Performance comparison across different regularization strategies on ViT-B/16
Model
ImageNet-R
EUROSAT
RESISC45
Zero-shot
77.72
49.48
66.02
Non-linear FT
82.32
71.21
73.85
Linear FT
81.66
70.40
72.28
Linear FT w. Diag. GN
81.64
73.94
74.04
7jP Yoshida al. 20255)
81.28
84.36
84.83
TAK. Ours (naive penalty)
82.64
79.64
78.91
TAK, Ours (aggregated penalty)
82.63
79.64
78.30
approximation. This variant is the most promising among those we evaluated, as it consistently provides the best trade-off between memory savings and accuracy.
The results, shown in Fig. 14 reveal that the block-based KFAC approximation preserves the same localization behavior the full KFAC model. Even with only eight diagonal blocks per factor, the model continues to sharply distinguish in-distribution from out-of-distribution samples. The com- pression therefore appears to have negligible impact on this diagnostic, suggesting that curvature- based task localization is robust to coarse, semory-frienddy KFAC approximations.
F.7 EXPERIMENT ON OTHER VISION DOMAINS
In Tab. we present additional experiments a different vision domain further assess the ef- fectiveness of KFAC regularization on less trivial tasks. Following (Porrello et al. (025). each dataset is split into partitions containing distinct classes. This procedure ensures task diversity while keeping the domain consistent, since all partitions originate from the same dataset. The number of classes per partition depends on the dataset: ImageNet-R (Hendrycks et al.) [021] divided into 10 tasks of 20 classes each, RESISC45 (Krizhevsky & Hinton] (2009) into 9 tasks of 5 classes each, and AT (Helber et al.) 2019) into 5 tasks classes each. After fine-tuning the base model on each partition, the resulting models are merged and evaluated on the full test set, considering the union of all classes across tasks rather than restricting evaluation to the classes of the training task only, done in the 8 Vision benchmark. Accuracy is then reported this joint classification prob- lem, following the protocol of (Porrello et al., 2025). These experiments demonstrate that KFAC regularization achieves state-of-the-art performance under this more challenging setting.
24

---

Published as a conference paper at ICLR 2026
Method
Dataless
Abs.
Norm.
Individual
85.9
Non-lin. FT
65.5
75.9
Linear FT
76.1
92.0
Attn-Only FT TaLos
67.0
78.3
75.8
92.8
TJp
81.0
99.5
KFAC, Ours
78.6
98.7
Table 8:
Task addition results for T5-base with a = 1.
F.8 TEXT DOMAIN: RESULTS FOR a
Results for 1. We follow the setup described in the main text for language tasks and evaluate T5-base using the fixed hyperparameter value a = 1. As reported in Tab. [, our method exhibits consistently strong performance in the text domain, mirroring the trends observed in the vision setting.
G RELATED WORKS ON LINEARIZED -INN-TUNING
Linearized models offer a principled lens for analyzing fine-tuning by considering first-order ex- pansions around pre-trained initialization. Foundational work (Arora al., 2019 Jacot et al,, 2018) showed that infinitely wide networks trained with gradient descent follow kernel gradient flow under the Neural Tangent Kernel (NTK), yielding exact functional characterizations of training dynamics. This perspective has since been extended to more realistic settings, including representa- tion learning (Mu al., 2020), small-data regimes (Arora al. 2020), and random-matris studies of generalization (Wei et al. (022). Building on these insights, several linearized fine-tuning ap- proaches have been proposed to improve efficiency and stability, such as LQF (Achille al.) 2021 privacy-preserving updates (Golatkar et al., (021), improved task-head initialization (Ren et al.) 2023), continual learning (Shon et al. [2022, and language-mone adaptation (Malladi et al. 2023).
More recent work explores model composition and ensembling through tangent-space operations
(Liu & Soatto
2023
Tang et al. 2024).
The linearized regime has also become central to task arithmetic. Tangent-space representations have been linked to weight disentanglement and reliable task editing (Ortiz-Jmmenez et al. 2023] Porrello et al.] 2025; Yoshida al. 2025) Liu et al. 2024). Within this framework, NTK based approximations enhance task separability and make linear combinations of task vectors more pre- dictable, further underscoring the versatility of model linearization for fine-tuning, composition, and editing.
25


Reviews:
[{"review_id": "8UVeYV1T6N", "reviewer": "Reviewer_KRUA", "summary": "The paper addresses cross-task interference in model merging/task arithmetic by proposing a method for enhancing weight disentanglement. The authors demonstrate that *representation drift regularization*, a data-dependent solution, can be reformulated in a linearized setting as a quadratic form involving the Jacobian's Gramian matrix (G). They then use a tractable approximation (KFAC) of G to propose a new, data-free regularizer. The authors also propose a heuristic to merge KFAC matrices with O(1) complexity. The resulting KFAC-regularized task vectors are shown to be more disentangled, leading to better performance when merging models that have been fine-tuned linearly.", "questions": "- What are the performance results of SOTA post-hoc methods (TIES, TSV, etc.) when applied to task vectors from standard non-linear fine-tuning? How does the KFAC-regularized framework compare in that more practical scenario?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The theoretical derivation is elegant, connecting representation drift in a linearized model to a practical, data-free regularizer via the GGN and KFAC.\n- In the linearized fine-tuning setting, the proposed method performs on-par with or better than the data-dependent $\\tau$Jp baseline.\n- KFAC regularization at training time allows simple weight averaging (task arithmetic) to outperform more complex, SOTA post-hoc merging methods (like TSV) that are applied to unregularized vectors if training was in the linearized regime.\n- The method is shown to be computationally fast.", "weaknesses": "- The entire experimental validation is confined to the linearized finetuning framework (see, e.g., Ortiz-Jimenez et al., 2023). The authors don't test their KFAC regularizer on full, non-linear finetuning. Is the regularizer only effective in a linearized regime?\n- The paper's validation against SOTA merging methods (TIES, TSV, etc.) is limited to this niche linearized FT setting (Figure 4). This avoids the most practical question: how do these methods perform on task vectors from standard, non-linear fine-tuning? These non-linear task vectors are the default, readily available artifacts in most real-world scenarios. If SOTA post-hoc methods applied to these standard vectors already outperform the entire KFAC/linearized-FT framework, the practical motivation for this work is unclear. The paper fails to demonstrate that its regularizer is necessary or effective in the standard, non-linear setting.\n- (Minor) The performance gains on T5 are less impressive, and the data-dependent alternative ($\\tau$Jp) is better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "StHTqN4xCZ", "reviewer": "Reviewer_1ihd", "summary": "The paper proposes a data-free regularizer for Task Arithmetic (TA) that penalizes cross-task representation drift using a curvature surrogate. Starting from the linearized network around the pre-trained weights, the drift penalty becomes a quadratic form in the Jacobian Gram / GGN matrix; the authors approximate it with KFAC, yielding a per-layer Kronecker structure that is cheap to evaluate during fine-tuning (Eqs. (3), (6)). They further introduce a merged regularizer that aggregates per-task KFAC factors into a single surrogate to keep cost constant in the number of tasks (Eq. (8)). Experiments on CLIP (8-Vision) and T5 show strong gains for TA addition/negation and competitiveness with τ-JP—without needing other tasks’ data. Notably, ViT-L/14 reaches 91.6 abs / 99.3 norm with α=1, and their method excels at task negation while preserving control accuracy. The paper also claims minimal training overhead and code release.", "questions": "1.\tTLSF baseline: Can you add results against Efficient Model Editing with Task-Localized Sparse Fine-tuning on the same 8-Vision and T5 setups, plus an editing-localization metric (e.g., Δ outside-task vs in-task) to demonstrate task localization superiority?\n\n2.\tAttention-only FT: Could you replicate the Fine-Tuning Attention Modules Only protocol (datasets, α-grids, disentanglement error maps) to enable a direct SOTA comparison under its claims? \n\n3.\tTheory for Eq. (8): Any conditions (e.g., approximate independence of factors across tasks) under which the Kronecker factor summation is a principled approximation to (\\sum_t B_t\\otimes A_t)? \n\n4.\tMemory/precision trade-offs: How do FP16/8-bit factor storage and block-sparsity affect accuracy and the normalcy score? Could you move factors off-GPU during training without bottlenecks? \n\n5.\tScale to LLMs: What are the empirical costs when applying the method to 7B–70B-parameter LMs (rough factor sizes, setup time), and do your MC/num-examples findings (Fig. 7) still hold?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tConceptual clarity: The drift penalty is cleanly derived from the linearization, tying representation drift to the GGN and enabling reuse of mature curvature approximations (Sec. 3.1–3.3). \n\n•\tScalability: The merged KFAC surrogate (Eq. 8) achieves O(1) cost in tasks and empirically matches the naïve O(T) sum. Table 3 shows near-parity on ViT-B/16 and T5. \n\n•\tStrong empirical results: On 8-Vision, KFAC-regularized TA outperforms linear and non-linear FT, matches or beats τ-JP in several settings without external task data, and yields robust α=1 behavior (Table 1). It also improves task negation (Table 2). \n\n•\tTask-localization signal: The paper links its regularizer to a practical normalcy score (|J_\\theta f(x,\\theta_0)\\tau_t|_2^2) that separates in- vs out-of-task inputs (Fig. 5). \n\n•\tPracticality: Overhead analysis and ablations (Fig. 6–7) are useful; MC=1 with ~128–256 examples often suffices for KFAC estimation.", "weaknesses": "•\tMissing baseline: No comparison to Task-Localized Sparse FT; given the shared goal (localized updates with low interference), this absence limits claims of state-of-the-art effectiveness on task-local editing.\n\n•\tAttention-only baseline not fully aligned: While “Non-linear (Attn.)” appears, the paper doesn’t replicate the full protocol and metrics of Fine-Tuning Attention Modules Only, so the reader can’t conclude whether KFAC-regularized FT beats that specific method under its strengths (e.g., disentanglement maps/metrics across α-sweeps, stability under merging). \n\n•\tHeuristic merge lacks theory: The Kronecker aggregation (Eq. 8) is a heuristic; no approximation error bounds or conditions are provided, though it works empirically. \n\n•\tMemory footprint: The Limitations section concedes KFAC’s quadratic-in-units per layer memory cost, which could bite for very large LMs; compression or structured factors are left to future work. \n\n•\tLanguage results trail τ-JP: On T5, the approach narrows the gap but τ-JP still tops the leaderboard in places (Fig. 3a), hinting the curvature surrogate could be further improved for text. \n\n•\tPrivacy/deployment details: Since the method proposes sharing curvature stats instead of data, the paper would benefit from concrete guidance on what to share (sizes, precisions, any leakage risks), especially beyond CLIP/T5 scales. (Code release is stated but details aren’t in the main text.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKvq2e1gxR", "reviewer": "Reviewer_RdRs", "summary": "This paper introduces a data-free approach to improving task arithmetic grounded in Kronecker-Factored Approximate Curvature (K-FAC), a second-order optimization approximation that captures local curvature information in parameter space.", "questions": "1. I got confused by the naming of “Dataless” because even linear finetuning requires data, and gradients/Jacobian require data.\n2. In Eq(5), is $R(\\theta)$ also removed in nonlinear finetuning?\n3. Can you discuss the relationship of GGN and other Fisher or Hessian matrix based methods like “Efficient Model Editing with Task-Localized Sparse Fine-tuning.” (Iurada, Leonardo, et al. 2025) and “Merging Models with Fisher-Weighted Averaging” (Matena, Raffel, 2022)?\n4. Notation: In line 179, what does the coefficient matrix refer to here? In line 218, what is $s_{n,m}$?\n5. Eq 8: What’s the memory and runtime cost of this approximation and can you explain any intuition of this approximation?\n6. In Figure 5, wonder why some of the 8 standard vision tasks like SVHN are missing.\n7. In Table 1 & 2, include the results of Porrello et al. (2025)’s numbers for a more straightforward comparison.\n8. Line 369 “naive” typo.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Quality: Good. The paper presents a well-motivated and technically sound contribution, with strong theoretical grounding in second-order optimization and empirical evidence supporting its claims. Experimental evaluation is relatively comprehensive, covering diverse architectures and tasks, with consistent metrics and clear ablation studies isolating the contribution of curvature-based disentanglement.\n2. Clarity: The paper is very well written and easy to read. \n3. Significance: The proposed solution is efficient and effective (on par with the best provided TA method for vision datasets, yet slightly lower performance on language datasets). Its performance on both standard finetuning and linear finetuning shows its flexibility.", "weaknesses": "1. Minor Novelty concern because this paper seems relatively incremental as it’s built upon the key ideas of Ortiz-Jimenez et al., 2023 and Yoshida et al. (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6OhZGFkr70", "reviewer": "Reviewer_K5Dr", "summary": "This paper proposes a new method for weight disentanglement in task arithmetic. To address the limitations of prior constraints on cross-task representation drift—which often require access to auxiliary data during fine-tuning and incur high computational cost—the authors approximate the penalty via Kronecker-factored decomposition. This lets fine-tuning proceed without explicit data, needing only each layer’s input covariance and output-gradient covariance, while substantially reducing compute. Experiments across both vision and NLP tasks show that the approach matches or surpasses the strong baseline $\\tau$Jp.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear presentation: The paper is well structured, easy to follow, and provides accessible explanations of the theoretical derivations.\n\n- Novelty of bringing KFAC into task arithmetic: It reframes the traditional representation-drift penalty as a curvature (GGN) approximation under squared loss, then applies a KFAC approximation to make the method more data-efficient during fine-tuning and significantly cheaper computationally.\n\n- Comprehensive empirical evaluation: Beyond merge performance versus recent state-of-the-art methods, the experiments also cover cost analysis, comparisons with linearization-based approximations, and ablations over different KFAC estimation choices.", "weaknesses": "- Dependence on other-task statistics (A, B): While the method mitigates the need for raw auxiliary data required by τJp, it still relies on access to precomputed input/gradient covariance factors from the other tasks to be disentangled. This dependence may limit applicability in settings with privacy or access constraints and introduces coupling to external task-specific artifacts.\n\n- Missing non-linear regime comparisons to SOTA: The paper compares against recent methods such as TSV and ISO only in the linear regime, not in the non-linear regime where those methods are primarily positioned. If performance is comparable to—or worse than—those SOTA approaches, the practical advantage of the proposed method remains unclear, especially since TSV/ISO can merge using parameters alone whereas the proposed approach requires auxiliary task statistics.\n\n- Underspecified regularization strength selection: The KFAC regularizer’s weight is fixed per task/dataset, but the paper does not clearly describe how these values were chosen nor provide sensitivity or robustness analyses. This gap raises concerns about reproducibility and about how performance depends on the strength of the regularization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)