A Second-Order Perspective on Model Compositionality and Incremental Learning
Authors: Angelo Porrello
Lorenzo Bonicelli
Pietro Buzzega
Monica Millunzi
Simone Calderara
Rita Cucchiara
Venue: ICLR 2025 Spotlight (2025)
Keywords: Continual Learning
Model Compositionality
Ensemble Learning
Task Arithmetic
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We explore compositionality in fine-tuning non-linear deep models, revealing that staying within the pre-training basin is key to creating effective incremental learners.

Abstract: The fine-tuning of deep pre-trained models has revealed compositional properties, with multiple specialized modules that can be arbitrarily composed into a single, multi-task model. However, identifying the conditions that promote compositionality remains an open issue, with recent efforts concentrating mainly on linearized networks. We conduct a theoretical study that attempts to demystify compositionality in standard non-linear networks through the second-order Taylor approximation of the loss function. The proposed formulation highlights the importance of staying within the pre-training basin to achieve composable modules. Moreover, it provides the basis for two dual incremental training algorithms: the one from the perspective of multiple models trained individually, while the other aims to optimize the composed model as a whole. We probe their application in incremental classification tasks and highlight some valuable skills. In fact, the pool of incrementally learned modules not only supports the creation of an effective multi-task model but also enables unlearning and specialization in certain tasks. Code available at <https://github.com/aimagelab/mammoth>

Published as a conference paper at ICLR 2025
A SECOND-ORDER PERSPECTIVE ON MODEL COMPOSITIONALITY AND INCREMENTAL LEARNING
Angelo Porrello', Lorenzo Bonicelli', Pietro Buzzega', Monica Millunzi!
Simone Calderara' Rita Cucchiara 1.3
University of Modena and Reggio Emilia, Italy AxyonAI, Italy SIT--CR, Italy
ABSTRACT
The fine-tuning of deep pre- pre-trained models has revealed compositional properties, with multiple specialized modules that can be arbitrarily composed into single, multi-task model. However, identifying the conditions that promote composition- ality remains an open issue, with recent efforts concentrating mainly on linearized networks. We conduct a theoretical study that attempts to demystify composition- ality in standard non-linear networks through the second-order Taylor approxima- tion of the loss function. The proposed formulation highlights the importance of staying within the pre- pre-training basin to achieve composable modules. Moreover, it provides the basis for two dual incremental training algorithms: the one from the perspective of multiple models trained individually, while the other aims to optimize the composed model as whole. We probe their application in incre- mental classification tasks and highlight some valuable skills. In fact, the pool of incrementally learned modules not only supports the creation of an effective multi-task model but also enables unlearning and specialization in certain tasks. Code available at phttps://gitte
com/ a i image ab/ mammot.h
INTRODUCTION
In the last two decades, AI technologies have predominantly been viewed as monoliths, centered around single model trained on a single (albeit large) dataset. This paradigm has undeniably yielded impressive results and is likely to continue doing so. However, to meet the numerous chal- lenges of business, more modular and flexible approaches, such as model averaging (McMahan et al. (017) and Mixture Of Experts (Yadav et al., (024a), have experienced a renewed interest. In fact, the life cycle of monolithic models demands incremental and intensive updates to accommodate the diversity of markets, platforms, and customer needs. Moreover, many companies lack the finan- cial means to assemble a proficient Al team or access GPU clusters to support the rapid evolution of these factors. In an effort to democratize AI, advocate for modern paradigms (Li al. 20236] Pfeiffer et al. 2023) prioritizing fast adaptability, model customization, and data heterogeneity.
Recent frameworks (Liu & Soatto, 2023 Bowman et al. 2023) support these principles through model compoxitionality, allowing the creation of bespoke models through cheap editing opera- tions. Surprisingly, fine-tuning pre-trained models not only enables efficient transfer with limited data iZhuang et al.||2020) but also offers insights into compositionality (Ilaacco et al. 2023). In fact, it has been shown (Wortsman al. 2022) that simple linear combination of individually fine-tuned weights yields meaningful outcomes. The resulting composed model exhibits rich and robust repre- sentations (Zhang & Bottou, 2023) without incurring additional inference or memory costs. Such a simple but powerful schema has been primarily employed in model soups (Wortsman et al.] 2022).
con- trast, Task arithmetic (Ilharco et al. 2023 Liu & Soatto 2023 -Ortiz-Jimennn et al.) (024) addresses a cross- cross-d setting with each individual model focusing on a distinct task. Such a framework has also been recently explored for language modeling through ColD) fusion fusDon (Don- Yehiya et al.) 2023).
where individual models are trained on the same task but with different hyperr-paaaeetee
In this context, we elaborate on two points. Firstly, considering multiple models trained individually on different tasks, we aim to understand the conditions that allow the successful combination of their weights. This finding has been predominantly explored empirically (Ilaarco et al. (202)., with a few notable exceptions (Ortiz-Jimenez et al. 2024) that aim to provide more theoretical understanding.

---

Published as a conference paper at ICLR 2025
However, these approaches only focus linearized networks and tangent fine-tuning (Liu & Soatto 2023). We instead aim for a formulation that generalizes to standard non-linear deep networks and diverse fine-tuning strategies (e.g.. LoRA (Hu et al. 2022)). ). To do so, we propose leveraging the second-order Taylor approximation of the loss function around the pre-training weights. This enables to derive relationship between the capabilities of individual models and those of their composition, offering a key takeaway. Specifically, for the composed model to be accurate, each component should attain decent performance on examples outside its individual training distribution.
The second point of our study is a derivation of the former and revolves around learning compos- able models in an incremental fashion, dedicating an individual module to each incoming task. In this respect, previous works (Bowman et al.] 2023] Liu & Soatto (023) build on the intuitive idea that modularity provides a natural foundation for incremental training. Through model editing and composition, in fact, one can simply introduce novel concepts without having to re-train the model. In this work, we turn the perspective and argue that compositional capabilities require incremental learning skills. Indeed, mentioned above, preserving for compositionality. Nevertheless, this can be framed as a continual learning problem (Kirkpatrick et al. 2017), where the objective of each individual model is to maintain the pre-training general capabilities on examples that fall outside its specific training distribution.
performance is essential
On this basis, we propose two algorithms that tackle the incremental fine-tuning of composable models. Both use the same second-order formulation but approach the problem from opposed per- spectives: optimizing the loss of each model individually vs. optimizing the loss of the composed model. Evaluated in the class-incremental setting (Van de Ven et al. (022). our approaches lead to an accurate multi-task model with further editing capabilities, which allow the efficient specializa- tion of the model on certain tasks, as well as the ability of removing some others (unlearning).
2 FRAMEWORK
We consider [:; 0) : :X - as a twice-differentiable deep network with weights 0 E θ C R takes inputs x E X' C Rd and yields a conditional distribution pa myyla (z)) over the targets y E y R In this paper, we focus on incremental training, which progresses sequentially through a series of T classification tasks T =1,2, (1.2..... ........ T.. Each task t T is characterized by a dataset Dj with n, training samples x. y ~ (x, y) drawn from a distribution varying across tasks. We assume that tasks share the same loss function {(0 ar, y), i.e,, the negative log-likelihood loglo(
In this setting, we maintain a pool of composable networks p (:) 0, _ +.+TE where each model is fine-tuned from a common set of pre-training weights 0o. The task vector (Iharcco let al., 2023) Tr indicates the displacement in weight space w.r.t. Og after training on task t. We obtain the weights of the composed model fo by averaging the weights within the pool:
fp A f(;0p) s.t. Op 0o DT WiTe, Di, =1
where Wy balances the contribution of the t-th learner. While some works (Asadi et al. 2024 Huang ct al. 2024) optimize these coefficients, we devise uniform weights we 1/T in our algorithms.
Scope. How can we learn multiple disjoint tasks through a pool P of models, so that the composed model performs well on their union? To answer this question, we introduce the concept of empirical risk, i.e., the average loss (())D), computed over the union D = U D. of all training tasks:
C-T ((( Σو. En.yED (00 x y) a.y-
[E(0 x. y)]
To simplify notation, we will henceforth omit the explicit dependence of the loss the data, de- noting the individual loss ( y) simply as {(0), and the empirical risk DD) as ((0). On this basis, the rest of this section will delve into the following research questions.
Question i) Given the empirical risk of each individual model, what can we say about the composed model ({:0p)? Question ii) How can we train individual learners f(: 6,) on distinct tasks (individual training) to still achieve reliable composition ʃ(: 0p)? Question üij Instead of optimizing each model on its individual loss, could we optimize each model based on the loss of the whole composed model f(; 0p) (ensemble training)?
2

---

Published as a conference paper at ICLR 2025
2.1 INDIVIDUAL LEARNERS VS. THE COMPOSED MODEL: A PRE-TRAINING PERSPECTIVE
We now relate the composed model fp to the individual components ( 0₂) of the pool. To do so, we introduce the second-order Taylor approximation (0) of the loss around the pre-trained weights 0g:
((0) teur (0) + 0 0o113)
(3)
1 (0 0n) Ho)) On).
where lour (0) O0) + (0 0o)² V +
(4)
Ve(0o) A Vol(On) xr y) and (,) - V (0 mr y) are the gradient and the Hessian around 0o. Similarly, we can define the second-order approximation (o) ~ {(0) of the empirical risk, which corresponds to averaging the approximated loss across examples from all tasks.
Assumption. We now assume that O = 0n is a point of local minimum of the empirical risk ( across all tasks (Fq. 2. Under this hypothesis, the Hessian of the empirical risk H2(0,) > 0 is positive semidefinite. In light of this and the quadratic nature of Leur (0). we can state that the second- order approximation of the empirical risk (0) is convex. Therefore, we apply the Jensen's inequality to derive a relation between the composed model and the individual components:
(5) The relation states that, under the second-order approximation. the empirical risk (0p) of the composed model is upper-bounded by the convex combination of its individuals. In other words, if each individual model trained to the optimum with near-zero loss value, there are some guarantees on the loss function attained by the composed model. Notably, this relation could help reduce the computational footprint during inference, as it enables the reduction of forward passes, from multiple (one for each individual) to singular pass (performed the composed model).
Leur (Op = 0. + EL=w) I Σ We (eu = 0u + Tt).
At a first glance, the result of Eq. appears similar to the statement of Eq. 2 in (Liu & Soatto 2023
i(pp 0g + Σ Ei= WiTe S Ei w/ 0, + +T)) given that fr 0,) _ fün (3 0.) (((;0)) - (0, 0o) 'v "V [1 On) (langentness)
However, some notable distinctions remain. Their inequality applies to the exact risk but is valid only for linearized models (i.e., fine-tuned in the tangent space of pre-training weights). In contrast, our result pertains to the second-orde approximation leur of the risk and applies to any fine-tuning strategy (e.g.. LoRA, adapters, etc.). Intuitively, our inequality provides more flexibility to the train- ing of individual learners, as long as: i) the learners remain the pre-training basin, such that Ol1 0n (| + O and four can be considered good proxy of é; üi) 0 is local minimum of f.
2.2 ENABLING INDIVIDUAL TRAINING IN INCREMENTAL SCENARIOS
As mentioned above. a possible application of Eq. is to devote each learner to a distinct task and optimize them in isolation (i.e., individual training). Indeed, the upper bound in Eq. 5 describes sort of worst-case scenario for the risk of the composed model: at worst, collapses to that given by the upper bound. Nonetheless, if every individual model were accurate all tasks, the right-side would yield an appealing upper-bound to the risk of the composed model.
term Eq. B
Limits of Eq. 5 Under the constraints of individual training, each individual learner f(): can be optimized only on the current distribution (x y). Therefore, when considering examples from other data distributions &r (a:, y), the loss four (0, (ar, yy of the I-th individual learner is likely to be much higher for examples outside its training distribution Pe Pr(a: (x, y). As a consequence, the upper bound delivered by the right-side of Eq. iss likely to increase one task after the other, and we cannot rely on it to recover reliable composed model f(: Op).
In this respect, our proposal is to tighten the upper bound of Eq. through explicit regularization, which we devise during the optimization of each individual learner. In practice, each model is
As shown in Ostapenko et al. (2022). techniques like linear probing. latent replay, and incremental linear discriminant analysis can he employed to enforce the optimality of the base model fg. along with Instruction Tuning for language modeling (Yadav al. (0224c). See Sec. for additional implementation details.
3

---

Published as a conference paper at ICLR 2025
provided with learning objective that extends beyond minimizing the loss on the assigned data dis- tribution (r. y). Specifically, to ensure decent performance on external distributions pu (z, y), anchor the model to the pre-training knowledge for out-of-distribution examples. We do so by encouraging the predictions of [(; to be close to those generated by the pre-trained model f(; 0o). given examples x. y (u y). Denoting the pre-training posterior distribution as pe. (y)x), we have:
minimize 0:
(r. [euur a y) + PKL (po, (3)(z) (ylz))
(8)
It is noted that the computation of the KL term (-L(-) requires sampling from the external distribu- tions p (Ir, y). Theoretically, this clashes with the constraints of individual training. Thankfully,
if + 0, the KI. term can be approximated (Chaudhry et al. 2018) with the distance between O, and pre-training weights 0o:
DKL (pe, (3) (/)z)) pe, (3yla)) ~ (0, 0o) TF. (0, 0o) 1
(9)
The distance is computed in the Riemannian manifold (Lee 2006) induced by the Fisher Information Matrix (FIM) (Huszár 2018); namely, (0] x [0] positive semi-definite matrix given by:
Fe. E T-T (m.
Ey Eyrpa, (w)
V log pa, (y ar) V log Pa, (y]
(10)
Under the local minimum hypothesis on pre-training weights, the FIM at Og equals the expected" Hessian of the negative log- ligelikeliood (Martens, 2020 Fo, = Ez (0.). Due to this connection, the FIM yields insights on the sensitivity of each parameter to changes in the data distribution.
Application to incremental learning. Given that optimizing the regularization above does not necessitate data from external tasks, it can be readily adapted to incremental learning. Follow- ing Chaudhry al. (2018); [Schwarz al. (2018), Firstly, we limit to estimate the diagonal Fe, of the FIM, thus avoiding the prohibitive footprint required to treat a |0] x [0] matrix. Basically, the diagonal Fo, consists of Monte Carlo estimate of the (squared) gradients of the log-likelihood. Secondly, we note that the expectation Ez (H, in the FIM cannot be directly estimated, as examples from all tasks are not available simultancously but rather sequentially. We hence compute the expectation incrementally (Chaudhry et al. 2018 Schwarz et al., 2018) one task at a time. As each new task becomes available, we calculate a lo- cal Fisher matrix the data of the new task and then accumulate it into global running Fisher estimate. As outlined by Alg. [ the accumulation can be thought as a simple summation, net of re-normalizing operations reflecting the number of samples of each task (see App-Ei Eb.
introduce few additional approximations.
Given Eq. and the diagonal FIM, the augmented optimization problem for the f-th learner becomes:
minimize 6x
Leeur (0 xr 9) + EWCo,
(11)
(w..))
EWC,, (0.) = E! P( (0 -
where
(12)
where 2 0 is an hyper-parameter and EWC,, (-) indicates the Riemannian distance from the pre-training weights Og. The acronym EWCa, highlights the strong analogy with Elastic Weight Consolidation (Kirkpatrick et al. 2017). well-established approach against catastrophic forget- ting (McCloskey & Cohen 1989). In sense, term prevents forgetting pre-training knowledge; however, while our anchor is fixed at og. the anchor of EWC instead shifts and focuses on the weights learned during the preceding task.
2.3 JOINT TRAINING OF THE COMPOSED MODEL IN INCREMENTAL SCENARIOS
Individual training profitably aligns with decentralized learning (Bowman et al. 2023), emphasizing minimal interactions between learners and privacy preservation. Nevertheless, it might be inefficient when these constraints are not of interest and the goal is simply to create an accurate model. In fact, individual training prevents a learner from leveraging the knowledge of other ensemble members, climinating the potential for beneficial mutual transfer. For these reasons, we adopt the dual per- spective, in which each model directly optimized using the loss of the composed model f(; 0p) (ensemble training). Since the inequality in Eq. 3 does not provide much help for the explicit opti- mization of Lur (0p), we quantify the exact gap between the two sides of the Jensen's inequality:
Precisely, we take the expectation w.r.t. data from other tasks I' of to reflect the regularization in Eq.
4

---

Published as a conference paper at ICLR 2025
Theorem 1. Let us assume pool P with T 2 2 models, with the t-th model parameterized by 0, 0, + Tt. If we compose them through coefficients wi:--., WT s.1. wt E [0, 1] and and_ >i=1 We 1,
the 2nd order approximation (cur((p) evaluated on composed weights 0p = 0n + =-1=1 WT is:
Lour (0p) + 2((1....,, = -i=1
(13)
where 22 Qr) = -i= E - -.)) "H Tr).
(14)
Proof in App. B The equality introduces term S2(-) that is non-negative (due to Hz)O,) > 0) and
depends on weights oi...., 0y. Therefore, 2(-) is proportional to the cumulative distance between every pair of learners, within the Riemannian manifold induced by the Hessian H H(())) (Lee) 2
Notably, Eq. 13 permits to draw the following insights: i) optimizing both the loss of the composed model for (p) and the term 1(-) is equivalent to individual training; ii) during inference, if (?(-)
tends towards zero, performing a prediction with the composed model is akin to conducting multiple forward passes, each corresponding to an individual learner. Based on that interpretation, we now transition to setting that considers the explicit auxiliary minimization of 52(-), as follows:
minimize 01....0rr
E t-T [Peur (cur(077 /z, y) + B Ii .......................... 0r)]
(15)
a.y--p(2x.y)
where B 2 0 is an parpmeerr is noted that minimizing 2(-) encourages alignment among task vectors, especially for those weights that are sensitive/impootant in the pre-training loss land- scape.
Similarly to Jeffares et al. (2024), the objective of Eq. 15 can be interpreted as smooth transition between individual and ensemble training. Indeed, given that S2(-) = We - Leur Lo))
Eq. 15 can be restated:
(enr(pp) + 8 (2(0))......... θT) (1- B) Lour (0p) + We 1 (0.)
(16)
This suggests that by minimizing both the loss (0p) of the joint composed model and ((-), we also implicitly optimize the individual models. Notably, this result not only paves the way for a well perforrrfg ensemble but also for components that are reliable when considered individually.
Incremental ensembles. We now refine the problem in Eq. 15 account for incremental settings. We firstly bring the expectation inside 22(-) (Eq. [4] and replace the Hessian (Oo) with its expec- tation, taken across data points æ, y ~ pe( y from all tasks up to the current one. Afterwards, capitalize on property mentioned in Sec. 2.2 which states that, for point of maximum like- lihood like = o. the expected Hessian of the negative log- ligeeeillhood coincides with the Fisher Feo- As a result, we can approximate Ex (0))) with the tractable diagonal Fisher Fe.- Based on He = Ee, and further steps (see App. B we can rearrange !?(-) as: 22(-) ~ (0p .r) = 1
w.) EWC,, (0.) (00))_-. EE WONT To, Te. (17)
Intuitively, (.) aims to preserve pre-training knowledge embodied by Og through the first term; through the second one, instead, it encourages pairwise alignment between task vectors. Also, we highlight that the summations depend on the number of models T. Therefore, the more components we manage within the ensemble, the more crucial it becomes to minimize (p(() within Eq. 15
Finally, plug the approximated barrier term (p(+) into the optimization problem outlined in Eq. [5] As learning proceeds in subsequent tasks, we can optimize the composed model only one task a time. To prevent biasing the previously learned components of the pool toward current data, each round we optimize only the weights O, A On + Ti of the corresponding t-th learner, while freezing the others components of the pool. This yields the following form for the 1-th learning task:
minimize (mz.y-p(( [eerr (0p) æ. 3)] + B (0.. 0.).
(18)
0. Finally, if we optimize only one 0, at a time and w; = 7. then several terms in Eq. 17 can be ignored, such as FWCo, (0p) ) for t' < f. Hence, minimizing Eq. 18 is equivalent minimize:
Ex.y-p. (z. v) Heur (0p je, y)] + (1 +) EWC,, (0.) - Ee T. T
(19)
As shown in App. for both full and LoRA fine-tuning, the gradients of the regularization term in Eq. can be computed in closed form. Importantly, this derivation enables a lightweight optimiza- tion, as the analytical gradients maintain constant complexity w.r.t. the number of tasks.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Incremental Task Arithmetic (TTA) vs. Incremental Ensemble Learning (IEL.
Input: T disjoint classification tasks D: [ me a pre-trained DNN f(; learning rate hyper-parameters and 3, initialized pool p 0 and the diagonal Fisher Fo, 0.
for cach task & E 21, T do
h + Linear Probing on D, with the pre-trained f(; 0o)
Task pre-consolidation
0g < 0o U 1hg
add the pre-training classification head
Feo - Fop + +
b update the global Fisher with the local Fisher Fi(t) estimated on D.
Ti .N (Hiuii, dinit)
Fine-tuning extend with the weights the t-th learner
P - P (f O. Go + T
for each example (r,, y) in D, do
po(y) +- (=; 0.)
> predict with the 1-th learner 0g <- 0o + T
Po x)) +- (w: 0p)
predict with the composed model Op <- Oo + Ei-
+ log po (z))
T. - TI - 1r Vr, [e + EWCo, (O.))
b arithmetic-oriented regularization (Eq.
Tt - T: -r V [e + (Pp(
> ensemble-oriented regularization (Eqs. 18 and 19
ALGORITHM(S)
We present Incremental Task Arithmetic (ITA) and Incremental Ensemble Learning (IEL), two distinct algorithms for individual and ensemble incremental learning respectively. As shown in Alg. ! they divide each task into the pre-consolidation and fine-tuning stages, with differences occurring in the latter. We direct the reader to App. E for comprehensive implementation guidelines.
Task pre-consolidation. Due to the pivotal role of the local optimality of 0u (Sec. 2.1 we now follow up on this aspect and consider closed vocabulary models. Unlike models like CLIP (Radford) et al. 2021 closed vocabulary models require the addition of tailored classification head to handle novel classes. In line with Ortizz-Jimenzz et al. 20244,, denote the process of fine-tuning only the added classification head as linear probing (LP): specifically, LP keeps the rest of the layers fixed the pre-training Og. In our work, we basically exploit linear probing to enforce the optimality of pre-training weights 0g. Namely, during the mre-consolidation phase, we conduct few preliminary training epochs on the t-th incoming task, with the sole purpose of fine-tuning the new classification head. From that point onward, the fine-tuned head h is regarded as a part of pre-training weights Og. Finally, the pre-consolidation stage concludes with the update of the diagonal Fisher matrix Fo.-
Fine-tuning, While the pre-consolidation step is identical for both ITA and IEL, they differ during the fine-tuning phase. The shared goal is to learn task vector т: s.t. O, On- + for the current task. However, ITA treats 7; as the weights of an individual model, whereas IEL interprets it as a new learablee component of the composition. In other words, ITA computes predictions through the t-th learner (a; 0z). while IEL leverages the composed function f(æ; 0p). Moreover, their regularizing objectives differ: ITA builds upon Eq. 11 (i.e., the additional EWC-like term computed w.r.t. Oo), while IEL exploit Eqs. [8 and 19 to train the composed model. Notably, both approaches can be applied to any fine-tuning strategy in the form of 0, + AO. We conduct experiments on Full Fine- gFooo Foor XFIN) ), Low-Rank Adaptation (LoRA) (Hu et al. 2022) (i.e., = A, and on (IA) (Liu et al. 2022). About the latter, let h represent the hidden dimension and E R denote the (IA) task specific vector: it can be shown that (IA)3 is equivalent to 0 = Op + T with T. = 0. O ((1 - 1s) @ 1x), where 1, is the all-ones vector. For each fine-tuning strategy, we initialize their parameters so that the resulting task vector T starts null vector at the beginning.
Tuning (i.e., T, T,E R
Computational analysis. As outlined in App. D by treating the composed model as a cumulative average of individual models, both training raaining/iffrrence stages of ITA/IEL maintain constant complex- ity (1) with respect to the number of tasks T. Indeed, a single forward pass is sufficient to compute the output of the composed model (constant time). Moreover, we do not need to store separate set of weights for each task (constant memory), provided we are not interested in more complex forms of composition than the simplest uniform average (as required for specialization and unlearning).

---

Published as a conference paper at ICLR 2025
4 RELATION WITH EXISTING WORKS
Task arithmetic. Standard and Aarameter-Efficient (PEFT) fine-tuning have been shown to support addition/subbraationn of task vectors. However, while the evidence in (Zhang et al. 2024; Ilharco et al. 2023) is primarily empirical, we derive theoretical insights about the pre-conditions for task arithmetic, emphasizing the importance of staying close to the pre-training basin. In this respect, our derivations ground previous findings regarding the efficacy of low learning rates (Ilharco et al. 2023 Ortiz-Jimenez et al. 2024). Remarkably, staying within the pre-training basin has also been proved beneficial in (Sadrtdinov et al. (0224) for ensemble learning. The conditions for compositionality are also studied by "Ortiz-Jimenee et al. (222)) on linearized networks (Eq. 7p. Albeit considering this work inspirational, we see the pros of task arithmetic in the non-linear regime. Firstly, non-linear models surpass their linearized counterparts in single-task accuracy, making them more attractive. To reduce the gap, iinerization-awar fine-tuning has to be used (Liu & Soatto, 2023), contrarily to our approach that is compatible with the prevalent fine-tuning techniques. Secondly, linearized inference requires the demanding Jacobian-vector product (three times slower than a forward pass).
Ensemble learning. While original model soups (Wortsman et al. 2022) combine multiple weights fine-tuned on the same dataset, we herein managed to unlock running model soups in cross-dataset incremental settings, with possible returns in terms of forward transfer (Lopez- Paz & Ranzato 2017). The optimization of the whole deep ensemble is also discussed in Jeffares et al. (2024) for standard ensembles, i.e., averaging the outputs of different models. Their derivation is similar to ours and regards the decomposition of the ensemble loss into the strength of the individual learners and their diversity. However, Jeffares et al. (2024) use this result to elucidate the shortcomings of jointly Hfofever,Jeffares
trained deep ensembles, whereas we leverage it to provide effective regularization for model soups in incremental scenarios. Similar to our IEL, several works (Li et al.) 2022, 2023a] [Schmidt et al., 2023) build an ensemble through cumulative mean of intermediate checkpoints sampled along the training trajectory, with benefits in terms of generalization and preservation of zero-shot pre-training capabilities (Zheng et 20233. of models trained with varying configurations (e.g., data augmentations). They also gradually push each weight toward the population average, thus encouraging alignment across individual models. Notably, such a behaviour is also positively rewarded by our second-order formulation (see Eq. 17
Jolicoouu Martineau et al. (2024) maintain population
Incremental learning. Adding new knowledge in deep networks often degrades the ability on ear- lier tasks. Approaches against forgetting can be divided into three groups: those involving regular- ization (Kirkpatrick et al. 2017 Zenke et al., (017), those retaining old data for rehearsal (Lopez- Paz & Ranzato| 2017 Aljundi et 2019)) and those allocating new modules (Mallya & Lazebnik] 2018 Abati et al., 2020). Among the latter, SEED (Rypeśé et al., 2024) manages an ensemble of expert networks learned incrementally, sharing similarities with our IEL. However, SEED stores separate models and combines their outputs, whereas our approach maintains a cumulative weight average of past experts, minimizing memory and computational overhead. This running average can also be achieved by InfLoRA (Liang & Li 2024), which addresses interference among LoRA modules through tailored initialization. In contrast, our method addresses this issue through explicit regularization. In addition, a recent trend capitalizes on prompt-tuning (Wang et al., 2022a b) Smith et al. 2023). ) devising pool of learnable prompts. Notably, the extent to which these models sup- port compositionality is investigated in Perera et al. (2023) Bowman et al. (023) propose A-la-carte Prompt Tuning (APT), an attention mechanism that enables the creation of bespoke models by com- posing arbitrary prompts. Their setting called à-la-carte aligns with the scope of our ITA; however, our work extends to a broader range of PEFT modules, unlike APT that is limited to soft prompts.
NaK-based Incremental learning. The authors of Tangent Model Composition (TMC) (Liu & Soatto| 2023) build on the foundational work of (Ortiz-Jimenee et al. (2024) to address incremental learning. They enforce task arithmetic across subsequent tasks through linearization-wwanl fine- tuning, which entails a first-order Taylor approximation of the output function around 0g. In this context, each task of TMC is effectively equivalent to training a kernel predictor using the Neural Tangent Kernel (NTK) (Jacot et al. 20118), defined as KNTK (z, x²) Vaf(I; 0o) Vof(z'; 0o). In contrast, our approach employs a higher-order approximation of the empirical risk, thereby focus- ing on the geometry of the loss landscape (Chaudhry et al. 2018). Notably, other recent works (Liul ct al. 2024) have adopted the NTK framework to tackle incremental learning: e.g., TKIL (Xiang & Shlizerman, 2023) exploit the NTK formulation to align representations for current and past tasks.
TV)
7

---

Published as a conference paper at ICLR 2025
Table Comparison with SOTA (Final Accuracy [t]). Best results in bold, second-best underlined. EWC, LwF-MC, DER++ (buffer size of 1, 000 examples), SEED, and TMC rely on full fine-tumne -unnttnn L,P,, CODA, and APT utilize prompt-based based learning. Finally, InfLoRA adopts LoRA fine-tuning.
Model
IN-R
C-100
CUB
Caltech
MIT
RESISC
CropDis.
Joint
86.08
91.74
87.12
93.21
87.19
96.79
99.71
Finetune
22.31
21.57
29.35
44.03
18.85
12.90
20.54
EWC
58.64
73.49
40.33
73.23
64.44
58.80
70.33
LWF-MC
50.93
72.16
21.88
79.73
67.04
68.09
78.28
DER+
60.53
83.02
76.10
86.11
68.88
67.23
98.77
L2P
67.17
87.32
78.95
91.22
83.17
63.47
75.18
CODA
74.12
86.48
78.54
90.57
77.73
69.50
74.65
SEED
55.87
83.39
85.35
90.04
86.34
74.81
92.77
APT
65.32
86.19
69.51
87.71
75.83
49.99
59.37
InfLoRA
76.97
87.17
79.14
90.53
79.14
79.92
89.05
TMC
60.01
78.42
71.72
82.30
68.66
60.66
66.56
ITA-FFT
76.43
89.38
84.80
92.32
85.35
80.50
91.81
ITA-LORA
77.79
89.96
85.55
92.65
86.60
82.00
95.85
ITA-(IA)
77.04
90.66
85.67
92.67
84.74
83.73
95.41
IEL-FFT
80.09
89.38
84.89
92.23
82.79
81.42
95.83
IEL-LORA
79.93
89.53
84.95
92.19
84.49
82.53
95.88
IEL- (IA)
77.86
89.72
84.57
92.70
85.54
81.50
95.68
EXPERIMENTS
Datasets. Following affine works (Wang al. 2022b Bowman al. 2023 Liu & Soatto
2023
we evaluate on these class-incremental benchmarks: Split ImageNet-R (Hendrycks et al. 2021
(10 tasks 20 classes each). Split CIFAR-100 100 (Krizhevsky al., 2009) (10 tasks x 10 classes),
Split CUB-200 (Wah et al. 2011) (10 tasks x 20 classes), Split Caltech-256 (Grifin et al. 2007)
(10 tasks, in (Liu & Soatto| 2023)), and Split MIT-67 (Quattoni & Torralba 2009) (10 tasks, as
in (Liu & Soatto, (023)). We conduct further tests on the aerial and medical domains using Split
RESISC45 (Cheng al. 2017) (9 tasks classes) and Split CropDiseases (Hughes et al.) 2015 (7
tasks x classes). They provide a challenging benchmark for our proposals due to their low domain similarity with the ImageNet pre-training (Oh et 2022). Further details can be found in App. G
Benchmarking. We compare against recognized incremental methods as EWC (Kirkpatrick et al, 2017). LwF-MC (Rebuffi al.
2017), DER++ (Buzzega et al. 2020b), I.2P (Wang al. (022h) and CODA (Smith et al.] 2023) We also asses SEED (Rypesé et al. 2024). ofRA (Liang & 2023). and TMC (Liu & Soatto 2023). four approaches featur- ing compositionality and ensemble learning. Their description and the main differences from our approaches are provided App. FF. All methods, including ours, utilize the same backbone
Li 2024). APT (Bowman et al.
ViT-
B/16 (Dosovitskiy et al., 2021) with supervised pre- pre-training on ImageNet2IK (Ridnik et al.] (021) and the same batch size (128). We compute the accuracy on all classes at the end of the final task (Final Accuracy, FA). Following Buzzega et al.) (2020b), the hyperparameters are chosen through a grid search on validation set (i.e., 10% the training set). This procedure was repeated for each method, ensuring careful tuning of the search space. See the supplementary for 1) the standard deviation of the FA (the results are averaged over three runs, see App. H.2); 2) the results expressed H.1 3) the selected hyper- papperrerraeterss (App.[).
Final Forgetting (Chaudhry et al. 2018) (App.
Comparison with SOTA. As reported in Tab. 1 ITA and IEL. outperform existing approaches on all datasets except MIT-67 and CropDisease. At times, they match SEED; however, its reliance on Mixture of Experts makes its inference computationally demanding. Our results far exceed those of TMC, showcasing the potential of the non-linear regime over linearization. Considering the good re- sults on RESISC and CropDis., ITA and IEL do not seem affected by large domain shifts, indicating that our formulation remains effective even when the pre-training optimality is challenged. Finally, given the comparable results ITA and IEL, we recommend ITA as the preferred starting point for future research, in light of the greater flexibility offered by the individual training paradigm.
8

---

Published as a conference paper at ICLR 2025
Table 2: For ITA, analysis of the impact of the proposed regularization loss (FA [t].
Model
IN-R
C-100
CUB
Caltech
MIT
RESISC
CropDis.
ITA-FFT (reg)
76.43
89.38
84.80
92.32
85.35
80.50
91.81
without Eq. 12 reg.
8.61
17.59
10.47
12.76
12.01
17.17
20.64
Eq. 12 only on CLS
76.00
87.60
83.54
91.04
81.45
75.26
77.00
ITA-LoRA (reg)
77.79
89.96
85.55
92.65
86.60
82.00
95.85
without Eq. 12 reg.
50.17
66.58
60.58
74.87
52.74
37.59
55.86
Eq. 12 only on CLS
77.33
90.03
85.55
92.59
84.86
80.64
96.22
On the impact of regularization. Tab. reports the results of ITA when the regularization term in Eq. [2 is removed (the same analysis for (TA) and IEL is available in H).. To evaluate the effect on distinct layers, we additionally assess the model with only the last classification layer regularized (see Eq. 12 only on CLS in Tab. 2. As observed: i) applying Eq. i2 beneficial for all examined fine-tune tuuing strategies; ii) although regularizing all layers is the most consistent approach, applying Eq. 12 only on the classification head already yields good accuracy. This suggests that compositionality in closed-set models is largely dependent on the learning dynamics of the last layer. Finally, full fine-tuning (FFT) struggles the most when no regularization or partial regularization applied, in contrast to PEFT modules like LoRA and (IA)3 that still manage to achieve decent results. We ascribe this evidence to the tendency of PEFT modules to forget less of the pre-trained knowledge (Biderman et al. 2024), feature we identify as beneficial for model compositionality.
To gain insights into our regularization, we re- visit the foundations of ITA: specifically the inequality in Eq. [ which states that the risk
Impact of ITA reg. (Eq.
40-
of the composed model (0pp) is upper bounded by the weighted risk of the individ- ual models Ewreeu((B) In practice, there is
Empirical risk
a5-
Upp: bound
3.
no guarantee that this upper bound is tight: as
LG
these individual models are trained on disjoint tasks, their risk is likely high for examples out- side their training distribution. We hence ex- pect the upper bound to be loose: this is shown in Fig. T where we measure the effect of our regularization on the empirical risk of the com- posed model = and on the upper bound zzz, From this lens, ITA significantly tightens the upper bound, with a remarkable reduction of the risk associated with the composed model (0p)) Notably, the latter consistently surpasses the pre-trained model f (0.) (obtained with linear probing). This indicates that there a good margin to find point in parameter space that improves pre-training Vice versa, the upper bound Σ Eurkeur (f.) does not exceed pre-training, even with regularization applied, raising the theoretical possibility of regret between the composed model and pre-training. This work provides empirical evidence to address it but a theoretical analysis is needed, which we leave for future works.
0.5
IN.R
C-100
CUB
Caliehh
MTT
RESISC
Cropt
((6)
Σ (2)) (w/o meg))
Σ (1 (- meg
1(66) [w/o rep.)
((p) (+ reg))
Figure 1: Effect of ITA. Best viewed in color.
On the compositional skills of ITA and IEL. We herein investigate specialization and unlearning capabilities of our approaches. Specifically, given the task vectors trained on the sequence of tasks, freeze and use them to edit the composed model. As these adaptations involve simple additions and subtractions of task vectors, we can perform specialization and unlearning in a zero-shot fashion requiring no examples. In particular, through the former test on specialization. we examine whether composing only subset of the task vectors can boost the composed model on the respective selected tasks. In doing so, we focus on three tasks of the incremental sequence i.e., the first, the central, and the last one. The performance is then measured considering the average FA across the tasks we wish to specialize in (FATGT) ) and the average FA across the others (FACTRL). ). In the second test, instead, our goal is to eliminate knowledge of specific task without degrading performance on other tasks (unlearning). Specifically, to unlearn a given task, we subtract its associated task vector from the composition, which still includes the weights of the tasks that need to be preserved. Afterwards: i) we measure the accuracy for the unlearned task (FATeT ) and the others (FACTRI.) ); ii) the procedure is then repeated for all tasks, with the performance averaged across them.

---

Published as a conference paper at ICLR 2025
Table 3: Analysis of compositional capabilities App. Horr results on other datasets. In paren- theses, we report the gain (or loss) in accuracy on the target task.
zero-shot specialization
zero-shot unlearning
Dataset
Model
FATCT [t]
FACTRI.
FATGT [4]
FACTRI.
ITA-LORA
80.83 (+11.40)
50.52
22.77 (-55.02)
52.72 ( (-25.07)
IN-R
IEL-LoRA
73.46 (-06.68)
38.46
18,55 (-61.88)
41.97 (-37.96)
TMC
69.93 (+-08.36)
34.08
45.77 (- 14.24)
54.37 (- 06.64)
ITA-LORA
92.80 (+-1169)) 66)
60.06
28.67 (-61.29)
71.96 (-17.99)
C-100
IEL-
77.77 (- 13.22)
37.90
19.48 (- 70.05)
56.52 (- 33.01)
TMC
87.53 (+-06.49)
45.75
55.63 (-22.79)
71.83 06.59)
The results are in Tab. and Tab. of App. H. where we compare with TMC by Liu & Soatto 202.3) ie.. model linearization and tangent fine-tuning. Regarding specialization, both ITA and TMC improve the accuracy the target tasks. IEL, instead, leads severe drop during specialization. This yields an interesting finding: while Tab. hhihhlgghs its performance as satisfying, we observe that the ensemble struggles when any of its members removed. In an era where ensemble models are experiencing renewed attention (Liang & Li, 2024) Marouf et al.) 2024), this evidence should be sobering. On the other hand, underline the specialization capabilities of ITA, with the best absolute performance on the target tasks. We ascribe it to the devised regularization objective and the natural advantages of the non-linear regime. When focusing on unlearning, ITA shows mixed results. Notably, both ITA and TMC consistently reduce the performance of the task to be forgotten. While this also affects the tasks that should be preserved, ITA exhibits a greater distinction between the target and control tasks, indicating a more effective disentanglement of knowledge across tasks.
6 DISCUSSION OF LIMITATIONS AND FUTURE DIRECTIONS
Second-order Taylor approximations are valid tools make the theory more tractable. For instance, a similar approach was used by Mirzadeh et al.] (2020) to support the importance of wider local minima against forgetting. Nonetheless, these approximations often rely on certain concessions, the first regarding their validity during training. Since our approximation is performed at the pre-training weights, it may become inaccurate as parameters drift. While importance-baset partially mitigate the drift, other techniques could be used (e.g., L2-norm penalty task vectors or low learning rate). Even without explicit countermeasures, existing studies suggest that deep networks often fall into a regime where the loss tends to be almost convex in neighborhood around their local minima (Goodfellow et al., 2015)] Lucas et al. 2021: Yunis et al., (022)). Also, under some conditions, their training dynamics can enter the lazy regime (Chizat et al. 2019 Jacot al., 2018), where these models rapidly achieve near-zero loss with minimal changes to their weights.
terms like ours may
Moreover, two other concessions warrant a discussion: approx. we implicitly model the weight distribution with a Gaussian (Chaudhry et al. 2018) (Eq. 9): approx. ii) the Fisher matrix is ap- proximated by its diagonal (Eq. [Tp. Regarding approx. i). while non-Gaussian posteriors are often cumbersome to manage, Farquhar et al. (2200) challenge the common belief that mean-field approx- imations are overly restrictive for deep networks. Moreover, although approx. ii) may seem crude, the diagonal approximation is efficient, with a low memory footprint. Still, our approach could profit from more accurate estimations of the Hessian, like the Kronecker factored approximation (Martens) & Grosse 2015), which considers the interactions between weights of the same layer.
Future work. Through theoretical and empirical analyses, we support the importance of remaining within the pre-training basin to achieve composable deep networks. However, there is still much to explore along this path. We mainly focus on closed-set classification models but it would be note- worthy to extend our analysis to self-supervised pre-training and open-vocabulary models like CLIP. Indeed, recent works (Zheng et al. 2023) have shown their tendency to forget zero-shot pre-training capabilities while fine-tuning. In this respect, our recond-order regularization could significantly aid compositionality in CLiP-based models. Finally, we believe that staying within the pre- pre-train basin is only one aspect to consider; future research should emphasize the exploration of the pre-train basin, to achieve composable modules with higher degree of specialization on their respective tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work has been supported by the PNRR-M4C2 (PE00000013) project "FAIR - Future Artificial Intelligence Research", funded by the European Commission, and by the Italian Ministerial grant PRIN 2020 "LEGO.AI: LEarning the Geometry of knOwledge in Al systems", 2020TA3K9N We acknowledge the CINECA award under the ISCRA initiative, for the availability of high- performance computing resources and support.
REFERENCES
Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learn- ing. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2020.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 2019.
Nader Asadi, Mahdi Beitollahi, Yasser Khalil, Yinchuan Li, Guojun Zhang, and Xi Chen. Does combining parameter-efficient modules improve few-shot transfer accuracy? arXiv preprint arXiv:2402.15414, 2024.
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Con- nor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2022.
Benjamin Bowman, Alessandro Achille, Luca Zancato, Matthew Trager, Pramuditha Perera, Gio- vanni Paolini, and Stefano Soatto. A-la-carte prompt tuning (apt): Combining distinct data via composable prompting. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2023.
Pietro Buzzega, Matteo Boschini, Lorenzo Bonicelli, and Angelo Porrello. Mammoth ex- tendible (general) continual learning framework for pytorch, 2020a.
com/ aaaaammaaaellll mammmoth
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe- rience for general continual learning: strong, simple baseline. Advances in Neural Information Processing Systems, 2020b.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Bugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In Interna- tional Conference on Learning Representations, 2022.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European Conference on Computer Vision, 2018.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 105(10), 2017.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 2019.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine- -ne-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018.

---

Published as a conference paper at ICLR 2025
Shachar Don- Yehhya, Elad Venezian, Colin Raffel, Noam Slonim, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Reprexentations, 2021.
Sebastian Farquhar, Lewis Smith, and Yarin Gal. Liberty or depth: not need complex weight posterior approximations. Advances in Neural Information Processing Systems, 2020.
Deep bayesian neural do
Thomas George. NNGeometry: Easy and Fast Fisher Information Matrices and Neural Tangent Kernels in PyTorch, 2021. URL htLLhttp://ooio www.10...rg/..... 5281/ wttp:////.../////.o......... zenodo. 4532597
lan J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. In International Conference on Learning Representations, 2015.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. CalTech Report, 2007.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. IEEE International Conference on Computer Vision, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du and Min Lin. Lorahub: Effi cient cross-task generalization via dynamic loRA composition. In First Conference on Language Modeling, 2024.
David Hughes, Marcel Salathé, al. An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv preprint arXiv: 2015.
Ferene Huszár. Note on the quadratic penalties in elastic weight consolidation. Proceedings the National Academy of Sciences, 115(11), 2018.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in Neural Information Processing Systems, 2018.
Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela van der Schaar. Joint training of deep ensembles fails due to learner collusion. Advances in Neural Information Processing Systems, 2024.
Xisen Jin, Xiang Ren, Daniel Preotiuc- Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations, 2023.
Alexia Martienaa,, Emy Gervais, Kilian Fatras, Yan Zhang, and Simon Lacoste-Julies Population parameter averaging (PAPA). Transactions on Machine Learning Research, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 11 114(13), 2017.
12

---

Published as a conference paper at ICLR 2025
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approxi- mation for natural gradient descent. Advances in Neural Information Processing Systems, 2019.
John M Lee. Riemannian manifolds: introduction to curvature, volume 176. Springer Science & Business Media, 2006.
Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable weight averaging: Efficient training by optimizing historical solutions. In International Conference on Learning Representations, 2022.
Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable weight averaging:
A general approach for subspace training. arXiv preprint arXiv:2205.33104. 2023a.
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A 2023b.
survey, arXiv preprint arXiv:2309.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-fin low-rank adaptation for continual learn- ing. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2024.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 2022.
Jingren Liu, Zhong Ji, YunLong Yu, Jiale Cao, Yanwei Pang. Jungong Han, and Xuelong Li. Parameter-efficient fine-tuning for continual learning: A neural tangent kernel perspective. arXiv preprint arXiv:2407.17120 2024.
Tian Yu Liu and Stefano Soatto. Tangent model composition for ensembling and continual fine- tuning. In IEEE International Conference on Computer Vision, 2023.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
James Lucas, Juhan Bae, Michael Zhang, Stanislav Fort, Richard Zemel, and Roger Grosse. Analyzing monotonic linear interpolation in neural network loss landscapes. In International Conference on Machine Learning, 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018.
Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted en- semble models are strong continual learners. In Proceedings of the European Conference on Computer Vision, 2024.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21, 2020.
James Martens and Roger Grosse. Optimizing neural networks with kronecker- approximate curvature. In International Conference on Machine Learning, 2015.
Michael S Matena and Colin A Raffel. Merging models with weighte averaging. Advances in Neural Information Processing Systems, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
13

---

Published as a conference paper at ICLR 2025
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication learning of deep networks from decentralized data. In International Conference Artificial Intelligence and Statistics, 2017.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. Advances in Neural Information Pro- cessing Systems, 2020.
Jachoon Oh, Sungnyun Kim, Namgyu Ho, Jin-Hwa Kim, Hwanjun Song, and Se- Young Yun. Un- derstanding cross-domain few-shot learning based on domain similarity and few-shot difficulty. Advances in Neural Information Processing Systems, 2022.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-traincs models. Advances in Neural Information Processing Sys- tems, 2024.
Oleksiy Ostapenko, Timothee Lesort, Pau Rodriguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, and Laurent Charlin. Continual learning with foundation models: An empirical study of latent replay. In Conference on lifelong learning agents, 2022.
Pramuditha Perera, Matthew Trager, Luca Zancato, Alessandro Achille, and Stefano Soatto. Prompt algebra for task composition. arXiv preprint arXiv:2306.003 2023.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Ponti. Modular deep learning, Transactions Machine Learning Research, 2023.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. IEEE, 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Sylvestre-AIvis Alvise Rebulli, Alexander Kolesnikov, Georg Sperl, and Christoph H Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2017.
Lampert. icarl:
Tal Ridnik, Emanuel Ben-Baruch Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k, pretraining for the masses. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Grzegorz Rypešč, Sebastian Cygert, Valeriya Khan, Tomasz Trzcinski, Bartosz Michal Zieliński, and Bartlomiej Twardowski. Divide and not forget: Ensemble of selectively trained experts in continual learning. In International Conference on Learning Representations, 2024.
Ildus Sadrtdinov, Dmitrii Pozdeev, Dmitry P Vetrov, and Ekaterina Lobacheva. To stay not to stay in the pre-train basin: Insights on ensembling in transfer learning. Advances in Neural Information Processing Systems, 2024.
Fabian David Schmidt, Ivan Vulić, and Goran Glavaš. Free lunch: Robust cross-lingual transfer via model checkpoint averaging. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), 2023.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barwiissa, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for contin- ual learning. In International Conference Machine Learning, 2018.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonillla Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention based prompting for rehearsal free continual learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2023.
14

---

Published as a conference paper at ICLR 2025
George Stoica, Daniel Bolya, Jakob Bjomer, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. In International Conference on Learning Represen- tations, 2024.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12), 2022.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-200 dataset. Technical Report CNS-TR-2011-00 1-001, California Institute Technology, 2011.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Proceedings of the European Conference on Computer Vi- sion, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2022b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022.
Jinlin Xiang and Eli Shlizerman. Tkil: Tangent kernel optimization for class balanced incremental learning. In IEEE International Conference on Compuler Vision, 2023.
Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. arXiv preprint 0770072408.000557 2024a.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 2024b.
Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint arXiv:2410.03611 2024c.
David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT: Optimization for Machine Learning (NeurIPS 2022 Workshop). 2022.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on pre-trained model. In IEEE International Conference on Computer Vision, 2023.
Jianyu Zhang and Léon Bottou. Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning, 2023.
Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arith- metic operation. Advances in Neural Information Processing Systems, 2024.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vinion-language models. In IEEE Interna- tional Conference on Computer Vision, 2023.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1), 2020.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX / SUPPLEMENTAL MATERIAL
The appendix is organized via the following contributions:
App.
provides the proofs for Theorem and Eq.
App-
illustrates that gradients of Eq. T9 be derived in closed-form.
App. D
reports the computational analysis for our approaches Incremental Task Arithmetic
(ITA) and Incremental Ensemble Learning (IEL..
App. E describes the essential details required reproduce the methods described in the
main paper.
Apps. and G supplements the main paper with additional information regarding the rela- tion with existing works and the adopted benchmarks.
App- provides additional experimental results: the forgetting metric, the standard devia- tion measured in each experiment, an additional ablation study regarding IEI,, an extensive evaluation of model compositionality across multiple datasets, analysis of the similari- ties between task vectors learned by ITA and IEL, and a timing analysis.
App-Doutlines the hyperparameter configurations adopted in our experiments.
B PROOFS
B. PROOF OF THEOREM I
Proof. For the sake of notation, we will use the shortcuts t A ((O)), J, ve(0)). and H, A H. (6a). Given setting with only two learners and B, the second-order approximation of the loss function of the composed model P is:
Leur (0p) Eo + -0pp 0o) To)Jee
2 (0p 0o) TT) He 0o)
1 (0p (H( 0n)
(wa wb) to 0p 0u Je +
2
W tisTT
walo + wolo + (WaTa + (.)))) Thp +
TwaTa + wbTT) "e + wST)
2
wwlo + wplo + wa J, WWT TJ,
+ (WaT T + ubT H + wT)))
wallo + To. Jrl + wb L4 To² Je] +
(wa Ta TTe + WbT H. (Ua u.
= =walo + T.- Jæl + wr + To JJl +
(w Ta HET. + H
w. WbT T
1 H Tu + Wa WTb HeTa)
Given that To H.T. is a scalar and (ABC)T- = (CT A
- = (TT) (T/ He TTa) Tb H.T
symm.
(cur(pp)) -Walfo J[] wb (eo To
T Jel
? H.T. w HHT 2w, W. H.T))
2
w.llo + - wy
1lo + To Jel
+
w HeTa tw HETD + w6 HHT.)
2
w w. (1- =. un2
1
= =waloo + + + T. HeTal + wplko + -. Jzl
(wa - HeT,
2
WTTTT HHTT. w. TH H 2w. wh 7
(., H.
16

---

Published as a conference paper at ICLR 2025
1 To H H + 1 2T. HeTal + wbl6o +To² J + wallo + Jo
Ty
2
1
2 Wol HeT + HeT 2TT H(T.)
= Walcur (0a) + Wylcur (0g)
w Wb T HeTo + Th H.T, - 2T H(T))
wy (0a) + Wrlcur (0g) WoWblTo - - " - -T))
In the multiple learning setting, we have that 0p = 00 Tp = 0p + Σ We with Ei Wt = 1. Therefore:
1
tour (0p) = lo + (0p- - 0n) Jr +
-0p 0o)
"H - 0o)
2
T
W/ to + Tp JJ
1
TP
"HTT
Σ
w +
wiTh
TJ +
1 TP HHT
2
1=1
t=1
T
1 Wt [lo + Tr "J]] + TTP *HTTp.
Let us now focus on the quadratic term:
T
T
Tp
HTTP
WeTs
H
WiTi)
W
wiTc)
t=1
1=1
1=1
T
T
T
Wei (W)
+
Σ WeTe))
= Σ
Wi
T
HeT + wT He Σ WoTe
1=1
tyt
1=1
T
T
T
Σ
Wi T:
H +
Σ
W
He Σ WH
t=1
t'
T
Σ w 2 Tt H.T. Σ
W. T HATH'
Since T, T "HTTY
T.. T
H.T. (symmetry)
w 2 "HTT +
Σ
W WeWp T HeT
Given that w.² WT - Wi w,(1
T Leee we) Wt wi Etrge we:
W
W:
Wp
Ti
HH, +2
W.Wp Ti H
t' e'
1=
Σ W.. H
Σ
Wh.
Wy
Ti TH +2
Σ
Wowe T2 HETH
t=1
t'ft
t=,
T
T
Σ W+ H Ti Σ>
Will T H Ti + 2
Σ
WPwe Ti H
1'stt
t=
Σ T H
Σ
W.Wy (T. T HET: + T H +2 Σ
wowe T HeT
t=
t=
17

---

Published as a conference paper at ICLR 2025
T
10
H,
Σ
Wy T H + T H,Te) - 2w, Ti T HH
t=,,'<t
T H
Σ
Willy (T: T HOT: + Te HOTe - 2T''Ho HeTe)
t=,
WiTi
H.T
Σ
wiyy (Tr (TT-T))) TP)
11,
Therefore:
1 lour (0p) = to t - 000) +
=00 Ho) - 0o)
2
Σ Wt (bo + Ti JJ] + TT
1
"HTTP.
T
T Σ
Σ We (oo + Ti [J]] +
Σ
W
HeT:
W.Wr - -rr) Tx)
2
t=1
t=,,t<t
1
Σ
Wt Lo + Tr TJ
+
w HeTe
Σ
WrWr - - Tp)
2
[=,,'<t
WeLour (0.)
Σ
Wilf (Tr - -r²) TH(TT Tr),
1=1,'<
which ends the proof of Theorem
B.2 PROOF OF EQ.
Proof. In the dual- setting we have:
A(A, B) 2nth WAWB(TA "B) (H,()) TB).
we replace the Hessian matrix with the diagonal Fisher matrix Fe.. we get:
2(A, B) W WAWWB(T
- -TB) - - (TA-TB) =
2
TA To, 2TA Fo, TB + TB T FF.
w WAWB
we focus on the first term inside the parenthesis, we obtain:
TA TF 0.
1/2 8o
(F
(F//2
TA) = =11F 0o TAl2
TA
TA
0o
TA =
00 TA
[0al
10
Σ
10
((())
00) EWC,, (0A)-
6 A
i=1
Therefore, rewrite Ω(A, B)
as:
1
2(A, B) 2 WAWB EWC,, (0A) EWC,, (0z) 2TA TF TB
We now generalize to > 2. Starting from Eq, 14 and replacing the Hessian with the diagonal Fisher approximation Fe,, we obtain:
Σ (l
WT: TV) ) He(T: Tc)
18

---

Published as a conference paper at ICLR 2025
T
( + EWC,, - -2T,² TF Foo Tt
Wy Wy
(=,,'<t
Σ
MyWp [EWCa, (0.) + EWC,. (0:)) - Σ
w.We T Fo. Tr
2
|=,,'<t
f=1,/<t
w.Wp EWCo,
Σ
wiWH Fo,
11,
t=,,
Σ Wy E.Co,, (0.)
Σ
wwwe Ti F. TP
Wy
t'pht
t= 1,,
1-
T
T
Σ we - wx) EWCo,
Σ
wiWp
Tv:
2
which aligns with the result of Eq. 17
C CLOSED FORM GRADIENTS FOR EQ. 19
We start by deriving the gradients of Sg (-) (Eq. 14 w.r.t. the generic task vector Tk- We initially consider the case of full-fine tuning, and then generalize the results to LoRA and (IA)
To simplify the calculations, first rearrange (-) as:
(0,.......... ... 0r)
.r) = 2 1=1 Pet
wwwe (T. Tt. TFo (Te Te)
we wx) (O.)
Σ
W.w To, Tr
(see Eq.
t=1,00
Therefore, write:
02p 0 06 Ei=1 wo(1 w.) EWCo, (0:)
0 2
e/E "f To
ÔTk
0
1xx 012
wk) EWCo, (6x)
(20)
Ôn. alErerwwuy F. TV + S Fo. Te
FTh
3 F.Co. (0c) w/ (1 wx)
FT.
WKWF
ÔTk
0Tk
t'
T
ww (1 [[w) Foo
Tk
Wk
Wy Fe.
T
T
wk (1 wx) F Foo 2 Tk Σ
W. F 0o
Tt
t'
wk Foo
[11
wk)
Tk
we
T']
(21)
19

---

Published as a conference paper at ICLR 2025
We now suppose that training is performed incrementally a sequence of 1, 1.2.... .......
..... T tasks.
During the k-th task, IEL optimizes only Tx. namely the task vector instantiated at the beginning of the k-th task. Indeed, as discussed in Sec. (.3 [9), the task vectors Ti, T2..... 7k-1 introduced in preceding tasks are kept frozen. Moreover, at each round, we devise uniform weights, such that wy E with t. < k. On top of that, we can rewrite the gradients in Eq. 20:
a ÔT
Foo @
[11
1 T TH
Σ t<k
(22)
Notably, the term 7 Eick Te is a running average the preceding task vectors. As also discussed in App. D this reduces the memory/time computational cost for computing the gradients of S2p to (1).
As final step, and (IA) For LoRA, we have that:
exploit the result in Eq. 22 to compute the gradients of LoRA (Tk BBAA))
on On θTk
Fo. 2
[1
Σ
To
AT
0B
ÔTk 0Bk
t<
02
Oe θTk
BBT B"
F @ [[1
1) -
ET1
0A
ÔTg dA,
k
In the case of (IA) 3, where Tk 0o
((4
1a) 2 1d):
an
8n θTk
1
Fo.
@
[(1
T T
Σ
To
3
0
la
ÔTz 04
h
T
t<k
where 1a is a d-dimensional column vector of ones with shape d 1.
D COMPUTATIONAL ANALYSIS
Analysis of ITA. Regarding ITA (i.e., individual training), the learning phase has constant ((1) time and memory complexity. As each model is optimized in isolation, the training cost is similar that of EWC (Kirkpatrick et al., (017)) and stems from the computation and storage of the Fisher Information Matrix, along with the calculations of the penalty term. During the evaluation phase, the time complexity of ITA remains ((1), as it involves a single forward pass on the composed model f(: 0p). Memory complexity, with distinct expertise is desired (e.g., for later re-composition). Otherwise, this can be avoided by considering the composed model as cumulative average of individual models (see later), resulting in memory cost of (()).
the other hand, is O(T)) if maintaining separate models
Analysis of IEL. Referring to IEL (i.e., ensemble training), it might initially appear expensive due to the joint training of the ensemble. However, the complexity of IEL. remains constant to (L)) in terms of both memory and time. This reduction is intuitive when we view the ensemble as a cumulative average of individual weights. To demonstrate this, we begin by considering the following cascade of models to be learned:
Task - f(; 0p 0p TI)
1 2 T2)
Task - f(; 0p = 0
n
+
2
1
Task #: - f(:0p 0 -
T-- T()
At each task, only the latter component of the composed model learnable, while the preceding components are frozen:
Task + f(:0pp = Ou + Ti +
+
T
T learnable
frozen components
20

---

Published as a conference paper at ICLR 2025
Therefore, the preceding #t 1 components are kept frozen, incorporate them into the initialization weights 0, - 0 and optimize:
1 1-1
1 Ta f(: Op 00" + 7.) where 00¹h Ou + Σ Te.
Under this perspective, the learning of the 1-th task is comparable to standard fine-tuning with re-scaling factor 1/t. Therefore, provided that 6 is computed once the beginning of the t-th task, the learning of the t-th task has constant (1) time complexity.
noted that optimizing Eq. 19 via gradient descent can be similarly simplified, resulting constant (1) time complexity. In fact, the gradients derived in App. [ involve averaging the weights learned during previous tasks, specifically 1 Er Tr'. Since the weights from previous tasks are frozen during the current task, we can compute this average once and cache it.
To reduce memory complexity to (1), must avoid storing Tı. Ti..... T--1 separately. This can be accomplished by assuming an initial null displacement To = O and redefining 08* as:
1-1
t-
1-1
0 Oo
Σ
Te Go + TO Σ P=1
Tr Oo + 7
T
1=1
V'=0
0o
(t) TAVG
(1) The term TAVG = 7 -0 Tr is basically the cumulative average of the displacements up to the
current task (excluded). The cumulative average is straightforward to compute and, as is well- known, eliminates the need to store all previous values appearing in the sum, with resulting memory complexity (1))
E IMPLEMENTATION DETAILS OF ITA AND IEL
Task pre-consolidation Linear Probing. consolidation phase, we train new classification head to account for the classes introduced by the current task. While the rest of the backbone remains frozen, new linear classification layer is then trained with standard Stochastic Gradient Descent (SGD) for varying number of epochs, depending on the dataset (typically either 3 or 8 epochs, as detailed in App. [..
the beginning of each task, during the pre-
With standard linear probing, the newly trained classification head can be considered reliable only for the classes of the current task. However, it may be miscalibrated for classes from previous tasks. This misalignment could undermine the role of pre-training in regularization. To mitigate this and enforce the hypothesis of pre-training optimality for the global empirical risk across all tasks, we adopt classifier alignment approach inspired by Zhang al. (202)). For each new class, fit a class-specific Mixture multivariate Gaussians (MoG) model on the feature representations ob- tained from the frozen pre-trained model. To capture intra-class variations, we use K 5 Gaussian components per MoG. In subsequent tasks, we generate N 256 synthetic feature vectors using the respective MoGs. These generated feature vectors, encompassing both past and present classes, are then used to fine-tune the new classification head. This approach allows us to enforce the hypothesis of pre-training optimality without relying on input data from other tasks.
Task pre-consolidation Update of the FIM. Afterward, the diagonal Fisher Information Matrix (FIM) has be updated to incorporate new information from the current task. Similar to Schwarz et al. 2018). we consider the estimated FIM as an online cumulative average of the squared gradi- ents of the negative ligelikeli Unlike Schwarz et al. (2018), we do not introduce the hyper- parameter y to down-weight the importance of the previous estimate. Moreover, following Kunstner ct al. (2019), we compute the true FIM as defined in Eq. 10 i.e., taking the expected gradient on the prediction vector ý. This approach differs from the majority of existing methods (Schwarz al. 2018 Kirkpatrick et al. 2017 Chaudhry et al. 2018), which apply a further approximation by rely- ing the empirical FIM: i.e., only considering the gradient of the ground truth label. Consequently, our estimate is more accurate but requires multiple backward passes, but the computational impact of this operation can be significantly reduced through batch-wise parallelization in George (202))
21

---

Published as a conference paper at ICLR 2025
Fine-tuning Initialization. All methods utilize the same pre-training (supervised) Ima- geNet21K ('vit.base.patcc 6_224. augreg.innnnne from the "timm" library). Following the original works, we initialize the learnable parameters such that task vectors start from the pre- train initialization:
Full Fine-Tuning: we apply zero-initialization.
LoRA: we use Gaussian initialization for matrix A and zero initialization for matrix B.
(IA) ; we initialize the vectors / with ones.
Fine-tuning Loss function. Importantly, while our derivations regard the second-order ap- proximation feuz. the full loss l' instead employed in our algorithms. Indeed, as common in frameworks similar ours (Chaudhry al. 2018 Mirzadeh et al. 2020), the Taylor approximation is employed to build a surrogate of the exact loss that both accurate and mathematically tractable. However, when it comes practice, this proxy often relaxed, and the full target function is used instead for simplicity.
Following existing works (Smith al., 2023) Wang al. 2022b), employ the local cross- entropy loss as the learning objective. Given an example from the current task, while the standard cross-entropy loss considers logits related to all classes, including those learned in previous tasks, the local cross-entropy focuses only on logits corresponding to the classes introduced in the current task. This approach prevents the logits of past classes from being overly penalized during the current task (Caccia et al.] [0222] Boschini et al. 2022). To ensure fair comparison in our experiments, we apply this modification to other competing methods (e.g., EWC (Kirkpatriek et al. 2017
Fine-tuning Optimization. In each experiment, we use the AdamW optimizer (Loshchilov & Hutter 2019) with a learning rate of x 10 4 for LoRA and (IA)a fine-tuning, and 10 for full fine-tuning, Importantly, in both ITA and IEL, we employ decoupled strategy (Loshchilov Hutter 2019) incorporate the gradients of the regularization term. Specifically, we apply the gradients of the regularizing objectives directly to the parameters before the gradient update step, ensuring that the regularization term does not interfere with momentum the optimization dynam- ics. By adopting this approach, we observe empirical improvement in final accuracy, attributed to a more effective minimization of the Riemannian distance relative to the pre-training initialization (see Eq. [2). Finally, we apply this decoupled gradient update exclusively to LoRA and (IA) For full finettunig, we refrain from using it as we observed numerical instabilities (i.e., exploding loss).
Furthermore, decouple the regularization strength applied to the final classification layer from that applied to the rest of the learnable parameters. This introduces two additional byper-parametent OCLS for ITA and BCLS for IEL. Intuitively, by decoupling the regularization weights, we can increase the regularization strength of intermediate layers without causing numerical instabilities, which often stem from the final classification layer.
F EXTENDED DISCUSSION ON RELATED WORKS AND COMPETING METHODS
To deliver the most comprehensive and significant comparison with the state of the art in incremental learning, we chose a combination of well-established standard approaches (such as EWC, DER+, L2P, and CODA) and recent proposals emphasizing compositionality skills (e.g., TMC and APT) and ensemble learning (i.e., SEED). It is important to note that these methods have been heavily influenced by the research trends prevalent the time they were originally proposed, and therefore, they tend to employ different strategies for fine-tuning the model. To sum up:
EWC, LwF, DER++, SEED and TMC leverage full-fine tuning.
- L2P, CODA, APT resort instead to prompting.
Therefore, achieving direct apple-to-appl apple comparison is challenging, as the approaches present in the literature are themselves prone to this issue.
We herein summarize the main aspects of the more important recent methods we compared with.
Elastic Weight Consolidation (EWC) As discussed in the main paper, while our approach regu- larizes the distance in parameter space with respect to On. the regularization in EWC (Kirkpatrick
22

---

Published as a conference paper at ICLR 2025
et al. 2017) instead focuses on the weights learned during the preceding task. However, beyond the different regularizing strategies, there is another significant difference between ITA and EWC. While ITA fine-tunes each task starting from the same original pre-trained model 8,. EWC begins with the weights of the previous task.
It is noted that an EWC-like term that protects the last task weights could work as well in framework based on task vectors. We chose to anchor the model to the pre-training weights to allow for more flexible decentralized learning, wherein multiple task vectors can be trained on different tasks in parallel, with minimal interactions. In contrast, an EWC-like term, which regularizes each successor based on its predecessor, would necessitate training the multiple task vectors in sequence.
Learning to Prompt (L.PP) Wang et al. 2022b) and CODA-Prompt (Smith et al. 2023) are two continual learning techniques based on prompting. They fine-tune a pre-trained model through few learnable parameters stored in a prompt pool, which can be either shared or tied to different tasks. At inference time, the prompts are retrieved from the pool through query-key search. In terms of memory complexity, our ITA is in line with L2P and CODA when coupled with aurameter-efficien fine-tuning technique such as IA3, Moreover, ITA does not require the additional forward pass on the frozen pre-trained model required to retrieve the prompts.
A-la-carte Prompt Tuning (APT) [Bowman et al. 2023) is prompt-based strategy similar to L2P and CODA. The prompts are trained in isolation (similarly to ITA) and concatenated at inference time to create the composed predictor (no prior query-key search is required). To avoid destructive interference, tailored masking mechanism is employed in self-attention layers. Differently, our approaches fuse parameters through linear combinations.
Tangent Model Composition (TMC) (Liu & Soatto, 2023) is recent approach addressing con- tinual learning through task arithmetic in the tangent space. TMC builds upon task vectors and is similar ITA (full fine-tuning). They differ in two aspects: i) TMC applies a first-order approxi- mation of the forward pass to support compositionality, making it two to three times slower than a non-linear forward pass; ii) TMC does not include auxiliary regularization during training.
Selection of Experts for Ensemble Diversification (SEED) (Rypešć et al. 2024) is a recent ap- proach that trains an ensemble of models incrementally. For each incoming task, an expert model is chosen from the pool and trained. The major difference with respect to our IEL concerns the infer- ence stage: while IEL performs an ensemble prediction in (1) time (thanks to weight averaging), SEED makes inference on all models at test time and averages their predictions.
Model merging. While address compositionality during training, other approaches focus on post-training techniques, as simple averaging leads to interference (Yadav et al. (0224b when pa- rameters are redundant or have conflicting signs. TIES (Yadav al. 2024b) discards uninformative parameters and addresses conflicts via majority voting. Zipit! (Stoica et al. 2024) merges redundant parameters that produce similar features, while RegMean (Tin al. 2023) exploits a closed-fome form solution for linear layers. Notably, (Matena & Raffel (022) weighs the contribution of each param- eter through the Fisher matrix, computed by each individual model at its optimum. This differs from our approach that evaluates the FIM at the pre-training optimum.
G DATASETS
We conduct a comprehensive evaluation using variety of benchmarks. Following the current liter- ature on pre-trained CL models (Wang al., 2022b a Smith et al.) (2023), we include conventional image datasets such as Split CIFAR- 100 and Split IetgeNet-R. We also include Split CUB-200, Split Caltech-255 256 and Split MIT-67, recently used in the context of composable incremental meth- ods (Bowman et al.] 20233 Liu & Soatto) 2023). Finally, we assess the adaptability of these pre- trained methods in settings with decreasing domain similarity to the ImageNet pre-training utilized by our backbone model (Oh et al. 2022, Cui et al., 2018). Specifically, these settings include the satellite and medical domains, represented by Split RESISC45 and Split CropDiseases, respectively. In the following, outline the due details:
Standard domains: Split CIFAR-100 Krizhevsky et al. 2009 and Split ImageNet-
R (Hendryeks et al. 202T), with respectively 100 and 200 classes split into 10 tasks. We
23

---

Published as a conference paper at ICLR 2025
Table 4: Comparison with the SOTA across several benchmarks (Final Forgetting (JD.
Model
IN-R
C-100
CUB
Caltech
MIT
RESISC
CropDis.
Joint
X
K
X
x
X
X
Finetune
75.5
85.7
65.79
58.76
87.89
97.6
92.04
EWC (ON)
32.43
25.37
53.04
23.52
34.65
43.12
31.58
LWF-MC
10.39
6.13
26.2
5.23
3.56
5.66
7.94
DER++
35.81
15.66
13.58
11.19
31.51
43.04
0.76
L2P
5.6
6.08
6.63
1.78
5.55
26.29
17.81
CODA
4.97
5.28
8.82
2.64
7.49
22
15.37
SEED
8.96
6.39
5.12
2.67
4.53
8.66
1.19
InfLoRA
5.67
4.05
4.58
1.89
6.85
10.77
4.79
APT
8.99
6.71
8.82
3.64
6.33
27.74
14.01
TMC
11.66
9.2
7.37
6.39
12.43
16.43
16.94
ITA-FFT
7.72
3.89
1.39
2.44
4.80
6.19
0.95
ITA-LORA
8.00
3.22
2.08
2.38
4.68
8.30
0.68
ITA (IA)
8.72
3.57
2.23
2.39
4.48
7.33
0.76
IEL-FFT 5.40
2,20
2.04
1.50
2.29
3.46
1.40
IEL-LORA 7.63
5.14
3.46
2.46
8.34
8.30
2.14
IEL- 8.34
3.27
1.69
2.20
5.82
6.86
1.04
train each task Split ImageNet-R for 30 epochs and each task of Split CIFAR-100 for 20 epochs. In particular, IN-R is variant of the ImageNet dataset that includes artistic renditions such as sketches, cartoons, and paintings. It is used to evaluate the robustness and generalization capabilities of models trained the original ImageNet when tested on out-of-distribution data. Following (Liu & Soatto 2023). we also employ Split Caltech- 256 (Griffin al. 2007) and Split MIT-67 (Quattoni & Torralba, 2009) dividing both into 10 tasks (5 epoch each).
Specialized domain: We adopt Split CUB-200 (Wah et
2011 to evaluate compositional
capabilities in more fine-grained classification scenario, namely recognizing 200 species of birds. The classes are split across 10 tasks, each lasting for 50 epochs.
Aerial domain: we use Split RESISC45 (Cheng et al. (2017). which comprises 30000 RGB satellite images for land use and land cover classification. The dataset contains 45 classes (e.g.. airport, cloud, island, and so on) divided into 9 tasks, with each task lasting 30 epochs. Medical domain: we finally explore the medical setting (i.e., plant diseases) and conduct experiments on Split CropDiseases (Hughes et al. 2015). It regards infected leaves with tasks classes each (5 epochs).
We base our code on Mammoth (Buzzega et 2020b widely adopted framework in the class-
incremental learning literature.
H ADDITIONAL RESULTS
H.
FORGETTING
Tab.
reports the Final Forgetting metric (Chaudhry et al. 2018) for our experiments.
H.2 STANDARD DEVIATION OF FA
Tab.
reports the standard deviation of the Final Accuracy metrics reported in Tab.
H.3 ABLATION STUDY ON IEL
We herein present an ablative analysis regarding the ensemble-orientee regularization applied to the proposed IRI. approach. Specifically, evaluate the impact of Eq. 19 on the final accuracy and
24

---

Published as a conference paper at ICLR 2025
Table 5: For each experiment of Tab.
the standard deviation of the Final Accuracy (FA).
Model
IN-R
C-100
CUB
Caltech
MIT
RESISC
CropDis.
Joint
0.87
0.16
0.41
0.43
0.31
0.22
0.21
Finetune
4.53
2.83
1.65
2.57
3.49
1.12
0.43
EWC
0.73
1.20
0.87
1.29
1.45
0.61
0.99
LWF-MC
0.89
1.12
3.23
3.94
2.98
2.85
2.85
DER ++
1.12
1.14
0.82
1.27
0.95
1.79
1.29
L2P
0.56
1.23
2.12
0.63
0.81
3.42
0.45
CODA
0.63
0.95
2.87
0.82
0.69
4.61
0.98
SEED
0.18
0.47
0.65
0.55
0.41
0.86
0.23
APT
0.84
1.30
2.45
0.95
0.73
1.94
0.68
InfLoRA
0.10
0.21
0.89
0.68
2.03
1.16
0.42
TMC
0.45
0.85
1.67
0.71
0.92
0.82
0.81
ITA-FFT
0.27
0.27
0.39
0.18
0.24
0.56
2.10
ITA-LORA
0.26
0.20
0.13
0.16
0.40
0.89
0.55
ITA- 3
0.28
0.04
0.18
0.18
0.69
0.68
0.27
IEL-FFT
0.11
0.17
0.21
0.09
1.12
0.77
0.90
IEL-LoRA
0.36
0.27
0.37
0.41
0.93
0.30
0.43
IEL-(IA)
0.23
0.20
0.32
0.21
0.55
0.48
0.41
Table Ablation study for ITA- 3 and IEL on several benchmarks (FA [t)).
Model
IN-R
C-100
CUB
Caltech
MIT
RESISC
CropDis.
ITA- (IA)² 3 (reg)
77.04
90.66
85.67
92.67
84.74
83.73
95.41
without Eq. 12 reg.
71.82
88.43
77.61
90.66
69.14
69.01
63.72
Eq. 12 only on CLS
76.72
90.48
85.56
92.56
85.25
84.37
95.45
IEL-FFT (reg)
80.09
89.38
84.89
92.23
82.79
81.42
95.83
withou Eq.
19 reg.
40.85
52.56
14.02
53.76
47.63
39.20
31.24
Eq. reg. only on CLS
77.99
85.82
85.30
91.43
77.58
76.87
96.18
IEL- (reg)
79.93
89.53
84.95
92.19
84.49
82.53
95.88
without Eq.
19 reg.
51.15
66.
60.39
70.71
55.38
42.72
45.25
Eq. 19 reg. only on CLS
76.14
86. 86.11
84.43
91.77
82.50
70.05
95.54
IEL- 3IA)3 3 (reg)
77.86
89.72
84.57
92.70
85.54
81.50
95.68
withou Eg. reg.
73.72
84.00
74.72
89.58
69.82
62.52
66.29
Eq. 19
reg. only on CLS
77.23
89.38
84.70
92.76
85.43
81.60
95.72
report the results in Tab, 6 After examining them, we can draw conclusions similar those made for ITA.. In particular, the regularization driven by the second-order formulation proves beneficial for achieving effective composition (especially the full-fine tuning), with the final classification layer playing an important role in attaining good performance.
H.4 ADDITIONAL EXPERIMENTS ON INCREMENTAL MODEL COMPOSITIONALITY
Tab. 7 reports the results of ITA, IEL and TMC additional datasets, with focus their compo- sitionality capabilities (i.e., specialization and unlearning).
H.5 ALIGNMENT BETWEEN TASK VECTORS OF ITA AND IEL
To better understand the relationship between the task vectors learned by ITA and IEL, in Fig. conduct an additional evaluation by measuring the alignment in terms of cosine similarity between the parameters learned by the two proposed approaches. The analysis is twofold: first, the alignment is evaluated between individual task vectors of !TA and JLL by averaging the similarity
25

---

Published as a conference paper at ICLR 2025
Table 7: Analysis of compositional capabilities. In parentheses, report the gain (or loss) in accuracy on the target task.
zero-shot specialization
zero-shot untearning
Dataset
Model
FATGT [T]
FACTRL
FATGT [1]
FACTRL
ITA-LoRA
80.83 (+-11.40)
50.52
22.77 (-55.02)
52. 72 (-26.07)
IN-R
IEL-LORA
73.46 (-06.68)
38.46
18.55 (-61.38)
41.97 (-37.96)
TMC
69.93 (+00.36)
34.08
45.77 (-14.24)
54.37 (-05.64)
ITA-LoRA
92.80 (+-1.63)
60.06
28.67 (-61.29)
71.96
(-17.99)
C-100
IEL-LoRA
77.77 (- 13.22
37.90
19.48 (-70.05)
56.52 (-33.01)
TMC
87.53 (406.49)
45.75
55.63 (-22.79)
71.83 (-06.59)
ITA-LoRA
90.46 (+05.21)
57.87
68.19 (-17.36)
74.63 (-10)92)
CUB
IEL-LoRA
74.03 (- 10.67)
47.95
22.44 (-62.51)
30.99 (-53.96)
TMC
71.06 (+-0.99)
44.93
67.91 (-03.81)
53.22 (- 18.50)
ITA-LoRA
89.84 (-01.21)
65.47
80.02 (- 12.63)
82.40 (- 10.
Caltech
IEL-LORA
79.70 12.99)
62.45
32.00 (-60.19)
42.03 (-50.
TMC
89.23 (0.6))
49.46
64.52 (-17.78)
73.33 (-08.97
ITA-LoRA
89.66 (+-08.7)
57.30
36.99 (-49.61)
74.75 (-11.85)
MIT
IEL-LoRA
56.14 (- 19.38)
37.54
06.39
78.100)
32.57 (-52.02)
TMC
88.03 (+ 10)
30.56
29.77
(-38.89)
62.03 (-06 63)
ITA-LoRA
89.4 19.47 (+00677)
49.75
33.38
(-48.22)
64.06 (-17.94)
RESISC
IEL-LoRA
90.13 (4-0..92)
48.06
32.77
(-49.76)
65.19
(- 17.34)
TMC
75.77 (+27.63)
17.00
06.64
(-54.02)
52.31
08.33)
ITA-LoRA
97.63 (+00.11)
53.05
65.87
(-29.98)
79.24 16.61)
CropDis.
IEL-LORA
90.12 (-04.68)
48.31
34.01 (-61.87)
54.43 (- (-1.45)
TMC 73.60 (+00.23)
15.95
06.24
60.32)
53.21 (- 13.35)
across tasks; second, it is assessed between composed models T.TA and IEL, To eliminate potential confounding effects due to PEFT strategies, we perform the experiment using full fine-tuning.
Our results reveal mostly positive alignment, even for out-of-distribution datasets such as RESISC45 and CropDisease, and low-resolution datasets like CIFAR-100. On the other hand, we observe smaller similarities between the task vectors of the individual models learned more fine- fine-grained and complex datasets such as CUB-200 and MIT67. Indeed, CUB-200 demands highly detailed distinctions between bird species, while MIT67 involves subtle contextual differences, such dis- tinguishing children's rooms from living rooms or art studios from classrooms.
Despite the lower alignment for individual task vectors in certain datasets, observe that the similarity is consistently higher when comparing the composed task vectors to the individual ones.
H.6 TIMING
We herein include a wall-clock time analysis of the training algorithms. Specifically, considering Split ImageNet-R, Split RESISC45, and Split CropDisease, we measured the per-task runtime of ITA and IEL (both trained with full fine-tuning), as well as that of three existing approaches to incremental learning: DER+ (rehearsal), TMC (model compositionality) and SEED (ensemble learning).
Among the methods compared, SEED appears to the most efficient. However, the runtimes of both ITA and IEL are comparable to, or even better than, those of baseline methods like DER++ and TMC. Specifically, on ImageNet-R and RESISC45, the average runtime of ITA/IEL is approxi- mately 7/3 that of DER++. We attribute the reasonable training times of IEL/ITA to the efficient pro- cedure we employed for estimating the FIM, which leverages batch-wiss parallelization (see App. and George (202T)), combined with the use of closed-form gradient computation (refer App.
26

---

Published as a conference paper at ICLR 2025
Split ImageNet-R
Split CIFAR100
Split CUB-200
Split MIT-67
Split RESISC45
Individual
0.2
Composed
0.0
10
0
10
10
Layers
Figure 2: Alignment- - ie., cosine similarity - between the task vectors produced by ITA and IEL for both the composed model Op and individual learners O, (averaged across tasks 1).
Split ImageNet-R R
Split RESISC4S
Split CropDisease
50
50
Training Time -iiuttes
TI
DER T1C
ITA
DEV11
ITA
DERI
40
I8L
TMC
TMC
SIED
SEED
SEED
00
11
2
D2
D0
D4
Do
Dio
12
D2
Dr
D1
D5
17
D1
14
14
Figure 3: Comparative timing analysis (in minutes). The plot illustrates the per-task runtime of ITA and IEL, alongside baseline methods (DER++, TMC, and SEED). Runtimes include both the setup phase (e.g., steps required to compute FIM statistics) and the training phase.
I HYPERPARAMETERS
The hyperparameters employed for each experiment are reported in the following subsections (one for each dataset).
L.I IMAGENET-R
CODA-Prompt: 1.0 10-3
DER++: a 3.0 10-1:3 8.0 10 1r-- 1.0 x 10-3;
EWC: 10::-= 1.0; 1.0 10-2:
L2P: Ir = 2.5 x 10-3;
LWF-MC: wd 0.0; Ir 1.0 10
SEED: Ir = 3.0 x 10-4;
SGD: Ir 3.0 10-2
TMC: Ir = 1.0 x 10-4
InfLoRA: lr 5.0 10 4 , 10; e 0.98;
For both ITA and IEL: epochhspe-tuuniie pre-tuning 3; Ir pre-ttuning 1.0 x 10
ITA-FFT: 5.0 10 ocis 1.0 10-1, OCLS-pprior 1.0 10-2; 2; Ir 1.0 10-4
27

---

Published as a conference paper at ICLR 2025
ITA-LoRA: 2.0 10 acis 1.0 10-1; aCLS-prior 1.0 10 3, 1
= 3.0 10
ITA) a 7.0 x 10-1; acis = 1.0 x 10 10-1; OCIS-prrior = 1.0 x 10 3.0 10-3,
IEL-FFT: 3 = 5.0 x 10"; SCLS 1.0 x 10-1; PCLS-prior 3.0 10 3; Ir 1.0 10-4;
IEI.-LoRA: 8 10'; fcss = 1.0 x 10-1; PrIi-prrorr x.0 10-3; Ir 3.0 x 10-1
IEL-(IA) 3 2.0 10,; BaLs 1.0 10-1; PCTS-prior
1.0 10-2; Ir 3.0 10-3;
1.2 CIFAR-100
CODA-Prompt: Ir = x 10-3;
DER++: a 3.0 10-1;3---0000000 10-1:1r 1.0 x 10-4,
EWC: = 1.0 x 10:7-= 1.0; Ir 1.0 10-3;
L2P: Ir 2.5 10-3;
LWF-MC: wd = 0.0; Ir = 1.0 x 10-3;
SEED: lr 3.0 x 10"4;
SGD: = 1.0 x 10 10"
TMC: Ir = 1.0 x 10"4 4:
InfLoRA: = 5.0 x 10 10-4,- 10; 0.95;
For both ITA and IEL: # appochspre-tuniae 3; Ir pree-tuning 1.0 10-2;
ITA-FFT: a 2.0 10a; acLS 1.0 x 10-1; OCCLS-prior = 3.0 x 10 1.0 10-4;
ITA-LoRA: a 2.0 10-2; acts 1.0 10-1; acT.s-prior 1.0 30- -
10-4:
ITA- a 2.0 10 2; acis 1.0 10-1; ACIS-prior 3.0 x = 10-3;
IEL-FFT: 3 = 2.0 x 10;; :eLs 2.5 x 10-2; BCLS-prior 1.0 x 10 1r 10-4
IEL-LoRA: 3 = 2.0 10'; fLLS = 1.0 x 10 10-1; SCLS-prior = 1.0 x 10 3, 1 = 10
IEI- 3 = 2.0 10 BCLS 2.5 x 2; SCLS.prior
1.0 10-3, Ir 3.0 10-4;
I.3 CUB-200
CODA-Prompt: Ir = 1.00 10-3
DER++ a 3.0 10 8.0 1.0 x 10-3;
EWC: = 1.0 - 9.0 0.0x
1r- 1.0 x 10-3;
L2P: Ir 2.5 10-3
LWF-MC: wd = 1.0 x 10 1r-
- 1.0 10-2
SEED: 3.0 x 1044;
SGD: Ir = 3.0 x 10 10-
TMC: Ir 1.0 10 4;
InfLoRA: Ir = 5.0 x 10 - 10; = 0.98;
For both ITA and IEL: lrpre-tuning = 1.0 x 10-2; # epochs pre-tuning
8,
ITA-FFT: a = 5.0 x 102: acis = 1.0 x 10 acLS-prior = 1.0 x 10 10-2: 2: 1.0 x 10
ITA-LoRA: a 5.0; aCLS 1.0 10-1; aCIS-prior
1.0 10 2, Ir 3.0 10-4;
ITA-(IA)3: 7.0 10-1; ocus 1.0 10-1: OCIS-
1.0 10-2: 3.0 10-3
28

---

Published as a conference paper at ICLR 2025
IEL-FFT: 3 = 5.0 x 109; BCLS 1.0 10-1; PCLS-prior 1.0 10 2: 1r 1.0 10-5;
IEL-LoRA: 3 = 2.0 x 10"; ACLS = 1.0 10-1; SCIS- = 1.0 10 20 3.0 10-4:
IEI-(IA) 3 5.0 x 10"; BCLS = 1.0 x 10 PCLS.prior
1.0 10-2; lr 3.0 x 10"
1.4 25L
CODA-Prompt: Ir = 1.0 10-3;
DER++: a = 3.0 30-1;3 8.0 10-00:1-1000 1.0 x 10-3;
EWC: 1.0 10:-- 1.0: Ir 1.0 10-3;
L2P: Ir 2.5 10-3;
LWF-MC: wd = 0.0; Ir = 1.0 x 10-3;
SEED: 3.0 10"4;
SGD: Ir = 1.0 x 10 10-3
TMC: Ir 1.0 10-4,
InfLoRA: Ir = 5.0 x 1 10; = 0.99;
For both ITA and IEL: Ir preettuning 1.0 x 10-2
ITA-FFT: # epochs pre-tuning 3: a 2.0 x 103; acis = 1.0 10-1; aCIS-prror = 1.0 10-2;
Ir= 1.0 x 10-4;
ITA-LoRA: # epochs pre-tuning 3; a 2.0x10 acts 1.0x10 OCTS-prror 3.0xx00-.
3.0 x 10-4;
ITA)(IA) # epochs
pre-tuning prppccsspr--tunng I = 3.0 x 10-3;
8; a = 7.0 acls 1.0x10 OCLS.prior = 1.0x10-?;
IEL-FFT: # epochs pre-tuning 3; = 5.0 10¹; BCLS = 1.0 x 10-1; BCLS-prior 1.0 x 10
-- 1.0 x 10-4;
IEI.-LOR:: # epochsspretuunnngg 3; 2.0 104 Bers 5.0; BCIS. 3.0 10-3;
3.0 x 10-4;
IEI, 3: # #epochspee-uningg pre-tuning = 3 = = 10 BCLS = 1.0 x 10 1000:,,0s B.LLS-prior = 3.0 10-3;
Ir 3.0 x 10
1.5 MIT-67
CODA-Prompt: = x 3:
DER++: a = 3.0 x 10-1;3- 8.0 x 10 10-1;1r= 1r= 1.0 x 10-3;
EWC: 1.0 10 1.0: Ir 1.0 10-3;
L2P: Ir 2.5 x 10-3;
IWF-MC: wd 0.0; Ir 1.0 10-2;
SEED: 3.0 x 10-4
SGD: Ir 1.0 10-2
TMC: = 1.0 x 10 4;
InfLoRA: lr 1.0 x 3 10; 0.95;
For both ITA and IEL:: Ir prpe-taniig 1.0 x 1
29

---

Published as a conference paper at ICLR 2025
ITA-FFT: # epochs
spochspre-tuning pre-tuning
8; 5.0 x 109;
accs acis 1.0 x 10"1; aCLS-prior 1.0 x 10-1;
Ir 1.0 x 10-1;
ITA-LoRA: cpochs pre-tuning 8: a 5.0; aCtS = 1.0 x 10-,; OCLS-prior = 1.0 x 10-2;
I 3.0 x 10 10-
# ITA-(IA) 1- 3.0 x 10-4;
pre-tuning a 5.0; aCLS
1.0 10 &CLS-prior
3.0 10-3;
IEL-FFF: # epochs
Spoch&pre-tuning pre-tuning 8 = 5.0 x 109; fLLS 1.0 x 10-1; SCIS-prior 1.0 10-2;
Ir -- 1.0 x 10-4;
IEL-LoRA: # epochs pro-tuning =
2.0 x 10, BCLS = 1.0 1011; BCLS-prior 1.0 2; 3 =3;
Ir = 3.0 x 10-4;
IEL-(IA)*, 9: # epochs pre-tuiig = 10 CLs 1.0 10-1; PCLS-prio 1.0 10-2;
=== 3.0 x 10-1;
1.6 RESISC
CODA-Prompt: Ir = 1.0 10-3
DER++: a = 3.0 x = 100 1.0 x 10-3;
EWC: 1.0 10:;-- 9.0 10-1, x 10-2;
L2P: Ir = 2.5 10 10:3;
LWF-MC: wd = 0.0; Ir 1.0 30
SEED: 3.0 x 40"4;
SGD: Ir 1.0 x 10-2,
TMC: 3.0 x 10-4;
InfT.RAA: Ir 1- 5.0 x 10 10; 0.98;
For both ITA and IEL: # preochhpre-tunigg
8; prpe-tuning 1.0 x 10
ITA-FFT: 1.0 10 ocis 1.0 10 OCIS-prior
1.0 10 1.0 10-4
ITA-LORA: a 2.0 10 10-2; acts = 1.0 x 10-1; actssprior = 110 - 10-4
ITA-(IA) a = 5.0; acts = 1.0 x 10-1; OCT.S-prio 3.0 10-3: x
IEL-FFT: 3 = 5.0 x 10:: BCLS 1.0 x 10-1; BCLS-prior 3.0 10 3, Ir 1.0 10-4
IEL-LORA: 3 2.0 10:: fCLS 1.0 10 2; SCLS-prior 1.0 10 3:1- 3.0 10
IEI-(IA) 3 2.0; BCtS 1.0 10-2, PCIS
1.0 10 10 Ir 3.0 10-4
1.7 CROPDISRASE
CODA-Prompt: Ir 1.0 10-3;
DER++: a 3.0 10-1: 8.0 10-1;1 1.0 x 10-3;
EWC: c = 1.0; = 1.0; lr 1.0 10-2
L2P: Ir 2.5 10 10-3, 3,
LWF-MC: wd = 1.0 x 10
Ir = 1.0 10
SEED: 3.0 x 40
SGD: Ir = 1.0 10-2.
TMC: Ir 1.0 10-4.
30

---

Published as a conference paper at ICLR 2025
InfLoRA: Ir = 5.0 x 10 10; 0.98;
For both ITA and IEL: appochspr-ttunnn 8; pre-tnuning 1.0 x 10
ITA-FFT: 5.0 10:; actS 2.5 10 OCT.S-prior x 10 10- lr 1.0 x 10-,;
ITA-LoRA: 5.0; acIS 1.0 10-1; aCLS-prior 1.0 1003 3: Ir 3.0 x 10"+
ITAA a 2.0 x acis
acis = 1.0 x 10-1; OCIS-prior = 1.0 x 10 10-2; 1r 3.0 x 10-4;
IEL-FFT: 3 = 2.0 x 10:: BCLS = 1.0 x 10 1 PCLS-prior = 1.0 x 10 2; Ir 1.0 10 10-5;
IEI-LLRA: 3 5.0 10,; fcss 1.0 x 10-1; BCLS- 1.0 10 10:211 = 3.0 10-4
(EI-(IA) 3 = 2.0; BCLs 2.5 10-2; SCI.S-prior
1.0 x 10- == = 3.0 x 10-4;
31


Reviews:
[{"review_id": "Ld0jUSz3mh", "reviewer": "Reviewer_yT3o", "summary": "The paper conducts a theoretical analysis of model fusion in parameter space through the analysis of the second-order Taylor approximation of the multitask loss function. \n\nThe authors begin by showing that the 2nd order approximation of the multitask (MT) loss, when evaluated on the $w$-weighted averaged of individual models ($\\theta_P = w_1\\theta_1 + ... + w_t\\theta_t$), is upper bounded by $w$-weighted sum of individual models evaluated on the same multitask data. From this observation, the authors argue that, if each individual model could perform well on all tasks, this would drive the MT loss on $\\theta_P$ down. \n\nGiven that individual models cannot be optimized on data outside their training distribution, the authors instead propose to regularize individual models towards the base pretrained model $\\theta_0$, for such samples. This leads the authors to propose a EWC-like regularizer between $\\theta_0$ and $\\theta_t$ (when optimizing $\\theta_t$) where the Empirical Fisher Information Matrix can be computed iteratively, accumulating gradients from $\\theta_0$ on data from task $t$. \n\nFurthermore, the authors proceed to quantify the exact gap between the previous inequality, showing that term is a function of the weighted alignment across task vectors. The authors then proceed to propose a new learning algorithm, which extends the previous EWC regularizer to also penalize not only differences between $\\theta_0$ and $\\theta_t$, but also difference between $\\theta_t$ and subsequent  learned task vectors $\\theta_{<t}$. \n\nThe authors proceed to analyze their methods on class-incremental learning benchmarks, showing favorable results. An ablation showing the importance of the EWC regularizer is also provided. Finally, akin to Task Arithmetic manipulations, the authors show that task vectors can be used to enhance or unlearn specific task knowledge.", "questions": "1. How different are the task vectors learned with ITA vs IEL ? Do you find them to be well-aligned ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel theoretical analysis of the multitask loss through a second-order Taylor expansion, leading to interesting insights (namely equation 5 & 14)\n2. The authors leverage this insight to propose new methods for incremental learning with weight-space merging, showing favorable results. \n3. The paper is well-structured, and the experimental section contains relevant ablations showing the effectiveness of the proposed components", "weaknesses": "I have some reservations as to the motivation of the proposed methods. \n\n1. Regarding equation 5, I agree that should all individual models perform well on all tasks, then  $\\theta_P$ would likely perform quite well. That being said, I find that approximating this condition by regularizing the task vectors to stay close to the base model somewhat of a big conceptual leap. By construction, the base model performs sub-optimally on the tasks for which we optimize a task vector, otherwise the task-specific optimization does not serve much purpose. Therefore, in Eq. 5, wouldn't replacing $\\theta_t$ with $\\theta_0$ for any $x \\sim p_{t'}, t' \\neq t$, still make the upper bound loose, and therefore not useful ?\n\n2. The authors motivate ITA vs IEL as an approach that can be executed in a decentralized fashion. However, I am not sure this is the case; given that the FIM is updated sequentially, how would one train several tasks in a decentralized manner without communicating FIM statistics ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N6zBfxvHRR", "reviewer": "Reviewer_tsHk", "summary": "The paper explores how incremental learning can be addressed through incremental model merging. The theoretical framework is based on the following intuition(s):\n- if the base model is already well suited for the downstream asks (\\theta_0 is a local min.), fine-tuned versions of this base model will merge better. This connects well to the recent work [5], which shows that stronger base models (instruction tuned, or more params.) lead to better merging.\nThis intuition serves as a starting point for their theoretical analysis and the derivation of the **ITA** method, which is similar in spirit to ColD Fusion (which does not include the EWC term) [1]. ITA ensures that each new fine-tuned model lies in the viscidity of the base model (as measured by Riemannian distance), which intuitively results in better merging.\n\nIt has to be noted, that because this work addressed image classification setting where each task has a different output space, in order to make sure that the base model is a local minimum of the downstream tasks, authors propose the so-called \"task pre-consolidation\" step, which involves training a task-specific on a fixed backbone before the backbone's params. are learned. The importance of this type of alignment has been recently explored in [2].\n\n**ITA method**: in essence, this method is similar to EWC, with the difference that for each new task we start from the base model (\\theta_0) and not from the model learned for the previous task (I have some questions regarding the motivation of this approach, see questions part)\n\nWhile ITA enables completely independent model training in the spirit of \"MoErging\" methods [3], one problem with ITA is the fact that each new task's model only transfers knowledge from the base model and not from other tasks. Motivated by this, the paper proposes **IEL** method. IEL optimizes each new task vector assuming access to the merged model on tasks seen so far (somewhat standard in CL literature). This amounts to using both EWC regularization w.r.t to the base model and task alignment term ensuring the new task vector is similar to the previous ones. Importantly, the gradients of the regularization term in IEL can be computed efficiently. \n\nThis paper presents a solid empirical section with several interesting insights. All the theoretical insights are substantiated with empirical validations.\n\n\n[1] Don-Yehiya, Shachar, et al. \"Cold fusion: Collaborative descent for distributed multitask finetuning.\" _arXiv preprint arXiv:2212.01378_ (2022).\n\n[2] Schmidt, Fabian David, Ivan Vulić, and Goran Glavaš. \"Free lunch: Robust cross-lingual transfer via model checkpoint averaging.\" _arXiv preprint arXiv:2305.16834_ (2023).\n\n[3] Yadav, Prateek, et al. \"A survey on model moerging: Recycling and routing among specialized experts for collaborative learning.\" _arXiv preprint arXiv:2408.07057_ (2024).\n\n[4] Ostapenko, Oleksiy, et al. \"Continual learning with foundation models: An empirical study of latent replay.\" _Conference on lifelong learning agents_. PMLR, 2022.\n\n[5] Yadav, Prateek, et al. \"What Matters for Model Merging at Scale?.\" _arXiv preprint arXiv:2410.03617_ (2024).", "questions": "- ll. 028: \"Recent AI technologies are predominantly being viewed as monoliths\": while I understand the point authors are making here, it must be acknowledged that modular architectures, such as MoE have become increasingly widely used in recent years (maybe as wide as monolithic models)\n- ll. 152-157: the paragraph seems to argue that the right-hand side of equation 5 will increase as new tasks are learned. But in equation 5's right-hand side, each individual loss starts from the base model and does not depend on any form of continual learning. So I am not sure I understand the argument being made here.\n- would the pre-consolidation step be necessary in language modelling, where the output space is usually shared across tasks? \nEmpirical section:\n- are the different classification heads of the task-specific models also merged together? if so, does it mean that all tasks must have the same number of classes (i.e. share the output space)? \n- since IN-R, C-100 and CUB are very much id-distribution w.r.t pre-training on imagenet, I wonder whether simple fine-tuning of the final classification layer, which can be a metric-based classifier with no forgetting, can be sufficient to achieve good performance? (e.g. see [4])\n- Did the authors investigate the effect of pre-consolidation on the CL performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, while the idea of using model merging for continual learning has been explored in previous works, see e.g. [1], this paper does a good job of first formalizing intuitive ideas into an appealing theoretical framework, and subsequently validating these intuitions (and theoretical insights) empirically.  While I have several questions regarding several formulations made in the paper, the overall quality of the work is commendable. \n\nAnother strength of this work is its empirical section. Even though experiments are performed only on the image classification domain, a broad range of datasets is covered including also datasets which are OOD w.r.t the backbone.", "weaknesses": "- This works connects to several existing works in the realm of model merging, see the \"summary\" section of my review. These works are not cited in the current version, maybe because these connections were not obvious to the authors. I suggest the authors revisit this potential connection and cite the related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JFRfNe8N5u", "reviewer": "Reviewer_YXTt", "summary": "In this work, the author(s) examines the compositional properties of fine-tuning deep pre-trained models, focusing on how specialized modules within these models can be combined to create a single, multi-task model. To be specific, the author leverage a second-order Taylor approximation to understand what promotes compositionality in non-linear networks.", "questions": "see weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This approach reveals the importance of staying within the pre-training 'basin' to create composable modules. To achieve this, the authors introduce two incremental training algorithms: 1) training multiple models individually, and 2) optimizing the entire composed model.\n\nThe theoretical analysis section is well-developed, and the experiments are clearly presented.\n\nTested on incremental classification tasks, this approach enables the creation of an effective multi-task model, with capabilities for unlearning and specialization in specific tasks.", "weaknesses": "Major:\n\nThe motivation for using the second-order linear approximation is unclear. To my understanding, this work employs a similar approach to neural tangent kernel (NTK) analyses. However, in the linear approximation, NTK typically uses the first-order Taylor expansion, here the author employs the second order. Why not use the first-order Taylor expansion or perhaps a third-order expansion? So, it would be great the author add a section and comparing their work with [1-2], especially the Eq.3-4 with Lemma 3.1 in [2].\n\nAlso, there are NTK-based class incremental learning works [3-5]. The authors could include a section in the introduction or discussion to clearly point out the differences and benefits of using their approach compared to these works.\n\nMinor issue: \n\nSome notation is hard to follow and not clearly defined, such as $ L_{\\text{cur}}$. The definition of in incremental learning $t,T,\\tau$ might also be confusing. The dataset $D_t$ contains $n_t$ training samples $x, y) \\sim p_t(x, y)$, drawn from a distribution that varies across tasks. The authors use $\\tau$ for $ T $ from $1 \\ldots t $, so it is likely that $D_T$ would be more appropriate.\n\n\n[1] Neural tangent kernel: Convergence and generalization in neural networks.\n\n[2] On Exact Computation with an Infinitely Wide Neural Net\n\n[3] Tkil: Tangent kernel optimization for class balanced incremental learning\n\n[4] Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective\n\n[5] A Continual Learning Algorithm Based on Orthogonal Gradient Descent Beyond Neural Tangent Kernel Regime", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gQvGX7vAKD", "reviewer": "Reviewer_hFQw", "summary": "The paper studies model compositionality applied under class-incremental continual learning settings, where individual models fine-tuned on disjoint tasks are composed via weight averaging. By studying the second-order Taylor expansion of deep networks about a strong initial condition, the paper motivates two modifications to existing training objectives. The first encourages local linearity in model weights by minimizing the change in weights (from the pre-trained point) resulting from fine-tuning, which is shown to penalize deviations between the pre-trained and fine-tuned model outputs. The second operates in the regime where individual model weight perturbations are fine-tuned using the loss computed from the _composed_ model. The paper proposes adding a regularizer to minimize the deviation between the loss resulting from weight composition, and the loss resulting from output composition (ensembling).", "questions": "- I see some replay-based methods like DER++ are included in Table 1, but I don't see any mention of the experimental setup used for these comparisons (e.g. assumptions on buffer size)\n- Equation (13) suggests that by minimizing $\\Omega$, IEL also seeks to encourage linearity in the loss, as done in ITA.  Can you provide some intuition (formal or informal) regarding under what regimes IEL fairs better than ITA, and vice-versa?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper explores the model composition approach for continual learning, which has strong advantages in more general settings, for instance in federated setups where individual models can be trained in parallel on disjoint datasets and then composed at the end.\n- I appreciate the idea of enforcing local convexity or linearity via regularization, since this circumvents the need to do so explicitly via model linearization techniques like in TMC, and it appears to work very well in practice (Table 1).\n- The composed model training objective proposed in IEL is also intuitive, essentially \"learning to compose\" since each new task perturbation is conditioned on all previously seen ones.\n- Experiments are comprehensive and a variety of datasets are explored at least in the class-incremental setting. Method performs well across all datasets.", "weaknesses": "- The regularization condition proposed in Sec 2.2 does not seem to necessitate the second-order analysis, since it simply aims to enforce a local (first-order) linear structure by making perturbation smaller.\n- The wording on \"incrementally train(ing) the whole composed model rather than its individual components\" used throughout the paper is misleading. I only understood what this statement is trying to convey after seeing equation (15) on Page 5. It appears to me that what is being trained is still individual perturbation components ($\\tau_1, \\ldots, \\tau_T$) , but only that the _loss_ is computed based on the composed model. I would suggest modifying the terminology for better clarity.\n- L120 assumes that the pre-trained initialization $\\theta_0$ is globally optimal for a quadratic model trained on the union of all tasks. This seems to be a very strong assumption that likely would never hold in practice (why train/compose when the initialization is already optimal?), contrary to what is claimed in footnote 1. This weakens the theoretical analysis.\n- Lacks wall-clock time comparison ($\\mathcal{O}(1)$ does not shed any light on the actual cost incurred), for instance that used to update the FIM, which is important to faithfully evaluate the method w.r.t. to other works.\n- The need for the task pre-consolidation (linear probing, etc.) stage requires additional training steps, so comparisons to most existing methods that do not require this would not be completely fair. If the paper did not implement control steps to make fairer comparisons, then this should be mentioned explicitly in the limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)