A Second-Order Perspective on Model Compositionality and Incremental Learning

## Abstract

Abstract The fine-tuning of deep pre-trained models has revealed compositional properties, with multiple specialized modules that can be arbitrarily composed into a single, multi-task model. However, identifying the conditions that promote compositionality remains an open issue, with recent efforts concentrating mainly on linearized networks. We conduct a theoretical study that attempts to demystify compositionality in standard non-linear networks through the second-order Taylor approximation of the loss function. The proposed formulation highlights the importance of staying within the pre-training basin to achieve composable modules. Moreover, it provides the basis for two dual incremental training algorithms: the one from the perspective of multiple models trained individually, while the other aims to optimize the composed model as a whole. We probe their application in incremental classification tasks and highlight some valuable skills. In fact, the pool of incrementally learned modules not only supports the creation of an effective multi-task model but also enables unlearning and specialization in certain tasks.

## 1 Introduction

Recent AI technologies are predominantly being viewed as monoliths, centered around a single model trained on a single (albeit large) dataset. This paradigm has undeniably yielded impressive results and is likely to continue doing so. However, despite a genuine interest in these technologies, businesses face numerous challenges. In fact, the life-cycle of these monolithic models demands incremental and intensive updates to accommodate the diversity of markets, platforms, and customer needs. Moreover, many companies lack the financial means to assemble a proficient AI team or access GPU clusters to support the rapid evolution of these factors. In an effort to democratize AI, we advocate for modern paradigms (Li et al., ; Pfeiffer et al., ) prioritizing fast adaptability, model customization, and data heterogeneity.

Recent frameworks (Liu & Soatto, ; Bowman et al., ) support these principles through model compositionality, allowing the creation of bespoke models through cheap editing operations. Surprisingly, fine-tuning pre-trained models not only enables efficient transfer with limited data (Zhuang et al., ) but also offers insights into compositionality (Ilharco et al., ). In fact, it has been shown (Wortsman et al., ) that a simple linear combination of individually fine-tuned weights yields meaningful outcomes. The resulting composed model exhibits rich and robust representations (Zhang & Bottou, ) without incurring additional inference or memory costs. Such a simple but powerful schema has been primarily employed in model soups (Wortsman et al., ), where individual models are trained on the same task but with different hyper-parameters. In contrast, Task arithmetic (Ilharco et al., ; Liu & Soatto, ; Ortiz-Jimenez et al., ) addresses instead a cross-dataset setting, with each individual model focusing on a distinct task.

In this context, our research elaborates on two points. Firstly, considering multiple models trained individually on different tasks, we aim to understand the conditions that allow the successful combination of their weights. This finding has been predominantly discussed through empirical evaluations (Ilharco et al., ); with some notable exceptions (Ortiz-Jimenez et al., ) that strive for a more theoretical understanding. However, these approaches only focus on linearized networks and tangent fine-tuning (Liu & Soatto, ). We instead aim for a formulation that generalizes to standard non-linear deep networks and diverse fine-tuning strategies (e.g., LoRA (Hu et al., )). To do so, we propose leveraging the second-order Taylor approximation of the loss function around the pre-training weights. This enables us to derive a relationship between the capabilities of individual models and those of their composition, offering a key takeaway. Specifically, for the composed model to be accurate, each component should attain decent performance on examples outside its individual training distribution.

The second point of our study is a derivation of the former and revolves around learning composable models in an incremental fashion, dedicating an individual module to each incoming task. In this respect, previous works (Bowman et al., ; Liu & Soatto, ) build on the intuitive idea that modularity provides a natural foundation for incremental training. Through model editing and composition, in fact, one can simply introduce novel concepts without having to re-train the model. In this work, we turn the perspective and argue that compositional capabilities require incremental learning skills. Indeed, as mentioned above, preserving out-of-distribution performance is essential for compositionality. Nevertheless, this can be framed as a continual learning problem (Kirkpatrick et al., ), where the objective of each individual model is to maintain the pre-training general capabilities on examples that fall outside its specific training distribution.

On this basis, we propose two algorithms that tackle the incremental fine-tuning of composable models. The two algorithms build upon the same second-order formulation but approach the problem from opposed perspectives: optimizing each model individually vs. optimizing the whole composed model directly. We assess them on several class-incremental settings (Van de Ven & Tolias, ), differing in terms of alignment with pre-training knowledge. Notably, our approaches lead to an accurate multi-task model with further editing capabilities. These allow the efficient specialization of the model on certain tasks, as well as the ability of removing some others (unlearning).

## 2 Framework

We consider $f(\cdot;\bm{\theta}):\mathcal{X}\to\mathcal{Y}$ as a twice-differentiable deep network with weights $\bm{\theta}\in\Theta\subseteq\mathbb{R}^{m}$. It takes inputs $\bm{x}\in\mathcal{X}\subseteq\mathbb{R}^{d}$ and yields a conditional distribution $p_{\theta}(\bm{y}|\bm{x})$ over the targets $\bm{y}\in\mathcal{Y}\subseteq\mathbb{R}^{c}$. In this paper, we focus on incremental training, which progresses sequentially through a series of $T$ classification tasks $\mathcal{T}=\{1,2,\dots,T\}$. Each task $t\sim\mathcal{T}$ is characterized by a dataset $\mathcal{D}_{t}$ with $n_{t}$ training samples $\bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})$ drawn from a distribution varying across tasks. We assume that tasks share the same loss function $\ell(\bm{\theta}|\bm{x},\bm{y})$, i.e., the negative log-likelihood $-\log p_{\theta}(\bm{y}|\bm{x})$.

In this setting, we build upon a pool of fine-tuned composable networks $\mathcal{P}=\{f(\cdot;\bm{\theta}_{t})\mid\bm{\theta}_{t}\triangleq\bm{\theta}
_{0}+\bm{\tau}_{t}\}_{t\in\mathcal{T}}$. In detail, each model is fine-tuned from a common set of pre-training weights $\bm{\theta}_{0}$. The $\bm{\tau}_{t}$ indicates the displacement in parameter space w.r.t. $\bm{\theta}_{0}$ after training on task $t$, often referred as task vector (Ilharco et al., ; Ortiz-Jimenez et al., ). We obtain the weights of the composed model $f_{\mathcal{P}}$ by averaging the weights within the pool, as follows:

$$ $\displaystyle\textstyle f_{\mathcal{P}}$ $\displaystyle\triangleq f(\cdot;\bm{\theta}_{\mathcal{P}})\quad\text{s.t.} \quad\bm{\theta}_{\mathcal{P}}=\bm{\theta}_{0}+{{\textstyle\sum}}_{t=1}^{T}w_{ t}\bm{\tau}_{t},\quad{{\textstyle\sum}}_{t=1}^{T}w_{t}=1$ (1) $$

where $w_{t}$ balances the contribution of the $t$-th learner to the aggregated model. While some works (Asadi et al., ; Huang et al., ) optimize these coefficients, we devise uniform weights $w_{t}=\nicefrac{{1}}{{T}}$ in our algorithms.

Scope. How can we learn multiple disjoint tasks through a pool $\mathcal{P}$ of models, so that the composed model performs well on their union? To answer this question, we introduce the concept of empirical risk, i.e., the average loss $\hat{\ell}(\bm{\theta}|\mathcal{D})$, computed over the union $\mathcal{D}=\bigcup_{t=1}^{T}\mathcal{D}_{t}$ of all training tasks:

$$ $\textstyle\hat{\ell}(\bm{\theta}|\mathcal{D})=\frac{1}{\sum_{t=1}^{T}n_{t}} \sum_{\bm{x},\bm{y}\in\mathcal{D}}\ell(\bm{\theta}|\bm{x},\bm{y})\approx \mathop{\mathbb{E}}_{\begin{subarray}{c}t\sim\mathcal{T}\\ \bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})\end{subarray}}\left[\ell(\bm{\theta}| \bm{x},\bm{y})\right]$ (2) $$

To simplify notation, we will henceforth omit the explicit dependence of the loss on the data, denoting the individual loss $\ell(\bm{\theta}|\bm{x},\bm{y})$ simply as $\ell(\bm{\theta})$, and the empirical risk $\hat{\ell}(\bm{\theta}|\mathcal{D})$ as $\hat{\ell}(\bm{\theta})$. On this basis, the rest of this section will delve into the following aspects.

### 2.1 Individual learners vs. the composed model: a pre-training perspective

We now relate the composed model $f_{\mathcal{P}}$ to the individual components $f(\cdot;\bm{\theta}_{t})$ of the pool. To do so, we introduce the second-order Taylor expansion of the loss around the pre-trained weights $\bm{\theta}_{0}$:

$$ $\displaystyle\ell(\bm{\theta})$ $\displaystyle=\ell_{\operatorname{cur}}(\bm{\theta})+\mathcal{O}(\|\bm{\theta} -\bm{\theta}_{0}\|^{3})$ (3) $\displaystyle\text{where}\quad\ell_{\operatorname{cur}}(\bm{\theta})$ $\displaystyle=\ell(\bm{\theta}_{0})+(\bm{\theta}-\bm{\theta}_{0})^{\mathrm{T}} \nabla\ell(\bm{\theta}_{0})+\frac{1}{2}(\bm{\theta}-\bm{\theta}_{0})^{\mathrm{ T}}{\bf{H}_{\ell}}(\bm{\theta}_{0})(\bm{\theta}-\bm{\theta}_{0}).$ (4) $$

$\nabla\ell(\bm{\theta}_{0})\triangleq\nabla_{\bm{\theta}}\ell(\bm{\theta}_{0}|
\bm{x},\bm{y})$ and ${\bf{H}_{\ell}}(\bm{\theta}_{0})\triangleq\nabla_{\bm{\theta}}^{2}\ell(\bm{
\theta}_{0}|\bm{x},\bm{y})$ are the gradient and the Hessian around $\bm{\theta}_{0}$. Similarly, we can define the second-order approximation $\hat{\ell}_{\operatorname{cur}}(\bm{\theta})\approx\hat{\ell}(\bm{\theta})$ of the empirical risk, which corresponds to averaging the approximated loss across examples from all tasks.

Assumption. We now assume that $\bm{\theta}=\bm{\theta}_{0}$ is a point of local minimum of the empirical risk $\hat{\ell}(\bm{\theta})$(^1^11We refer the reader to Sec. 3 for a practical strategy to enforce the optimality of $\bm{\theta}_{0}$. It is noted that such a condition can be easily satisfied with over-parameterized deep learning models.) across all tasks (Eq. 2). Under this hypothesis, the Hessian of the empirical risk ${\bf{H}_{\hat{\ell}}}(\bm{\theta}_{0})\succeq 0$ is positive semidefinite. In light of this and the quadratic nature of $\hat{\ell}_{\operatorname{cur}}(\bm{\theta})$, we can state that the second-order approximation of the empirical risk $\hat{\ell}_{\operatorname{cur}}(\bm{\theta})$ is convex. Therefore, we apply the Jensen’s inequality to derive a relation between the composed model and the individual components:

$$ $\textstyle\hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}}=\bm{\theta }_{0}+{{\textstyle\sum}}_{t=1}^{T}w_{t}\bm{\tau}_{t})\leq\sum_{t=1}^{T}w_{t}\ \hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{t}=\bm{\theta}_{0}+\bm{\tau}_{t}).$ (5) $$

The relation states that, under the second-order approximation, the empirical risk $\hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ of the composed model is upper-bounded by the convex combination of its individuals. In other words, if each individual model is trained to the optimum with near-zero loss value, there are some guarantees on the loss function attained by the composed model. Notably, this relation could help reduce the computational footprint during inference, as it enables the reduction of forward passes, from multiple (one for each individual) to a singular pass (performed on the composed model).

At a first glance, the result of Eq. 5 is somewhat similar to what is stated by Liu & Soatto () in Eq. 2 of the respective paper:

$$ $\displaystyle\hat{\ell}(\bm{\theta}_{\mathcal{P}}=\bm{\theta}_{0}+{{\textstyle \sum}}_{t=1}^{T}w_{t}\bm{\tau}_{t})\leq$ $\displaystyle\ {{\textstyle\sum}}_{t=1}^{T}w_{t}\hat{\ell}(\bm{\theta}_{t}=\bm {\theta}_{0}+\bm{\tau}_{t})$ (6) $\displaystyle\text{given that }f_{t}(\cdot;\bm{\theta}_{t})\triangleq f_{ \operatorname{lin}}(\cdot;\bm{\theta}_{t})=$ $\displaystyle f(\cdot;\bm{\theta}_{0})+(\bm{\theta}_{t}-\bm{\theta}_{0})^{ \mathrm{T}}\nabla f(\cdot;\bm{\theta}_{0})\quad\text{({tangentness})}$ (7) $$

However, some notable distinctions remain. Their inequality applies to the exact risk $\hat{\ell}$ but is valid only for linearized models (i.e., fine-tuned in the tangent space of pre-training weights). In contrast, our result pertains to the second-order approximation $\hat{\ell}_{\operatorname{cur}}$ of the risk and applies to any fine-tuning strategy (e.g., LoRA, adapters, etc.). Intuitively, our inequality provides more flexibility to the training of individual learners, as long as: i) the learners remain in the pre-training basin, such that $\mathcal{O}(\|\bm{\theta}-\bm{\theta}_{0}\|^{3})\rightarrow 0$ and $\ell_{\operatorname{cur}}$ can be considered a good proxy of $\ell$; ii) $\bm{\theta}_{0}$ is a local minimum of $\ell$.

### 2.2 Enabling individual training in incremental scenarios

As mentioned above, a possible application of Eq. 5 is to devote each learner to a distinct task and optimize them in isolation (i.e., individual training). Indeed, the upper bound in Eq. 5 describes a sort of worst-case scenario for the risk of the composed model: at worst, it collapses to that given by the upper bound. Nonetheless, if every individual model were accurate on all tasks, the right-side term of Eq. 5 would yield an appealing upper-bound to the risk of the composed model.

Limits of Eq. 5. Under the constraints of individual training, each individual learner $f(\cdot;\bm{\theta}_{t})$ can be optimized only on the current distribution $p_{t}(\bm{x},\bm{y})$. Therefore, when considering examples from other data distributions $p_{t^{\prime}\neq t}(\bm{x},\bm{y})$, the loss $\ell_{\operatorname{cur}}(\bm{\theta}_{t}|\bm{x},\bm{y})$ of the $t$-th individual learner is likely to be much higher for examples outside its training distribution $p_{t}(\bm{x},\bm{y})$. As a consequence, the upper bound delivered by the right-side of Eq. 5 is likely to increase one task after the other, and we cannot rely on it to recover a reliable composed model $f(\cdot;\bm{\theta}_{\mathcal{P}})$.

In this respect, our proposal is to tighten the upper bound of Eq. 5 through explicit regularization, which we devise during the optimization of each individual learner. In practice, each model is provided with a learning objective that extends beyond minimizing the loss on the assigned data distribution $p_{t}(\bm{x},\bm{y})$. Specifically, to ensure decent performance on external distributions $p_{t^{\prime}\neq t}(\bm{x},\bm{y})$, we anchor the model to the pre-training knowledge for out-of-distribution examples. We do so by encouraging the predictions of $f(\cdot;\bm{\theta}_{t})$ to be close to those generated by the pre-trained model $f(\cdot;\bm{\theta}_{0})$, given examples $\bm{x},\bm{y}\sim p_{t^{\prime}\neq t}(\bm{x},\bm{y})$. Denoting the pre-training posterior distribution as $p_{\bm{\theta}_{0}}(\bm{y}|\bm{x})$, we have:

$$ $\operatorname*{minimize}_{\bm{\theta}_{t}}\quad\textstyle{\mathop{\mathbb{E}}_ {\bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})}\left[\ell_{\operatorname{cur}}(\bm{ \theta}_{t}|\bm{x},\bm{y})\right]+\mathcal{D}_{\operatorname{KL}}(p_{\bm{ \theta}_{0}}(\bm{y}|\bm{x})||p_{\bm{\theta}_{t}}(\bm{y}|\bm{x})).}$ (8) $$

It is noted that the computation of the KL term $\mathcal{D}_{\operatorname{KL}}(\cdot)$ requires sampling from the external distributions $p_{t^{\prime}\neq t}(\bm{x},\bm{y})$. Theoretically, this clashes with the constraints of individual training. Thankfully, if $\bm{\tau}_{t}\to 0$, the KL term can be approximated (Chaudhry et al., ) with the distance between $\bm{\theta}_{t}$ and pre-training weights $\bm{\theta}_{0}$:

$$ $\textstyle\mathcal{D}_{\operatorname{KL}}(p_{\bm{\theta}_{0}}(\bm{y}|\bm{x})\ ||\ p_{\bm{\theta}_{t}}(\bm{y}|\bm{x}))\approx\frac{1}{2}(\bm{\theta}_{t}-\bm{ \theta}_{0})^{\mathrm{T}}\mathrm{F}_{\bm{\theta}_{0}}(\bm{\theta}_{t}-\bm{ \theta}_{0})=\frac{1}{2}\lVert\bm{\tau}_{t}\rVert^{2}_{\mathrm{F}_{\bm{\theta} _{0}}}.$ (9) $$

The distance is computed in the Riemannian manifold (Lee, ) induced by the Fisher Information Matrix (FIM) (Huszár, ); namely, a $|\theta|\times|\theta|$ positive semi-definite matrix given by:

$$ $\displaystyle\textstyle\mathrm{F}_{\bm{\theta}_{0}}=\mathop{\mathbb{E}}_{ \begin{subarray}{c}t\sim\mathcal{T}\\ \bm{x}\sim p_{t}(\bm{x},\bm{y})\end{subarray}}\left[\mathop{\mathbb{E}}_{\bm{y }\sim p_{\bm{\theta}_{0}}(\bm{y}|\bm{x})}\left[\nabla\log p_{\bm{\theta}_{0}}( \bm{y}|\bm{x})\>\nabla\log p_{\bm{\theta}_{0}}(\bm{y}|\bm{x})^{\top}\right] \right].$ (10) $$

Under the local minimum hypothesis on pre-training weights, the FIM at $\bm{\theta}_{0}$ equals the expected(^2^22Precisely, we take the expectation w.r.t. data from other tasks $t^{\prime}\neq t$ to reflect the regularization in Eq. 8.) Hessian of the negative log-likelihood (Martens, ): $\mathrm{F}_{\bm{\theta}_{0}}=\mathop{\mathbb{E}}_{\bm{x}}\left[{\bf{H}_{\ell}}
(\bm{\theta}_{0})\right]$. Due to this connection, the FIM yields insights on the sensitivity of each parameter to changes in the data distribution.

Application to incremental learning. Given that optimizing the regularization above does not necessitate data from external tasks, it can be readily adapted to incremental learning. Following Chaudhry et al. (); Schwarz et al. (), we introduce a few additional approximations. Firstly, we limit to estimate the diagonal $\hat{\mathrm{F}}_{\bm{\theta}_{0}}$ of the FIM, thus avoiding the prohibitive footprint required to treat a $|\theta|\times|\theta|$ matrix. Basically, the diagonal $\hat{\mathrm{F}}_{\bm{\theta}_{0}}$ consists of a Monte Carlo estimate of the (squared) gradients of the log-likelihood. Secondly, we note that the expectation $\mathop{\mathbb{E}}_{\bm{x}}\left[{\bf{H}_{\ell}}(\bm{\theta}_{0})\right]$ in the FIM cannot be directly estimated, as examples from all tasks are not available simultaneously but rather sequentially. We hence compute the expectation incrementally (Chaudhry et al., ; Schwarz et al., ) one task at a time. As each new task becomes available, we calculate a local Fisher matrix on the data of the new task and then accumulate it into a global running Fisher estimate. As outlined by Alg. 1, the accumulation can be thought as a simple summation, net of re-normalizing operations reflecting the number of samples of each task (see App. E).

Given Eq. 9 and the diagonal FIM, the augmented optimization problem for the $t$-th learner becomes:

$$ $\displaystyle\operatorname*{minimize}_{\bm{\theta}_{t}}$ $\displaystyle\quad\mathop{\mathbb{E}}_{\bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})} \left[\ell_{\operatorname{cur}}(\bm{\theta}_{t}|\bm{x},\bm{y})\right]+\frac{ \alpha}{2}\operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t})$ (11) where $\displaystyle\operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t})={{ \textstyle\sum}}_{i}^{|\theta|}\hat{\mathrm{F}}_{\bm{\theta}_{0}}^{(i)}(\bm{ \theta}_{t}^{(i)}-\bm{\theta}_{0}^{(i)})^{2}.$ (12) $$

where $\alpha\geq 0$ is an hyper-parameter and $\operatorname{EWC}_{\bm{\theta}_{0}}(\cdot)$ indicates the Riemannian distance from the pre-training weights $\bm{\theta}_{0}$. The acronym $\operatorname{EWC}_{\bm{\theta}_{0}}$ highlights the strong analogy with Elastic Weight Consolidation (Kirkpatrick et al., ), a well-established approach against catastrophic forgetting (McCloskey & Cohen, ). In a sense, our term prevents forgetting pre-training knowledge; however, while our anchor is fixed at $\bm{\theta}_{0}$, the anchor of EWC instead shifts and focuses on the weights learned during the preceding task.

### 2.3 Joint training of the composed model in incremental scenarios

Individual training profitably aligns with decentralized learning (Bowman et al., ), emphasizing minimal interactions between learners and privacy preservation. Nevertheless, it might be inefficient when these constraints are not of interest and the goal is simply to create a powerful accurate model. In fact, individual training prevents a learner from leveraging the knowledge of other ensemble members, eliminating the potential for beneficial mutual transfer. For these reasons, we conduct a theoretical analysis of the dual perspective: the incremental training of the composed model $f(\cdot;\bm{\theta}_{\mathcal{P}})$ (ensemble training). Since the inequality in Eq. 5 does not provide much help for the explicit optimization of $\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$, we quantify the exact gap between the two sides of the Jensen’s inequality:

###### Theorem 1 .

Let us assume a pool $\mathcal{P}$ with $T\geq 2$ models, with the $t$-th model parameterized by $\bm{\theta}_{t}=\bm{\theta}_{0}+\bm{\tau}_{t}$. If we compose them through coefficients $w_{1},\dots,w_{T}$ s.t. $w_{t}\in[0,1]$ and ${{\textstyle\sum}}_{t=1}^{T}w_{t}=1$, the 2nd order approximation $\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ evaluated on composed weights $\bm{\theta}_{\mathcal{P}}=\bm{\theta}_{0}+{{\textstyle\sum}}_{t=1}^{T}w_{t}\bm
{\tau}_{t}$ is:

$$ $\displaystyle\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ $\displaystyle+\Omega(\bm{\theta}_{1},\dots,\bm{\theta}_{T})={{\textstyle\sum}} _{t=1}^{T}w_{t}\ell_{\operatorname{cur}}(\bm{\theta}_{t})$ (13) $\displaystyle\textnormal{{where}}\quad\Omega(\bm{\theta}_{1},\dots,\bm{\theta} _{T})$ $\displaystyle=\frac{1}{2}{{\textstyle\sum}}_{t=1}^{T}{{\textstyle\sum}}_{t^{ \prime}<t}w_{t}w_{t^{\prime}}(\bm{\tau}_{t}-\bm{\tau}_{t^{\prime}})^{\mathrm{T }}{\bf{H}_{\ell}}(\bm{\theta}_{0})(\bm{\tau}_{t}-\bm{\tau}_{t^{\prime}}).$ (14) $$

Proof in App. B. The equality introduces a term $\Omega(\cdot)$ that is non-negative (due to ${\bf{H}_{\ell}}(\bm{\theta}_{0})\succeq 0$) and depends on weights $\bm{\theta}_{1},\dots,\bm{\theta}_{T}$. Therefore, $\Omega(\cdot)$ is proportional to the cumulative distance between every pair of learners, within the Riemannian manifold induced by the Hessian ${\bf{H}_{\ell}}(\bm{\theta}_{0})$ (Lee, ). Notably, Eq. 13 permits to draw the following insights: i) optimizing both the loss of the composed model $\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ and the term $\Omega(\cdot)$ is equivalent to individual training; ii) during inference, if $\Omega(\cdot)$ tends towards zero, performing a prediction with the composed model is akin to conducting multiple forward passes, each corresponding to an individual learner. Based on that interpretation, we now transition to a setting that considers the explicit auxiliary minimization of $\Omega(\cdot)$, as follows:

$$ $\operatorname*{minimize}_{\bm{\theta}_{1},\dots,\bm{\theta}_{T}}\quad{ \textstyle\mathop{\mathbb{E}}_{\begin{subarray}{c}t\sim\mathcal{T}\\ \bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})\end{subarray}}\left[\ell_{\operatorname {cur}}(\bm{\theta}_{\mathcal{P}}|\bm{x},\bm{y})+\beta\ \Omega(\bm{\theta}_{1}, \dots,\bm{\theta}_{T})\right]}$ (15) $$

where $\beta\geq 0$ is an hyper-parameter. It is noted that minimizing $\Omega(\cdot)$ encourages alignment among task vectors, especially for those weights that are sensitive/important in the pre-training loss landscape.

Similarly to Jeffares et al. (), the objective of Eq. 15 can be interpreted as a smooth transition between individual and ensemble training. Indeed, given that $\Omega(\cdot)={{\textstyle\sum}}_{t}w_{t}\ell_{\operatorname{cur}}(\bm{\theta}
_{t})-\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$, Eq. 15 can be restated:

$$ $\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})+\beta\ \Omega(\bm{\theta} _{1},\dots,\bm{\theta}_{T})\stackrel{{\scriptstyle\text{\lx@cref{ creftype~refnum}{eq:theorem_one}}}}{{=}}(1-\beta)\ \ell_{\operatorname{cur}}( \bm{\theta}_{\mathcal{P}})+\beta\ {{\textstyle\sum}}_{t=1}^{T}w_{t}\ell_{ \operatorname{cur}}(\bm{\theta}_{t})$ (16) $$

This suggests that by minimizing both the loss $\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ of the joint composed model and $\Omega(\cdot)$, we also implicitly optimize the individual models. Notably, this result not only paves the way for a well-performing ensemble but also for components that are reliable when considered individually.

Incremental ensembles. We now refine the problem in Eq. 15 to account for incremental settings. We firstly bring the expectation inside $\Omega(\cdot)$ (Eq. 14) and replace the Hessian ${\bf{H}_{\ell}}(\bm{\theta}_{0})$ with its expectation, taken across data points $\bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})$ from all tasks up to the current one. Afterwards, we capitalize on a property mentioned in Sec. 2.2, which states that, for a point of maximum likelihood like $\bm{\theta}=\bm{\theta}_{0}$, the expected Hessian of the negative log-likelihood coincides with the Fisher $\mathrm{F}_{\bm{\theta}_{0}}$. As a result, we can approximate $\mathop{\mathbb{E}}_{\bm{x}}\left[{\bf{H}_{\ell}}(\bm{\theta}_{0})\right]$ with the tractable diagonal Fisher $\hat{\mathrm{F}}_{\bm{\theta}_{0}}$. Based on ${\bf{H}_{\ell}}(\bm{\theta}_{0})\approx\hat{\mathrm{F}}_{\bm{\theta}_{0}}$ and further steps (see App. B) , we can rearrange $\Omega(\cdot)$ as:

$$ $\textstyle\Omega(\cdot)\approx\Omega_{\hat{\mathrm{F}}}(\bm{\theta}_{1},\dots, \bm{\theta}_{T})=\frac{1}{2}{{\textstyle\sum}}_{t=1}^{T}w_{t}(1-w_{t}) \underbrace{\operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t})}_{\text{see \lx@cref{creftype~refnum}{eq:ewcdef}}}-{{\textstyle\sum}}_{t=1}^{T}{{ \textstyle\sum}}_{t^{\prime}<t}w_{t}w_{t^{\prime}}{\bm{\tau}_{t}}^{\mathrm{T}} \hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{t^{\prime}}.$ (17) $$

Intuitively, $\Omega_{\hat{\mathrm{F}}}(\cdot)$ aims to preserve pre-training knowledge embodied by $\bm{\theta}_{0}$ through the first term; through the second one, instead, it encourages pairwise alignment between task vectors. Also, we highlight that the summations depend on the number of models $T$. Therefore, the more components we manage within the ensemble, the more crucial it becomes to minimize $\Omega_{\hat{\mathrm{F}}}(\cdot)$ within Eq. 15.

Finally, we plug the approximated barrier term $\Omega_{\hat{\mathrm{F}}}(\cdot)$ into the optimization problem outlined in Eq. 15. As learning proceeds in subsequent tasks, we can optimize the composed model only one task at a time. To prevent biasing the previously learned components of the pool toward current data, at each round $t$ we optimize only the weights $\bm{\theta}_{t}\triangleq\bm{\theta}_{0}+\bm{\tau}_{t}$ of the corresponding $t$-th learner, while freezing the others components of the pool. This yields the following form for the $t$-th learning task:

$$ $\operatorname*{minimize}_{\bm{\theta}_{t}}\quad\textstyle{\mathop{\mathbb{E}}_ {\bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})}\left[\ell_{\operatorname{cur}}(\bm{ \theta}_{\mathcal{P}}|\bm{x},\bm{y})\right]+\beta\ \Omega_{\hat{\mathrm{F}}}( \bm{\theta}_{1},\dots,\bm{\theta}_{t})}.$ (18) $$

Finally, if we optimize only one $\bm{\theta}_{t}$ at a time and $w_{t}=\frac{1}{t}$, then several terms in  Eq. 17 can be ignored, such as $\operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t^{\prime}})\text{ for }t^{
\prime}<t$. Hence, minimizing Eq. 18 is equivalent to minimize:

$$ $\textstyle\mathop{\mathbb{E}}_{\bm{x},\bm{y}\sim p_{t}(\bm{x},\bm{y})}\left[ \ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}}|\bm{x},\bm{y})\right]+\ \frac{\beta}{t}\left[(1-\frac{1}{t})\frac{1}{2}\operatorname{EWC}_{\bm{\theta} _{0}}(\bm{\theta}_{t})-\frac{1}{t}\sum_{t^{\prime}<t}{\bm{\tau}_{t}}^{\mathrm{ T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{t^{\prime}}\right].$ (19) $$

As shown in App. C for both full and LoRA fine-tuning, the gradients of the regularization term in Eq. 19 can be computed in closed form. Importantly, this derivation enables a lightweight optimization, as the analytical gradients maintain constant complexity w.r.t. the number of tasks.

###### Theorem 1 .

## 3 Algorithm(s)

Figure: Algorithm 1 Incremental Task Arithmetic (ITA) vs. Incremental Ensemble Learning (IEL)

We present Incremental Task Arithmetic(ITA) and Incremental Ensemble Learning(IEL), two distinct algorithms for individual and ensemble incremental learning respectively. As shown in Alg. 1, they divide each task into the pre-consolidation and fine-tuning stages, with differences occurring in the latter. We direct the reader to App. E for comprehensive implementation guidelines.

Task pre-consolidation. Due to the pivotal role of the local optimality of $\bm{\theta}_{0}$ (Sec. 2.1), we now follow up on this aspect and consider closed vocabulary models. Unlike open models like CLIP (Radford et al., ), closed vocabulary models require the addition of a tailored classification head on top of the existing layers to handle novel classes. In line with Ortiz-Jimenez et al. (), we denote the process of fine-tuning only the added classification head as linear probing (LP): specifically, LP keeps the rest of the layers fixed to the pre-training $\bm{\theta}_{0}$. In our work, we basically exploit linear probing to enforce the optimality of pre-training weights $\bm{\theta}_{0}$. Namely, during the pre-consolidation phase, we conduct a few preliminary training epochs on the $t$-th incoming task, with the sole purpose of fine-tuning the new classification head. From that point onward, the fine-tuned head $\bm{h}_{0}^{t}$ is regarded as a part of the set of pre-training weights $\bm{\theta}_{0}$. Finally, the pre-consolidation stage concludes with the update of the diagonal Fisher matrix $\hat{\mathrm{F}}_{\bm{\theta}_{0}}$.

Fine-tuning. While the pre-consolidation step is identical for both ITA and IEL, they differ during the fine-tuning phase. The shared goal is to learn a task vector $\bm{\tau}_{t}$ s.t. $\bm{\theta}_{t}\equiv\bm{\theta}_{0}+\bm{\tau}_{t}$ for the current task. However, ITA treats $\bm{\tau}_{t}$ as the weights of an individual model, whereas IEL interprets it as a new learnable component of the composition. In other words, ITA computes predictions through the $t$-th learner $f(\bm{x};\bm{\theta}_{t})$, while IEL leverages the composed function $f(\bm{x};\bm{\theta}_{\mathcal{P}})$. Moreover, their regularizing objectives differ: ITA builds upon Eq. 11 (i.e., the additional EWC-like term computed w.r.t. $\bm{\theta}_{0}$), while IEL exploit Eqs. 18 and 19 to train the composed model. Notably, both approaches can be applied to any fine-tuning strategy in the form of $\bm{\theta}_{0}+\Delta\bm{\theta}$. We conduct experiments on Full Fine-Tuning (i.e., $\tau_{t}\in\mathbb{R}^{F_{OUT}\times F_{IN}}$), Low-Rank Adaptation (LoRA) (Hu et al., ) (i.e., $\tau_{t}=B_{t}A_{t}$), and on (IA)^3 (Liu et al., ). About the latter, let $h$ represent the hidden dimension and $l\in\mathbb{R}^{h}$ denote the (IA)^3 task-specific vector: it can be shown that (IA)^3 is equivalent to $\bm{\theta}_{t}\equiv\bm{\theta}_{0}+\bm{\tau}_{t}$ with $\bm{\tau}_{t}=\bm{\theta}_{0}\odot((l-\mathbf{1}_{h})\otimes\mathbf{1}_{h})$, where $\mathbf{1}_{h}$ is the all-ones vector. For each fine-tuning strategy, we initialize their parameters so that the resulting task vector $\tau_{t}$ starts as a null vector at the beginning.

Importantly, while our derivations regard the second-order approximation $\ell_{\operatorname{cur}}$, the full loss $\ell$ is instead employed in our algorithms. Indeed, as is common in frameworks similar to ours (Chaudhry et al., ; Mirzadeh et al., ), the Taylor approximation is employed to build a surrogate of the exact loss that is both accurate and mathematically tractable. However, when it comes to practice, this proxy is often relaxed, and the full target function is used instead for simplicity.

Computational analysis. As outlined in App. D, both training/inference stages of ITA/IEL maintain constant complexity $\mathcal{O}(1)$ with respect to the number of tasks $T$. Indeed, a single forward pass is sufficient to compute the output of the composed model (constant time). Moreover, we do not need to store a separate set of weights for each task (constant memory), provided we are not interested in more complex forms of composition than the simplest uniform average (as required for model customization and unlearning). Intuitively, the constant complexity can be achieved by treating the composed model as a running cumulative average of the individual models.

## 4 Relation with existing works

Task arithmetic. Standard and Parameter-Efficient (PEFT) fine-tuning have been shown to support addition/subtraction of task vectors. However, while the evidence in (Zhang et al., ; Ilharco et al., ) is primarily empirical, we derive theoretical insights about the pre-conditions for task arithmetic, emphasizing the importance of staying close to the pre-training basin. In this respect, our derivations ground previous findings regarding the efficacy of low learning rates (Ilharco et al., ; Ortiz-Jimenez et al., ). Remarkably, staying within the pre-training basin has also been proved beneficial in (Sadrtdinov et al., ) for ensemble learning. The conditions for compositionality are also studied by Ortiz-Jimenez et al. () on linearized networks (Eq. 7). Albeit considering this work inspirational, we see the pros of task arithmetic in the non-linear regime. Firstly, non-linear models surpass their linearized counterparts in single-task accuracy, making them more attractive. To reduce the gap, linearization-aware fine-tuning has to be used (Liu & Soatto, ), contrarily to our approach that is compatible with the prevalent fine-tuning techniques. Secondly, linearized inference requires the demanding Jacobian-vector product (three times slower than a forward pass).

Ensemble learning. While original model soups (Wortsman et al., ) combine multiple weights fine-tuned on the same dataset, we herein managed to unlock running model soups in cross-dataset incremental settings, with possible returns in terms of forward transfer (Lopez-Paz & Ranzato, ). Moreover, the optimization of the whole deep ensemble is also discussed in Jeffares et al. () for standard ensembles, i.e., averaging the outputs of different models. Their derivation is similar to ours and regards the decomposition of the ensemble loss into the strength of the individual learners and their diversity. However, Jeffares et al. () use this result to elucidate the shortcomings of jointly trained deep ensembles, whereas we leverage it to provide effective regularization for model soups in incremental scenarios. Similar to our IEL, several works (Li et al., ; ) build an ensemble through a cumulative mean of intermediate checkpoints sampled along the training trajectory, with benefits in terms of generalization and preservation of zero-shot pre-training capabilities (Zheng et al., ). Differently, Jolicoeur-Martineau et al. () maintain a population of models trained with varying configurations (e.g., data augmentations). They also gradually push each weight toward the population average, thus encouraging alignment across individual models. Notably, such a behaviour is also positively rewarded by our second-order formulation (see Eq. 17).

Incremental learning. Adding new knowledge in deep networks often degrades the ability on earlier tasks. Approaches against forgetting can be divided into three groups: those involving regularization (Kirkpatrick et al., ; Zenke et al., ), those retaining old data for rehearsal (Lopez-Paz & Ranzato, ; Aljundi et al., ), and those allocating new modules (Mallya & Lazebnik, ; Abati et al., ). Among the latter, the recent SEED (Rypeść et al., ) manages an ensemble of expert networks learned incrementally, sharing similarities with our IEL.
However, SEED stores separate models and combines their outputs, whereas our approach maintains a cumulative weight average of past experts, minimizing memory and computational overhead. This running average can also be achieved by InfLoRA (Liang & Li, ), which addresses interference among LoRA modules through tailored initialization. In contrast, our method addresses this issue through explicit regularization. In addition, a recent trend capitalizes on prompt-tuning (Wang et al., ; ; Smith et al., ), devising a pool of learnable prompts. Notably, the extent to which these models support compositionality is investigated in Perera et al. (). Bowman et al. () propose À-la-carte Prompt Tuning (APT), an attention mechanism that enables the creation of bespoke models by composing arbitrary prompts. Their setting called à-la-carte aligns with the scope of our ITA; however, our work extends to a broader range of PEFT modules, unlike APT that is limited to soft prompts. Finally, the use of linearized networks in Tangent Model Composition (TMC) (Liu & Soatto, ) establishes a further baseline for incremental compositionality.

## 5 Experiments

Datasets. Following affine works (Wang et al., ; Bowman et al., ; Liu & Soatto, ), we evaluate on these class-incremental benchmarks: Split ImageNet-R (Hendrycks et al., ) (10 tasks $\times$ 20 classes each), Split CIFAR-100 (Krizhevsky et al., ) (10 tasks $\times$ 10 classes), Split CUB-200 (Wah et al., ) (10 tasks $\times$ 20 classes), Split Caltech-256 (Griffin et al., ) (10 tasks, as in (Liu & Soatto, )), and Split MIT-67 (Quattoni & Torralba, ) (10 tasks, as in (Liu & Soatto, )). We conduct further tests on the aerial and medical domains using Split RESISC45 (Cheng et al., ) (9 tasks × 5 classes) and Split CropDiseases (Hughes et al., ) (7 tasks × 5 classes). They provide a challenging benchmark for our proposals due to their low domain similarity with the ImageNet pre-training (Oh et al., ). Further details can be found in App. G.

Benchmarking. We compare against recognized incremental methods as EWC (Kirkpatrick et al., ), LwF-MC (Rebuffi et al., ), DER++ (Buzzega et al., ), L2P (Wang et al., ), and CODA (Smith et al., ). We also asses SEED (Rypeść et al., ), InfLoRA (Liang & Li, ), APT (Bowman et al., ), and TMC (Liu & Soatto, ), four approaches featuring compositionality and ensemble learning. Their description and the main differences from our approaches are provided in App. F. All methods, including ours, utilize the same backbone – a ViT-B/16 (Dosovitskiy et al., ) with supervised pre-training on ImageNet21K (Ridnik et al., ) – and the same batch size ($128$). We compute the accuracy on all classes at the end of the final task (Final Accuracy, FA). Following Buzzega et al. (), the hyperparameters are chosen through a grid search on a validation set (i.e., 10% of the training set). This procedure was repeated for each method, ensuring careful tuning of the search space. See the supplementary for 1) the standard deviation of the FA (the results are averaged over three runs, see Sec. H.2); 2) the results expressed as Final Forgetting (Chaudhry et al., ) (Sec. H.1); 3) the selected hyper-parameters (App. I).

**Table 1: Comparison with SOTA (Final Accuracy [$\uparrow$]). Best results in bold, second-best underlined. For a comprehensive analysis, EWC, LwF-MC, DER++, SEED, and TMC rely on full fine-tuning; L2P, CODA, and APT utilize prompt-based learning. Finally, InfLoRA adopts LoRA fine-tuning.**
| Model | IN-R | C-100 | CUB | Caltech | MIT | RESISC | CropDis. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Joint | $86.08$ | $91.74$ | $87.12$ | $93.21$ | $87.19$ | $96.79$ | $99.71$ |
| Finetune | $22.31$ | $21.57$ | $29.35$ | $44.03$ | $18.85$ | $12.90$ | $20.54$ |
| EWC | $58.64$ | $73.49$ | $40.33$ | $73.23$ | $64.44$ | $58.80$ | $70.33$ |
| LWF-MC | $50.93$ | $72.16$ | $21.88$ | $79.73$ | $67.04$ | $68.09$ | $78.28$ |
| DER++ | $60.53$ | $83.02$ | $76.10$ | $86.11$ | $68.88$ | $67.23$ | 98.77 |
| L2P | $67.17$ | $87.32$ | $78.95$ | $91.22$ | $83.17$ | $63.47$ | $75.18$ |
| CODA | $74.12$ | $86.48$ | $78.54$ | $90.57$ | $77.73$ | $69.50$ | $74.65$ |
| SEED | $55.87$ | $83.39$ | $85.35$ | $90.04$ | $\underline{86.34}$ | $74.81$ | $92.77$ |
| APT | $65.32$ | $86.19$ | $69.51$ | $87.71$ | $75.83$ | $49.99$ | $59.37$ |
| InfLoRA | $76.97$ | $87.17$ | $79.14$ | $90.53$ | $79.14$ | $79.92$ | $89.05$ |
| TMC | $60.01$ | $78.42$ | $71.72$ | $82.30$ | $68.66$ | $60.66$ | $66.56$ |
| ITA-FFT | $76.43$ | $89.38$ | $84.80$ | $92.32$ | $85.35$ | $80.50$ | $91.81$ |
| ITA-LoRA | $77.79$ | $\underline{89.96}$ | $\underline{85.55}$ | $92.65$ | 86.60 | $82.00$ | $95.85$ |
| ITA-(IA)^3 | $77.04$ | 90.66 | 85.67 | $\underline{92.67}$ | $84.74$ | 83.73 | $95.41$ |
| IEL-FFT | 80.09 | $89.38$ | $84.89$ | $92.23$ | $82.79$ | $81.42$ | $95.83$ |
| IEL-LoRA | $\underline{79.93}$ | $89.53$ | $84.95$ | $92.19$ | $84.49$ | $\underline{82.53}$ | $\underline{95.88}$ |
| IEL-(IA)^3 | $77.86$ | $89.72$ | $84.57$ | 92.70 | $85.54$ | $81.50$ | $95.68$ |

Comparison with SOTA. As reported in Tab. 1, ITA and IEL outperform existing approaches on all datasets except MIT-67 and CropDisease. At times, they match SEED; however, its reliance on Mixture of Experts makes its inference computationally demanding. Our results far exceed those of TMC, showcasing the potential of the non-linear regime over linearization. Considering the good results on RESISC and CropDis., ITA and IEL do not seem affected by large domain shifts, indicating that our formulation remains effective even when the pre-training optimality is challenged. Finally, given the comparable results of ITA and IEL, we recommend ITA as the preferred starting point for future research, in light of the greater flexibility offered by the individual training paradigm.

**Table 2: For ITA, analysis of the impact of the proposed regularization loss (FA [$\uparrow$]).**
| Model | IN-R | C-100 | CUB | Caltech | MIT | RESISC | CropDis. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ITA-FFT (reg) | 76.43 | 89.38 | 84.80 | 92.32 | 85.35 | 80.50 | 91.81 |
| without Eq. 12 reg. | $8.61$ | $17.59$ | $10.47$ | $12.76$ | $12.01$ | $17.17$ | $20.64$ |
| Eq. 12 only on $\operatorname{CLS}$ | $76.00$ | $87.60$ | $83.54$ | $91.04$ | $81.45$ | $75.26$ | $77.00$ |
| ITA-LoRA (reg) | 77.79 | $89.96$ | 85.55 | 92.65 | 86.60 | 82.00 | $95.85$ |
| without Eq. 12 reg. | $50.17$ | $66.58$ | $60.58$ | $74.87$ | $52.74$ | $37.59$ | $55.86$ |
| Eq. 12 only on $\operatorname{CLS}$ | $77.33$ | 90.03 | 85.55 | $92.59$ | $84.86$ | $80.64$ | 96.22 |

On the impact of regularization. Tab. 2 reports the results of ITA when the regularization term in Eq. 12 is removed (the same analysis for ITA-(IA)^3 and IEL is available in Tab. 7). To evaluate the effect on distinct layers, we additionally assess the model with only the last classification layer regularized (see Eq. 12 only on CLS in Tab. 2). As observed: i) applying Eq. 12 is beneficial for all examined fine-tuning strategies; ii) although regularizing all layers is the most consistent approach, applying Eq. 12 only on the classification head already yields good accuracy. This suggests that compositionality in closed-set models is largely dependent on the learning dynamics of the last layer. Finally, full fine-tuning (FFT) struggles the most when no regularization or partial regularization is applied, in contrast to PEFT modules like LoRA and (IA)^3 that still manage to achieve decent results. We ascribe this evidence to the tendency of PEFT modules to forget less of the pre-trained knowledge (Biderman et al., ), a feature we identify as beneficial for model compositionality.

Figure: Figure 1: Effect of ITA. Best viewed in color.
Refer to caption: x1.png

To gain insights into our regularization, we revisit the foundations of ITA: specifically the inequality in Eq. 5, which states that the risk of the composed model $\hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ is upper bounded by the weighted risk of the individual models $\sum w_{t}\hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{t})$. In practice, there is no guarantee that this upper bound is tight: as these individual models are trained on disjoint tasks, their risk is likely high for examples outside their training distribution. We hence expect the upper bound to be loose: this is shown in Fig. 1, where we measure the effect of our regularization on the empirical risk of the composed model and on the upper bound . From this lens, ITA significantly tightens the upper bound, with a remarkable reduction of the risk associated with the composed model $\hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$. Notably, the latter consistently outperforms the pre-trained model $\hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{0})$ (obtained with linear probing). This indicates that there is a good margin to find a point in parameter space that improves pre-training. Vice versa, the upper bound $\sum w_{t}\hat{\ell}_{\operatorname{cur}}(\bm{\theta}_{t})$ does not always exceed pre-training, even with regularization applied. This raises the theoretical possibility of regret between the composed model and pre-training. This work provides empirical evidence to address it but a theoretical analysis is needed, which we leave for future works.

On the compositional skills of ITA and IEL.  We herein investigate specialization and unlearning capabilities of our approaches. Specifically, given the task vectors trained on the sequence of tasks, we freeze and use them to edit the composed model. As these adaptations involve simple additions and subtractions of task vectors, we can perform specialization and unlearning in a zero-shot fashion requiring no examples. In particular, through the former test on specialization, we examine whether composing only a subset of the task vectors can boost the composed model on the respective selected tasks. In doing so, we focus on three tasks of the incremental sequence i.e., the first, the central, and the last one. The performance is then measured considering the average FA across the tasks we wish to specialize in ($\operatorname{FA}_{\operatorname{TGT}}{}$) and the average FA across the others ($\operatorname{FA}_{\operatorname{CTRL}}{}$). In the second test, instead, our goal is to eliminate knowledge of a specific task without degrading performance on other tasks (unlearning). Specifically, to unlearn a given task, we subtract its associated task vector from the composition, which still includes the weights of the tasks that need to be preserved. Afterwards: i) we measure the accuracy for the unlearned task ($\operatorname{FA}_{\operatorname{TGT}}{}$) and the others ($\operatorname{FA}_{\operatorname{CTRL}}{}$); ii) the procedure is then repeated for all tasks, with the performance averaged across them.

**Table 3: Analysis of compositional capabilities – see Tab. 7 for results on other datasets. In parentheses, we report the gain (or loss) in accuracy on the target task.**
| Dataset | Model | zero-shot specialization | zero-shot unlearning |  |  |
| --- | --- | --- | --- | --- | --- |
| $\operatorname{FA}_{\operatorname{TGT}}{}$ $[\uparrow]$ | $\operatorname{FA}_{\operatorname{CTRL}}{}$ | $\operatorname{FA}_{\operatorname{TGT}}{}$ $[\downarrow]$ | $\operatorname{FA}_{\operatorname{CTRL}}{}$ |  |  |
|  | ITA-LoRA | $80.83$ ($+11.40$) | $50.52$ | $22.77$ ($-55.02$) | $52.72$ ($-25.07$) |
|  | IEL-LoRA | $73.46$ ($-06.68$) | $38.46$ | $18.55$ ($-61.38$) | $41.97$ ($-37.96$) |
| IN-R | TMC | $69.93$ ($+08.36$) | $34.08$ | $45.77$ ($-14.24$) | $54.37$ ($-05.64$) |
|  | ITA-LoRA | $92.80$ ($+01.63$) | $60.06$ | $28.67$ ($-61.29$) | $71.96$ ($-17.99$) |
|  | IEL-LoRA | $77.77$ ($-13.22$) | $37.90$ | $19.48$ ($-70.05$) | $56.52$ ($-33.01$) |
| C-100 | TMC | $87.53$ ($+06.49$) | $45.75$ | $55.63$ ($-22.79$) | $71.83$ ($-06.59$) |

The results are in Tab. 3 and Tab. 7 of Tab. 7, where we compare with TMC by Liu & Soatto (), i.e., model linearization and tangent fine-tuning. Regarding specialization, both ITA and TMC improve the accuracy on the target tasks. IEL, instead, leads to a severe drop during specialization. This yields an interesting finding: while Tab. 1 highlights its performance as satisfying, we observe that the ensemble struggles when any of its members are removed. In an era where ensemble models are experiencing renewed attention (Liang & Li, ; Marouf et al., ), this evidence should be sobering. On the other hand, we underline the specialization capabilities of ITA, with the best absolute performance on the target tasks. We ascribe it to the devised regularization objective and the natural advantages of the non-linear regime. When focusing on unlearning, ITA shows mixed results. Notably, both ITA and TMC consistently reduce the performance of the task to be forgotten. While this also affects the tasks that should be preserved, ITA exhibits a greater distinction between the target and control tasks, indicating a more effective disentanglement of knowledge across tasks.

## 6 Discussion of limitations and future directions

Second-order Taylor approximations are valid tools to make the theory more tractable. For instance, a similar approach was used by Mirzadeh et al. () to support the importance of wider local minima against forgetting. Nonetheless, these approximations often rely on certain concessions, the first regarding their validity during training. Since our approximation is performed at the pre-training weights, it may become inaccurate as parameters drift. While importance-based terms like ours may partially mitigate the drift, other techniques could be used (e.g., a L2-norm penalty on task vectors or low learning rate). Even without explicit countermeasures, existing studies suggest that deep networks often fall into a regime where the loss tends to be almost convex in a neighborhood around their local minima (Goodfellow et al., ; Lucas et al., ; Yunis et al., ). Also, under some conditions, their training dynamics can enter the lazy regime (Chizat et al., ; Jacot et al., ), where these models rapidly achieve near-zero loss with minimal changes to their weights.

Moreover, two other concessions warrant a discussion: approx. i) we implicitly model the weight distribution with a Gaussian (Chaudhry et al., ) (Eq. 9); approx. ii) the Fisher matrix is approximated by its diagonal (Eq. 11). Regarding approx. i), while non-Gaussian posteriors are often cumbersome to manage, Farquhar et al. () challenge the common belief that mean-field approximations are overly restrictive for deep networks. Moreover, although approx. ii) may seem crude, the diagonal approximation is efficient, with a low memory footprint. Still, our approach could profit from more accurate estimations of the Hessian, like the Kronecker factored approximation (Martens & Grosse, ), which considers the interactions between weights of the same layer.

Future work. Through theoretical and empirical analyses, we support the importance of remaining within the pre-training basin to achieve composable deep networks. However, there is still much to explore along this path. We mainly focus on closed-set classification models but it would be noteworthy to extend our analysis to self-supervised pre-training and open-vocabulary models like CLIP. Indeed, recent works (Zheng et al., ) have shown their tendency to forget zero-shot pre-training capabilities while fine-tuning. In this respect, our second-order regularization could significantly aid compositionality in CLIP-based models. Finally, we believe that staying within the pre-train basin is only one aspect to consider; future research should emphasize the exploration of the pre-train basin, to achieve composable modules with a higher degree of specialization on their respective tasks.

## Appendix A Appendix / supplemental material

The appendix is organized via the following contributions:

- •
App. B provides the proofs for Theorems 1 and 17.
- •
App. C illustrates that gradients of Eq. 19 can be derived in closed-form.
- •
App. D reports the computational analysis for our approaches Incremental Task Arithmetic (ITA) and Incremental Ensemble Learning (IEL).
- •
App. E describes the essential details required to reproduce the methods described in the main paper.
- •
Apps. F and G supplements the main paper with additional information regarding the relation with existing works and the adopted benchmarks.
- •
Tab. 7 provides additional experimental results: the forgetting metric, the standard deviation measured in each experiment, an additional ablation study regarding IEL, and finally an extensive evaluation of model compositionality in disparate incremental learning settings.
- •
App. I outlines the hyperparameter configurations adopted in our experiments.

## Appendix B Proofs

### B.1 Proof of Theorem 1

###### Proof.

For the sake of notation, we will use the shortcuts $\ell_{0}\triangleq\ell(\bm{\theta}_{0})$, ${\bf{J}_{\ell}}\triangleq\nabla\ell(\bm{\theta}_{0})$, and ${\bf{H}_{\ell}}\triangleq{\bf{H}_{\ell}}(\bm{\theta}_{0})$. Given a setting with only two learners $A$ and $B$, the second-order approximation of the loss function of the composed model $\mathcal{P}$ is:

$$ $\displaystyle\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ $\displaystyle=\ell_{0}+(\bm{\theta}_{\mathcal{P}}-\bm{\theta}_{0})^{\mathrm{T} }{\bf{J}_{\ell}}+\frac{1}{2}(\bm{\theta}_{\mathcal{P}}-\bm{\theta}_{0})^{ \mathrm{T}}{\bf{H}_{\ell}}(\bm{\theta}-\bm{\theta}_{0})$ $\displaystyle=\underbrace{(w_{a}+w_{b})}_{=1}\ell_{0}+(\underbrace{\bm{\theta} _{\mathcal{P}}-\bm{\theta}_{0}}_{w_{a}\tau_{a}+w_{b}\tau_{b}})^{\mathrm{T}}{ \bf{J}_{\ell}}+\frac{1}{2}(\bm{\theta}_{\mathcal{P}}-\bm{\theta}_{0})^{\mathrm {T}}{\bf{H}_{\ell}}(\bm{\theta}-\bm{\theta}_{0})$ $\displaystyle=w_{a}\ell_{0}+w_{b}\ell_{0}+(w_{a}\tau_{a}+w_{b}\tau_{b})^{ \mathrm{T}}{\bf{J}_{\ell}}+\frac{1}{2}(w_{a}\tau_{a}+w_{b}\tau_{b})^{\mathrm{T }}{\bf{H}_{\ell}}(w_{a}\tau_{a}+w_{b}\tau_{b})$ $\displaystyle=w_{a}\ell_{0}+w_{b}\ell_{0}+w_{a}\tau_{a}^{\mathrm{T}}{\bf{J}_{ \ell}}+w_{b}\tau_{b}^{\mathrm{T}}{\bf{J}_{\ell}}+\frac{1}{2}(w_{a}\tau_{a}^{ \mathrm{T}}+w_{b}\tau_{b}^{\mathrm{T}}){\bf{H}_{\ell}}(w_{a}\tau_{a}+w_{b}\tau _{b})$ $\displaystyle=w_{a}[\ell_{0}+\tau_{a}^{\mathrm{T}}{\bf{J}_{\ell}}]+w_{b}[\ell_ {0}+\tau_{b}^{\mathrm{T}}{\bf{J}_{\ell}}]+\frac{1}{2}(w_{a}\tau_{a}^{\mathrm{T }}{\bf{H}_{\ell}}+w_{b}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}})(w_{a}\tau_{a}+w_{ b}\tau_{b})$ $\displaystyle=w_{a}[\ell_{0}+\tau_{a}^{\mathrm{T}}{\bf{J}_{\ell}}]+w_{b}[\ell_ {0}+\tau_{b}^{\mathrm{T}}{\bf{J}_{\ell}}]+\frac{1}{2}(w_{a}^{2}\tau_{a}^{ \mathrm{T}}{\bf{H}_{\ell}}\tau_{a}+w_{b}^{2}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell }}\tau_{b}+$ $\displaystyle\quad+\ w_{a}w_{b}\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}+w_ {a}w_{b}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{a})$ $$

$$ Given that $\displaystyle\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}\text{ is a scalar and }(ABC)^{\mathrm{T}}=(C^{\mathrm{T}}B^{\mathrm{T}}A^{\mathrm{T}})$ $\displaystyle\rightarrow\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}=(\tau_{a} ^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b})^{\mathrm{T}}=(\tau_{b}^{\mathrm{T}}{ \underbrace{{\bf{H}_{\ell}}}_{\text{symm.}}}^{\mathrm{T}}\tau_{a})=\tau_{b}^{ \mathrm{T}}{\bf{H}_{\ell}}\tau_{a}$ $\displaystyle\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ $\displaystyle=w_{a}[\ell_{0}+\tau_{a}^{\mathrm{T}}{\bf{J}_{\ell}}]+w_{b}[\ell_ {0}+\tau_{b}^{\mathrm{T}}{\bf{J}_{\ell}}]+\frac{1}{2}(w_{a}^{2}\tau_{a}^{ \mathrm{T}}{\bf{H}_{\ell}}\tau_{a}+w_{b}^{2}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell }}\tau_{b}+2w_{a}w_{b}\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b})$ $\displaystyle=w_{a}[\ell_{0}+\tau_{a}^{\mathrm{T}}{\bf{J}_{\ell}}]+w_{b}[\ell_ {0}+\tau_{b}^{\mathrm{T}}{\bf{J}_{\ell}}]+\frac{1}{2}(\underbrace{w_{a}^{2} \tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{a}}_{\begin{subarray}{c}w_{a}^{2}=w_ {a}(1-w_{b})\\ =w_{a}-w_{a}w_{b}\end{subarray}}+w_{b}^{2}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}} \tau_{b}+2w_{a}w_{b}\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b})$ $\displaystyle=w_{a}[\ell_{0}+\tau_{a}^{\mathrm{T}}{\bf{J}_{\ell}}+\frac{1}{2} \tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{a}]+w_{b}[\ell_{0}+\tau_{b}^{\mathrm {T}}{\bf{J}_{\ell}}]+\frac{1}{2}(w_{a}\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau _{a}+w_{b}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}+$ $\displaystyle\quad-w_{a}w_{b}\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{a}-w_{a }w_{b}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}+2w_{a}w_{b}\tau_{a}^{ \mathrm{T}}{\bf{H}_{\ell}}\tau_{b})$ $\displaystyle=w_{a}[\ell_{0}+\tau_{a}^{\mathrm{T}}{\bf{J}_{\ell}}+\frac{1}{2} \tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{a}]+w_{b}[\ell_{0}+\tau_{b}^{\mathrm {T}}{\bf{J}_{\ell}}+\frac{1}{2}\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}]\ +$ $\displaystyle\quad-\frac{1}{2}w_{a}w_{b}(\tau_{a}^{\mathrm{T}}{\bf{H}_{\ell}} \tau_{a}+\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}-2\tau_{a}^{\mathrm{T}}{ \bf{H}_{\ell}}\tau_{b})$ $\displaystyle=w_{a}\ell_{\operatorname{cur}}(\bm{\theta}_{A})+w_{b}\ell_{ \operatorname{cur}}(\bm{\theta}_{B})-\frac{1}{2}w_{a}w_{b}(\tau_{a}^{\mathrm{T }}{\bf{H}_{\ell}}\tau_{a}+\tau_{b}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b}-2\tau_{ a}^{\mathrm{T}}{\bf{H}_{\ell}}\tau_{b})$ $\displaystyle=w_{a}\ell_{\operatorname{cur}}(\bm{\theta}_{A})+w_{b}\ell_{ \operatorname{cur}}(\bm{\theta}_{B})-\frac{1}{2}w_{a}w_{b}(\tau_{a}-\tau_{b})^ {\mathrm{T}}{\bf{H}_{\ell}}(\tau_{a}-\tau_{b}).$ $$

In the multiple learning setting, we have that $\bm{\theta}_{\mathcal{P}}=\bm{\theta}_{0}+\bm{\tau}_{\mathcal{P}}=\bm{\theta}_
{0}+{{\textstyle\sum}}_{t=1}^{T}w_{t}\bm{\tau}_{t}$ with ${{\textstyle\sum}}_{t=1}^{T}w_{t}=1$. Therefore:

$$ $\displaystyle\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ $\displaystyle=\ell_{0}+(\bm{\theta}_{\mathcal{P}}-\bm{\theta}_{0})^{\mathrm{T} }{\bf{J}_{\ell}}+\frac{1}{2}(\bm{\theta}_{\mathcal{P}}-\bm{\theta}_{0})^{ \mathrm{T}}{\bf{H}_{\ell}}(\bm{\theta}-\bm{\theta}_{0})$ $\displaystyle=\underbrace{\sum_{t=1}^{T}w_{t}}_{=1}\ell_{0}+{\bm{\tau}_{ \mathcal{P}}}^{\mathrm{T}}{\bf{J}_{\ell}}+\frac{1}{2}{\bm{\tau}_{\mathcal{P}}} ^{\mathrm{T}}{\bf{H}_{\ell}}{\bm{\tau}_{\mathcal{P}}}=\sum_{t=1}^{T}w_{t}\ell_ {0}+(\sum_{t=1}^{T}w_{t}\bm{\tau}_{t})^{\mathrm{T}}{\bf{J}_{\ell}}+\frac{1}{2} {\bm{\tau}_{\mathcal{P}}}^{\mathrm{T}}{\bf{H}_{\ell}}{\bm{\tau}_{\mathcal{P}}}$ $\displaystyle=\sum_{t=1}^{T}w_{t}\left[\ell_{0}+{\bm{\tau}_{t}}^{\mathrm{T}}{ \bf{J}_{\ell}}\right]+\frac{1}{2}{\bm{\tau}_{\mathcal{P}}}^{\mathrm{T}}{\bf{H} _{\ell}}{\bm{\tau}_{\mathcal{P}}}.$ $$

Let us now focus on the quadratic term:

$$ $\displaystyle{\bm{\tau}_{\mathcal{P}}}^{\mathrm{T}}{\bf{H}_{\ell}}{\bm{\tau}_{ \mathcal{P}}}$ $\displaystyle=(\sum_{t=1}^{T}w_{t}\bm{\tau}_{t})^{\mathrm{T}}{\bf{H}_{\ell}}( \sum_{t=1}^{T}w_{t}\bm{\tau}_{t})=(\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T }}{\bf{H}_{\ell}})(\sum_{t=1}^{T}w_{t}\bm{\tau}_{t})$ $\displaystyle=(\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}})(w _{t}\bm{\tau}_{t}+\sum_{t^{\prime}\neq t}^{T}w_{t^{\prime}}\bm{\tau}_{t^{ \prime}})=\sum_{t=1}^{T}{w_{t}}^{2}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}} \bm{\tau}_{t}+w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\sum_{t^{\prime} \neq t}^{T}w_{t^{\prime}}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\sum_{t=1}^{T}{w_{t}}^{2}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell }}\bm{\tau}_{t}+\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}} \sum_{t^{\prime}\neq t}^{T}w_{t^{\prime}}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\sum_{t=1}^{T}{w_{t}}^{2}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell }}\bm{\tau}_{t}+\sum_{t=1}^{T}\sum_{t^{\prime}\neq t}^{T}w_{t}w_{t^{\prime}} \bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t^{\prime}}$ $\displaystyle\text{Since }\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{ t^{\prime}}=\bm{\tau}_{t^{\prime}}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t} \text{ (symmetry)}$ $\displaystyle=\sum_{t=1}^{T}{w_{t}}^{2}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell }}\bm{\tau}_{t}+2\sum_{t,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}\bm{\tau}_{t}^{ \mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t^{\prime}}$ $$

Given that ${w_{t}}^{2}=w_{t}\cdot w_{t}=w_{t}(1-\sum_{t^{\prime}\neq t}^{T}w_{t^{\prime}}
)=w_{t}-w_{t}\sum_{t^{\prime}\neq t}^{T}w_{t^{\prime}}$:

$$ $\displaystyle=\sum_{t=1}^{T}\left(w_{t}-w_{t}\sum_{t^{\prime}\neq t}^{T}w_{t^{ \prime}}\right)\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t}+2\sum_{t =1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell }}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t}-\sum_{t=1}^{T}\left(w_{t}\sum_{t^{\prime}\neq t}^{T}w_{t^{\prime}} \right)\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t}+2\sum_{t=1,t^{ \prime}<t}^{T}w_{t}w_{t^{\prime}}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t^{\prime}}$ $\displaystyle=\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t}-\sum_{t=1}^{T}\sum_{t^{\prime}\neq t}^{T}w_{t}w_{t^{\prime}}\bm{\tau }_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t}+2\sum_{t=1,t^{\prime}<t}^{T}w_{ t}w_{t^{\prime}}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t}-\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}(\bm{\tau}_{t}^{ \mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t}+\bm{\tau}_{t^{\prime}}^{\mathrm{T}}{ \bf{H}_{\ell}}\bm{\tau}_{t^{\prime}})+2\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{ \prime}}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t}-\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}(\bm{\tau}_{t}^{ \mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t}+\bm{\tau}_{t^{\prime}}^{\mathrm{T}}{ \bf{H}_{\ell}}\bm{\tau}_{t^{\prime}})-2w_{t}w_{t^{\prime}}\bm{\tau}_{t}^{ \mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t}-\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}(\bm{\tau}_{t}^{ \mathrm{T}}{\bf{H}_{\ell}}\bm{\tau}_{t}+\bm{\tau}_{t^{\prime}}^{\mathrm{T}}{ \bf{H}_{\ell}}\bm{\tau}_{t^{\prime}}-2\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell} }\bm{\tau}_{t^{\prime}})$ $\displaystyle=\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t}-\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}(\bm{\tau}_{t}-\bm{ \tau}_{t^{\prime}})^{\mathrm{T}}{\bf{H}_{\ell}}(\bm{\tau}_{t}-\bm{\tau}_{t^{ \prime}})$ $$

Therefore:

$$ $\displaystyle\ell_{\operatorname{cur}}(\bm{\theta}_{\mathcal{P}})$ $\displaystyle=\ell_{0}+(\bm{\theta}_{\mathcal{P}}-\bm{\theta}_{0})^{\mathrm{T} }{\bf{J}_{\ell}}+\frac{1}{2}(\bm{\theta}_{\mathcal{P}}-\bm{\theta}_{0})^{ \mathrm{T}}{\bf{H}_{\ell}}(\bm{\theta}-\bm{\theta}_{0})$ $\displaystyle=\sum_{t=1}^{T}w_{t}\left[\ell_{0}+{\bm{\tau}_{t}}^{\mathrm{T}}{ \bf{J}_{\ell}}\right]+\frac{1}{2}{\bm{\tau}_{\mathcal{P}}}^{\mathrm{T}}{\bf{H} _{\ell}}{\bm{\tau}_{\mathcal{P}}}.$ $\displaystyle=\sum_{t=1}^{T}w_{t}\left[\ell_{0}+{\bm{\tau}_{t}}^{\mathrm{T}}{ \bf{J}_{\ell}}\right]+\frac{1}{2}\sum_{t=1}^{T}w_{t}\bm{\tau}_{t}^{\mathrm{T}} {\bf{H}_{\ell}}\bm{\tau}_{t}-\frac{1}{2}\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{ \prime}}(\bm{\tau}_{t}-\bm{\tau}_{t^{\prime}})^{\mathrm{T}}{\bf{H}_{\ell}}(\bm {\tau}_{t}-\bm{\tau}_{t^{\prime}})$ $\displaystyle=\sum_{t=1}^{T}w_{t}\left[\ell_{0}+{\bm{\tau}_{t}}^{\mathrm{T}}{ \bf{J}_{\ell}}+\frac{1}{2}w_{t}\bm{\tau}_{t}^{\mathrm{T}}{\bf{H}_{\ell}}\bm{ \tau}_{t}\right]-\frac{1}{2}\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}(\bm {\tau}_{t}-\bm{\tau}_{t^{\prime}})^{\mathrm{T}}{\bf{H}_{\ell}}(\bm{\tau}_{t}- \bm{\tau}_{t^{\prime}})$ $\displaystyle=\sum_{t=1}^{T}w_{t}\ell_{\operatorname{cur}}(\bm{\theta}_{t})- \frac{1}{2}\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}(\bm{\tau}_{t}-\bm{ \tau}_{t^{\prime}})^{\mathrm{T}}{\bf{H}_{\ell}}(\bm{\tau}_{t}-\bm{\tau}_{t^{ \prime}}),$ $$

which ends the proof of Theorem 1.
∎

###### Proof.

### B.2 Proof of Eq. 17

###### Proof.

In the dual-learner setting we have:

$$ $\Omega(A,B)=\frac{1}{2}w_{A}w_{B}(\bm{\tau}_{A}-\bm{\tau}_{B})^{\mathrm{T}}{ \bf{H}_{\ell}}(\bm{\theta}_{0})(\bm{\tau}_{A}-\bm{\tau}_{B}).$ $$

If we replace the Hessian matrix with the diagonal Fisher matrix $\hat{\mathrm{F}}_{\bm{\theta}_{0}}$, we get:

$$ $\displaystyle\Omega(A,B)$ $\displaystyle=\frac{1}{2}w_{A}w_{B}(\bm{\tau}_{A}-\bm{\tau}_{B})^{\mathrm{T}} \hat{\mathrm{F}}_{\bm{\theta}_{0}}(\bm{\tau}_{A}-\bm{\tau}_{B})=$ $\displaystyle=\frac{1}{2}w_{A}w_{B}\left[{\bm{\tau}_{A}}^{\mathrm{T}}\hat{ \mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{A}-2{\bm{\tau}_{A}}^{\mathrm{T}}\hat{ \mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{B}+{\bm{\tau}_{B}}^{\mathrm{T}}\hat{ \mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{B}\right].$ $$

If we focus on the first term inside the parenthesis, we obtain:

$$ $\displaystyle{\bm{\tau}_{A}}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm {\tau}_{A}$ $\displaystyle={\bm{\tau}_{A}}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}^{ \nicefrac{{1}}{{2}}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}^{\nicefrac{{1}}{{2}}} \bm{\tau}_{A}=(\hat{\mathrm{F}}_{\bm{\theta}_{0}}^{\nicefrac{{1}}{{2}}}{\bm{ \tau}_{A}})^{\mathrm{T}}(\hat{\mathrm{F}}_{\bm{\theta}_{0}}^{\nicefrac{{1}}{{2 }}}\bm{\tau}_{A})=\lVert\hat{\mathrm{F}}_{\bm{\theta}_{0}}^{\nicefrac{{1}}{{2} }}{\bm{\tau}_{A}}\lVert_{2}^{2}$ $\displaystyle=\sum_{i=1}^{|\theta_{A}|}\hat{\mathrm{F}}_{\bm{\theta}_{0}}^{(i) }(\bm{\tau}^{(i)}_{A})^{2}=\sum_{i=1}^{|\theta_{A}|}\hat{\mathrm{F}}_{\bm{ \theta}_{0}}^{(i)}(\bm{\theta}^{(i)}_{A}-\bm{\theta}_{0}^{(i)})^{2}= \operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{A}).$ $$

Therefore, we can rewrite $\Omega(A,B)$ as:

$$ $\displaystyle\Omega(A,B)$ $\displaystyle=\frac{1}{2}w_{A}w_{B}\left[\operatorname{EWC}_{\bm{\theta}_{0}}( \bm{\theta}_{A})+\operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{B})-2{\bm{ \tau}_{A}}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{B}\right].$ $$

We now generalize to $T\geq 2$. Starting from Eq. 14 and replacing the Hessian with the diagonal Fisher approximation $\hat{\mathrm{F}}_{\bm{\theta}_{0}}$, we obtain:

$$ $\displaystyle\frac{1}{2}\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}$ $\displaystyle(\bm{\tau}_{t}-\bm{\tau}_{t^{\prime}})^{\mathrm{T}}{\bf{H}_{\ell} }(\bm{\tau}_{t}-\bm{\tau}_{t^{\prime}})\approx$ $\displaystyle\approx\frac{1}{2}\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}} \left[\operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t})+\operatorname{EWC} _{\bm{\theta}_{0}}(\bm{\theta}_{t^{\prime}})-2\bm{\tau}_{t}^{\mathrm{T}}\hat{ \mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{t^{\prime}}\right]$ $\displaystyle=\frac{1}{2}\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}\left[ \operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t})+\operatorname{EWC}_{\bm{ \theta}_{0}}(\bm{\theta}_{t^{\prime}})\right]-\sum_{t=1,t^{\prime}<t}^{T}w_{t} w_{t^{\prime}}\bm{\tau}_{t}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{ \tau}_{t^{\prime}}$ $\displaystyle=\frac{1}{2}\sum_{t=1,t^{\prime}\neq t}^{T}w_{t}w_{t^{\prime}} \operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t})-\sum_{t=1,t^{\prime}<t}^ {T}w_{t}w_{t^{\prime}}\bm{\tau}_{t}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_ {0}}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\frac{1}{2}\sum_{t=1}^{T}w_{t}\operatorname{EWC}_{\bm{\theta}_{0 }}(\bm{\theta}_{t})\underbrace{\sum_{t^{\prime}\neq t}^{T}w_{t^{\prime}}}_{1-w _{t}}-\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}}\bm{\tau}_{t}^{\mathrm{T}} \hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{t^{\prime}}$ $\displaystyle=\frac{1}{2}\sum_{t=1}^{T}w_{t}(1-w_{t})\operatorname{EWC}_{\bm{ \theta}_{0}}(\bm{\theta}_{t})-\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime}} \bm{\tau}_{t}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{t^{ \prime}},$ $$

which aligns with the result of Eq. 17.
∎

###### Proof.

## Appendix C Closed form gradients for Eq. 19

We start by deriving the gradients of $\Omega_{\hat{\mathrm{F}}}(\cdot)$ (Eq. 14) w.r.t. the generic task vector $\bm{\tau}_{k}$. We initially consider the case of full-fine tuning, and then generalize the results to LoRA and (IA)^3.

To simplify the calculations, we first rearrange $\Omega_{\hat{\mathrm{F}}}(\cdot)$ as:

$$ $\displaystyle\Omega_{\hat{\mathrm{F}}}(\bm{\theta}_{1},\dots,\bm{\theta}_{T})$ $\displaystyle=\frac{1}{2}\sum_{t=1}^{T}\sum_{t^{\prime}<t}w_{t}w_{t^{\prime}}( \bm{\tau}_{t}-\bm{\tau}_{t^{\prime}})^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta }_{0}}(\bm{\tau}_{t}-\bm{\tau}_{t^{\prime}})$ $\displaystyle\approx\frac{1}{2}\sum_{t=1}^{T}w_{t}(1-w_{t})\operatorname{EWC}_ {\bm{\theta}_{0}}(\bm{\theta}_{t})-\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{ \prime}}\bm{\tau}_{t}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_ {t^{\prime}}\quad\text{(see \lx@cref{creftype~refnum}{eq:fisher_ensemble})},$ $$

Therefore, we can write:

$$ $\displaystyle\frac{\partial{\Omega_{\hat{\mathrm{F}}}}}{\partial{\bm{\tau}_{k}}}$ $\displaystyle=\frac{\partial{\big{[}\frac{1}{2}\sum_{t=1}^{T}w_{t}(1-w_{t}) \operatorname{EWC}_{\bm{\theta}_{0}}(\bm{\theta}_{t})\big{]}}}{\partial{\bm{ \tau}_{k}}}-\frac{\partial{\big{[}\sum_{t=1,t^{\prime}<t}^{T}w_{t}w_{t^{\prime }}\bm{\tau}_{t}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{t^{ \prime}}\big{]}}}{\partial{\bm{\tau}_{k}}}$ $\displaystyle=\frac{\partial{\big{[}\frac{1}{2}w_{k}(1-w_{k})\operatorname{EWC }_{\bm{\theta}_{0}}(\bm{\theta}_{k})\big{]}}}{\partial{\bm{\tau}_{k}}}-$ (20) $\displaystyle\quad\quad\quad\quad-\frac{\partial{\big{[}{{\textstyle\sum}}_{t^ {\prime}<k}w_{k}w_{t^{\prime}}\bm{\tau}_{k}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{ \theta}_{0}}\bm{\tau}_{t^{\prime}}+{{\textstyle\sum}}_{t\neq k,t^{\prime}<t}w_ {t}w_{t^{\prime}}\bm{\tau}_{t}^{\mathrm{T}}\hat{\mathrm{F}}_{\bm{\theta}_{0}} \bm{\tau}_{t^{\prime}}\big{]}}}{\partial{\bm{\tau}_{k}}}$ $\displaystyle=w_{k}(1-w_{k})\frac{\partial{\big{[}\frac{1}{2}\operatorname{EWC }_{\bm{\theta}_{0}}(\bm{\theta}_{k})\big{]}}}{\partial{\bm{\tau}_{k}}}-\sum_{t ^{\prime}\neq k}^{T}w_{k}w_{t^{\prime}}\frac{\partial{\bm{\tau}_{k}^{\mathrm{T }}\hat{\mathrm{F}}_{\bm{\theta}_{0}}\bm{\tau}_{t^{\prime}}}}{\partial{\bm{\tau }_{k}}}$ $\displaystyle=w_{k}(1-w_{k})\ \hat{\mathrm{F}}_{\bm{\theta}_{0}}\odot\bm{\tau} _{k}-w_{k}\sum_{t^{\prime}\neq k}^{T}w_{t^{\prime}}\hat{\mathrm{F}}_{\bm{ \theta}_{0}}\odot\bm{\tau}_{t}^{\prime}$ $\displaystyle=w_{k}\Big{[}(1-w_{k})\ \hat{\mathrm{F}}_{\bm{\theta}_{0}}\odot \bm{\tau}_{k}-\sum_{t^{\prime}\neq k}^{T}w_{t^{\prime}}\hat{\mathrm{F}}_{\bm{ \theta}_{0}}\odot\bm{\tau}_{t}^{\prime}\Big{]}$ $\displaystyle=w_{k}\left[\hat{\mathrm{F}}_{\bm{\theta}_{0}}\odot\big{[}(1-w_{k })\bm{\tau}_{k}-\sum_{t^{\prime}\neq k}^{T}w_{t^{\prime}}\bm{\tau}_{t}^{\prime }\big{]}\right].$ (21) $$

We now suppose that training is performed incrementally on a sequence of $1,2,\dots,k,\dots,T$ tasks. During the $k$-th task, IEL optimizes only $\bm{\tau}_{k}$, namely the task vector instantiated at the beginning of the $k$-th task. Indeed, as discussed in Sec. 2.3 (Eq. 19), the task vectors $\bm{\tau}_{1},\bm{\tau}_{2},\dots,\bm{\tau}_{k-1}$ introduced in preceding tasks are kept frozen. Moreover, at each round, we devise uniform weights, such that $w_{t}=\frac{1}{k}$, with $t<k$. On top of that, we can rewrite the gradients in App. C as:

$$ $\frac{\partial{\Omega_{\hat{\mathrm{F}}}}}{\partial{\bm{\tau}_{k}}}=\frac{1}{k }\left[\hat{\mathrm{F}}_{\bm{\theta}_{0}}\odot\big{[}(1-\frac{1}{k})\bm{\tau}_ {k}-\frac{1}{k}\sum_{t<k}\bm{\tau}_{t}\big{]}\right]$ (22) $$

Notably, the term $\frac{1}{k}\sum_{t<k}\bm{\tau}_{t}$ is a running average of the preceding task vectors. As also discussed in App. D, this reduces the memory/time computational cost for computing the gradients of $\Omega_{\hat{\mathrm{F}}}$ to $\mathcal{O}(1)$.

As a final step, we can exploit the result in Eq. 22 to compute the gradients of LoRA ($\bm{\tau}_{k}=B_{k}A_{k}$) and (IA)^3. For LoRA, we have that:

$$ $\displaystyle\frac{\partial{\Omega}}{\partial{B_{k}}}$ $\displaystyle=\frac{\partial{\Omega}}{\partial{\bm{\tau}_{k}}}\frac{\partial{ \bm{\tau}_{k}}}{\partial{B_{k}}}=\frac{1}{k}\left[\hat{\mathrm{F}}_{\bm{\theta }_{0}}\odot\big{[}(1-\frac{1}{k})\bm{\tau}_{k}-\frac{1}{k}\sum_{t<k}\bm{\tau}_ {t}\big{]}\right]A^{\mathrm{T}}$ $$

$$ $\displaystyle\frac{\partial{\Omega}}{\partial{A_{k}}}$ $\displaystyle=\frac{\partial{\Omega}}{\partial{\bm{\tau}_{k}}}\frac{\partial{ \bm{\tau}_{k}}}{\partial{A_{k}}}=\frac{1}{k}B^{\mathrm{T}}\left[\hat{\mathrm{F }}_{\bm{\theta}_{0}}\odot\big{[}(1-\frac{1}{k})\bm{\tau}_{k}-\frac{1}{k}\sum_{ t<k}\bm{\tau}_{t}\big{]}\right]$ $$

In the case of (IA)^3, where $\bm{\tau}_{k}=\bm{\theta}_{0}\odot((l_{k}-\mathbf{1}_{d})\otimes\mathbf{1}_{d})$:

$$ $\displaystyle\frac{\partial{\Omega}}{\partial{l_{k}}}$ $\displaystyle=\frac{\partial{\Omega}}{\partial{\bm{\tau}_{k}}}\frac{\partial{ \bm{\tau}_{k}}}{\partial{l_{k}}}=\left[\frac{1}{k}\left[\hat{\mathrm{F}}_{\bm{ \theta}_{0}}\odot\big{[}(1-\frac{1}{k})\bm{\tau}_{k}-\frac{1}{k}\sum_{t<k}\bm{ \tau}_{t}\big{]}\right]\odot\bm{\theta}_{0}^{\mathrm{T}}\right]\mathbf{1}_{d}$ $$

where $\mathbf{1}_{d}$ is a $d$-dimensional column vector of ones with shape $d\times 1$.

## Appendix D Computational analysis

Analysis of ITA. Regarding ITA (i.e., individual training), the learning phase has constant $\mathcal{O}(1)$ time and memory complexity. As each model is optimized in isolation, the training cost is similar to that of EWC (Kirkpatrick et al., ) and stems from the computation and storage of the Fisher Information Matrix, along with the calculations of the penalty term. During the evaluation phase, the time complexity of ITA remains $\mathcal{O}(1)$, as it involves a single forward pass on the composed model $f(\cdot;\bm{\theta}_{\mathcal{P}})$. Memory complexity, on the other hand, is $\mathcal{O}(T)$ if maintaining separate models with distinct expertise is desired (e.g., for later re-composition). Otherwise, this can be avoided by considering the composed model as a cumulative average of individual models (see later), resulting in a memory cost of $\mathcal{O}(1)$.

Analysis of IEL. Referring to IEL (i.e., ensemble training), it might initially appear expensive due to the joint training of the ensemble. However, the complexity of IEL remains constant to $\mathcal{O}(1)$ in terms of both memory and time. This reduction is intuitive when we view the ensemble as a cumulative average of individual weights. To demonstrate this, we begin by considering the following cascade of models to be learned:

$$ Task #1 $\displaystyle\rightarrow f(\cdot;\bm{\theta}_{\mathcal{P}}=\bm{\theta}_{0}+ \tau_{1})$ Task #2 $\displaystyle\rightarrow f(\cdot;\bm{\theta}_{\mathcal{P}}=\bm{\theta}_{0}+ \frac{1}{2}\tau_{1}+\frac{1}{2}\tau_{2})$ $\displaystyle\dots$ Task #t $\displaystyle\rightarrow f(\cdot;\bm{\theta}_{\mathcal{P}}=\bm{\theta}_{0}+ \frac{1}{t}\tau_{1}+\frac{1}{t}\tau_{2}+\dots+\frac{1}{t}\tau_{t-1}+\frac{1}{t }\tau_{t})$ $$

At each task, only the latter component of the composed model is learnable, while the preceding components are frozen:

$$ $\text{Task${}_{\#t}$}\rightarrow f(\cdot;\bm{\theta}_{\mathcal{P}}=\bm{\theta} _{0}+\underbrace{\frac{1}{t}\tau_{1}+\frac{1}{t}\tau_{2}+\dots\frac{1}{t}\tau_ {t-1}}_{\text{frozen components}}+\frac{1}{t}\underbrace{\tau_{t}}_{\text{ learnable}}).$ $$

Therefore, as the preceding $\#t-1$ components are kept frozen, we can incorporate them into the initialization weights $\bm{\theta}_{0}\rightarrow\bm{\theta}_{0}^{(t)}$ and optimize:

$$ $\text{Task${}_{\#t}$}\rightarrow f(\cdot;\bm{\theta}_{\mathcal{P}}=\bm{\theta} _{0}^{(t)}+\frac{1}{t}\tau_{t})\quad\text{where}\quad\bm{\theta}_{0}^{(t)}=\bm {\theta}_{0}+\frac{1}{t}\sum_{t^{\prime}=1}^{t-1}\tau_{t^{\prime}}.$ $$

Under this perspective, the learning of the $t$-th task is comparable to standard fine-tuning with a re-scaling factor $=\nicefrac{{1}}{{t}}$. Therefore, provided that $\bm{\theta}_{0}^{(t)}$ is computed once at the beginning of the $t$-th task, the learning of the $t$-th task has constant $\mathcal{O}(1)$ time complexity.

It is noted that optimizing Eq. 19 via gradient descent can be similarly simplified, resulting in a constant $\mathcal{O}(1)$ time complexity. In fact, the gradients derived in App. C involve averaging the weights learned during previous tasks, specifically $\frac{1}{t}\sum_{t^{\prime}=1}^{t-1}\tau_{t^{\prime}}$. Since the weights from previous tasks are frozen during the current task, we can compute this average once and cache it.

To reduce memory complexity to $\mathcal{O}(1)$, we must avoid storing $\tau_{1},\dots,\tau_{t-1}$ separately. This can be accomplished by assuming an initial null displacement $\tau_{\bm{0}}=\bm{0}$ and redefining $\bm{\theta}_{0}^{(t)}$ as:

$$ $\displaystyle\bm{\theta}_{0}^{(t)}$ $\displaystyle=\bm{\theta}_{0}+\frac{1}{t}\sum_{t^{\prime}=1}^{t-1}\tau_{t^{ \prime}}=\bm{\theta}_{0}+\tau_{\bm{0}}+\frac{1}{t}\sum_{t^{\prime}=1}^{t-1} \tau_{t^{\prime}}=\bm{\theta}_{0}+\frac{1}{t}\sum_{t^{\prime}=0}^{t-1}\tau_{t^ {\prime}}=$ $\displaystyle=\bm{\theta}_{0}+\tau_{\operatorname{AVG}}^{(t)}$ $$

The term $\tau_{\operatorname{AVG}}^{(t)}=\frac{1}{t}\sum_{t^{\prime}=0}^{t-1}\tau_{t^{
\prime}}$ is basically the cumulative average of the displacements up to the current task (excluded). The cumulative average is straightforward to compute and, as is well-known, eliminates the need to store all previous values appearing in the sum, with resulting memory complexity $\mathcal{O}(1)$.

## Appendix E Implementation details of ITA and IEL

Task pre-consolidation – Linear Probing. At the beginning of each task, during the pre-consolidation phase, we train a new classification head to account for the classes introduced by the current task. While the rest of the backbone remains frozen, a new linear classification layer is then trained with standard Stochastic Gradient Descent (SGD) for a varying number of epochs, depending on the dataset (typically either $3$ or $8$ epochs, as detailed in App. I).

Task pre-consolidation – Update of the FIM. Afterward, the diagonal Fisher Information Matrix (FIM) has to be updated to incorporate new information from the current task. Similar to Schwarz et al. (), we consider the estimated FIM as an online cumulative average of the squared gradients of the negative log-likelihood. Unlike Schwarz et al. (), we do not introduce the hyper-parameter $\gamma$ to down-weight the importance of the previous estimate. Moreover, following Kunstner et al. (), we compute the true FIM as defined in Eq. 10: i.e., taking the expected gradient on the prediction vector $\hat{y}$. This approach differs from the majority of existing methods (Schwarz et al., ; Kirkpatrick et al., ; Chaudhry et al., ), which apply a further approximation by relying on the empirical FIM: i.e., only considering the gradient of the ground truth label. Consequently, our estimate is more accurate but requires multiple backward passes, but the computational impact of this operation can be significantly reduced through batch-wise parallelization as in George ().

Fine-tuning – Initialization. All methods utilize the same pre-training (supervised) on ImageNet21K (‘vit_base_patch16_224.augreg_in21k‘ from the ‘timm‘ library). Following the original works, we initialize the learnable parameters such that task vectors start from the pre-train initialization:

- •
Full Fine-Tuning: we apply zero-initialization.
- •
LoRA: we use Gaussian initialization for matrix $A$ and zero initialization for matrix $B$.
- •
(IA)^3: we initialize the vectors $l$ with ones.

Fine-tuning – Loss function. Following existing works (Smith et al., ; Wang et al., ), we employ the local cross-entropy loss as the learning objective. Given an example from the current task, while the standard cross-entropy loss considers logits related to all classes, including those learned in previous tasks, the local cross-entropy focuses only on logits corresponding to the classes introduced in the current task. This approach prevents the logits of past classes from being overly penalized during the current task (Caccia et al., ; Boschini et al., ). To ensure a fair comparison in our experiments, we apply this modification to other competing methods (e.g., EWC (Kirkpatrick et al., )).

Fine-tuning – Optimization. In each experiment, we use the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of $3\times 10^{-4}$ for LoRA and (IA)^3 fine-tuning, and $1\times 10^{-4}$ for full fine-tuning. Importantly, in both ITA and IEL, we employ a decoupled strategy (Loshchilov & Hutter, ) to incorporate the gradients of the regularization term. Specifically, we apply the gradients of the regularizing objectives directly to the parameters before the gradient update step, ensuring that the regularization term does not interfere with momentum in the optimization dynamics. By adopting this approach, we observe an empirical improvement in final accuracy, attributed to a more effective minimization of the Riemannian distance relative to the pre-training initialization (see Eq. 12). Finally, we apply this decoupled gradient update exclusively to LoRA and (IA)^3. For full fine-tuning, we refrain from using it as we observed numerical instabilities (i.e., exploding loss).

Furthermore, we decouple the regularization strength applied to the final classification layer from that applied to the rest of the learnable parameters. This introduces two additional hyper-parameters: $\alpha_{\operatorname{CLS}}$ for ITA and $\beta_{\operatorname{CLS}}$ for IEL. Intuitively, by decoupling the regularization weights, we can increase the regularization strength of intermediate layers without causing numerical instabilities, which often stem from the final classification layer.

## Appendix F Extended discussion on related works and competing methods

To deliver the most comprehensive and significant comparison with the state of the art in incremental learning, we chose a combination of well-established standard approaches (such as EWC, DER++, L2P, and CODA) and recent proposals emphasizing compositionality skills (e.g., TMC and APT) and ensemble learning (i.e., SEED). It is important to note that these methods have been heavily influenced by the research trends prevalent at the time they were originally proposed, and therefore, they tend to employ different strategies for fine-tuning the model. To sum up:

- •
EWC, LwF, DER++, SEED and TMC leverage on full-fine tuning.
- •
L2P, CODA, APT resort instead to prompting.

Therefore, achieving a direct apple-to-apple comparison is challenging, as the approaches present in the literature are themselves prone to this issue.

We herein summarize the main aspects of the more important recent methods we compared with.

Elastic Weight Consolidation (EWC) As discussed in the main paper, while our approach regularizes the distance in parameter space with respect to $\bm{\theta}_{0}$, the regularization in EWC (Kirkpatrick et al., ) instead focuses on the weights learned during the preceding task. However, beyond the different regularizing strategies, there is another significant difference between ITA and EWC. While ITA fine-tunes each task starting from the same original pre-trained model $\theta_{0}$, EWC begins with the weights of the previous task.

It is noted that an EWC-like term that protects the last task weights could work as well in our framework based on task vectors. We chose to anchor the model to the pre-training weights to allow for more flexible decentralized learning, wherein multiple task vectors can be trained on different tasks in parallel, with minimal interactions. In contrast, an EWC-like term, which regularizes each successor based on its predecessor, would necessitate training the multiple task vectors in sequence.

Learning to Prompt (L2P)  (Wang et al., ) and CODA-Prompt (Smith et al., ) are two continual learning techniques based on prompting. They fine-tune a pre-trained model through a few learnable parameters stored in a prompt pool, which can be either shared or tied to different tasks. At inference time, the prompts are retrieved from the pool through a query-key search. In terms of memory complexity, our ITA is in line with L2P and CODA when coupled with a parameter-efficient fine-tuning technique such as IA3. Moreover, ITA does not require the additional forward pass on the frozen pre-trained model required to retrieve the prompts.

À-la-carte Prompt Tuning (APT)  (Bowman et al., ) is a prompt-based strategy similar to L2P and CODA. The prompts are trained in isolation (similarly to ITA) and concatenated at inference time to create the composed predictor (no prior query-key search is required). To avoid destructive interference, a tailored masking mechanism is employed in self-attention layers. Differently, our approaches fuse parameters through linear combinations.

Tangent Model Composition (TMC)  (Liu & Soatto, ) is a recent approach addressing continual learning through task arithmetic in the tangent space. TMC builds upon task vectors and is similar to ITA (full fine-tuning). They differ in two aspects: i) TMC applies a first-order approximation of the forward pass to support compositionality, making it two to three times slower than a non-linear forward pass; ii) TMC does not include auxiliary regularization during training.

Selection of Experts for Ensemble Diversification (SEED)  (Rypeść et al., ) is a recent approach that trains an ensemble of models incrementally. For each incoming task, an expert model is chosen from the pool and trained. The major difference with respect to our IEL concerns the inference stage: while IEL performs an ensemble prediction in $\mathcal{O}(1)$ time (thanks to weight averaging), SEED makes inference on all models at test time and averages their predictions.

Model merging. While we address compositionality during training, other approaches focus on post-training techniques, as simple averaging leads to interference (Yadav et al., ) when parameters are redundant or have conflicting signs. TIES (Yadav et al., ) discards uninformative parameters and addresses conflicts via majority voting. Zipit! (Stoica et al., ) merges redundant parameters that produce similar features, while RegMean (Jin et al., ) exploits a closed-form solution for linear layers. Notably, (Matena & Raffel, ) weighs the contribution of each parameter through the Fisher matrix, computed by each individual model at its optimum. This differs from our approach that evaluates the FIM at the pre-training optimum.

## Appendix G Datasets

We conduct a comprehensive evaluation using a variety of benchmarks. Following the current literature on pre-trained CL models (Wang et al., ; ; Smith et al., ), we include conventional image datasets such as Split CIFAR-100 and Split ImageNet-R. We also include Split CUB-200, Split Caltech-256 and Split MIT-67, recently used in the context of composable incremental methods (Bowman et al., ; Liu & Soatto, ). Finally, we assess the adaptability of these pre-trained methods in settings with decreasing domain similarity to the ImageNet pre-training utilized by our backbone model (Oh et al., ; Cui et al., ). Specifically, these settings include the satellite and medical domains, represented by Split RESISC45 and Split CropDiseases, respectively. In the following, we outline the due details:

- •
Standard domains: Split CIFAR-100 (Krizhevsky et al., ) and Split ImageNet-R (Hendrycks et al., ), with respectively $100$ and $200$ classes split into $10$ tasks. We train each task of Split ImageNet-R for 30 epochs and each task of Split CIFAR-100 for 20 epochs. In particular, IN-R is a variant of the ImageNet dataset that includes artistic renditions such as sketches, cartoons, and paintings. It is used to evaluate the robustness and generalization capabilities of models trained on the original ImageNet when tested on out-of-distribution data. Following (Liu & Soatto, ), we also employ Split Caltech-256 (Griffin et al., ) and Split MIT-67 (Quattoni & Torralba, ), dividing both into $10$ tasks ($5$ epoch each).
- •
Specialized domain: We adopt Split CUB-200 (Wah et al., ) to evaluate compositional capabilities in a more fine-grained classification scenario, namely recognizing 200 species of birds. The classes are split across $10$ tasks, each lasting for 50 epochs.
- •
Aerial domain: we use Split RESISC45 (Cheng et al., ), which comprises $30000$ RGB satellite images for land use and land cover classification. The dataset contains $45$ classes (e.g., airport, cloud, island, and so on) divided into $9$ tasks, with each task lasting $30$ epochs.
- •
Medical domain: we finally explore the medical setting (i.e., plant diseases) and conduct experiments on Split CropDiseases (Hughes et al., ). It regards infected leaves with $7$ tasks of $5$ classes each ($5$ epochs).

We base our code on Mammoth (Buzzega et al., ; ), a widely adopted framework in the class-incremental learning literature.

## Appendix H Additional results

**Table 4: Comparison with the SOTA across several benchmarks (Final Forgetting [$\downarrow$]).**
| Model | IN-R | C-100 | CUB | Caltech | MIT | RESISC | CropDis. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Joint | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Finetune | $75.5$ | $85.7$ | $65.79$ | $58.76$ | $87.89$ | $97.6$ | $92.04$ |
| EWC (ON) | $32.43$ | $25.37$ | $53.04$ | $23.52$ | $34.65$ | $43.12$ | $31.58$ |
| LWF-MC | $10.39$ | $6.13$ | $26.2$ | $5.23$ | $3.56$ | $5.66$ | $7.94$ |
| DER++ | $35.81$ | $15.66$ | $13.58$ | $11.19$ | $31.51$ | $43.04$ | $0.76$ |
| L2P | $5.6$ | $6.08$ | $6.63$ | $1.78$ | $5.55$ | $26.29$ | $17.81$ |
| CODA | 4.97 | $5.28$ | $8.82$ | $2.64$ | $7.49$ | $22$ | $15.37$ |
| SEED | $8.96$ | $6.39$ | $5.12$ | $2.67$ | $4.53$ | $8.66$ | $1.19$ |
| InfLoRA | $5.67$ | $4.05$ | $4.58$ | $1.89$ | $6.85$ | $10.77$ | $4.79$ |
| APT | $8.99$ | $6.71$ | $8.82$ | $3.64$ | $6.33$ | $27.74$ | $14.01$ |
| TMC | $11.66$ | $9.2$ | $7.37$ | $6.39$ | $12.43$ | $16.43$ | $16.94$ |
| ITA-FFT | $7.72$ | $3.89$ | 1.39 | $2.44$ | $4.80$ | $6.19$ | $0.95$ |
| ITA-LoRA | $8.00$ | $3.22$ | $2.08$ | $2.38$ | $4.68$ | $8.30$ | $0.68$ |
| ITA-(IA)^3 | $8.72$ | $3.57$ | $2.23$ | $2.39$ | $4.48$ | $7.33$ | $0.76$ |
| IEL-FFT | $5.40$ | 2.20 | $2.04$ | 1.50 | 2.29 | 3.46 | $1.40$ |
| IEL-LoRA | $7.63$ | $5.14$ | 3.46 | $2.46$ | $8.34$ | $8.30$ | $2.14$ |
| IEL-(IA)^3 | $8.34$ | $3.27$ | $1.69$ | $2.20$ | $5.82$ | $6.86$ | $1.04$ |

**Table 5: For each experiment of Tab. 1, the standard deviation of the Final Accuracy (FA).**
| Model | IN-R | C-100 | CUB | Caltech | MIT | RESISC | CropDis. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Joint | $0.87$ | $0.16$ | $0.41$ | $0.43$ | $0.31$ | $0.22$ | $0.21$ |
| Finetune | $4.53$ | $2.83$ | $1.65$ | $2.57$ | $3.49$ | $1.12$ | $0.43$ |
| EWC | $0.73$ | $1.20$ | $0.87$ | $1.29$ | $1.45$ | $0.61$ | $0.99$ |
| LWF-MC | $0.89$ | $1.12$ | $3.23$ | $3.94$ | $2.98$ | $2.85$ | $2.85$ |
| DER++ | $1.12$ | $1.14$ | $0.82$ | $1.27$ | $0.95$ | $1.79$ | $1.29$ |
| L2P | $0.56$ | $1.23$ | $2.12$ | $0.63$ | $0.81$ | $3.42$ | $0.45$ |
| CODA | $0.63$ | $0.95$ | $2.87$ | $0.82$ | $0.69$ | $4.61$ | $0.98$ |
| SEED | $0.18$ | $0.47$ | $0.65$ | $0.55$ | $0.41$ | $0.86$ | $0.23$ |
| APT | $0.84$ | $1.30$ | $2.45$ | $0.95$ | $0.73$ | $1.94$ | $0.68$ |
| InfLoRA | $0.10$ | $0.21$ | $0.89$ | $0.68$ | $2.03$ | $1.16$ | $0.42$ |
| TMC | $0.45$ | $0.85$ | $1.67$ | $0.71$ | $0.92$ | $0.82$ | $0.81$ |
| ITA-FFT | $0.27$ | $0.27$ | $0.39$ | $0.18$ | $0.24$ | $0.56$ | $2.10$ |
| ITA-LoRA | $0.26$ | $0.20$ | $0.13$ | $0.16$ | $0.40$ | $0.89$ | $0.55$ |
| ITA-(IA)^3 | $0.28$ | $0.04$ | $0.18$ | $0.18$ | $0.69$ | $0.68$ | $0.27$ |
| IEL-FFT | $0.11$ | $0.17$ | $0.21$ | $0.09$ | $1.12$ | $0.77$ | $0.90$ |
| IEL-LoRA | $0.36$ | $0.27$ | $0.37$ | $0.41$ | $0.93$ | $0.30$ | $0.43$ |
| IEL-(IA)^3 | $0.23$ | $0.20$ | $0.32$ | $0.21$ | $0.55$ | $0.48$ | $0.41$ |

**Table 6: Ablation study for ITA-(IA)^3 and IEL on several benchmarks (FA [$\uparrow$]).**
| Model | IN-R | C-100 | CUB | Caltech | MIT | RESISC | CropDis. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ITA-(IA)^3 (reg) | 77.04 | 90.66 | 85.67 | 92.67 | $84.74$ | $83.73$ | $95.41$ |
| without Eq. 12 reg. | $71.82$ | $88.43$ | $77.61$ | $90.66$ | $69.14$ | $69.01$ | $63.72$ |
| Eq. 12 only on $\operatorname{CLS}$ | $76.72$ | $90.48$ | $85.56$ | $92.56$ | 85.25 | 84.37 | 95.45 |
| IEL-FFT (reg) | 80.09 | 89.38 | $84.89$ | 92.23 | 82.79 | 81.42 | $95.83$ |
| without Eq. 19 reg. | $40.85$ | $52.56$ | $14.02$ | $53.76$ | $47.63$ | $39.20$ | $31.24$ |
| Eq. 19 reg. only on $\operatorname{CLS}$ | $77.99$ | $85.82$ | 85.30 | $91.43$ | $77.58$ | $76.87$ | 96.18 |
| IEL-LoRA (reg) | 79.93 | 89.53 | 84.95 | 92.19 | 84.49 | 82.53 | 95.88 |
| without Eq. 19 reg. | $51.15$ | $66.01$ | $60.39$ | $70.71$ | $55.38$ | $42.72$ | $45.25$ |
| Eq. 19 reg. only on $\operatorname{CLS}$ | $76.14$ | $86.11$ | $84.43$ | $91.77$ | $82.50$ | $70.05$ | $95.54$ |
| IEL-(IA)^3 (reg) | 77.86 | 89.72 | $84.57$ | $92.70$ | 85.54 | $81.50$ | $95.68$ |
| without Eq. 19 reg. | $73.72$ | $84.00$ | $74.72$ | $89.58$ | $69.82$ | $62.52$ | $66.29$ |
| Eq. 19 reg. only on $\operatorname{CLS}$ | $77.23$ | $89.38$ | 84.70 | 92.76 | $85.43$ | 81.60 | 95.72 |

**Table 7: Analysis of compositional capabilities. In parentheses, we report the gain (or loss) in accuracy on the target task.**
| Dataset | Model | zero-shot specialization | zero-shot unlearning |  |  |
| --- | --- | --- | --- | --- | --- |
| $\operatorname{FA}_{\operatorname{TGT}}{}$ $[\uparrow]$ | $\operatorname{FA}_{\operatorname{CTRL}}{}$ | $\operatorname{FA}_{\operatorname{TGT}}{}$ $[\downarrow]$ | $\operatorname{FA}_{\operatorname{CTRL}}{}$ |  |  |
|  | ITA-LoRA | $80.83$ ($+11.40$) | $50.52$ | $22.77$ ($-55.02$) | $52.72$ ($-25.07$) |
|  | IEL-LoRA | $73.46$ ($-06.68$) | $38.46$ | $18.55$ ($-61.38$) | $41.97$ ($-37.96$) |
| IN-R | TMC | $69.93$ ($+08.36$) | $34.08$ | $45.77$ ($-14.24$) | $54.37$ ($-05.64$) |
|  | ITA-LoRA | $92.80$ ($+01.63$) | $60.06$ | $28.67$ ($-61.29$) | $71.96$ ($-17.99$) |
|  | IEL-LoRA | $77.77$ ($-13.22$) | $37.90$ | $19.48$ ($-70.05$) | $56.52$ ($-33.01$) |
| C-100 | TMC | $87.53$ ($+06.49$) | $45.75$ | $55.63$ ($-22.79$) | $71.83$ ($-06.59$) |
|  | ITA-LoRA | $90.46$ ($+05.21$) | $57.87$ | $68.19$ ($-17.36$) | $74.63$ ($-10.92$) |
|  | IEL-LoRA | $74.03$ ($-10.57$) | $47.95$ | $22.44$ ($-62.51$) | $30.99$ ($-53.96$) |
| CUB | TMC | $71.06$ ($+09.92$) | $44.93$ | $67.91$ ($-03.81$) | $53.22$ ($-18.50$) |
|  | ITA-LoRA | $89.84$ ($-01.21$) | $65.47$ | $80.02$ ($-12.63$) | $82.40$ ($-10.25$) |
|  | IEL-LoRA | $79.70$ ($-12.99$) | $62.45$ | $32.00$ ($-60.19$) | $42.03$ ($-50.16$) |
| Caltech | TMC | $89.23$ ($+10.63$) | $49.46$ | $64.52$ ($-17.78$) | $73.33$ ($-08.97$) |
|  | ITA-LoRA | $89.66$ ($+08.17$) | $57.30$ | $36.99$ ($-49.61$) | $74.75$ ($-11.85$) |
|  | IEL-LoRA | $56.14$ ($-19.38$) | $37.54$ | $06.39$ ($-78.10$) | $32.57$ ($-52.02$) |
| MIT | TMC | $88.03$ ($+10.11$) | $30.56$ | $29.77$ ($-38.89$) | $62.03$ ($-06.63$) |
|  | ITA-LoRA | $89.47$ ($+06.70$) | $49.75$ | $33.38$ ($-48.62$) | $64.06$ ($-17.94$) |
|  | IEL-LoRA | $90.13$ ($+04.92$) | $48.06$ | $32.77$ ($-49.76$) | $65.19$ ($-17.34$) |
| RESISC | TMC | $75.77$ ($+27.63$) | $17.00$ | $06.64$ ($-54.02$) | $52.31$ ($-08.35$) |
|  | ITA-LoRA | $97.63$ ($+00.11$) | $53.05$ | $65.87$ ($-29.98$) | $79.24$ ($-16.61$) |
|  | IEL-LoRA | $90.12$ ($-04.68$) | $48.31$ | $34.01$ ($-61.87$) | $54.43$ ($-41.45$) |
| CropDis. | TMC | $73.60$ ($+09.23$) | $15.95$ | $06.24$ ($-60.32$) | $53.21$ ($-13.35$) |

### H.1 Forgetting

Tab. 4 reports the Final Forgetting metric (Chaudhry et al., ) for our experiments.

### H.2 Standard deviation of FA

Tab. 5 reports the standard deviation of the Final Accuracy metrics reported in Tab. 1.

### H.3 Ablation study on IEL

We herein present an ablative analysis regarding the ensemble-oriented regularization applied to the proposed IEL approach. Specifically, we evaluate the impact of Eq. 19 on the final accuracy and report the results in Tab. 6. After examining them, we can draw conclusions similar to those made for ITA. In particular, the regularization driven by the second-order formulation proves beneficial for achieving effective composition (especially the full-fine tuning), with the final classification layer playing an important role in attaining good performance.

### H.4 Additional experiments on incremental model compositionality

Tab. 7 reports the results of ITA, IEL and TMC on additional datasets, with a focus on their compositionality capabilities (i.e., specialization and unlearning).

## Appendix I Hyperparameters

The hyperparameters employed for each experiment are reported in the following subsections (one for each dataset).

### I.1 ImageNet-R

CODA-Prompt: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

DER++: $\alpha=$3.0\text{\times}{10}^{-01}$$; $\beta=$8.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

EWC: $\epsilon=$1.0\text{\times}{10}^{02}$$; $\gamma=1.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

L2P: $\operatorname{lr}=$2.5\text{\times}{10}^{-03}$$;

LWF-MC: $\operatorname{wd}=0.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

SEED: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

SGD: $\operatorname{lr}=$3.0\text{\times}{10}^{-02}$$;

TMC: $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

InfLoRA: $\operatorname{lr}=$5.0\text{\times}{10}^{-04}$$ $r=$10$$; $\epsilon=0.98$;

For both ITA and IEL: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\operatorname{lr}_{\operatorname{pre-tuning}}=$1.0\text{\times}{10}^{-02}$$;

ITA-FFT: $\alpha=$5.0\text{\times}{10}^{01}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

ITA-LoRA: $\alpha=$2.0\text{\times}{10}^{-02}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

ITA-(IA)^3: $\alpha=$7.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-03}$$;

IEL-FFT: $\beta=$5.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

IEL-LoRA: $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-(IA)^3: $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-03}$$;

### I.2 CIFAR-100

CODA-Prompt: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

DER++: $\alpha=$3.0\text{\times}{10}^{-01}$$; $\beta=$8.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

EWC: $\epsilon=$1.0\text{\times}{10}^{02}$$; $\gamma=1.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

L2P: $\operatorname{lr}=$2.5\text{\times}{10}^{-03}$$;

LWF-MC: $\operatorname{wd}=0.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

SEED: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

SGD: $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

TMC: $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

InfLoRA: $\operatorname{lr}=$5.0\text{\times}{10}^{-04}$$ $r=$10$$; $\epsilon=0.95$;

For both ITA and IEL: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\operatorname{lr}_{\operatorname{pre-tuning}}=$1.0\text{\times}{10}^{-02}$$;

ITA-FFT: $\alpha=$2.0\text{\times}{10}^{03}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

ITA-LoRA: $\alpha=$2.0\text{\times}{10}^{-02}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

ITA-(IA)^3: $\alpha=$2.0\text{\times}{10}^{-02}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-03}$$;

IEL-FFT: $\beta=$2.0\text{\times}{10}^{03}$$; $\beta_{\operatorname{CLS}}=$2.5\text{\times}{10}^{-02}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

IEL-LoRA: $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-(IA)^3: $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$2.5\text{\times}{10}^{-02}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

### I.3 CUB-200

CODA-Prompt: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

DER++: $\alpha=$3.0\text{\times}{10}^{-01}$$; $\beta=$8.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

EWC: $\epsilon=$1.0\text{\times}{10}^{01}$$; $\gamma=$9.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

L2P: $\operatorname{lr}=$2.5\text{\times}{10}^{-03}$$;

LWF-MC: $\operatorname{wd}=$1.0\text{\times}{10}^{-04}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

SEED: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

SGD: $\operatorname{lr}=$3.0\text{\times}{10}^{-02}$$;

TMC: $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

InfLoRA: $\operatorname{lr}=$5.0\text{\times}{10}^{-04}$$ $r=$10$$; $\epsilon=0.98$;

For both ITA and IEL: $\operatorname{lr}_{\operatorname{pre-tuning}}=$1.0\text{\times}{10}^{-02}$$; $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$.

ITA-FFT: $\alpha=$5.0\text{\times}{10}^{02}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

ITA-LoRA: $\alpha=$5.0\text{\times}{10}^{00}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

ITA-(IA)^3: $\alpha=$7.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-03}$$;

IEL-FFT: $\beta=$5.0\text{\times}{10}^{03}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-05}$$;

IEL-LoRA: $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-(IA)^3: $\beta=$5.0\text{\times}{10}^{02}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

### I.4 Caltech-256

CODA-Prompt: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

DER++: $\alpha=$3.0\text{\times}{10}^{-01}$$; $\beta=$8.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

EWC: $\epsilon=$1.0\text{\times}{10}^{02}$$; $\gamma=1.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

L2P: $\operatorname{lr}=$2.5\text{\times}{10}^{-03}$$;

LWF-MC: $\operatorname{wd}=0.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

SEED: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

SGD: $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

TMC: $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

InfLoRA: $\operatorname{lr}=$5.0\text{\times}{10}^{-04}$$ $r=$10$$; $\epsilon=0.99$;

For both ITA and IEL: $\operatorname{lr}_{\operatorname{pre-tuning}}=$1.0\text{\times}{10}^{-02}$$.

ITA-FFT: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\alpha=$2.0\text{\times}{10}^{03}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

ITA-LoRA: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\alpha=$2.0\text{\times}{10}^{-02}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

ITA-(IA)^3: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$; $\alpha=$7.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-03}$$;

IEL-FFT: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\beta=$5.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

IEL-LoRA: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$5.0\text{\times}{10}^{00}$$; $\beta_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-(IA)^3: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

### I.5 MIT-67

CODA-Prompt: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

DER++: $\alpha=$3.0\text{\times}{10}^{-01}$$; $\beta=$8.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

EWC: $\epsilon=$1.0\text{\times}{10}^{02}$$; $\gamma=1.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

L2P: $\operatorname{lr}=$2.5\text{\times}{10}^{-03}$$;

LWF-MC: $\operatorname{wd}=0.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

SEED: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

SGD: $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

TMC: $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

InfLoRA: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$ $r=$10$$; $\epsilon=0.95$;

For both ITA and IEL: $\operatorname{lr}_{\operatorname{pre-tuning}}=$1.0\text{\times}{10}^{-02}$$.

ITA-FFT: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$; $\alpha=$5.0\text{\times}{10}^{03}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

ITA-LoRA: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$; $\alpha=$5.0\text{\times}{10}^{00}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

ITA-(IA)^3: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$; $\alpha=$5.0\text{\times}{10}^{00}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-FFT: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$; $\beta=$5.0\text{\times}{10}^{03}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

IEL-LoRA: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-(IA)^3: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=3$; $\beta=$2.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

### I.6 RESISC

CODA-Prompt: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

DER++: $\alpha=$3.0\text{\times}{10}^{-01}$$; $\beta=$8.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

EWC: $\epsilon=$1.0\text{\times}{10}^{02}$$; $\gamma=$9.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

L2P: $\operatorname{lr}=$2.5\text{\times}{10}^{-03}$$;

LWF-MC: $\operatorname{wd}=0.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

SEED: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

SGD: $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

TMC: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

InfLoRA: $\operatorname{lr}=$5.0\text{\times}{10}^{-04}$$ $r=$10$$; $\epsilon=0.98$;

For both ITA and IEL: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$; $\operatorname{lr}_{\operatorname{pre-tuning}}=$1.0\text{\times}{10}^{-02}$$.

ITA-FFT: $\alpha=$1.0\text{\times}{10}^{04}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

ITA-LoRA: $\alpha=$2.0\text{\times}{10}^{-02}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

ITA-(IA)^3: $\alpha=$5.0\text{\times}{10}^{00}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-03}$$;

IEL-FFT: $\beta=$5.0\text{\times}{10}^{03}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$3.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

IEL-LoRA: $\beta=$2.0\text{\times}{10}^{02}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-02}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-(IA)^3: $\beta=$2.0\text{\times}{10}^{00}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-02}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-03}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

### I.7 CropDisease

CODA-Prompt: $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

DER++: $\alpha=$3.0\text{\times}{10}^{-01}$$; $\beta=$8.0\text{\times}{10}^{-01}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-03}$$;

EWC: $\epsilon=$1.0\text{\times}{10}^{00}$$; $\gamma=1.0$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

L2P: $\operatorname{lr}=$2.5\text{\times}{10}^{-03}$$;

LWF-MC: $\operatorname{wd}=$1.0\text{\times}{10}^{-04}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

SEED: $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

SGD: $\operatorname{lr}=$1.0\text{\times}{10}^{-02}$$;

TMC: $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

InfLoRA: $\operatorname{lr}=$5.0\text{\times}{10}^{-04}$$ $r=$10$$; $\epsilon=0.98$;

For both ITA and IEL: $\#\operatorname{epochs}_{\operatorname{pre-tuning}}=8$; $\operatorname{lr}_{\operatorname{pre-tuning}}=$1.0\text{\times}{10}^{-02}$$;.

ITA-FFT: $\alpha=$5.0\text{\times}{10}^{03}$$; $\alpha_{\operatorname{CLS}}=$2.5\text{\times}{10}^{-02}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-04}$$;

ITA-LoRA: $\alpha=$5.0\text{\times}{10}^{00}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

ITA-(IA)^3: $\alpha=$2.0\text{\times}{10}^{-02}$$; $\alpha_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\alpha_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-FFT: $\beta=$2.0\text{\times}{10}^{02}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$1.0\text{\times}{10}^{-05}$$;

IEL-LoRA: $\beta=$5.0\text{\times}{10}^{01}$$; $\beta_{\operatorname{CLS}}=$1.0\text{\times}{10}^{-01}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

IEL-(IA)^3: $\beta=$2.0\text{\times}{10}^{00}$$; $\beta_{\operatorname{CLS}}=$2.5\text{\times}{10}^{-02}$$; $\beta_{\operatorname{CLS-prior}}=$1.0\text{\times}{10}^{-02}$$; $\operatorname{lr}=$3.0\text{\times}{10}^{-04}$$;

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How different are the task vectors learned with ITA vs IEL ? Do you find them to be well-aligned ?
soundness: 4
strengths: 1. The paper proposes a novel theoretical analysis of the multitask loss through a second-order Taylor expansion, leading to interesting insights (namely equation 5 & 14)
2. The authors leverage this insight to propose new methods for incremental learning with weight-space merging, showing favorable results. 
3. The paper is well-structured, and the experimental section contains relevant ablations showing the effectiveness of the proposed components
summary: The paper conducts a theoretical analysis of model fusion in parameter space through the analysis of the second-order Taylor approximation of the multitask loss function. 

The authors begin by showing that the 2nd order approximation of the multitask (MT) loss, when evaluated on the $w$-weighted averaged of individual models ($\theta_P = w_1\theta_1 + ... + w_t\theta_t$), is upper bounded by $w$-weighted sum of individual models evaluated on the same multitask data. From this observation, the authors argue that, if each individual model could perform well on all tasks, this would drive the MT loss on $\theta_P$ down. 

Given that individual models cannot be optimized on data outside their training distribution, the authors instead propose to regularize individual models towards the base pretrained model $\theta_0$, for such samples. This leads the authors to propose a EWC-like regularizer between $\theta_0$ and $\theta_t$ (when optimizing $\theta_t$) where the Empirical Fisher Information Matrix can be computed iteratively, accumulating gradients from $\theta_0$ on data from task $t$. 

Furthermore, the authors proceed to quantify the exact gap between the previous inequality, showing that term is a function of the weighted alignment across task vectors. The authors then proceed to propose a new learning algorithm, which extends the previous EWC regularizer to also penalize not only differences between $\theta_0$ and $\theta_t$, but also difference between $\theta_t$ and subsequent  learned task vectors $\theta_{<t}$. 

The authors proceed to analyze their methods on class-incremental learning benchmarks, showing favorable results. An ablation showing the importance of the EWC regularizer is also provided. Finally, akin to Task Arithmetic manipulations, the authors show that task vectors can be used to enhance or unlearn specific task knowledge.
weaknesses: I have some reservations as to the motivation of the proposed methods. 

1. Regarding equation 5, I agree that should all individual models perform well on all tasks, then  $\theta_P$ would likely perform quite well. That being said, I find that approximating this condition by regularizing the task vectors to stay close to the base model somewhat of a big conceptual leap. By construction, the base model performs sub-optimally on the tasks for which we optimize a task vector, otherwise the task-specific optimization does not serve much purpose. Therefore, in Eq. 5, wouldn't replacing $\theta_t$ with $\theta_0$ for any $x \sim p_{t'}, t' \neq t$, still make the upper bound loose, and therefore not useful ?

2. The authors motivate ITA vs IEL as an approach that can be executed in a decentralized fashion. However, I am not sure this is the case; given that the FIM is updated sequentially, how would one train several tasks in a decentralized manner without communicating FIM statistics ?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - ll. 028: "Recent AI technologies are predominantly being viewed as monoliths": while I understand the point authors are making here, it must be acknowledged that modular architectures, such as MoE have become increasingly widely used in recent years (maybe as wide as monolithic models)
- ll. 152-157: the paragraph seems to argue that the right-hand side of equation 5 will increase as new tasks are learned. But in equation 5's right-hand side, each individual loss starts from the base model and does not depend on any form of continual learning. So I am not sure I understand the argument being made here.
- would the pre-consolidation step be necessary in language modelling, where the output space is usually shared across tasks? 
Empirical section:
- are the different classification heads of the task-specific models also merged together? if so, does it mean that all tasks must have the same number of classes (i.e. share the output space)? 
- since IN-R, C-100 and CUB are very much id-distribution w.r.t pre-training on imagenet, I wonder whether simple fine-tuning of the final classification layer, which can be a metric-based classifier with no forgetting, can be sufficient to achieve good performance? (e.g. see [4])
- Did the authors investigate the effect of pre-consolidation on the CL performance?
soundness: 3
strengths: Overall, while the idea of using model merging for continual learning has been explored in previous works, see e.g. [1], this paper does a good job of first formalizing intuitive ideas into an appealing theoretical framework, and subsequently validating these intuitions (and theoretical insights) empirically.  While I have several questions regarding several formulations made in the paper, the overall quality of the work is commendable. 

Another strength of this work is its empirical section. Even though experiments are performed only on the image classification domain, a broad range of datasets is covered including also datasets which are OOD w.r.t the backbone.
summary: The paper explores how incremental learning can be addressed through incremental model merging. The theoretical framework is based on the following intuition(s):
- if the base model is already well suited for the downstream asks (\theta_0 is a local min.), fine-tuned versions of this base model will merge better. This connects well to the recent work [5], which shows that stronger base models (instruction tuned, or more params.) lead to better merging.
This intuition serves as a starting point for their theoretical analysis and the derivation of the **ITA** method, which is similar in spirit to ColD Fusion (which does not include the EWC term) [1]. ITA ensures that each new fine-tuned model lies in the viscidity of the base model (as measured by Riemannian distance), which intuitively results in better merging.

It has to be noted, that because this work addressed image classification setting where each task has a different output space, in order to make sure that the base model is a local minimum of the downstream tasks, authors propose the so-called "task pre-consolidation" step, which involves training a task-specific on a fixed backbone before the backbone's params. are learned. The importance of this type of alignment has been recently explored in [2].

**ITA method**: in essence, this method is similar to EWC, with the difference that for each new task we start from the base model (\theta_0) and not from the model learned for the previous task (I have some questions regarding the motivation of this approach, see questions part)

While ITA enables completely independent model training in the spirit of "MoErging" methods [3], one problem with ITA is the fact that each new task's model only transfers knowledge from the base model and not from other tasks. Motivated by this, the paper proposes **IEL** method. IEL optimizes each new task vector assuming access to the merged model on tasks seen so far (somewhat standard in CL literature). This amounts to using both EWC regularization w.r.t to the base model and task alignment term ensuring the new task vector is similar to the previous ones. Importantly, the gradients of the regularization term in IEL can be computed efficiently. 

This paper presents a solid empirical section with several interesting insights. All the theoretical insights are substantiated with empirical validations.


[1] Don-Yehiya, Shachar, et al. "Cold fusion: Collaborative descent for distributed multitask finetuning." _arXiv preprint arXiv:2212.01378_ (2022).

[2] Schmidt, Fabian David, Ivan Vulić, and Goran Glavaš. "Free lunch: Robust cross-lingual transfer via model checkpoint averaging." _arXiv preprint arXiv:2305.16834_ (2023).

[3] Yadav, Prateek, et al. "A survey on model moerging: Recycling and routing among specialized experts for collaborative learning." _arXiv preprint arXiv:2408.07057_ (2024).

[4] Ostapenko, Oleksiy, et al. "Continual learning with foundation models: An empirical study of latent replay." _Conference on lifelong learning agents_. PMLR, 2022.

[5] Yadav, Prateek, et al. "What Matters for Model Merging at Scale?." _arXiv preprint arXiv:2410.03617_ (2024).
weaknesses: - This works connects to several existing works in the realm of model merging, see the "summary" section of my review. These works are not cited in the current version, maybe because these connections were not obvious to the authors. I suggest the authors revisit this potential connection and cite the related works.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: see weakness.
soundness: 2
strengths: This approach reveals the importance of staying within the pre-training 'basin' to create composable modules. To achieve this, the authors introduce two incremental training algorithms: 1) training multiple models individually, and 2) optimizing the entire composed model.

The theoretical analysis section is well-developed, and the experiments are clearly presented.

Tested on incremental classification tasks, this approach enables the creation of an effective multi-task model, with capabilities for unlearning and specialization in specific tasks.
summary: In this work, the author(s) examines the compositional properties of fine-tuning deep pre-trained models, focusing on how specialized modules within these models can be combined to create a single, multi-task model. To be specific, the author leverage a second-order Taylor approximation to understand what promotes compositionality in non-linear networks.
weaknesses: Major:

The motivation for using the second-order linear approximation is unclear. To my understanding, this work employs a similar approach to neural tangent kernel (NTK) analyses. However, in the linear approximation, NTK typically uses the first-order Taylor expansion, here the author employs the second order. Why not use the first-order Taylor expansion or perhaps a third-order expansion? So, it would be great the author add a section and comparing their work with [1-2], especially the Eq.3-4 with Lemma 3.1 in [2].

Also, there are NTK-based class incremental learning works [3-5]. The authors could include a section in the introduction or discussion to clearly point out the differences and benefits of using their approach compared to these works.

Minor issue: 

Some notation is hard to follow and not clearly defined, such as $ L_{\text{cur}}$. The definition of in incremental learning $t,T,\tau$ might also be confusing. The dataset $D_t$ contains $n_t$ training samples $x, y) \sim p_t(x, y)$, drawn from a distribution that varies across tasks. The authors use $\tau$ for $ T $ from $1 \ldots t $, so it is likely that $D_T$ would be more appropriate.


[1] Neural tangent kernel: Convergence and generalization in neural networks.

[2] On Exact Computation with an Infinitely Wide Neural Net

[3] Tkil: Tangent kernel optimization for class balanced incremental learning

[4] Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective

[5] A Continual Learning Algorithm Based on Orthogonal Gradient Descent Beyond Neural Tangent Kernel Regime

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I see some replay-based methods like DER++ are included in Table 1, but I don't see any mention of the experimental setup used for these comparisons (e.g. assumptions on buffer size)
- Equation (13) suggests that by minimizing $\Omega$, IEL also seeks to encourage linearity in the loss, as done in ITA.  Can you provide some intuition (formal or informal) regarding under what regimes IEL fairs better than ITA, and vice-versa?
soundness: 3
strengths: - The paper explores the model composition approach for continual learning, which has strong advantages in more general settings, for instance in federated setups where individual models can be trained in parallel on disjoint datasets and then composed at the end.
- I appreciate the idea of enforcing local convexity or linearity via regularization, since this circumvents the need to do so explicitly via model linearization techniques like in TMC, and it appears to work very well in practice (Table 1).
- The composed model training objective proposed in IEL is also intuitive, essentially "learning to compose" since each new task perturbation is conditioned on all previously seen ones.
- Experiments are comprehensive and a variety of datasets are explored at least in the class-incremental setting. Method performs well across all datasets.
summary: The paper studies model compositionality applied under class-incremental continual learning settings, where individual models fine-tuned on disjoint tasks are composed via weight averaging. By studying the second-order Taylor expansion of deep networks about a strong initial condition, the paper motivates two modifications to existing training objectives. The first encourages local linearity in model weights by minimizing the change in weights (from the pre-trained point) resulting from fine-tuning, which is shown to penalize deviations between the pre-trained and fine-tuned model outputs. The second operates in the regime where individual model weight perturbations are fine-tuned using the loss computed from the _composed_ model. The paper proposes adding a regularizer to minimize the deviation between the loss resulting from weight composition, and the loss resulting from output composition (ensembling).
weaknesses: - The regularization condition proposed in Sec 2.2 does not seem to necessitate the second-order analysis, since it simply aims to enforce a local (first-order) linear structure by making perturbation smaller.
- The wording on "incrementally train(ing) the whole composed model rather than its individual components" used throughout the paper is misleading. I only understood what this statement is trying to convey after seeing equation (15) on Page 5. It appears to me that what is being trained is still individual perturbation components ($\tau_1, \ldots, \tau_T$) , but only that the _loss_ is computed based on the composed model. I would suggest modifying the terminology for better clarity.
- L120 assumes that the pre-trained initialization $\theta_0$ is globally optimal for a quadratic model trained on the union of all tasks. This seems to be a very strong assumption that likely would never hold in practice (why train/compose when the initialization is already optimal?), contrary to what is claimed in footnote 1. This weakens the theoretical analysis.
- Lacks wall-clock time comparison ($\mathcal{O}(1)$ does not shed any light on the actual cost incurred), for instance that used to update the FIM, which is important to faithfully evaluate the method w.r.t. to other works.
- The need for the task pre-consolidation (linear probing, etc.) stage requires additional training steps, so comparisons to most existing methods that do not require this would not be completely fair. If the paper did not implement control steps to make fairer comparisons, then this should be mentioned explicitly in the limitations.

Meta-review: This work presents a theoretical analysis on model merging. Through the second-order Taylor decomposition of the multi-task loss, they find that it is important that the weights of the models to be composed lie within the pre-training basin. This motivates introducing a new incremental learning algorithm that keeps the weights of each task close to the original pre-trained weights in a similar way to EWC, achieving promising results.

All reviewers find this is a good work and there is consensus towards acceptance.

Decision: Accept (Spotlight)