Closed-Form Merging of Parameter-Efficient Modules for Federated Continual Learning

## Abstract

Abstract Model merging has emerged as a crucial technique in Deep Learning, enabling the integration of multiple models into a unified system while preserving performance and scalability. In this respect, the compositional properties of low-rank adaptation techniques (e.g., LoRA) have proven beneficial, as simple averaging LoRA modules yields a single model that mostly integrates the capabilities of all individual modules. Building on LoRA, we take a step further by imposing that the merged model matches the responses of all learned modules. Solving this objective in closed form yields an indeterminate system with A 𝐴 A italic_A and B 𝐵 B italic_B as unknown variables, indicating the existence of infinitely many closed-form solutions. To address this challenge, we introduce LoRM, an alternating optimization strategy that trains one LoRA matrix at a time. This allows solving for each unknown variable individually, thus finding a unique solution. We apply our proposed methodology to Federated Class-Incremental Learning (FCIL), ensuring alignment of model responses both between clients and across tasks. Our method demonstrates state-of-the-art performance across a range of FCIL scenarios.

## 1 Introduction

Humans naturally excel at learning a diverse array of skills independently, effortlessly acquiring knowledge across multiple domains throughout their lives. In contrast, the traditional paradigm for artificial neural networks relies on training a unified model on a single, large dataset. While this approach facilitates the simultaneous incorporation of different skills, it lacks the capacity for specialized or incremental learning, making it less adaptable and responsive to changes in the environment. To overcome this limitation and mimic human flexibility, various paradigms have been developed to enhance neural networks’ ability to manage diverse skills effectively. Multi-Task Learning (Caruana, ) involves training a model on several tasks simultaneously, promoting the sharing of representations across tasks, while Continual Learning (CL) (McCloskey & Cohen, ) focuses on enabling models to learn tasks incrementally without forgetting prior knowledge. Federated Learning (FL) (McMahan et al., ), on the other hand, focuses on decentralized training by distributing data and computation across separate clients, each specializing in their local task. While each of these scenarios has its own unique characteristics, they all share the common objective of integrating task-specific modules into a unified framework.

Recently, large pre-trained architectures have facilitated model editing (Ortiz-Jimenez et al., ) and specialization (Bowman et al., ), particularly for fine-tuning downstream tasks. In practice, deep models often leave their parameters fixed, leveraging Parameter-Efficient Fine-Tuning (PEFT) techniques to adapt to new tasks effectively. Among PEFT methods, Low-Rank Adaptation (LoRA) (Hu et al., ) has emerged as a prominent approach. LoRA introduces residual weights in the form of $\Delta{\bm{W}}={\bm{B}}{\bm{A}}$, where ${\bm{B}}$ and ${\bm{A}}$ are low-rank matrices. These residuals, commonly referred to as task vectors (Ilharco et al., ), form the foundation of the novel model merging literature, which has introduced various approaches for their integration. For example, Zhang et al. () explore the combination of task vectors through linear arithmetic operations, while other works focus on identifying optimal coefficients for weighting these modules during aggregation (Yadav et al., ; Yang et al., ; Wu et al., ). In contrast to these methods, we propose a novel solution that merges LoRA modules in a closed-form.

To evaluate the feasibility of our approach, we base our investigation on a well-defined empirical framework, situating our work at the intersection of Federated Learning and Continual Learning. These two paradigms are ideal for assessing the merging of task vectors, as they encompass both spatial (across clients) and temporal (over tasks) aggregation. Specifically, in Federated Class-Incremental Learning (FCIL) (Yoon et al., ), tasks are introduced incrementally, and data is distributed across multiple clients in a decentralized manner.

We introduce a novel approach, termed Low-rank Regression Mean (LoRM), tailored to the FCIL setting. Our method builds upon RegMean (Jin et al., ), a model merging technique derived from an exactly solvable regression problem. Starting from RegMean’s formulation, we develop a strategy to merge LoRA modules in closed form. Our derivations result in two key equations — one for merging matrix ${\bm{A}}$ and another one for matrix ${\bm{B}}$. During training, we propose an alternating optimization procedure, where one matrix is learned while the other remains fixed across models. In the Federated Class-Incremental Learning setting, spatial aggregation across clients is performed using this alternating procedure. Conversely, for temporal aggregation, task-specific modules are merged by applying the RegMean formulation directly to the full residual weights $\Delta{\bm{W}}$ of all tasks.

In summary, the key contributions of this work are as follows:

- •
We explore the feasibility of merging LoRA modules using a closed-form solution.
- •
We introduce LoRM, a novel FCIL approach that leverages insights from our exploration.
- •
We demonstrate the effectiveness of our method across diverse datasets and varying degrees of data distribution, achieving state-of-the-art results.

## 2 Background and Motivation

### 2.1 Preliminaries

LoRA (Hu et al., ) was introduced to reduce the number of trainable parameters when fine-tuning pre-trained models. Formally, let ${\bm{W}}_{0}\in\mathbb{R}^{d\times k}$ represent the matrix of pre-trained weights of a linear layer, and let ${\bm{x}}\in\mathbb{R}^{k\times 1}$ be the input vector for that layer. The output ${\bm{h}}$ is given by:

$$ ${\bm{h}}={\bm{W}}_{0}{\bm{x}}+\Delta{\bm{W}}{\bm{x}}={\bm{W}}_{0}{\bm{x}}+{\bm {B}}{\bm{A}}{\bm{x}},$ (1) $$

where $\Delta{\bm{W}}={\bm{B}}{\bm{A}}$ is the residual weight introduced by LoRA, with matrices ${\bm{A}}$ and ${\bm{B}}$ as the only components being trained. The efficiency of this approach stems from the low rank $r$ of the matrices, where ${\bm{B}}\in\mathbb{R}^{d\times r}$ and ${\bm{A}}\in\mathbb{R}^{r\times k}$, with $d$ and $k$ representing the number of output and input features of the layer, respectively. Consequently, the number of trainable parameters is $r\cdot(d+k)$, which, since $r\ll d$, constitutes only a fraction of the $d\cdot k$ parameters required for full fine-tuning. Additionally, ${\bm{B}}$ is initialized to 0 0: *i.e*., the first forward pass is equivalent to the absence of a LoRA residual.

RegMean (Jin et al., ) introduces a method for merging a collection of $N$ linear layers $\{{\bm{W}}_{i}\}_{i=1}^{N}$, each corresponding to $N$ distinct models trained on distinct inputs $\{{\bm{X}}_{i}\}_{i=1}^{N}$. The goal is to identify a single linear layer that produces responses that closely match those of the starting layers. Specifically, the objective function is defined as follows(^1^11We rework the original formulation to ease subsequent derivations.):

$$ $\operatorname*{minimize}~{}\Omega=\sum_{i=1}^{N}{\|{\bm{W}}_{M}{\bm{X}}_{i}-{ \bm{W}}_{i}{\bm{X}}_{i}\|^{2}_{2}}.$ (2) $$

By computing the gradient of $\Omega$ with respect to ${\bm{W}}_{M}$ and setting it equal to zero, a closed-form solution is obtained. Notably, the merged layer ${\bm{W}}_{M}$ is computed as follows:

$$ ${\bm{W}}_{M}=\left(\sum_{i=1}^{N}{\bm{W}}_{i}{\bm{X}}_{i}{\bm{X}}_{i}^{\top} \right)\left(\sum_{i=1}^{N}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\right)^{-1}.$ (3) $$

In this context, ${\bm{X}}_{i}\in\mathbb{R}^{k\times\text{samples}}$ represents the input to the $i$-th layer being merged. Hence, the Gram matrix ${\bm{X}}_{i}{\bm{X}}_{i}^{\top}$ has dimensions $k\times k$ features.

While RegMean finds a weight matrix that approximates the outputs of all layers considered, it does not take into account starting from pre-trained weights or the use of low-rank modules (*e.g*., LoRA). Given these considerations, we ask whether this method can be suitably adapted to merge LoRA modules. In other words:

To further explore this question from an experimental standpoint, we place our investigation in the context of a Federated Class-Incremental Learning scenario.

### 2.2 Problem Setting

In Federated Class-Incremental Learning, the dataset $\mathcal{D}$ is first divided into $T$ tasks, each consisting of a distinct set of classes. Then, each partition $\mathcal{D}^{t}$ corresponding to the $t$-th task is further distributed among $N$ clients, resulting in $\mathcal{D}^{t}_{i}$ for the $i$-th client. Similar to standard Class-Incremental Learning (Van de Ven et al., ), the task-specific partitions $\{\mathcal{D}^{t}\}_{t=1,\ldots,T}$ arrive sequentially.

In this federated scenario, the training for each task is conducted over multiple communication rounds. During each round, clients are restricted to learning only from their local dataset $\mathcal{D}^{t}_{i}$. The local optimization objective for client $i$, based on the loss function $\mathcal{L}$, can be formally expressed as:

$$ $\operatorname*{minimize}_{\theta_{i}}\;\mathbb{E}_{(x,y)\sim\mathcal{D}^{t}_{i }}\left[\mathcal{L}(f(x;\theta_{i}),y)\right],$ (4) $$

where $x$ and $y$ denote the inputs and corresponding labels, respectively, with $\theta_{i}$ representing the set of parameters for client $i$, and $f(\cdot;\theta_{i})$ denoting the associated model.

After completing local updates, each client sends its model parameters $\theta_{i}$ to the central server, where they are aggregated with those from other clients. The server then sends the global aggregated model back to the clients, marking the end of a communication round. This process repeats for several rounds until the training for task $t$ is completed. Once all rounds for task $t$ have finished, the system progresses to the next task $t+1$ using the corresponding dataset $\mathcal{D}^{t+1}$. The ultimate objective is to obtain a global model, derived from the aggregation of local models performed by the server, that functions well across all incremental tasks and successfully integrates the distributed knowledge.

## 3 Methodology

### 3.1 A closed-form solution for LoRA merging

Using the same notation as in Equation 2, let ${\bm{W}}$ denote the weight matrix of a given layer. Since LoRA (see Equation 2) is applied to each linear layer across all clients, we express the weight matrix for the $i$-th client as ${\bm{W}}_{i}={\bm{W}}_{0}+{\bm{B}}_{i}{\bm{A}}_{i}$, where ${\bm{W}}_{0}$ is the shared pre-trained weight matrix, and ${\bm{B}}_{i}$, ${\bm{A}}_{i}$ represent the low-rank matrices specific to the client. This formulation is consistent across clients, as they all utilize the same model architecture. At the end of each communication round, after conducting local training on the LoRA matrices, the goal is to merge the corresponding matrices (*i.e*., ${\bm{A}}$’s with ${\bm{A}}$’s and ${\bm{B}}$’s with ${\bm{B}}$’s) using the closed-form solution derived from RegMean. Starting from Equation 2, our objective becomes:

$$ $\Omega=\sum_{i=1}^{N}{\|({\bm{W}}_{0}+{\bm{B}}_{M}{\bm{A}}_{M}){\bm{X}}_{i}-({ \bm{W}}_{0}+{\bm{B}}_{i}{\bm{A}}_{i}){\bm{X}}_{i}\|^{2}_{2}}.$ (5) $$

To find the optimal ${\bm{A}}_{M}$ and ${\bm{B}}_{M}$ that minimize $\Omega$, we differentiate $\Omega$ with respect to each variable, one at a time, and set the gradients to zero:

$$ $\begin{cases}\displaystyle\frac{\partial\Omega}{\partial{\bm{B}}}=0\vspace{4pt }\\ \displaystyle\frac{\partial\Omega}{\partial{\bm{A}}}=0\end{cases}$ (6) $$

However, the system reveals indeterminate, as the two equations exhibit linear dependence on one another during the calculations. For further mathematical derivations illustrating the infeasibility of this approach, refer to Section A.1.

As a solution, we propose freezing one of the two matrices. This means that either ${\bm{A}}$ or ${\bm{B}}$ is shared across clients (*i.e*., treated as a constant), and a single closed-form equation determines how to merge the trainable matrices across clients. If we choose to share ${\bm{A}}$, the merged ${\bm{B}}_{M}$ is obtained as:

$$ ${\bm{B}}_{M}=\left(\sum_{i=1}^{N}{\bm{B}}_{i}{\bm{A}}{\bm{X}}_{i}{\bm{X}}_{i}^ {\top}\right){\bm{A}}^{\top}\left({\bm{A}}\sum_{i=1}^{N}{\bm{X}}_{i}{\bm{X}}_{ i}^{\top}{\bm{A}}^{\top}\right)^{-1}.$ (7) $$

Note that ${\bm{A}}$ does not have a subscript $i$, as it is identical for all clients. Instead, if we opt to share ${\bm{B}}$, the merged ${\bm{A}}_{M}$ is computed as:

$$ ${\bm{A}}_{M}=\left(\sum_{i=1}^{N}{\bm{A}}_{i}{\bm{X}}_{i}{\bm{X}}_{i}^{\top} \right)\left(\sum_{i=1}^{N}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\right)^{-1}.$ (8) $$

For the formal derivation of these equations, refer to Sections A.1.2 and A.1.1.

### 3.2 LoRM

Figure: Figure 1: Training and aggregation procedure of LoRM, alternating between rounds. The figure depicts both Federated and Incremental aggregation for a generic layer.
Refer to caption: x1.png

Having established closed-form solutions for weight merging, we now outline the full procedure of LoRM, illustrated in Figure 1. Each client $i$ begins by optimizing its own ${\bm{B}}_{i}$, which is learned during local training, and a shared ${\bm{A}}$, initialized and distributed by the server. In this first round, it is essential to freeze ${\bm{A}}$, as LoRA’s ${\bm{B}}$ is initialized to 0 0. Freezing ${\bm{B}}=0$ across all clients would render the training ineffective. At the end of each round, each client $i$ computes the Gram matrix ${\bm{X}}_{i}{\bm{X}}_{i}^{\top}$ with a forward pass on all examples(^2^22Note that this process provides a Gram matrix for every layer in the network. In this discussion, ${\bm{X}}_{i}{\bm{X}}_{i}^{\top}$ is related to the single generic linear layer.). Then, it sends ${\bm{B}}_{i}$ and ${\bm{X}}_{i}{\bm{X}}_{i}^{\top}$ to the server, where the merging operation (as described in Equation 7) is performed. The resulting ${\bm{B}}_{M}$ is then sent back to the clients, who begin the next communication round.

Alternated optimization. Empirically, we observe that consistently training only the matrix ${\bm{B}}$ while keeping ${\bm{A}}$ fixed can be limiting (see Section 4.3 for a detailed analysis). Therefore, to fully exploit LoRA’s representational potential, we introduce an alternating training approach, where the matrix to be updated changes at each round. Specifically, after the first round, we freeze ${\bm{B}}$, which is already synchronized across clients due to the previous server aggregation, and train ${\bm{A}}$ instead. Then, at the end of the second round, all local ${\bm{A}}$’s are aggregated using Equation 8. This strategy also improves efficiency, as only one of the two matrices needs to be communicated per round, offering a significant advantage compared to transmitting the whole LoRA module.

Merge task-specific modules. At the conclusion of the generic task $t$, the merged matrices ${\bm{B}}_{M}^{t}$ and ${\bm{A}}_{M}^{t}$ are multiplied to obtain the task-specific residual module: $\Delta{\bm{W}}^{t}={\bm{B}}_{M}^{t}{\bm{A}}_{M}^{t}$. On the server side, when aggregating modules from all tasks at the end of the training, we apply the standard RegMean formulation (Equation 3) to merge the aforementioned residuals as:

$$ $\Delta{\bm{W}}=\left(\sum_{t=1}^{T}{\bm{B}}_{M}^{t}{\bm{A}}_{M}^{t}{\bm{X}}^{t }{\bm{X}}^{t\top}\right)\left(\sum_{t=1}^{T}{\bm{X}}^{t}{\bm{X}}^{t\top}\right )^{-1}.$ (9) $$

To accomplish this, the server stores the task-specific weight $\{\Delta{\bm{W}}^{1},\ldots,\Delta{\bm{W}}^{T}\}$ throughout the training process, along with the global Gram matrices $\{{\bm{X}}^{t}{\bm{X}}^{1\top},\ldots,{\bm{X}}^{t}{\bm{X}}^{T\top}\}$. Due to the associative property of matrix multiplication, the computation of the latter is straightforward, achieved by summing the client-specific Gram matrices shared by the clients at the end of each task $t$: ${\bm{X}}^{t}{\bm{X}}^{t\top}=\sum_{i=1}^{N}{{\bm{X}}_{i}^{t}{\bm{X}}_{i}^{t
\top}}$. It is worth noting that, for what concerns the classification layer, concatenating the task-specific classification heads is equivalent to applying RegMean. We provide a mathematical derivation of this result in Section A.4.

The overall procedure yields a final weight residual $\Delta{\bm{W}}$ that promotes both generalization across tasks and robustness to knowledge distribution. The pseudocode for LoRM, applied to a generic linear layer, is presented in Algorithm 1.

Figure: Algorithm 1 – LoRM, for a generic linear layer Round j Round j + 1

### 3.3 LoRM Characteristics for Federated Class-Incremental Learning

In Federated Class-Incremental Learning (FCIL), data privacy, efficiency, and the rate of convergence are crucial concerns. LoRM is specifically designed to address these challenges.

Privacy-preserving. In FCIL, inputs ${\bm{X}}_{i}$ for the $i$-th client’s generic layer cannot be transmitted to the central server, as doing so, especially for the first layer, would compromise data privacy by effectively sharing the dataset. In LoRM, instead of transmitting ${\bm{X}}_{i}$ directly, we send the Gram matrix ${\bm{X}}_{i}{\bm{X}}_{i}^{\top}$, which obfuscates the original data. Furthermore, only the diagonal of the Gram matrix is communicated, ensuring that no local data is exposed.

Efficiency. The use of LoRA inherently improves efficiency compared to full fine-tuning, as it necessitates the communication of only two low-rank matrices for each layer. Additionally, LoRM’s alternating optimization procedure permits the transmission of only one matrix per communication round, further reducing overhead. The use of solely the diagonal of the Gram matrix mitigates communication costs even further, as it avoids the need to transmit the full matrix. As a result, at the conclusion of each communication round, each client transmits a low-rank matrix and a vector to the server for each layer.

Rate of convergence. Ultimately, LoRM demonstrates faster convergence compared to other FCIL baselines, as further discussed in Section 4.2.

## 4 Experimental Study

### 4.1 Evaluation settings

Datasets. The importance of evaluating pre-trained models on datasets that deviate significantly from their pre-training domain is well-established in the literature (Kolesnikov et al., ; Kornblith et al., ). Accordingly, we divide the evaluation of LoRM into two parts: one focusing on in-domain datasets and the other on an out-of-domain dataset. For in-domain evaluation, we use CIFAR-100 (Krizhevsky et al., ) and ImageNet-R (Hendrycks et al., ). For out-of-domain evaluation, we employ EuroSAT (Helber et al., ), a satellite dataset recognized by Oh et al. () as one of the most challenging for domain adaptation from ImageNet-21k pre-training.
CIFAR-100 and ImageNet-R are partitioned into $10$ incremental tasks, with each task consisting of $10$ and $20$ classes, respectively. EuroSAT is divided into $5$ tasks, each containing $2$ classes. The data is distributed across $10$ clients using the commonly adopted distribution-based label imbalance setting (Li et al., ; Yurochkin et al., ), where partitioning is governed by a Dirichlet distribution parameterized by $\beta$. A smaller $\beta$ value corresponds to a more challenging data distribution.
We evaluate all methods across three scenarios for each dataset, using $\beta\in{0.5,0.1,0.05}$ for the in-domain datasets and $\beta\in{1,0.5,0.2}$ for the out-of-domain one. For further details on data preprocessing, including the dataset-specific augmentations used, refer to appendix B.

Evaluated approaches. We compare LoRM against $10$ competing methods spanning different fields. From Continual Learning, we evaluate EWC (Kirkpatrick et al., ), LwF (Li & Hoiem, ), DER++ (Buzzega et al., ), L2P (Wang et al., ), and CODA-Prompt (Smith et al., ). Following previous studies (Zhang et al., ; Guo et al., ), we adapt these methods to the federated domain by merging client weights using the FedAvg algorithm (McMahan et al., ). We also include FisherAvg (Matena & Raffel, ) and RegMean (Jin et al., ), from the model merging literature, and CCVR (Luo et al., ), a method from Federated Learning. These approaches are adapted for Continual Learning by applying Asymmetric Cross-Entropy (ACE) (Caccia et al., ), where the classification heads are optimized separately for each task.
Additionally, we evaluate two algorithms specifically designed for Federated Class-Incremental Learning: TARGET (Zhang et al., ) and PLoRA (Guo et al., ), the latter representing the current State Of The Art. Finally, we include Joint results as an upper bound, achieved by training the backbone on the entire dataset without any federated or incremental partitioning.

Implementation Details. As the backbone for LoRM and all competing approaches, we employ a pre-trained ViT-B/16 model (Dosovitskiy et al., ). Specifically, the model is initialized with pre-trained weights from ImageNet-21K (Ridnik et al., ) for all evaluated datasets. Each task is trained over $5$ communication rounds, with each round consisting of $5$ epochs. For a complete overview of the method-specific hyperparameters, refer to appendix D.

### 4.2 Results

**Table 1: In-domain SOTA comparison. Results are reported in terms of FAA($\%$). Best-performing methods are indicated in bold, while the second bests are underlined.**
|  | CIFAR-100 | ImageNet-R |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Joint | $91.59$ | $86.31$ |  |  |  |  |
| Partition | $\beta=0.5$ | $\beta=0.1$ | $\beta=0.05$ | $\beta=0.5$ | $\beta=0.1$ | $\beta=0.05$ |
| EWC (Kirkpatrick et al., ) | $71.31$ | $71.75$ | $63.48$ | $56.77$ | $55.67$ | $53.90$ |
| LwF (Li & Hoiem, ) | $54.18$ | $55.12$ | $42.44$ | $49.78$ | $45.03$ | $41.60$ |
| DER++ (Buzzega et al., ) | $67.90$ | $61.97$ | $55.79$ | $55.83$ | $50.07$ | $45.52$ |
| L2P (Wang et al., ) | $\underline{76.30}$ | $52.82$ | $45.41$ | $57.58$ | $46.82$ | $37.57$ |
| CODA-P (Smith et al., ) | $62.00$ | $22.21$ | $16.12$ | $48.02$ | $29.68$ | $21.52$ |
| FisherAvg (Matena & Raffel, ) | $70.43$ | $70.10$ | $67.87$ | $57.05$ | $54.05$ | $51.55$ |
| RegMean (Jin et al., ) | $71.07$ | $64.56$ | $61.47$ | $61.05$ | $57.32$ | $49.22$ |
| CCVR (Luo et al., ) | $73.56$ | $\underline{78.04}$ | $\underline{79.47}$ | $\underline{68.68}$ | $\underline{69.55}$ | $\bm{71.83}$ |
| TARGET (Zhang et al., ) | $69.36$ | $69.18$ | $64.23$ | $57.32$ | $58.75$ | $55.70$ |
| PILoRA (Guo et al., ) | $75.34$ | $73.27$ | $68.34$ | $53.03$ | $51.40$ | $47.33$ |
| LoRM (ours) | $\bm{83.19}$ | $\bm{82.05}$ | $\bm{81.21}$ | $\bm{75.62}$ | $\bm{70.83}$ | $\underline{70.18}$ |

In Tables 1 and 3, we present the results of the evaluated approaches in terms of Final Average Accuracy (FAA). For a formal definition of this metric, please refer to Section B.4. The reported results are averaged over $3$ runs, with standard deviations provided in Section C.1.

In the context of in-domain datasets (Table 1), models tend to perform better on CIFAR-100 than on ImageNet-R, indicating the former as a simpler task, partially due to having fewer classes. Continual Learning techniques such as LwF and EWC exhibit low to moderate performance in scenarios with limited data heterogeneity. Surprisingly, employing a rehearsal buffer (DER++) offers minimal benefit. Prompt-based CL approaches similarly show rapid performance degradation as the distribution parameter $\beta$ decreases, although L2P demonstrates strong performance at $\beta=0.5$, ranking second overall. Among FL and model merging methods, which generally outperform CL approaches, CCVR emerges as the top performer, achieving state-of-the-art accuracy in one setting (ImageNet-R, $\beta=0.05$). Interestingly, CCVR exhibits a reverse trend with respect to increasing data heterogeneity: as heterogeneity grows, its performance improves. TARGET and PILoRA, despite being designed specifically for FCIL, show lower performance relative to the adapted CCVR, a result we attribute to CCVR’s server-side calibration procedure using centralized synthetic data. Finally, due to its closed-form merging technique, LoRM achieves state-of-the-art results across all settings except for one (*i.e*., Imagenet-R, $\beta=0.05$).

On the EuroSAT dataset (Table 3), the previous experimental findings are reversed for most methods. Specifically, DER++ emerges as the second-best performer across all levels of increasing heterogeneity, highlighting the value of utilizing a memory buffer when adapting to different distributions. L2P follows closely, benefiting from its prompt pool, which proves crucial when deviating from the pre-training weights. Conversely, CCVR’s advantage vanishes in this setting, as merely calibrating the classifier fails to extract meaningful representations. Target and PILoRA perform on par with CCVR but fall short of LoRM, which surpasses all approaches by a significant margin(^3^33Less relevant methodologies are omitted from Table 3 and are reported in Table C.).

Rate of convergence. Figure 3 illustrates the convergence rates of LoRM and the FCIL competitors on the first task of ImageNet-R with $\beta=0.05$. The steeper slope of the curve for LoRM, compared to that of Target and PILoRA, showcases the faster convergence of our approach. This behavior stems from the closed-form solution applied for merging LoRA at each communication round, which accelerates convergence by leveraging its optimality. In contrast, PILoRA applies simple averaging of LoRA modules, while Target does not utilize PEFT modules at all. Additionally, the slower convergence of PILoRA compared to Target can be attributed to its prototype-based classification procedure, which necessitates a higher number of training rounds to achieve well-refined prototypes.

**Table 2: Out-of-domain SOTA comparison. Results are reported in terms of FAA($\%$).**
| Joint | $93.72$ |  |  |
| --- | --- | --- | --- |
| Partition | $\beta=1.0$ | $\beta=0.5$ | $\beta=0.2$ |
| DER++ | $\underline{66.72}$ | $\underline{65.20}$ | $\underline{65.44}$ |
| L2P | $64.31$ | $62.75$ | $64.42$ |
| CCVR | $47.41$ | $36.07$ | $43.90$ |
| TARGET | $36.78$ | $37.81$ | $43.69$ |
| PILoRA | $40.93$ | $34.02$ | $39.64$ |
| LoRM | $\bm{80.44}$ | $\bm{77.37}$ | $\bm{75.12}$ |

### 4.3 Ablation Studies

Alternating vs. Only B. We investigate whether training the ${\bm{B}}$ matrix only (*i.e*., following the same procedure as LoRM but applying solely Equation 7 to merge ${\bm{B}}$ matrices) could lead to improved performance compared to our alternating optimization approach. We conduct an experiment on EuroSAT with $\beta=1.0$ (see Figure 3), showing that the alternating strategy outperforms the only B approach. This result underscores the importance of leveraging the entire LoRA representation capabilities to achieve superior performance.

LoRM components. To evaluate the contribution of each component to the overall performance of LoRM, we conduct an ablation study by incrementally introducing each part and comparing it with baseline methods. These experiments are performed on EuroSAT with $\beta=1.0$ and ImageNet-R with $\beta=0.5$. The initial baseline for both datasets is FedAvg (McMahan et al., ) combined with ACE (Caccia et al., ). Notably, leveraging LoRA leads to a remarkable performance improvement on EuroSAT but does not affect ImageNet-R. Meanwhile, using the closed-form aggregation of RegMean, enhanced with ACE, establishes an additional baseline. Building upon this, our alternating optimization strategy, when applied without Equation 9 (*i.e*., simply averaging task-specific residuals), substantially boosts performance on both datasets, with a more pronounced impact on the out-of-domain dataset. Finally, applying RegMean to merge task-specific residuals further enhances robustness in addressing FCIL challenges, resulting in the final form of LoRM.

## 5 Relation with prior works

### Parameter-Efficient Fine-Tuning

Adapting deep neural networks to new tasks typically requires full re-training of all parameters, which is computationally demanding. Parameter-Efficient Fine-Tuning (PEFT) addresses this by updating only a small subset of the model parameters, leaving the rest of the network unchanged. One of the earliest methods in this domain is Adapters (Houlsby et al., ), which are lightweight neural modules inserted between the layers of a pre-trained model to facilitate task adaptation. More recently, a prominent class of PEFT techniques includes Prompt Tuning (Lester et al., ) and Prefix Tuning (Li & Liang, ), both of which introduce learnable embeddings, or prompts, appended to the layers’ input tokens. Prompt Tuning prepends these embeddings directly to the initial input sequence (Wang et al., ; ), whereas Prefix Tuning concatenates them to specific attention layers (Smith et al., ). An alternative approach to prompting is Low-Rank Adaptation (LoRA) (Hu et al., ), which injects residual parameters into the pre-trained weights using low-rank matrices, significantly reducing the number of learnable parameters. While recent works have explored various alternatives to LoRA(Kopiczko et al., ; Zhang et al., ; Renduchintala et al., ; Liu et al., ), our research is grounded in the original LoRA framework, leveraging its efficiency and the extensive literature on model merging that goes with it.

### Model merging

Task-specific modules are typically deployed to address individual tasks. However, an alternative approach involves merging these modules to create a model capable of generalizing across all tasks. A notable line of research within model merging focuses on combining task vectors (Ilharco et al., ), specifically those generated by LoRA-like methods. Early studies have explored the use of linear arithmetic operations (*e.g*., addition and subtraction) to combine these modules (Zhang et al., ). Expanding on this, subsequent works have focused on optimizing the coefficients that determine the contribution of each LoRA module during aggregation (Huang et al., ; Yang et al., ; Yadav et al., ; Wu et al., ). Alternatively, RegMean (Jin et al., ) introduces a closed-form solution for merging model weights, though its application to LoRA module merging has not yet been explored. In this work, we present a novel methodology that leverages this closed-form solution, specifically tailored for the Federated Class-Incremental Learning setting.

### Federated Class-Incremental Learning

Federated Class-Incremental Learning (FCIL), introduced by Yoon et al. (), addresses the challenges inherent in Continual Learning, such as Catastrophic Forgetting (Robins, ), and Federated Learning, such as Client Drift (Zhao et al., ; Karimireddy et al., ), within a unified framework. Although the FCIL literature is still in its early stages, several studies have adapted popular Continual Learning methods to the federated paradigm, often by employing the FedAvg (McMahan et al., ) technique for aggregating client model weights. A few approaches, however, are specifically designed for the FCIL setting. For instance, GLFC (Dong et al., ) incorporates local buffers and a regularization technique to mitigate the effects of gradient updates related to novel classes, with further refinements in a subsequent work by the same authors (Dong et al., ). TARGET (Zhang et al., ) trains a centralized GAN (Goodfellow et al., ) to populate replay buffers with synthetic samples. Fed-CPrompt (Bagwe et al., ) combines local and global prompts to address both Continual Learning and Federated Learning challenges simultaneously. Most recently, PILoRA (Guo et al., ) proposed a method that integrates LoRA with prototypes (*i.e*., average feature vectors), using the latter for classification instead of relying on the traditional classification layer. In contrast, our proposed methodology leverages LoRA modules and aggregates them by aligning their output in both Continual and Federated scenarios in a closed-form.

Figure: Figure 2: Convergence rate.
Refer to caption: x2.png

## 6 Conclusions

At the outset of our research, we explored the possibility of a closed-form solution for merging LoRA modules. Motivated by this question, we identified and validated the existence of such a solution, demonstrating its practicality within the framework of Federated Class-Incremental Learning (FCIL). We introduced LoRM, a method specifically designed for the FCIL scenario, which sets a new State of the Art in this domain.

Looking forward, we aim to broaden our exploration to encompass a wider array of PEFT modules. For modules such as VeRA (Kopiczko et al., ) and (IA)^3 (Liu et al., ), we have already derived closed-form merging equations, which are detailed in Sections A.2 and A.3, respectively. Furthermore, given the generality of our derivations beyond the FCIL setting, we also intend to evaluate the robustness of our equations across various model merging scenarios. Ultimately, we hope this work will serve not only the Federated Learning and Continual Learning communities but also contribute to the growing body of research on efficient module compositionally.

## Supplementary Material Overview

This supplementary document provides detailed mathematical derivations and additional information to complement the main text. The contents are organized as follows:

- •
Section A – Mathematical Derivations: Presents the mathematical foundations and derivations related to the combination of different Parameter-Efficient Fine-Tuning methods with RegMean.
–
Section A.1 – Combination of LoRA and RegMean: Derives the optimization problem when integrating Low-Rank Adaptation (LoRA) with RegMean.
*
Section A.1.1 – Solving for ${\bm{A}}$ by Fixing ${\bm{B}}$: Details the solution for matrix ${\bm{A}}$ when matrix ${\bm{B}}$ is held constant.
*
Section A.1.2 – Solving for ${\bm{B}}$ by Fixing ${\bm{A}}$: Details the solution for matrix ${\bm{B}}$ when matrix ${\bm{A}}$ is held constant.
–
Section A.2 – Combination of VeRA and RegMean: Extends the derivations to the Vector-based Random matrix Adaptation (VeRA) method.
–
Section A.3 – Combination of (IA)^3 and RegMean: Explores how the (IA)^3 method can be combined with RegMean and provides the corresponding derivations.
–
Section A.4 – RegMean Applied to Classification Heads: Discusses the application of RegMean to classification heads in a class-incremental learning setting.
- •
Section B – Details on Datasets and Metrics: Offers comprehensive details about the datasets used and the evaluation metric employed.
- •
Section C – Additional Results: Presents supplementary experimental results and analyses.
–
Section C.1 – Standard Deviations: Reports the standard deviations for all evaluated approaches.
–
Section C.2 – Additional Results on EuroSAT: Provides extended results on the EuroSAT dataset.
- •
Section D – Hyperparameters: Details the hyperparameters used across different experiments.

## Appendix A Mathematical derivations

### A.1 Combination of LoRA and RegMean

We consider a minimization problem over a single linear layer where ${\bm{X}}_{i}$ represents the input to the layer. The objective function is defined as:

$$ $\operatorname*{minimize}~{}\Omega=\sum_{i=1}^{N}{\|{\bm{W}}{\bm{X}}_{i}-{\bm{W }}_{i}{\bm{X}}_{i}\|^{2}_{2}}.$ (10) $$

Here, ${\bm{W}}$ and ${\bm{W}}_{i}$ are the weight matrices for the global model (server) and the local models (clients) respectively. In the context of a LoRA residual module, we replace ${\bm{W}}$ with ${\bm{W}}_{0}+{\bm{B}}{\bm{A}}$, where ${\bm{W}}_{0}$ denotes the pre-trained weight matrix and ${\bm{B}}{\bm{A}}$ represents the learned low-rank difference. The objective function becomes:

$$ $\Omega=\sum_{i=1}^{N}{\|({\bm{W}}_{0}+{\bm{B}}{\bm{A}}){\bm{X}}_{i}-({\bm{W}}_ {0}+{\bm{B}}_{i}{\bm{A}}_{i}){\bm{X}}_{i}\|^{2}_{2}}.$ $$

Expanding the previous expression yields:

$$ $\Omega=\sum_{i=1}^{N}{\|{\bm{W}}_{\theta_{0}}{\bm{X}}_{i}-{\bm{W}}_{0}{\bm{X}} _{i}+{\bm{B}}{\bm{A}}{\bm{X}}_{i}-{\bm{B}}_{i}{\bm{A}}_{i}{\bm{X}}_{i}\|^{2}_{ 2}}.$ $$

Which simplifies the equation to:

$$ $\Omega=\sum_{i=1}^{N}{\|{\bm{B}}{\bm{A}}{\bm{X}}_{i}-{\bm{B}}_{i}{\bm{A}}_{i}{ \bm{X}}_{i}\|^{2}_{2}}.$ $$

It is important to note that ${\bm{B}}$ and ${\bm{A}}$ refer to the merged LoRA matrices (${\bm{B}}_{M}$ and ${\bm{A}}_{M}$ in the main paper). For simplicity, we omit the subscript M. To minimize this objective function, we compute the partial derivatives with respect to the unknown matrices ${\bm{A}}$ and ${\bm{B}}$, setting them equal to zero. This leads to the following system of equations:

$$ $\begin{cases}\displaystyle\frac{\partial\Omega}{\partial{\bm{A}}}=0\quad \Rightarrow\quad\sum_{i=1}^{N}{2{\bm{B}}^{\top}({\bm{B}}{\bm{A}}{\bm{X}}_{i}-{ \bm{B}}_{i}{\bm{A}}_{i}{\bm{X}}_{i}){\bm{X}}_{i}^{\top}}=0\\ \displaystyle\frac{\partial\Omega}{\partial{\bm{B}}}=0\quad\Rightarrow\quad \sum_{i=1}^{N}{2({\bm{B}}{\bm{A}}{\bm{X}}_{i}-{\bm{B}}_{i}{\bm{A}}_{i}{\bm{X}} _{i}){\bm{X}}_{i}^{\top}{\bm{A}}^{\top}}=0\end{cases}$ $$

We can rearrange the equations as:

$$ $\begin{cases}\displaystyle\sum_{i=1}^{N}{{\bm{B}}^{\top}{\bm{B}}{\bm{A}}{\bm{X }}_{i}{\bm{X}}_{i}^{\top}}=\sum_{i=1}^{N}{{\bm{B}}^{\top}{\bm{B}}_{i}{\bm{A}}_ {i}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\\ \displaystyle\sum_{i=1}^{N}{{\bm{B}}{\bm{A}}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}{ \bm{A}}^{\top}}=\sum_{i=1}^{N}{{\bm{B}}_{i}{\bm{A}}_{i}{\bm{X}}_{i}{\bm{X}}_{i }^{\top}{\bm{A}}^{\top}}\end{cases}$ $$

Next, we multiply the second equation on the left by ${\bm{B}}^{\top}$, in order to align it with the structure of the first equation:

$$ $\sum_{i=1}^{N}{{\bm{B}}^{\top}{\bm{B}}_{i}{\bm{A}}_{i}{\bm{X}}_{i}{\bm{X}}_{i} ^{\top}{\bm{A}}^{\top}}=\sum_{i=1}^{N}{{\bm{B}}^{\top}{\bm{B}}_{i}{\bm{A}}_{i} {\bm{X}}_{i}{\bm{X}}_{i}^{\top}{\bm{A}}^{\top}}.$ $$

By substituting the left-hand side of the first equation into the second, we obtain:

$$ $\sum_{i=1}^{N}{{\bm{B}}^{\top}{\bm{B}}_{i}{\bm{A}}_{i}{\bm{X}}_{i}{\bm{X}}_{i} ^{\top}{\bm{A}}^{\top}}-\sum_{i=1}^{N}{{\bm{B}}^{\top}{\bm{B}}_{i}{\bm{A}}_{i} {\bm{X}}_{i}{\bm{X}}_{i}^{\top}{\bm{A}}^{\top}}=0.$ $$

This simplifies to $0=0$, indicating that the system is indeterminate, with infinitely many possible solutions.

#### A.1.1 Solving for A 𝐴 A italic_A by Fixing B 𝐵 B italic_B

Assuming the matrix ${\bm{B}}$ is fixed, we can solve for ${\bm{A}}$. The objective function $\Omega$ becomes:

$$ $\Omega_{{\bm{A}}}=\sum_{i=1}^{N}{\|{\bm{B}}{\bm{A}}{\bm{X}}_{i}-{\bm{B}}{\bm{A }}_{i}{\bm{X}}_{i}\|^{2}_{2}}.$ $$

To minimize $\Omega_{{\bm{A}}}$, we set its derivative with respect to ${\bm{A}}$ to zero:

$$ $\frac{\partial\Omega_{{\bm{A}}}}{\partial{\bm{A}}}=0\quad\Rightarrow\quad\sum_ {i=1}^{N}{2{\bm{B}}^{\top}({\bm{B}}{\bm{A}}{\bm{X}}_{i}-{\bm{B}}{\bm{A}}_{i}{ \bm{X}}_{i}){\bm{X}}_{i}^{\top}}=0,$ $$

which simplifies to:

$$ $\sum_{i=1}^{N}{{\bm{B}}^{\top}{\bm{B}}({\bm{A}}{\bm{X}}_{i}-{\bm{A}}_{i}{\bm{X }}_{i}){\bm{X}}_{i}^{\top}}=0.$ $$

Since ${\bm{B}}{\bm{A}}$ is designed to be low-rank, the matrix ${\bm{B}}$ has more rows than columns. As a result, provided that no row or column of ${\bm{B}}$ is zero, the inverse $\left({\bm{B}}^{\top}{\bm{B}}\right)^{-1}$ exists. By multiplying both sides of the equation from the left by $\left({\bm{B}}^{\top}{\bm{B}}\right)^{-1}$, we obtain:

$$ $\sum_{i=1}^{N}{({\bm{A}}{\bm{X}}_{i}-{\bm{A}}_{i}{\bm{X}}_{i}){\bm{X}}_{i}^{ \top}}=0,\quad\Rightarrow\quad{\bm{A}}\sum_{i=1}^{N}{{\bm{X}}_{i}{\bm{X}}_{i}^ {\top}}=\sum_{i=1}^{N}{{\bm{A}}_{i}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}.$ $$

Finally, solving for ${\bm{A}}$, we get:

$$ ${\bm{A}}=\left(\sum_{i=1}^{N}{{\bm{A}}_{i}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}} \right)\left(\sum_{i=1}^{N}{{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)^{-1}.$ $$

#### A.1.2 Solving for B 𝐵 B italic_B by Fixing A 𝐴 A italic_A

When the matrix ${\bm{A}}$ is fixed, we can solve for ${\bm{B}}$. The objective function $\Omega$ becomes:

$$ $\Omega_{{\bm{B}}}=\sum_{i=1}^{N}{\|{\bm{B}}{\bm{A}}{\bm{X}}_{i}-{\bm{B}}_{i}{ \bm{A}}{\bm{X}}_{i}\|^{2}_{2}}.$ $$

To minimize $\Omega_{{\bm{B}}}$, we set its derivative with respect to ${\bm{B}}$ equal to zero:

$$ $\frac{\partial\Omega_{{\bm{B}}}}{\partial{\bm{B}}}=0\quad\Rightarrow\quad\sum_ {i=1}^{N}{2({\bm{B}}{\bm{A}}{\bm{X}}_{i}-{\bm{B}}_{i}{\bm{A}}{\bm{X}}_{i}){\bm {X}}_{i}^{\top}{\bm{A}}^{\top}}=0.$ $$

Expanding the equation, we get:

$$ $\sum_{i=1}^{N}{{\bm{B}}{\bm{A}}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}{\bm{A}}^{\top}} -\sum_{i=1}^{N}{{\bm{B}}_{i}{\bm{A}}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}{\bm{A}}^{ \top}}=0,$ $$

which simplifies to:

$$ ${\bm{B}}\sum_{i=1}^{N}{{\bm{A}}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}{\bm{A}}^{\top}} =\sum_{i=1}^{N}{{\bm{B}}_{i}{\bm{A}}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}{\bm{A}}^{ \top}}.$ $$

Finally, solving for ${\bm{B}}$, we obtain:

$$ ${\bm{B}}=\left(\sum_{i=1}^{N}{{\bm{B}}_{i}{\bm{A}}{\bm{X}}_{i}{\bm{X}}_{i}^{ \top}}\right){\bm{A}}^{\top}\left(\sum_{i=1}^{N}{{\bm{A}}{\bm{X}}_{i}{\bm{X}}_ {i}^{\top}{\bm{A}}^{\top}}\right)^{-1}.$ $$

### A.2 Combination of VeRA and RegMean

Following Section A.1, we consider the same minimization problem as Equation 10. In the context of a VeRA residual module, we replace ${\bm{W}}$ with ${\bm{W}}_{0}+\Lambda_{b}{\bm{B}}\Lambda_{d}{\bm{A}}$, where ${\bm{W}}_{0}$ denotes the pre-trained weight matrix and $\Lambda_{b}{\bm{B}}\Lambda_{d}{\bm{A}}$ represents the learned weight difference. More precisely, ${\bm{B}}$ and ${\bm{A}}$ are randomly initialized, while $\Lambda_{b}$ and $\Lambda_{d}$ are learned. The objective function becomes:

$$ $\Omega=\sum_{i=1}^{N}{\|({\bm{W}}_{0}+\Lambda_{b}{\bm{B}}\Lambda_{d}{\bm{A}}){ \bm{X}}_{i}-({\bm{W}}_{0}+\Lambda_{b,i}{\bm{B}}\Lambda_{d,i}{\bm{A}}){\bm{X}}_ {i}\|^{2}_{2}}.$ $$

While every component of LoRA is a full matrix, here we are considering $\Lambda_{b}$ and $\Lambda_{d}$, which are diagonal matrices. In order to enforce this constraint into the previous formulation, we can rewrite them as column vectors $\lambda^{b}$ and $\lambda^{d}$:

$\displaystyle\Omega=\sum_{i=1}^{N}{\|\left({\bm{W}}_{0}+\left(\left(\lambda^{b
}{\bm{1}}_{b}\right)\odot{\bm{B}}\right)\left(\left(\lambda^{d}{\bm{1}}_{d}
\right)\odot{\bm{A}}\right)\right){\bm{X}}_{i}-({\bm{W}}_{0}+\left(\left(
\lambda^{b}_{i}{\bm{1}}_{b}\right)\odot{\bm{B}}\right)\left(\left(\lambda^{d}_
{i}{\bm{1}}_{d}\right)\odot{\bm{A}}\right)){\bm{X}}_{i}\|^{2}_{2}},$

where ${\bm{1}}_{b}$ and ${\bm{1}}_{d}$ are two row vectors of ones with the same columns as ${\bm{B}}$ and ${\bm{A}}$, respectively. Please note that in our notation, indices of column vectors are denoted as superscripts, while those of row vectors are represented as subscripts. Expanding this objective, we can write:

$\displaystyle\Omega=\sum_{i=1}^{N}{\|{\bm{W}}_{0}{\bm{X}}_{i}-{\bm{W}}_{0}{\bm
{X}}_{i}+\left(\left(\lambda^{b}{\bm{1}}_{b}\right)\odot{\bm{B}}\right)\left(
\left(\lambda^{d}{\bm{1}}_{d}\right)\odot{\bm{A}}\right){\bm{X}}_{i}-\left(
\left(\lambda^{b}_{i}{\bm{1}}_{d}\right)\odot{\bm{B}}\right)\left(\left(
\lambda^{d}_{i}{\bm{1}}_{d}\right)\odot{\bm{A}}\right){\bm{X}}_{i}\|^{2}_{2}}.$

Which simplifies the equation to:

$$ $\Omega=\sum_{i=1}^{N}{\|\left(\left(\lambda^{b}{\bm{1}}_{b}\right)\odot{\bm{B} }\right)\left(\left(\lambda^{d}{\bm{1}}_{d}\right)\odot{\bm{A}}\right){\bm{X}} _{i}-\left(\left(\lambda^{b}_{i}{\bm{1}}_{d}\right)\odot{\bm{B}}\right)\left( \left(\lambda^{d}_{i}{\bm{1}}_{d}\right)\odot{\bm{A}}\right){\bm{X}}_{i}\|^{2} _{2}}.$ $$

If we compute the partial derivatives with respect to the unknown vectors $\lambda^{b}$ and $\lambda^{d}$ and set them equal to zero, we obtain an indeterminate system, as similarly discussed in Section A.1. The detailed derivation is left to the reader. However, it is still possible to solve for either $\lambda^{b}$ or $\lambda^{d}$ when the other is known.

#### A.2.1 Solving for λ d superscript 𝜆 𝑑 \lambda^{d} italic_λ start_POSTSUPERSCRIPT italic_d end_POSTSUPERSCRIPT with fixed λ b superscript 𝜆 𝑏 \lambda^{b} italic_λ start_POSTSUPERSCRIPT italic_b end_POSTSUPERSCRIPT

Assuming the vector $\lambda^{b}$ is fixed, we can solve for $\lambda^{d}$. The objective function $\Omega$ becomes:

$$ $\Omega_{\lambda^{d}}=\sum_{i=1}^{N}{\left\|\left(\left(\lambda^{b}{\bm{1}}_{b} \right)\odot{\bm{B}}\right)\left(\left(\lambda^{d}{\bm{1}}_{d}\right)\odot{\bm {A}}\right){\bm{X}}_{i}-\left(\left(\lambda^{b}{\bm{1}}_{b}\right)\odot{\bm{B} }\right)\left(\left(\lambda^{d}_{i}{\bm{1}}_{d}\right)\odot{\bm{A}}\right){\bm {X}}_{i}\right\|^{2}_{2}}.$ $$

This can be rearranged as:

$$ $\Omega_{\lambda^{d}}=\sum_{i=1}^{N}{\left\|\left(\left(\lambda^{b}{\bm{1}}_{b} \right)\odot{\bm{B}}\right)\left[\left(\left(\lambda^{d}{\bm{1}}_{d}\right) \odot{\bm{A}}\right){\bm{X}}_{i}-\left(\left(\lambda^{d}_{i}{\bm{1}}_{d}\right )\odot{\bm{A}}\right){\bm{X}}_{i}\right]\right\|^{2}_{2}}.$ $$

Since $\left(\lambda^{b}{\bm{1}}_{b}\right)\odot{\bm{B}}$ is a multiplicative constant, it can be factored out and disregarded, leading to the following simplified objective:

$$ $\Omega_{\lambda^{d}}=\sum_{i=1}^{N}{\left\|\left(\left(\lambda^{d}{\bm{1}}_{d} \right)\odot{\bm{A}}\right){\bm{X}}_{i}-\left(\left(\lambda^{d}_{i}{\bm{1}}_{d }\right)\odot{\bm{A}}\right){\bm{X}}_{i}\right\|^{2}_{2}}.$ $$

To minimize $\Omega_{\lambda^{d}}$, we take its derivative with respect to $\lambda^{d}$ and set it to zero:

$$ $\frac{\partial\Omega_{\lambda^{d}}}{\partial\lambda^{d}}=0\quad\Rightarrow \quad 2\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{\bm{A} }\big{)}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\odot{\bm{A}}{\bm{1}}_{d}^{\top}-\sum_ {i=1}^{N}{\big{(}\big{(}\lambda^{d}_{i}{\bm{1}}_{d}\big{)}\odot{\bm{A}}\big{)} {\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\odot{\bm{A}}{\bm{1}}_{d}^{\top}=0.$ $$

This simplifies to:

$$ $\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{\bm{A}}\big{)}\sum_{i=1}^{N} {{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}=\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{d}_{i} {\bm{1}}_{d}\big{)}\odot{\bm{A}}\big{)}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}.$ $$

We then solve for $\lambda^{d}$ as follows:

$$ $\displaystyle\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{\bm{A}}\big{)}$ $\displaystyle=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{d}_{i}{\bm{1}}_{d} \big{)}\odot{\bm{A}}\big{)}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{ i=1}^{N}{{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)^{-1},$ $\displaystyle\lambda^{d}{\bm{1}}_{d}$ $\displaystyle=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{d}_{i}{\bm{1}}_{d} \big{)}\odot{\bm{A}}\big{)}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{ i=1}^{N}{{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)^{-1}\odot\frac{1}{{\bm{A}}},$ $\displaystyle\lambda^{d}{\bm{1}}_{d}{\bm{1}}_{d}^{\top}$ $\displaystyle=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{d}_{i}{\bm{1}}_{d} \big{)}\odot{\bm{A}}\big{)}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{ i=1}^{N}{{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)^{-1}\odot\frac{1}{{\bm{A}}}{ \bm{1}}_{d}^{\top}.$ $$

Since ${\bm{1}}_{d}{\bm{1}}_{d}^{\top}=d$ is a scalar, we can finally find:

$$ $\lambda^{d}=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{d}_{i}{\bm{1}}_{d}\big {)}\odot{\bm{A}}\big{)}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{i=1} ^{N}{{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)^{-1}\odot\frac{1}{d{\bm{A}}}{\bm{ 1}}_{d}^{\top}.$ $$

#### A.2.2 Solving for λ b superscript 𝜆 𝑏 \lambda^{b} italic_λ start_POSTSUPERSCRIPT italic_b end_POSTSUPERSCRIPT with fixed λ d superscript 𝜆 𝑑 \lambda^{d} italic_λ start_POSTSUPERSCRIPT italic_d end_POSTSUPERSCRIPT

When the vector $\lambda^{d}$ is fixed, we solve for $\lambda^{b}$. The objective function $\Omega$ becomes:

$$ $\Omega_{\lambda^{b}}=\sum_{i=1}^{N}{\left\|\left(\left(\lambda^{b}{\bm{1}}_{b} \right)\odot{\bm{B}}\right)\left(\left(\lambda^{d}{\bm{1}}_{d}\right)\odot{\bm {A}}\right){\bm{X}}_{i}-\left(\left(\lambda^{b}_{i}{\bm{1}}_{b}\right)\odot{ \bm{B}}\right)\left(\left(\lambda^{d}{\bm{1}}_{d}\right)\odot{\bm{A}}\right){ \bm{X}}_{i}\right\|^{2}_{2}}.$ $$

To minimize $\Omega_{\lambda^{b}}$, we compute its derivative with respect to $\lambda^{b}$ and set it to zero:

$$ $\displaystyle\frac{\partial\Omega_{\lambda^{b}}}{\partial\lambda^{b}}=0\quad\Rightarrow\quad$ $\displaystyle 2\sum_{i=1}^{N}{\Big{[}\big{(}\big{(}\lambda^{b}{\bm{1}}_{b}\big {)}\odot{\bm{B}}\big{)}\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{\bm{A }}\big{)}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}}-$ $\displaystyle-$ $\displaystyle\big{(}\big{(}\lambda^{b}_{i}{\bm{1}}_{b}\big{)}\odot{\bm{B}}\big {)}\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{\bm{A}}\big{)}{\bm{X}}_{i }{\bm{X}}_{i}^{\top}\Big{]}\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{ \bm{A}}\big{)}^{\top}\odot{\bm{B}}{\bm{1}}_{b}^{\top}=0.$ $$

This simplifies to:

$$ $\big{(}\big{(}\lambda^{b}{\bm{1}}_{b}\big{)}\odot{\bm{B}}\big{)}\sum_{i=1}^{N} {\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\mathcal{A}^{\top}}=\sum_{i=1}^{N}{ \big{(}\big{(}\lambda^{b}_{i}{\bm{1}}_{b}\big{)}\odot{\bm{B}}\big{)}\mathcal{A }{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\mathcal{A}^{\top}},$ $$

where we use $\mathcal{A}$ to denote $\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{\bm{A}}\big{)}$ for simplicity. Solving for $\lambda^{b}$, we get:

$$ $\displaystyle\big{(}\big{(}\lambda^{b}{\bm{1}}_{b}\big{)}\odot{\bm{B}}\big{)}$ $\displaystyle=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{b}_{i}{\bm{1}}_{b} \big{)}\odot{\bm{B}}\big{)}\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\mathcal{ A}^{\top}}\right)\left(\sum_{i=1}^{N}{\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{ \top}\mathcal{A}^{\top}}\right)^{-1},$ $\displaystyle\lambda^{b}{\bm{1}}_{b}$ $\displaystyle=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{b}_{i}{\bm{1}}_{b} \big{)}\odot{\bm{B}}\big{)}\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\mathcal{ A}^{\top}}\right)\left(\sum_{i=1}^{N}{\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{ \top}\mathcal{A}^{\top}}\right)^{-1}\odot\frac{1}{{\bm{B}}},$ $\displaystyle\lambda^{b}{\bm{1}}_{b}{\bm{1}}_{d}^{\top}$ $\displaystyle=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{b}_{i}{\bm{1}}_{b} \big{)}\odot{\bm{B}}\big{)}\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\mathcal{ A}^{\top}}\right)\left(\sum_{i=1}^{N}{\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{ \top}\mathcal{A}^{\top}}\right)^{-1}\odot\frac{1}{{\bm{B}}}{\bm{1}}_{d}^{\top}.$ $$

Finally, since ${\bm{1}}_{d}{\bm{1}}_{d}^{\top}=d$ is a scalar, we can solve for $\lambda^{b}$:

$$ $\lambda^{b}=\left(\sum_{i=1}^{N}{\big{(}\big{(}\lambda^{b}_{i}{\bm{1}}_{b}\big {)}\odot{\bm{B}}\big{)}\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{\top}\mathcal{A}^{ \top}}\right)\left(\sum_{i=1}^{N}{\mathcal{A}{\bm{X}}_{i}{\bm{X}}_{i}^{\top} \mathcal{A}^{\top}}\right)^{-1}\odot\frac{1}{d{\bm{B}}}{\bm{1}}_{d}^{\top},$ $$

where $\mathcal{A}=\big{(}\big{(}\lambda^{d}{\bm{1}}_{d}\big{)}\odot{\bm{A}}\big{)}$.

### A.3 Combination of (IA) 3 and RegMean

(IA)^3 (Liu et al., ) adapts the pretrained model by learning multiplicative weight vectors $\ell^{k}$ that modulate the intermediate activations — where $k$ is the dimension of the considered activation. Specifically, for each attention layer, these weight vectors are applied to the input keys, values, and output. Notably, it can be demonstrated that, for a single linear layer, (IA)^3 can be expressed as a residual module (Porrello et al., ):

$$ $\ell^{k}\odot({\bm{W}}_{0}{\bm{X}})=\left[{\bm{W}}_{0}+((\ell^{k}-{\bm{1}}_{k} ^{\top}){\bm{1}}_{h})\odot{\bm{W}}_{0}\right]{\bm{X}},$ $$

where ${\bm{1}}_{o}$ and ${\bm{1}}_{h}$ represent row vectors of ones, respectively corresponding to the dimensions of the output and input of the considered layer. For the sake of simplicity, we deviate from the original formulation, which initializes $\ell^{k}$ as a column vector of ones. Instead, we begin with a column vector of zeroes at the beginning of training. This adjustment does not affect the outcome, as we simultaneously modify our formulation as follows:

$$ $\ell^{k}\odot({\bm{W}}_{0}{\bm{X}})=\left[{\bm{W}}_{0}+(\ell^{k}{\bm{1}}_{h}) \odot{\bm{W}}_{0}\right]{\bm{X}},$ $$

Then, following Section A.1, we consider the same minimization problem as Equation 10. This time, we replace ${\bm{W}}$ with ${\bm{W}}_{0}+(\ell^{k}{\bm{1}}_{h})\odot{\bm{W}}_{0}$, where ${\bm{W}}_{0}$ denotes the pre-trained weight matrix and $(\ell^{k}{\bm{1}}_{h})\odot{\bm{W}}_{0}$ represents the learned weight difference and $\ell^{k}$ is zero initialized. The objective function becomes:

$$ $\Omega_{\text{{(IA)\textsuperscript{3}}}}=\sum_{i=1}^{N}{\|({\bm{W}}_{0}+(\ell ^{k}{\bm{1}}_{h})\odot{\bm{W}}_{0}){\bm{X}}_{i}-({\bm{W}}_{0}+(\ell^{k}_{i}{ \bm{1}}_{h})\odot{\bm{W}}_{0}){\bm{X}}_{i}\|^{2}_{2}}.$ $$

Following Section A.1, we can rearrange $\Omega_{\text{{(IA)\textsuperscript{3}}}}$ as:

$$ $\Omega_{\text{{(IA)\textsuperscript{3}}}}=\sum_{i=1}^{N}{\|((\ell^{k}{\bm{1}}_ {h})\odot{\bm{W}}_{0}){\bm{X}}_{i}-((\ell^{k}_{i}{\bm{1}}_{h})\odot{\bm{W}}_{0 }){\bm{X}}_{i}\|^{2}_{2}}.$ $$

To minimize $\Omega_{\text{{(IA)\textsuperscript{3}}}}$, we take its derivative with respect to $\ell^{k}$ and set it to zero:

$$ $\frac{\partial\Omega_{\text{{(IA)\textsuperscript{3}}}}}{\partial\ell^{k}}=0 \quad\Rightarrow\quad 2\sum_{i=1}^{N}{\left[((\ell^{k}{\bm{1}}_{h})\odot{\bm{W }}_{0}){\bm{X}}_{i}{\bm{X}}_{i}^{\top}-((\ell^{k}_{i}{\bm{1}}_{h})\odot{\bm{W} }_{0}){\bm{X}}_{i}{\bm{X}}_{i}^{\top}\right]}\odot{\bm{W}}_{0}{\bm{1}}_{h}^{ \top}=0.$ $$

This simplifies to:

$$ $((\ell^{k}{\bm{1}}_{h})\odot{\bm{W}}_{0})\sum_{i=1}^{N}{{\bm{X}}_{i}{\bm{X}}_{ i}^{\top}}=\sum_{i=1}^{N}{((\ell^{k}_{i}{\bm{1}}_{h})\odot{\bm{W}}_{0}){\bm{X} }_{i}{\bm{X}}_{i}^{\top}}.$ $$

Solving for $\ell^{k}$, we get:

$$ $\displaystyle((\ell^{k}{\bm{1}}_{h})\odot{\bm{W}}_{0})$ $\displaystyle=\left(\sum_{i=1}^{N}{((\ell^{k}_{i}{\bm{1}}_{h})\odot{\bm{W}}_{0 }){\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{i=1}^{N}{{\bm{X}}_{i}{\bm {X}}_{i}^{\top}}\right)^{-1}.$ $\displaystyle\ell^{k}{\bm{1}}_{h}$ $\displaystyle=\left(\sum_{i=1}^{N}{((\ell^{k}_{i}{\bm{1}}_{h})\odot{\bm{W}}_{0 }){\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{i=1}^{N}{{\bm{X}}_{i}{\bm {X}}_{i}^{\top}}\right)^{-1}\odot\frac{1}{{\bm{W}}_{0}}.$ $\displaystyle\ell^{k}{\bm{1}}_{h}{\bm{1}}_{h}^{\top}$ $\displaystyle=\left(\sum_{i=1}^{N}{((\ell^{k}_{i}{\bm{1}}_{h})\odot{\bm{W}}_{0 }){\bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{i=1}^{N}{{\bm{X}}_{i}{\bm {X}}_{i}^{\top}}\right)^{-1}\odot\frac{1}{{\bm{W}}_{0}}{\bm{1}}_{h}^{\top}.$ $$

Finally, since ${\bm{1}}_{h}{\bm{1}}_{h}^{\top}=h$ is a scalar, we can solve for $\ell^{k}$:

$$ $\ell^{k}=\left(\sum_{i=1}^{N}{((\ell^{k}_{i}{\bm{1}}_{h})\odot{\bm{W}}_{0}){ \bm{X}}_{i}{\bm{X}}_{i}^{\top}}\right)\left(\sum_{i=1}^{N}{{\bm{X}}_{i}{\bm{X} }_{i}^{\top}}\right)^{-1}\odot\frac{1}{h{\bm{W}}_{0}}{\bm{1}}_{h}^{\top}.$ $$

### A.4 RegMean Applied to Classification Heads

While each client adapts the pretrained model to the specific task using residual parameter-efficient modules, the classification head must be re-initialized and trained from scratch. Specifically, following the approach of Caccia et al. (), we initialize a distinct classification head for each task. This practice has recently emerged as the de facto standard in Class-Incremental Learning, consistently providing a performance boost across various methodologies.

In the context of Class-Incremental Learning, all task-specific classification heads must eventually be merged into a unified classifier. To address this, we aim to minimize the RegMean problem, as defined in Equation 10, across the classifiers of different tasks. Rather than considering a separate set of parameters for each client, we now introduce a weight matrix for each task:

$$ $\operatorname*{minimize}~{}\Omega_{\text{cls}}=\sum_{t=1}^{T}{\|{\bm{W}}{\bm{X }}_{t}-{\bm{W}}_{t}{\bm{X}}_{t}\|^{2}_{2}}.$ $$

Notably, in this case, ${\bm{W}}$ and ${\bm{W}}_{t}$ differ in dimensionality, as the classification heads $\{{\bm{W}}_{1},{\bm{W}}_{2},\dots,{\bm{W}}_{T}\}$ are trained independently. Specifically, while ${\bm{W}}$ and each ${\bm{W}}_{t}$ share the same number of columns (input dimension), they differ in the number of rows (output dimension). The total number of rows of ${\bm{W}}$ corresponds to the sum of the rows of all ${\bm{W}}_{t}$, which is equivalent to the total number of classes $\mathcal{C}$; instead, each ${\bm{W}}_{t}$ has $\mathbbm{c}$ rows. This structure enables us to decompose $\Omega_{\text{cls}}$ into $T$ separate minimization problems:

$$ $\operatorname*{minimize}~{}\sum_{t=1}^{T}{\Omega_{\text{cls}}^{t}},\quad\text{ where}\quad{\Omega_{\text{cls}}^{t}}=\|{\bm{W}}_{\mathbbm{c}t:\mathbbm{c}(t+1) }{\bm{X}}_{t}-{\bm{W}}_{t}{\bm{X}}_{t}\|^{2}_{2}.$ $$

Since the tasks are class-disjoint, each $\{\Omega_{\text{cls}}^{1},\Omega_{\text{cls}}^{2},\dots,\Omega_{\text{cls}}^{T}\}$ operates on distinct rows of ${\bm{W}}$. Consequently, the optimal solution for each $\Omega_{\text{cls}}^{t}$ is to assign ${\bm{W}}_{\mathbbm{c}_{t}:\mathbbm{c}(t+1)}={\bm{W}}_{t}$ for each $t$.

## Appendix B Details on Datasets and Metrics

### B.1 CIFAR-100

For the incremental split of CIFAR-100, we follow the standard partitioning into $10$ tasks, each consisting of $10$ classes, as commonly employed in numerous Continual Learning studies (Wang et al., ; ; Smith et al., ). For data augmentation on the training set, we apply bicubic interpolation to resize each image from $32\times 32$ to $224\times 224$, followed by random horizontal flipping and normalization. For the test set, each image is first resized to $256\times 256$ using bicubic interpolation, followed by a center crop to $224\times 224$, and a final normalization step.

### B.2 ImageNet-R

For the incremental split of ImageNet-R, we employ the widely used partitioning into $10$ tasks, each comprising $20$ classes, as is common in various Continual Learning studies (Wang et al., ; Smith et al., ; Zhang et al., ). For data augmentation on the training set, we process each $224\times 224$ image with a random horizontal flip, followed by normalization. In the test set, we first resize each image to $256\times 256$ using bicubic interpolation, apply a center crop to $224\times 224$, and then perform normalization.

### B.3 EuroSAT

For the incremental split of EuroSAT, we follow a partitioning scheme of $5$ tasks, each consisting of $2$ classes, consistent with prior Continual Learning studies (Jung et al., ; Menabue et al., ). For data augmentation in the training set, each $64\times 64$ image is resized to $224\times 224$ using bicubic interpolation. The same interpolation process is applied to the test set.

### B.4 Final Average Accuracy (FAA)

We evaluate the performance of all methods using the Final Average Accuracy (FAA), a widely adopted metric in the FCIL literature.

FAA quantifies the mean accuracy across all tasks at the conclusion of the incremental training process. Formally, let $a^{t}$ represent the accuracy on the $t^{th}$ task after completing the incremental training. The FAA is defined as:

$$ $\text{FAA}=\frac{1}{T}\sum_{t=1}^{T}a^{t},$ (11) $$

where $T$ represents the total number of tasks.

## Appendix C Additional Results

### C.1 Standard Deviations

The standard deviations for all evaluated approaches, based on three runs, are reported for the in-domain datasets in Table A and for the out-of-domain dataset in Table B.

**Table A: In-domain standard deviations. Standard deviations for all evaluated approaches.**
|  | CIFAR-100 | ImageNet-R |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Joint | $\pm 0.66$ | $\pm 1.08$ |  |  |  |  |
| Partition | $\beta=0.5$ | $\beta=0.1$ | $\beta=0.05$ | $\beta=0.5$ | $\beta=0.1$ | $\beta=0.05$ |
| EWC (Kirkpatrick et al., ) | $\pm 0.95$ | $\pm 0.57$ | $\pm 0.83$ | $\pm 0.89$ | $\pm 1.09$ | $\pm 0.83$ |
| LwF (Li & Hoiem, ) | $\pm 0.90$ | $\pm 0.89$ | $\pm 0.53$ | $\pm 0.81$ | $\pm 0.80$ | $\pm 0.98$ |
| DER++ (Buzzega et al., ) | $\pm 0.61$ | $\pm 0.75$ | $\pm 1.41$ | $\pm 0.47$ | $\pm 0.58$ | $\pm 1.38$ |
| L2P (Wang et al., ) | $\pm 1.24$ | $\pm 1.37$ | $\pm 0.82$ | $\pm 1.10$ | $\pm 0.66$ | $\pm 1.15$ |
| CODA-P (Smith et al., ) | $\pm 0.82$ | $\pm 0.58$ | $\pm 0.63$ | $\pm 0.50$ | $\pm 1.53$ | $\pm 0.91$ |
| FisherAvg (Matena & Raffel, ) | $\pm 1.28$ | $\pm 0.67$ | $\pm 0.58$ | $\pm 1.09$ | $\pm 1.15$ | $\pm 0.03$ |
| RegMean (Jin et al., ) | $\pm 0.28$ | $\pm 0.36$ | $\pm 0.78$ | $\pm 1.41$ | $\pm 0.67$ | $\pm 0.56$ |
| CCVR (Luo et al., ) | $\pm 0.86$ | $\pm 0.49$ | $\pm 0.55$ | $\pm 0.39$ | $\pm 0.66$ | $\pm 0.72$ |
| TARGET (Zhang et al., ) | $\pm 0.54$ | $\pm 0.60$ | $\pm 0.47$ | $\pm 0.23$ | $\pm 1.33$ | $\pm 0.94$ |
| PILoRA (Guo et al., ) | $\pm 0.86$ | $\pm 0.48$ | $\pm 0.80$ | $\pm 1.57$ | $\pm 1.14$ | $\pm 1.56$ |
| LoRM (ours) | $\pm 0.73$ | $\pm 0.62$ | $\pm 0.59$ | $\pm 1.36$ | $\pm 0.70$ | $\pm 1.46$ |

**Table B: Out of domain standard deviations. Standard deviations for all evaluated approaches.**
| EuroSAT |  |  |  |
| --- | --- | --- | --- |
| Joint | $\pm 0.66$ |  |  |
| Partition | $\beta=1.0$ | $\beta=0.5$ | $\beta=0.2$ |
| EWC | $\pm 1.17$ | $\pm 0.74$ | $\pm 0.83$ |
| LwF | $\pm 0.67$ | $\pm 0.33$ | $\pm 0.87$ |
| DER++ | $\pm 0.94$ | $\pm 0.82$ | $\pm 0.85$ |
| L2P | $\pm 1.69$ | $\pm 1.18$ | $\pm 1.36$ |
| CODA-P | $\pm 0.57$ | $\pm 1.28$ | $\pm 0.73$ |
| FisherAvg | $\pm 1.38$ | $\pm 0.77$ | $\pm 0.62$ |
| RegMean | $\pm 0.17$ | $\pm 0.78$ | $\pm 0.98$ |
| CCVR | $\pm 0.92$ | $\pm 0.87$ | $\pm 1.56$ |
| TARGET | $\pm 0.73$ | $\pm 0.82$ | $\pm 0.86$ |
| PILoRA | $\pm 1.01$ | $\pm 0.87$ | $\pm 0.45$ |
| LoRM (ours) | $\pm 1.04$ | $\pm 0.73$ | $\pm 0.81$ |

### C.2 Additional Results on EuroSAT

Additional evaluated approaches are reported in Table C.

**Table C: Additional EuroSAT comparison. Results in terms of FAA($\%$).**
| Joint | $93.72$ |  |  |
| --- | --- | --- | --- |
| Partition | $\beta=1.0$ | $\beta=0.5$ | $\beta=0.2$ |
| EWC | $55.88$ | $46.23$ | $44.16$ |
| LwF | $23.91$ | $21.31$ | $21.64$ |
| CODA-P | $62.96$ | $59.20$ | $62.23$ |
| FisherAvg | $48.83$ | $43.19$ | $42.48$ |
| RegMean | $27.12$ | $30.80$ | $32.31$ |

## Appendix D Hyperparameters

All method-specific parameters not explicitly mentioned follow the settings described in their original papers.

### D.1 Acronyms

List of all the acronyms used:

- •
lr: learning rate,
- •
$\textit{lr}_{pr}$: learning rate for prototypes,
- •
$\lambda_{\text{KL}}$: Knowledge Distillation Loss multiplier,
- •
r: rank for low rank matrices,
- •
$\textit{n}_{p}$: number of prompts,
- •
$\textit{g}_{ep}$: epochs of training and generation for the generator network,
- •
$\gamma$: decay factor for off-diagonal elements of the Gram matrices.

### D.2 CIFAR-100

#### D.2.1 Distribution β 𝛽 \beta italic_β = 0.5

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 3e-06; |
| LwF: | lr: 3e-06; |
| DER++: | lr: 3e-06; $\alpha$:34; $\beta$:13 |
| L2P: | lr: 3e-02; |
| CODA-P: | lr: 1e-03; |
| FisherAvg: | lr: 3e-06; |
| RegMean: | lr: 3e-06; $\gamma$: 0.5 |
| CCVR: | lr: 3e-06; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 1e-03; r: 16; |

#### D.2.2 Distribution β 𝛽 \beta italic_β = 0.1

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 3e-06; |
| LwF: | lr: 3e-06; |
| DER++: | lr: 1e-4; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-02; |
| CODA-P: | lr: 1e-03; |
| FisherAvg: | lr: 3e-06; |
| RegMean: | lr: 3e-06; $\gamma$: 0.5 |
| CCVR: | lr: 3e-06; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 3e-04; r: 16; |

#### D.2.3 Distribution β 𝛽 \beta italic_β = 0.05

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 3e-06; |
| LwF: | lr: 1e-05; |
| DER++: | lr: 3e-06; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-02; |
| CODA-P: | lr: 1e-03; $\textit{n}_{p}$: 100 |
| FisherAvg: | lr: 3e-06; |
| RegMean: | lr: 3e-06; $\gamma$: 0.5 |
| CCVR: | lr: 3e-06; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 3e-04; r: 32; |

### D.3 ImageNet-R

#### D.3.1 Distribution β 𝛽 \beta italic_β = 0.5

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 1e-05; |
| LwF: | lr: 1e-05; |
| DER++: | lr: 1e-05; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-04; |
| CODA-P: | lr: 3e-04; $\textit{n}_{p}$: 200 |
| FisherAvg: | lr: 1e-05; |
| RegMean: | lr: 1e-05; $\gamma$: 0.5 |
| CCVR: | lr: 3e-06; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 1e-03; r: 16 |

#### D.3.2 Distribution β 𝛽 \beta italic_β = 0.1

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 1e-05; |
| LwF: | lr: 1e-05; |
| DER++: | lr: 1e-05; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-04; |
| CODA-P: | lr: 3e-04; $\textit{n}_{p}$: 200 |
| FisherAvg: | lr: 1e-05; |
| RegMean: | lr: 1e-05; $\gamma$: 0.5 |
| CCVR: | lr: 1e-04; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 1e-03; r: 32 |

#### D.3.3 Distribution β 𝛽 \beta italic_β = 0.05

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 1e-05; |
| LwF: | lr: 1e-05; |
| DER++: | lr: 1e-05; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-04; |
| CODA-P: | lr: 3e-04; $\textit{n}_{p}$: 200 |
| FisherAvg: | lr: 1e-05; |
| RegMean: | lr: 3e-06; $\gamma$: 0.5 |
| CCVR: | lr: 1e-05; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 1e-03; r: 16 |

### D.4 EuroSAT

#### D.4.1 Distribution β 𝛽 \beta italic_β = 1.0

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 1e-05; |
| LwF: | lr: 1e-05; |
| DER++: | lr: 1e-04; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-04; |
| CODA-P: | lr: 3e-04; $\textit{n}_{p}$: 10 |
| FisherAvg: | lr: 1e-05; |
| RegMean: | lr: 3e-06; $\gamma$: 0.5 |
| CCVR: | lr: 1e-05; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 1e-03; r: 16 |

#### D.4.2 Distribution β 𝛽 \beta italic_β = 0.5

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 1e-05; |
| LwF: | lr: 1e-05; |
| DER++: | lr: 3e-06; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-04; |
| CODA-P: | lr: 3e-04; $\textit{n}_{p}$: 10 |
| FisherAvg: | lr: 1e-05; |
| RegMean: | lr: 1e-05; $\gamma$: 0.5 |
| CCVR: | lr: 1e-05; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$: 25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 1e-03; r: 16 |

#### D.4.3 Distribution β 𝛽 \beta italic_β = 0.2

| Joint: | lr: 1e-03; |
| --- | --- |
| EWC: | lr: 1e-05; |
| LwF: | lr: 1e-05; |
| DER++: | lr: 1e-05; $\alpha$: 0.3; $\beta$: 0.8 |
| L2P: | lr: 3e-04; |
| CODA-P: | lr: 3e-04; $\textit{n}_{p}$: 10 |
| FisherAvg: | lr: 1e-05; |
| RegMean: | lr: 1e-05; $\gamma$: 0.5 |
| CCVR: | lr: 1e-05; |
| TARGET: | lr: 3e-04; $\lambda_{\text{KL}}$:25; $\textit{g}_{ep}$: 30 |
| PILoRA: | lr: 2e-02; $\textit{lr}_{pr}$: 1e-4; |
| LoRM: | lr: 1e-03; r: 16 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: The validity of the optimization solver should be checked.
The lack of evaluation metrics should be supplemented to show the effectiveness of the proposed algorithm.
soundness: 3
strengths: The target problem and the contribution are well presented in the paper, and the following mathetmatical derivation is also easy to follow. The experimental results verify the effectiveness of the proposed algorithm, and futher ablation studies are well designed to prove the component-wise effectiveness and the mentioned contributions.
summary: The paper presents a novel optimization method for Federated Class-incremental Learning (FCIL). Based on the LoRA approaches, the authors propose a optimization method to estimate the two low-rank matrices effiently. Rather than the previous approaches upon the various types of assumptions, the proposed algorithm employed the alternating optimization methods by optimizing the two low-rank matrices separately by fixing the other one. The optimization methods was evaluated in the scenario of FCIL, which shows the state-of-the-art performance for CIFAR-100 and ImageNet-R dataset even with the various evalutation settings.
weaknesses: Even with the above mentioned strengths, I have several concerns about the technical algorithm and evaluations as follows:

- Contribution (Alternative optimization vs. closed-form):
 The authors emphasized the contribution of this paper as the alternating optimization and the closed-form optimization. However, the relationship between the two different optimization methods is not precisely described. As I understood, the closed-form optimization is performed at each iteration for the respective low-rank matices, and the overall optimization solver follows the mechanism of alternating optimization. Since the two optimization approaches are definitely different to each other, the authors should specify the relationship of the two contributions.

- Convergence check for the optimization solver:
 When we design the optimization solver, it is important to check its convergence. When the optimization fails to convergence, the solution can be diverged, which causes serious problems. If the convergence check fails, the strategy to stop the iteration should be given. In the paper, both the convergence check and the stop criterion are not described.

- Validity of alternating optimization:
 The approach of alternating optimization is one of the most common optimization solvers. When we employ the alternating optimization approach by fixing one other of two parameters, it is important to check the relationship between the two parameters. When the two paraemters are not indenpendent to each other, the iteration with separated optimization can cause serious problem where the basic assumption for the two parameter can be broken. To prevent the issue, the conventional alternating optimization employs a penalty term where the relationship between two optimizing parameters can be preserved as the iteration goes. I hope to know that the proposed algorithm considers the relationship between the two low-rank matrices or the two low-rank matrices are independent to each other.

- Additional measurements:
 The quantitative results are present only by the average accuracy for entire classes. However, to evaluate the FCIL method, tthe measurements of he average forgetting and accuracy of all seen classes in the final task are meaningful. To show the effectiveness of the proposed algorithm for the last rounds and the final round, I recommend such additional measurements in the tables.

- Comparison fairness:
 It seems that the most recent study (PILoRA) shows low performance upon ImageNet-R dataset. Of course, the performance can be dramatically dropped in the new dataset, but it shows too large gap between its reported performance with the ImageNet-based dataset. I hope to know the reason why the large gap happens.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the Weaknesses section.
soundness: 3
strengths: 1. The extension of RegMean for merging the LoRA module is interesting. Given the extensive applicability of the LoRA for parameter-efficient fine-tuning for foundation models, the reviewer believes that the proposed method may benefit a broad range of applications in the future.

2. The proposed alternative optimization procedure for extending RegMean to Low-rank Regression Mean (LoRM) is intuitive and sound. It is promising that the algorithm can converge quickly in practice, as shown in Fig. 2, compared to the methods that achieved the state-of-the-art (SOTA) results previously.

3. The chosen testbed, i.e., the federal class incremental learning (FCIL), is a surging research direction in continual learning with real-world board applications. The author conducted extensive comparisons with different methods and showed the superiority of the LoRM, which vastly improved the SOTA of FCIL.

4. The code has been attached for replication, which will benefit future study.
summary: This paper studies the model merging, specifically focusing on merging the low-rank adaptation (LoRA) modules. The author proposed to extend the work of RegMean to merge LoRA modules in a closed form, where the challenge lies in the infeasibility of solving the indeterminate system introduced by the LoRA modules. The author further proposes an alternative optimization algorithm, i.e., freezing one of the two matrices, A and B, in the LoRA modules and optimizing each alternatively. The author chose the federal class incremental learning as the case study for the proposed method, conducted extensive experiments to verify the effectiveness of proposed method.
weaknesses: 1. Since the proposed method is general for model merging, the reviewer may want to see whether it can achieve good results in different applications, e.g., parameter-efficient fine-tuning and class-incremental learning in computer vision and/or natural language processing.

2. Although the author shows that the LoRM can converge quickly in practice, the reviewer still wonders whether there are any theoretical guarantees for its convergence.

---

confidence: 5
contribution: 1
initial_rating: 6
presentation: 2
questions: Please refer to the questions in the weakness section
soundness: 2
strengths: The method is privacy preserving and enables continual merging for the Image Classification experiments shown.
summary: Low-rank Regression Mean (LoRM) is a novel method to merge LoRA (Low-Rank Adaptation) modules in a closed-form solution specifically designed for Federated Class-Incremental Learning (FCIL) by alternatively freezing and merging A and B submatrices of LoRA method.
weaknesses: **Inadequate Experimental Diversity** LoRM builds on LoRA, a parameter efficient finetuning method which was introduced to finetune large LLMs and other large, computationally expensive models for diverse and complex tasks. However, LoRM is only tested on the task of image classification and on a single backbone. This counteracts the motivation of using LoRAs (and LoRM is specific to only LoRAs). Experiments with different backbones - LLM, text-image models and diverse tasks like commonsense reasoning, text to image generation - where LoRA modules are regualrly - are necessary to show the efficacy of LoRM, especially since the contributions in the work are backed by only empirical results. LoRM's effectiveness across other federated learning tasks also remains unclear. Testing LoRM in other tasks, such as reinforcement learning or generative tasks, would strengthen its generalizability​.

**Sensitivity to Data Distribution**: LoRM’s performance may degrade with extreme data heterogeneity. The method could benefit from further optimization or regularization to ensure robustness across highly imbalanced data distributions typical in federated settings​.

It is unclear why the alternating freezing is needed? Why not train another LoRA which combines the previous base model+ LoRA weights? That appears to be the simplest solution to the model merging problem and would avoid problems like highly imbalanced data distributions across different tasks which could lead to load imbalance on particular matrices. Ablation for this would also have been helpful.

Comparisons to model merging and lora merging methods like [1, 2] are needed since this method focuses on this very aspect.

[1] Stoica et al. - ZipIt! Merging Models from Different Tasks without Training
[2] Yadav et. al - TIES-Merging: Resolving Interference When Merging Models

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - Replicated results appear to be conflicting with previous works (L2P and TARGET’s relative ranking is switched in the PILoRA paper). Is there a reason for this discrepancy?  

- It is unusual to see L2P outperforming Coda-P, as Coda-P outperforms L2P in the standard CIL setting. What is the reason for this? How do RANPAC (https://arxiv.org/abs/2307.02251) and DualPrompt (https://arxiv.org/abs/2204.04799) compare? Does this inverse relationship between CIL accuracy and FCIL accuracy hold?  

- ImageNet-A, CUB, and Cars196 have been shown to be more challenging datasets for CIL methods, with wider gaps in accuracy when comparing techniques. How does LoRM perform on these datasets?
soundness: 3
strengths: - Adequate novelty is shown by introducing a training scheme that can properly update LoRA modules across multiple clients.  

- LoRM preserves privacy between devices by using Gram matrices 

- Ablations clearly show the importance of the proposed components.  

- Provided results show significant improvement over SOTA.  

- Extremely detailed list of parameters for all experiments is provided, greatly aiding replication.  

- Very straightforward organization, the paper flows logically and is easy to read.
summary: This paper proposes a solution for federated continual  learning, where information is learned across multiple devices, with each device continually learning sets of disjoint classes. The proposed method proposes a two stage approach. The first stage involves learning LoRA modules in an alternating cycle, where one of the A or B matrices is kept frozen at a time while the other is optimized across individual devices. Once this has been done for all tasks across all devices, the LoRA matrices are aggregated one final time using RegMean to create one model with knowledge from each device across all tasks. The proposed method outperforms presented CIL and FCIL
weaknesses: - The paper mentions  that there is relatively little work in FCIL, a brief diagram visualizing the problem setup would greatly aid in understanding the problem.  

- Figure diagrams, namely Figure 1 requires more detailed captions. - The current SOTA for exemplar free CIL, RANPAC, is missing from the comparison list, this method also uses Gram matrices collecting information across tasks, so a comparison where it is adapted to be a FCIL method would be relevant.  

- The Previous SOTA, PILoRA shows results for both Quantity-based label imbalance and Distribution-based label imbalance, but results in this paper are only shown for Distribution-based.

Meta-review: The paper introduces LoRM, a technique for model merging Federated Continual Learning that builds upon low-rank adaptation methods like LoRA. The authors propose to unify models by ensuring that the merged model matches the responses of all participating models. The authors address the challenge of indeterminate solutions through an alternating optimization strategy, which allows for training one LoRA matrix at a time. The method is applied to the Federated Class-Incremental Learning (FCIL) scenario and it demonstrates state-of-the-art performance in a variety of FCIL scenarios.

Reviewers variously praised the submission for its clear presentation of the target problem and contributions, as well as its easy-to-follow  mathematical derivations and overall logical organization of the paper. The majority of reviewers found the experimental results -- and especially the ablations -- to convincingly support the effectiveness of LoRM. The extension of RegMean to merge the LoRA module is considered by most reviewers to be interesting and potentially beneficial for a variety of applications. The alternating optimization algorithm for LoRM shows quick convergence compared to the previous state-of-the-art, and the paper addresses a relevant research (Federated Class Incremental Learning) and demonstrates significant improvement over existing methods.

Reviewer rSBD raised a number of critical issues in their review, particularly regarding the clarity of the relationship between alternating and closed-form optimization, a lack of discussion of convergence checks and the overall validity of the alternating optimization approach, and questions about fairness of comparative evaluations. The author rebuttal provided additional details regarding convergence of the alternating optimization method and clarifications regarding performance on ImageNet-R which presents a very large domain shift compared to other benchmark datasets. The authors should include these details regarding convergence in the final version of the paper.

Reviewer BHoB criticizes the paper for its lack of experimental diversity, since it concentrates exclusively on image classification tasks. In rebuttal, the authors correctly observe that the vast majority of work on FCIL concentrates on image recognition and thus the paper aligns with the current state-of-the-art (and thus ensures broad comparability with existing methods). The authors also provide additional experimental results on text classification which demonstrate the effectiveness of LoRM on text backbones. It is the AC's opinion that the positive and novel aspects of the contribution outweigh this limitation and the decision is to Accept.

Decision: Accept (Poster)