Closed-Form Merging of Parameter-Efficient Modules for Federated Continual Learning
Authors: Riccardo Salami
Pietro Buzzega
Matteo Mosconi
Jacopo Bonato
Luigi Sabetta
Simone Calderara
Venue: ICLR 2025 Poster (2025)
Keywords: model merging
federated continual learning
federated learning
continual learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce LoRM, an alternating optimization strategy for merging low-rank adapted models, achieving state-of-the-art performance in Federated Class-Incremental Learning by effectively integrating multiple models into a unified system.

Abstract: Model merging has emerged as a crucial technique in Deep Learning, enabling the integration of multiple models into a unified system while preserving performance and scalability. In this respect, the compositional properties of low-rank adaptation techniques (e.g., LoRA) have proven beneficial, as simple averaging LoRA modules yields a single model that mostly integrates the capabilities of all individual modules. Building on LoRA, we take a step further by imposing that the merged model matches the responses of all learned modules. Solving this ob-
jective in closed form yields an indeterminate system with A and B as unknown variables, indicating the existence of infinitely many closed-form solutions. To address this challenge, we introduce LoRM, an alternating optimization strategy that trains one LoRA matrix at a time. This allows solving for each unknown variable individually, thus finding a unique solution. We apply our proposed methodology to Federated Class-Incremental Learning (FCIL), ensuring alignment of model responses both between clients and across tasks. Our method demonstrates state-of-the-art performance across a range of FCIL scenarios. The code to reproduce our experiments is available at github.com/aimagelab/fed-mammoth.

Published as a conference paper at ICLR 2025
CLOSED-FORM MERGING OF PARAMETER-EFFICEEN MODULES FOR FEDERATED CONTINUAL LEARNING
Riccardo Salami'
Pietro Buzzega'
Matteo Mosconi'
Jacopo Bonato?
Luigi Sabetta²
Simone Calderara'
A Almagelab University of Modena and Reggio Emilia, Modena, Italy name Surnamee@unimrre @unimore.
2 Leonardo Labs, Roma, Italy name surname. ext ext@leonardo.com
ABSTRACT
Model merging has emerged as crucial technique in Deep Learning, enabling the integration of multiple models into a unified system while preserving perfor- mance and scalability. In this respect, the compositional properties of low-rank adaptation techniques (e.g., LoRA) have proven beneficial, as simple averaging LoRA modules yields a single model that mostly integrates the capabilities of all individual modules. Building on LoRA, we take a step further by imposing that the merged model matches the responses of all learned modules. Solving this ob- jective in closed form yields an indeterminate system with A and B as unknown variables, indicating the existence of infinitely many closed-form solutions. To ad- dress this challenge, we introduce LoRM, an alternating optimization strategy that trains one LoRA matrix time. This allows solving for each unknown variable individually, thus finding a unique solution. We apply our proposed methodology Federated Class- Incremental Learning (FCIL), ensuring alignment of model responses both between clients and across tasks. Our method demonstrates state- of-the-art performance across a range of FCII, scenarios. The code to reproduce our experiments is available at ggthhub.ommkinnggllll//-aaaaamma
INTRODUCTION
Humans naturally excel at learning a diverse array of skills independently, effortlessly acquiring knowledge across multiple domains throughout their lives. contrast, the traditional paradigm for artificial neural networks relies training unified model proach facilitates the simultancous incorporation of different skills, lacks the capacity for special- ized or incremental learning, making it less adaptable and responsive changes in the environment. To overcome this limitation and mimic human flexibility, various paradigms have been developed to enhance neural networks' ability to manage diverse skills effectively. Multi-Task Learning (Caru-l ana 1997) involves training a model on several tasks simultaneously. promoting the sharing of representations across tasks, while Continual Learning (CL) (McCloskey & Cohen 1989) focuses on enabling models to learn tasks incrementally without forgetting prior knowledge. Federated Learning (FL) (McMahan et al. 2017). on the other hand, focuses on decentralized training by dis- tributing data and computation across separate clients, each specializing in their local task. While each of these scenarios has own unique characteristics, they all share the common objective of integrating task modules into unified framework.
single, large dataset. While this ap-
Recently, large pre-trained architectures have facilitated model editing (Ortiz-limenez et al. 2024) and specialization (Bowman et al., 2023), particularly for fine-tuning downstream tasks. In prac- tice, deep models often leave their parameters fixed, leveraging Parameter-Efffiient Fine-Tuning (PEFT) techniques to adapt to tasks effecttvely. Among PEFT methods, Low-Rank Adaptation (LoRA) (Hu et al. 2022) has emerged as a prominent approach. LoRA introduces residual weights in the form of AW BA,, where B and A are low-rank matrices. These residuals, commonly

---

Published as a conference paper at ICLR 2025
referred to as task vectors (Ihaacco et al. 2023), form the foundation of the novel model merg- ing literature, which has introduced various approaches for their integration. For example, Zhang et al. (2023c) explore the combination of task vectors through linear arithmetic operations, while other works focus on identifying optimal coefficients for weighting these modules during aggrega- tion (Yadav et al. 2024 Yang et al. 2024 WWet al. (024). In contrast to these methods, we propose a novel solution that merges LoRA modules in a closed-form.
To evaluate the feasibility of our approach, base our investigation on well-defined empiri- cal framework, situating our work at the intersection of Federated Learning and Continual Learn- ing. These two paradigms are ideal for assessing the merging of task vectors, as they encompass both spatial (across clients) and temporal (over tasks) aggregation. Specifically, in Federated Class- Incremental Learning (FCIL) (Yoon er al.) 2021], tasks are introduced incrementally. and data is distributed across multiple clients in decentralized manner.
We introduce novel approach, termed Low-rank Regression Mean (LoRM), tailored to the FCIL setting. Our method builds upon RegMean (Jin et al. 2023), model merging technique derived from an exactly solvable regression problem. Starting from RegMean's formulation, we develop a strategy to merge LoRA modules in closed form. Our derivations result in two key equations one for merging matrix A and another one for matrix B. During training, we propose an alternating optimization procedure, where one matrix is learned while the other remains fixed across models. In the Federated Inaremmnccelmttll Learning setting, spatial aggregation across clients is performed using this alternating procedure. Conversely, for temporal aggregation, task-specific modules merged by applying the RegMean formulation directly to the full residual weights A WW of all tasks.
In summary, the key contributions of this work follows:
We explore the feasibility of merging LoRA modules using a closed-form solution.
We introduce LoRM, a novel FCIL approach that leverages insights from our exploration.
We demonstrate the effectiveness our method across diverse datasets and varying degrees of data distribution, achieving state-of-the-an results.
2 BACKGROUND AND MOTIVATION
2.1 PRELIMINARIES
LoRA (Hu et al. 2022 introduced to reduce the number of trainable parameters when fine- tuning treitreined models. Formally, let Wo E dd represent the matrix of pre-trained weights of linear layer, and let kx1 be the input vector for that layer. The output h is given by:
Wa + AW = Wa + BAx,
(I)
where AW= = BA is the residual weight introduced by LoRA, with matrices A and B as the only components being trained. The efficiency of this approach stems from the low rank r of the matrices, where B E gdxr and A E gg with d and & representing the number of output and input features of the layer, respectively. Consequently, the number of trainable parameters is r- (d+k), which, since d, constitutes only fraction of the d k parameters required for full fine-tuning. Additionally, B is initialized to 0: Le., the first forward pass is equivalent to the absence a LoRA residual.
Reg.eaan (Jin et al. 2023) introduces method for merging collection of N linear layers (W) each corresponding to N distinct models trained on distinct inputs (X, ) The goal is to identify
a single linear layer that produces responses that closely match those of the starting layers. Specifi- cally, the objective function is defined as follows!
N
minimize Ω =
W W.X,
(2)
By computing the gradient of Ω with respect Wi and setting it equal zero, a closed-form solution is obtained. Notably, the merged layer WM is computed as follows:
WM M Σ W
Σ ExX]]
(3)
We rework the original formulation ease subsequent derivations.
2

---

Published as a conference paper at ICLR 2025
In this context, X, E samplee represents the input to the i-th layer being merged. Hence, the Gram matrix X,X; has dimensions features.
While RegMean finds weight matrix that approximates the outputs of all layers considered, it does not take into account starting from pre-trained weights or the use of low-rank modules (c.g., LoRA). Given these considerations, ask whether this method can be suitably adapted to merge LoRA modules. other words:
Can we devise strategy to merge LoRA modules in closed form?
To further explore this question from an experimental standpoint, we place our investigation in the context of a Federated
Learning scenario.
2.2 PROBLEM SETTING
In Federated Class-Incremental Learning, the dataset is first divided into T tasks, each consisting of a distinct set of classes. Then, each partition 'D' corresponding to the 1-th task is further dis- tributed among N clients, resulting in D for the i-th client. Similar to standard Plass-Incremente Learning (Van de Ven al. 2022), the task-specific partitions Jt=1..... arrive sequentially.
In this federated scenario, the training for each task is conducted over multiple communication rounds. During each round, clients are restricted to learning only from their local dataset DJ. The local optimization objective for client i, based on the loss function L, can be formally expressed as:
minimize (r,))) [<(s(;;; 0s), v)],
6,
where æ and y denote the inputs and corresponding labels, respectively, with 0, representing the set of parameters for client i, and f-;:0;) denoting the associated model.
After completing local updates, each client sends its model parameters 0, the central server, where they are aggregated with those from other clients. The server then sends the global aggregated model back to the clients, marking the end of communication round. This process repeats for several rounds until the training for task / is completed. Once all rounds for task / have finished, the system progresses to the next task f + using the corresponding dataset Di+1, The ultimate objective is to obtain a global model, derived from the aggregation of local models performed by the server, that functions well across all incremental tasks and successfully integrates the distributed knowledge.
METHODOLOGY
3.1 CLOSED-FORM SOLUTION FOR LORA MERGING
Using the same notation in Equation let W denote the weight matrix of given layer. Since LoRA (see Equation is applied to each linear layer across all clients, we express the weight matrix for the i-th client W. Wo B,A,, where W, is the shared pre-trained weight matrix, and B,, A₁ represent the low-rank matrices specific to the client. This formulation consistent across clients, as they all utilize the same model architecture. At the end of each communication round, after conducting local training on the LoRA matrices, the goal to merge the corresponding matri- (i.e., A's with A's ''s and B's with B's) using the closed-form solution derived from RegMean. Starting from Equation 2 our objective becomes:
N
Ω
Σ I(W + AM - (Wo + B,A,)
(5)
To find the optimal An and B that minimize Ω we differentiate Ω with respect to each variable, one at a time, and set the gradients to zero:
iΩ
aB 0
(6)
in
0A 0
3

---

Published as a conference paper at ICLR 2025
Round
Training
Frozen
Diagonal
Merged
B
A
BM
A
Share Byr
M
A
B2
A
Round j+1
B
A
B
A
X
B
A2
B
A M
Share AM
B
Av
A
AW
Figure 1: Training and aggregation procedure of LoRM, alternating hetween rounds. The figure depicts both Federated and Incremental aggregation for generic layer.
However, the system reveals indeterminate, the two equations exhibit linear dependence one another during the calculations. For further mathematical derivations illustrating the infeasibility of this approach, refer appendix A.1
As solution, we propose freezing one of the two matrices. This means that either A or B is shared across clients (i.e., treated constant), and single closed-form equation determines how merge the trainable matrices across clients. If we choose to share A, the merged B is obtained as:
N BM LBA.X. AT
N
A
X;X
A
i=
Note that A does not have subscript i, as it is identical for all clients. Instead, if we opt to share B, the merged Am computed as:
N Σ EAXXxx
)
N
-
AM
For the formal derivation of these equations, refer appendices I. and A.1.2
3.2 LORM
Having established closed-form solutions for weight merging, we now outline the full procedure of LoRM, illustrated in Figure [ Each client i begins by optimizing its own B,, which is learned during local training, and a shared A, initialized and distributed by the server. In this first round, essential to freeze A, as LoRA's B is initialized to O. Freezing B 0 across all clients would render the training ineffective. At the end of each round, each client i computes the Gram matrix X:X,T with forward pass all examples") Then, it sends B, and X, X,X; to the server, where the merging operation (as described in Equation 7 is performed. The resulting B is then sent back to the clients, who begin the next communication round.
Alternated optimization. Empirically, we observe that consistently training only the matrix B while keeping A fixed be limiting (see Section 4.3 detailed analysis). Therefore, fully exploit LoRA's representational potential, we introduce an alternating training approach, where the matrix be updated changes each round. Specifically, after the first round, we freeze B, which is already synchronized across clients due to the previous server aggregation, and train A instead. Then, at the end of the second round, local A's are aggregated using Equation g This strategy also improves
Note that this process provides Gram matrix for every layer in the network. In this discussion, X,X! is related to the single generic linear layer.
4

---

Send B to the server or Send A to the server end for Published as a conference paper at ICLR 2025
Algorithm LoRM, for generic linear layer
Round j
Round j
1: Input: T tasks; .N clients residual modules AW; = AB, layer's pre-trained weights Wo.
2: for each task E {l..... ] do
3: for each communication round do
4: Clients side:
5: for each client i E {1, .,, in parallel do
6: A' A or B' B for each epoch do for each input x do
Fix either matrix
7:
8:
9:
h Wa A/ or h Wo B' A
D Forward pass
10:
Optimize B or Optimize A!
11:
end for
12:
end for
13:
Send to the server diag( diag(X(X)) X
14:
15:
16:
Server side:
Use Equation and distribute Bis or Use Equation and distribute Air
18:
end for
end for
20:
Use Equation 9 to compute the final residual module
RegMean across tasks
efficiency, only one of the two matrices needs be communicated per round, offering significant advantage compared to transmitting the whole LoRA module.
Merge task-specific modules. At the conclusion of the generic task 1,, the merged matrices B and A hs are multiplied to obtain the task-specifie residual module: AW' B A'r- On the server side, when aggregating modules from all tasks at the end the training, we apply the standard RegMean formulation (Equation 3 to merge the aforementioned residuals
(
X ExX--)
AW
Σ
BM AM
To accomplish this, the server stores the task-specifir weight (AW) AW"> throughout the training process, along with the global Gram matrices {X'X* X ciative property of matrix multiplication, the computation of the latter is straightforward, achieved by summing the client-specific Gram matrices shared by the clients at the end of cach task X = EE' X is worth noting that, for what concerns the classification layer, con- catenating the task-specific classification heads is equivalent applying RegMean. We provide mathematical derivation of this result in appendix A.4
X'XIT). Due the asso-
The overall procedure yields final weight residual A WW that promotes both generalization across tasks and robustness to knowledge distribution. The pseudocode for LoRM, applied to a generic linear layer, presented in Algorithm !
3.3 LoRM CHARACTERISTICS FOR FEDERATED CLASS- LASSEINCCEMENAAL LEARNING
In Federated Class-Incremental Learning (FCIL), data privacy, efficiency, and the rate of conver- gence are crucial concerns. LoRM is specifically designed to address these challenges.
Privacy--reeervingg In FCIL, inputs X; for the i-th client's generic layer cannot be transmitted to the central server, as doing so, especially for the first layer, would compromise data privacy by effectively sharing the dataset. In LoRM, instead of transmitting X, directly, we send the Gram matrix X;X,". which obfuscates the original data. Furthermore, only the diagonal the Gram matrix is communicated, ensuring that local data is exposed.
5

---

Published as a conference paper at ICLR 2025
Efficiency. The use of LoRA inherently improves efficiency compared to full fine-tuning, as it ne- cessitates the communication of only two low-rank matrices for each layer. Additionally, LoRM's altermating optimization procedure permits the transmission of only one matrix per communication round, further reducing overhead. The use of solely the diagonal of the Gram matrix mitigates com- munication costs even further, it avoids the need to transmit the full matrix. As result, the conclusion of each communication round, each client transmits a low-rank matrix and a vector to the server for each layer.
Rate of convergence. Ultimately, LoRM demonstrates faster convergence compared to other FCIL baselines, as further discussed in Section 4.2
4 EXPERIMENTAL STUDY
4.1 EVALUATION SETTINGS
Vision datasets. The importance of evaluating pre-trained models on datasets that deviate signif- icantly from their pre-training domain is well-established in the literature (Kolesnikov et al., 2020) Korbblith et al., 2019). Accordingly, our experimental section features three in-domain datasets, two specialization datasets, and one out-of-domain dataset. For in-domain evaluation, we use CIFAR- 100 (Krizhessky et al. 2009), ImageNet-R (Hendryeks et al. 2021a), and ImageNet-A (Hendrycks) et al. (021b); to assess specialization within single category, we utilize Cars-196 (Krause et al.) 2013). and CUB-200 (Wah et al.) 201I). Finally, for out-of-domain evaluation, we employ Eu- roSAT (Helber et al. 2018), satellite dataset recognized by Oh et al. (2022) as one of the most challenging for domain adaptation from ImageNet-2IK pre-training. CIFAR-106 100, ImageNet-R, ImageNet-A, CUB-C00 200, and Cars- 196 are divided into 10 incremental tasks, while EuroSAT is split into 5 due to its fewer classes. Each task includes an equal share, except for Cars- 196. where the final task contains 16 classes. The data is distributed across 10 clients using the commonly adopted distribution-based label imhalance setting (Li et al.] 2022 Yurochkin et al. (019), where partition- ing is governed by a Dirichlet distribution parameterized by B. A smaller value corresponds to a more challenging data distribution. We evaluate all methods across three scenarios per dataset, using 3 E {0.5, 0.1, 0.05] for CIFAR-100 and ImageNet-R, and 3 E (1, 0.5, 0.2) for the others to account for fewer examples or classes. For further details on data pre-processing, including the sataset-specific augmentations used, please refer to appendix B
Language datasets. To validate our methodology in completely different domain, we also include Of-Soppp (OOS) (Larson et al. 2019), a textual intent-classification dataset comprising 150 classes. These classes are evenly distributed across 5 incremental tasks, with B E (1.0, 0.5, 0.2).
We compare LoRM against 10 competing methods spanning different fields. From Continual Learning, we evaluate EWC (Kirkpatrick et al. 2017). LwF (Li & Hoiem) 2017]. L2P (Wang et al.] 2022b), and CODA-Prompt (Smith et al. 2023). Following previous studies (Zhang et al.J 2023b] Guo et al. 2024). we adapt these methods to the federated domain by merging client weights using the FedAvg algorithm (McMahan et al. (017). We also include FisherAvg (Matena & Raffel 2022) and RegMean (Jin al. (023), from the model merging liter- ature, along with CCVR (Luo et al. (021) and FedProto Tan et al. (2022) from Federated Learn- ing. These approaches are adapted for Continual Learning by applying Asymmetric Cross- Entrooy (ACE) (Caccia et al.) 2022), a standard method that optimizes classification heads independently for each lask. Additionally, we evaluate two algorithms specifically designed for Federated Class- Incremental Learning: TARGET (Zhang et al. 2023b) and PILoRA (Guo et al. 2024). the latter rep- resenting the current State Of The Art. For OOS, we also evaluate TIES-Merginn from the model merging literature. Finally, we include Joint results as an upper bound, achieved by training the backbone on the entire dataset without any federated or incremental partitioning.
Evaluated approaches.
(Yadav al. 2024)
Implementation Details. As the backbone for LoRM and all competing approaches, we employ a ViT-B/16 model (Dosovitskiy et al.) 202I) pre-trained on ImageNet-21K (Ridnik et al. 202 on all vision datasets. We set the number of epochs per communication round to 5 for all datasets, and the total number of rounds to 5 for CIFAR- 100, ImageNet-R, EuroSAT, and CUB-200. Given the increased difficulty of ImageNet-A and Cars- 196, all methods are allowed 10 communication rounds on these datasets. For Of-opee we use pre-trained Tm-l (Raffel et al. 2020). For complete overview of the method-specific hyperparameters, refer to appendix

---

Published as a conference paper at ICLR 2025
Table Evaluation on CIFAR-100, ImageNet-R and ImageNet-A. Results terms FAA [tt] Best results are highlighted in bold, second-best underlined.
CIFAR-100
ImageNet-R
ImageNet-A
Joint
92.75
84.02
54.64
Distrib. ,3
0.5
0.1
0.05
0.5
0.1
0.05
1.0
0.5
0.2
EWC
78.46
72.42
64.51
58.93
48.15
43.68
10.86
10.07
8.89
LWF
62.87
55.56
47.09
54.03
41.02
46.07
8.89
8.89
7.90
FisherAVG
76.10
74.43
65.31
58.68
50.82
47.33
11.59
11.06
10.14
RegMean
59.80
45.88
39.08
61.18
57.00
55.80
8.56
6.22
4.34
CCVR
79.95
75.14
65.30
70.00
62.60
60.38
39.50
36.27
35.94
L2P
83.88
61.54
55.00
42.08
23.85
16.98
20.14
17.31
16.85
CODA-P
82.25
61.82
46.74
61.18
36.73
25.82
18.30
14.48
7.31
FedProto
75.79
70.02
60.55
58.52
47.30
52.93
9.87
9.22
10.01
TARGET
74.72
72.32
62.60
54.65
45.83
41.32
10.27
11.39
10.73
PILoRA
76.48
75.81
74.80
53.67
51.62
49.37
19.62
18.70
20.01
LoRM (ours)
86.95
81.75
82.76
72.48
63.83
66.45
37.26
36.34
33.11
4.2 RESULTS
In Tables T to present the results of the evaluated approaches in terms of Final Average Ac- curacy (FAA). For a formal definition of this metric, please refer to appendix B.7] The results are averaged over 3 runs with different seeds, with standard deviations provided in appendix C
In the first two in-domain datasets (Table Continual Learning techniques such as LwF and EWC exhibit low moderate performance in scenarios with limited data heterogeneity. Similarly, prompt- based CL approaches suffer from rapid performance degradation as the distribution parameter B decreases, whereas L2P maintains strong performance merging and FL methodologies generally underperform compared to CL approaches on CIFAR-100 but achieve better results on ImageNet-R. Among these, CCVR stands out as the top performer, attaining the second-best accuracy on the latter dataset. Despite being specifically designed for FCIL, TARGET and PILoRA demonstrate lower performance than the adapted CCVR, likely due to CCVR's s server-side calibration with centralized synthetic data. Lastly, leveraging its closed-form merging technique, LoRM achieves state-of the-art performance across all settings.
0.5, ranking second overall. Model
Due to the challenging adversarial nature of ImageNet-A, all approaches experience significant drop in this setting, with only CCVR, PILoRA, and LoRM maintaining competitive accuracy. Among them, CCVR and LoRM perform comparably average, achieving the best results.
On the EuroSAT dataset (Table methodologies of the same category exhibit varying performance. Among prompt-base based approaches, Coda- Coda-Prompt stands out the second-bes best performer, demon- strating strong generalization to out-of-domain settings, whereas L2P falls far behind. Similarly, within Continual Learning methods, EWC significantly outperforms LwF, suggesting that distilla- tion is less effective in this context. In general, Model Merging and Federated Learning methodolo- gies achieve comparable results, with CCVR showing a slight advantage within this group, while LoRM surpasses all other methods by substantial margin.
In the same table, models evaluated based their ability to specialize in specific domains (cars and birds). In these scenarios, CL and Model Merging techniques achieve comparable over- all performance, while FL methodologies demonstrate greater specialization capabilities. Notably, FCIL-tarrgeed approaches achieve the best results, they explicitly account for forgetting. CCVR, PRRA and LoRM emerge as the top-performing methods across both datasets. They achieve sim- ilar performance CUB-200, while LoRM outperforms the others by large margin on Cars-196
Table 3 presents the results on the Out-Of-Scope dataset. While CCVR often ranks second in the vision domain due to its server-side rebalancing strategy, it performs the worst in this setting, likely suffering from catastrophic forgetting. TIES- TIES-Merging surpasses CCVR, benefiting from its spe- cialized merging strategy, while the best-performing methods leverage closed-fome solution for merging linear layers. Among these, LoRM outperforms RegMean across all three distribution- imbalance settings by incorporating LoRA, achieving tate-of-theeard performance.

---

Published as a conference paper at ICLR 2025
Table Evaluation on EuroSAT, Cars- 19ts-196,,de and CUB-200. Results in terms of FAA [1]. Best results are highlighted in bold, second-best underlined.
EuroSAT
CARS-196
CUB-200
Joint
98.42
85.62
86.04
Distrib. ,3
1.0
0.5
0.2
1.0
0.5
0.2
1.0
0.5
0.2
EWC
64.12
59.30
56.52
19.55
18.02
18.29
31.46
29.60
27.89
LWF
31.91
21.26
31.42
20.84
22.72
31.76
25.25
21.11
18.54
FisherAVG
58.84
59.94
55.86
26.03
24.60
21.58
30.45
28.39
25.06
RegMean
48.74
51.73
45.27
21.83
20.36
15.92
35.57
32.84
32.83
CCVR
64.44
57.93
62.69
38.99
37.81
35.31
62.67
59.48
56.33
L2P
40.63
51.78
45.46
35.49
31.00
20.01
56.23
47.31
38.16
CODA-P
73.38
69.42
66.69
28.04
20.83
14.53
42.53
37.71
29.19
FedProto
58.79
62.85
64.17
26.08
24.55
22.75
30.22
28.27
26.01
TARGET
52.74
52.74
45.11
28.65
27.20
26.13
39.30
38.40
34.79
PILoRA
48.35
32.89
31.22
37.57
37.92
36.95
61.11
60.68
60.39
LoRM (ours)
84.23
77.26
81.36
54.41
51.87
48.81
64.60
63.67
60.06
Table 3: Evaluation on Out-of-Scope.
Table Distinct LoRM components.
Out-of-Scope
ImageNet-R
EuroSAT
Joint
95.53
Distrib. |3
0.5
1.0
Distrib. ,3
1.0
0.5
0.2
FedAvg
57.65
41.04
CCVR
39.18
38.13
30.36
FedAvg w/ LoRA
57.00
53.14
TIES
58.84
54.13
47.71
RegMean
61.18
48.74
RegMean
74.16
74.80
70.78
LoRM w/o Eq.
75.08
71.40
LoRM (ours)
84.78
82.58
77.67
LoRM
75.62
77.37
4.3 ABLATION STUDIES
Alternating vs. Only We investigate whether training the matrix only (i.e., following the same procedure LoRM but applying solely Equation 7 to merge B matrices) could lead to improved performance compared to our alternating optimization approach. We conduct an experiment on EuroSAT with 3 1.0 (see Figure 3. showing that the alternating strategy outperforms the only approach. This result underscores the importance of leveraging the entire LoRA representation capabilities to achieve superior performance.
Rate of convergence. Figure illlstraaes convergence rates of LoRM and the FCIL competitors on the first task of ImageNet-R with 8 = 0.05. The steeper slope of the curve for LoRM, compared to that of Target and PILoRA, showcases the faster convergence of our approach. This behavior stems from the closed-form solution applied for merging LoRA each communication round, which accelerates convergence by leveraging its optimality. In contrast, PILoRA applies simple averaging of LoRA modules, while Target does utilize PEFT modules at all. Additionally, the slower convergence of PILoRA compared to Target can be attributed to its prototype-based classification procedure, which necessitates higher number of training rounds achieve well-refined prototypes.
LoRM components. To evaluate the contribution of each component to the overall performance of LoRM, we conduct an ablation study by incrementally introducing each part and comparing it with baseline methods. These experiments are performed on EuroSAT with = 1.0 and ImageNet-R with 3 = 0.5. The initial baseline for both datasets FedAvg (MeMahan et al. 2001 7 combined with ACE (Caccia et al., 2022). Notably, leveraging LoRA leads to remarkable performance improve- ment on AT but does not affect ImageNet-R. Meanwhile, using the closed-fone aggregation of RegMean, enhanced with ACE, establishes an additional baseline. Building upon this, our alter- nating optimization strategy, when applied without Equation (i.e., simply averaging task- specifcc residuals), substantially boosts performance both datasets, with more pronounced impact the out-of-domain dataset. Finally, applying RegMean to merge task specifcc residuals further enhances robustness in addressing FCIL challenges, resulting in the final form of LoRM.
8

---

Published as a conference paper at ICLR 2025
Convergence rale
Alternating vs. Only B
100
100
Accuracy (%)
90
90
FAA (%)
85
85
LoRM
75
Target
75
Altemating
PILORA
Only
70
10
20
70
Communication rounds
Tasks
Figure 2: Speed of convergence.
Figure Alternating optimization.
5 RELATION WITH PRIOR WORKS
CRRAMETER-EFFICIRE FINE- FINN-GUNING
Adapting deep neural networks to new tasks typically requires full re-training of all parameters, Celumeter-fffiiien Fine-Tuning (PEFT) addresses this by
which is computationally demanding. updating only a small subset of the model parameters, leaving the rest of the network unchanged. One of the earliest methods in this domain is Adapters (Houlsby al. (019), which are lightweight neural modules inserted between the layers a pre-trained model to facilitate task adaptation. More recently, a prominent class of PEFT techniques includes Prompt Tuning (Lester et al. 202I Prefix Tuning (Li & Liang 202T). and Context Optimization (Zhou et al. 2022); they all introduce learn- able embeddings, or prompts, appended the layers' input tokens. Specifically, Prompt Tuning prepends these embeddings directly to the initial input sequence, whereas Prefix Tuning concate- nates them to specific attention layers, and context optimization learns textual prompts Vision- Language models. Recent adaptation for continual learning include I.2P (Wang et al.] 20226), CODA-Prompt (Smith et al., 2023), and CGIL (Frascaroli et al.) (024).. Alternatively, Low-Rand Adaptation (LoRA) (Hu et al. 2022) injects residual parameters into the pre-trained weights using low-rank matrices, significantly reducing the number of learnable parameters. While recent works have explored various alternatives to LoRA(Kopiczko et al.) 2024 Zhang et al., 2023d Renduch- intalla et al.] 2024 Liu et al., 2022), our research is grounded in the original LoRA framework, leveraging its efficiency and the extensive literature on model merging that goes with it.
MODEL MERGING
Task- specific modules are typically deployed to address individual tasks. However, an alternative approach involves merging these modules to create model capable of generalizing across all tasks. A notable line of research within model merging focuses on combining task vectors (Ilharco et al.) 2023), specifically those generated by LoRA-like methods. Early studics have explored the use of linear arithmetic operations (e.g., addition and subtraction) combine these modules (Zhang ct al. 2023c). Expanding on this, subsequent works have focused on optimizing the coefficients that determine the contribution of each LoRA module during aggregation (Huang et al. 2024 Yang et al.] 2024 (0024)aaaa Yadav et al. 2024) Wu et al. 2024). Alternatively, RegMean (Jin et al. (0023 introduces a closed-form solution for merging model weights, though its application LoRA module merging has not yet been explored. In this work, we present a novel methodology that leverages this closed- form solution, specifically tailored for the Federated Class- Insremencll Learning setting.
FEDERATED CLASS DAAREMENNLLL LEARNING
Federated Class-Incremental Learning (FCIL), introduced by Yoon et al. (2021). addresses the challenges inherent in Continual Learning, such as Catastrophic Forgetting (Robins 1995). and Federated Learning, such Client Drift (Zhao al. 2018, Karimireddy unified framework. Although the FCIL literature is still in its early stages, several studies have adapted popular Continual Learning methods the federated paradigm, often by employing the
al. 2020), within

---

Published as a conference paper at ICLR 2025
FedAvg (McMahan al. 2017) technique for aggregating client model weights, A few approaches,
however, are specifically designed for the FCIL setting. For instance, GLFC (Dong et al. 2022)
incorporates local buffers and regularization technique to mitigate the effects of gradient updates related to novel classes, with further refinements in a subsequent work by the same authors (Dogg al. b.23). TARGET (Zhang al. (023b) trains centralized GAN (Goodfellow al. (014)) to
populate replay buffers with synthetic samples. Fed-CPrompt (Bagwe et al. (023) combines local and global prompts to address both Continual Learning and Federated Learning simultaneously. Re- cently, HGP (Salami et al. (024) and PILoRA (Guo et al. 2024) have integrated PEFT techniques with prototypes to mitigate forgetting and client drift. While PILoRA combines LoRA with proto- types for classification, our proposed methodology leverages LoRA modules and aggregates them by aligning their output in both Continual and Federated scenarios in closed-form.
CONCLUSIONS
At the outset of our research, we explored the possibility closed-form solution for merging LoRA modules. Motivated by this question, we identified and validated the existence of such so- lution, demonstrating its practicality within the framework of Federated Class-Incrementer Learning (FCIL). We introduced LoRM, method specifically designed for the FCIL scenario, which sets new State of the Art in this domain.
Looking forward, aim broaden exploration to encompass wider array of PEFT modules. For modules such as VeRA (opiczko et al., 2024) and (IA)* (Liu et al. 2022), we have already derived closed-form merging equations, which are detailed in appendices and and A.3] respectively. Furthermore, given the generality of our derivations beyond the FCIL setting, we also intend to evaluate the robustness of our equations across various model merging scenarios. Ultimately, hope this work will serve not only the Federated Learning and Continual Learning communities but also contribute the growing body of research efficient module compositionally.
ACKNOWLEDGMENTS
This article is the result of collaboration with the partners of the STORE project (htps:://edf- store.com) funded by the European Defence Fund (EDF) under grant agreement EDF-2022. 101121405-STORE
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Gaurav Bagwe, Xiaoyong Yuan, Miao Pan, and Lan Zhang. Fed-cprompt:
Contrastive prompt for rehearsal-free federated continual learning. International Conference on Machine Learning. 2023.
Benjamin Bowman, Alessandro Achille, Luca Zancato, Matthew Trager, Pramuditha Perera, Gio- vanni Paolini, and Stefano Soatto. a-la-carte prompt tuning (apt): Combining distinct data via composable prompting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights reducing abrupt representation change in online continual learning. International Conference on Learning Representations, 2022.
Rich Caruana. Multitask learning. Machine Learning, 1997.
Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class massemnncreett learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022.
Jiahua Dong, Hongliu Li, Yang Cong, Gan Sun, Yulun Zhang, and Luc Van Gool. No one left behind: Real-world federated ctuss-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x 16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2021.
Emanuele Frascaroli, Aniello Panariello, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, and Simone Calderara. Clip with generative latent replay: strong baseline for incremental learning. British Machine Vision Conference, 2024.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- fion Processing Systems, 2014.
Haiyang Guo, Fei Zhu, Wenzhuo Liu, Xu- Yao Zhang, and Cheng-Lin Liu. Pilora: Prototype guided incremental lora for federated class-incremental learning. In Proceedings of the European Con- ference Computer Vision, 2024.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IEEE International Geoscience and Remote Sensing Symposium, 2018
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In IEEE International Conference Computer Vision, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVE conference on computer vision and puttern recogni- tion, pp. 15262-15271, 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. International Conference on Learning Representations, 2022.
11

---

Published as a conference paper at ICLR 2025
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang. Chao Du, and Min Lin. Lorahub: Effi- cient cross-task generalization via dynamic lora composition. Conference on Language Modeling, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations, 2023.
Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In IEEE International Conference Computer Vision, 2023.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka maabska-Barwinska al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Proceedings of the European Conference on Computer Vision, 2020.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In International Conference Learning Representations, 2024.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ane-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University Tront, 2009.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. An evaluation dataset for intent classification and out-of-scope prediction. Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing (EMNLP), 2019.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for eframeter-efffcient prompt tuning. In Conference on Empirical Methods in Natural Language Processing, 2021.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning non-iid data silos:
An experimental study. In IEEE International Conference on Data Engineering, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In International Joint Conference on Natural Language Processing, 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot efffmeter-efficient tnn-ttning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2025
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 2021.
Michael S Matena and Colin A Raffel. Merging models with wisher-weighted averaging. Advances in Neural Information Processing Systems, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, 1989.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--fficceent learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics, 2017.
Martin Menabue, Emanuele Frascaroli, Matteo Boschini, Enver Sangineto, Lorenzo Bonicelli, An- gelo Porrello, and Simone Calderara. Semantic residual prompts for continual learning. In Pro- ceedings of the European Conference Computer Vision 2024.
Jaehoon Oh, Sungnyun Kim, Namgyu Ho, Jin-Hwa Kim, Hwanjun Song, and Se- Young Yun. Un- derstanding cross-domain few-shot learning based on domain similarity and few-shot difficulty. Advances in Neural Information Processing Systems, 2022.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Sys- tems, 2024.
Angelo Porrello, Lorenzo Bonicelli, Pietro Buzzega, Monica Millunzi, Simone Calderara, and Rita Cucchiara. second-order perspective on compositionality and incremental learning. Interna- tional Conference on Learning Representations, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 67, 2020.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhancing parameter effi- ciency of lora with weight tying. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2024.
Tal Ridnik, Emanuel Ben- Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor Imagenet-21k pretraining for the masses. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 1995.
Riccardo Salami, Pietro Buzzega, Matteo Mosconi, Mattia Verasani, and Simone Calderara. Federated class-incremental learning with hierarchical generative prototypes. arXiv preprint arXiv:2406.02447, 2024.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention- based prompting for rehearsal-free continual learning. In Proceedings of the JEEE Conference on Computer Vision and Pattern Recognition, 2023.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang, Fed- proto: Federated prototype learning across heterogeneous clients. In AAAI Conference on Artifi- cial Intelligence, 2022.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas Tolias. Three types of incremental learning. Nature Machine Intelligence, 2022.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. Caltech-ucsd birds-200-2011. Techni- cal Report CNS-TR-201 1-001, California Institute of Technology, 2011.
13

---

Published as a conference paper at ICLR 2025
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Proceedings of the European Conference Computer Vi- sion, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022b.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. In International Conference Learning Reprexentations, 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 2024.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024.
Jachong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated contin- ual learning with weighted inter-elient transfer. International Conference on Machine Learn- ing, 2021.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Interna- tional Conference on Machine Learning, 2019.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on pre-trained model. In IEEE International Conference on Computer Vision, 2023a.
Jie Zhang. Chen Chen, Weiming Zhuang, and Lingjuan Lyu. Target: Federated class-continull learning via exemplar-free distillation. In IEEE International Conference on Computer Vision, 2023b.
Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arith- metic operation. Advances in Neural Information Processing Systems, 2023c.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efffiient fine-tuning. In International Con- ference on Learning Representations, 2023d.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint 10X.:21806.00582, 2018.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, (30(9))2337-2344 2022.
14

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL OVERVIEW
This supplementary document provides detailed mathematical derivations and additional informa- tion complement the main text. The contents are organized follows:
Section A
Mathematical Derivations: presents the mathematical derivations related the Fine-Tuning Tuning methods with RegMean.
combination of different Parameter-Efficien Fine-
Section A. Combination of LoRA and RegMean: derives the optimization problem when integrating Low-Rank Adaptation (LoRA) with RegMean.
Section A. Solving for by Fixing details the solution for matrix A when matrix B is held constant.
* Section A.7.2 Solving for B by Fixing A: details the solution for matrix B when matrix A is held constant.
Secciin A.2
Combination of VeRA and RegMean: extends the derivations to the Vector-based Random matrix Adaptation (VeRA) method.
explores how the (IA)a method can be combined with RegMean and provides the corresponding derivations.
Section A.3
Combination of (IA)² and RegMean:
Seccion A.4 RegMean Applied to Classification Heads: discusses the application of Mea to classification heads in a class-incremental learning setting.
Details Datasets and Metrics: offers comprehensive details about the datasets used and the evaluation metric employed.
Section
Section
Standard Deviations: presents the standard deviations for all experimental
results.
Section
Hyperparameters: details the hyperparameters used across all experiments.
15

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL DERIVATIONS
A.1 COMBINATION OF LORA AND REGMEAN
We consider a minimization problem over a single linear layer where X; represents the input the layer. The objective function is defined as:
minimize = Σ WX W.
(10)
Here, W and W; the weight matrices for the global model (server) and the local models (clients) respectively, while N represents the number of clients. In the context of a LoRA residual module, we replace W with Wo+ BA, where Wo denotes the pre-trained weight matrix and A represents the learned low-rank difference. The objective function becomes:
Ω = Σ IIW + BA)X. (Wo + B.A.)
Expanding the previous expression yields:
Ω Σ I,,; WoX, BAX, BAA,
Which simplifies the equation to:
N Ω Σ
BAX, BA
It is important to note that B and A refer to the merged LoRA matrices (BM and AM in the main paper). For simplicity, we omit the subscript To minimize this objective function, we compute the partial derivatives with respect the unknown matrices A and B, setting them equal to zero This leads to the following system of equations:
8N
N
=0
B (BAX, - BAA, =0
0A
ON
=
(AA, - BAX)X =
=0
0B
1=1
We can rearrange the equations as:
B BAXXX Σ B BAXX
BAX A Σ BAXXAA A
==1
Next, multiply the second equation the left by BT, in order align it with the structure of the first equation:
N ABAAAAAA = E B
By substituting the left-hand side of the first equation into the second, we obtain:
Σ B.A...A EB T 0.
1=1
This simplifies to = indicating that the system is indeterminate, with infinitely many possible solutions.
16

---

Published as a conference paper at ICLR 2025
A.1.1 SOLVING FOR A BY FIXING B
Assuming the matrix B is fixed, we can solve for A. The objective function becomes:
ΩA Σ BAX: BA
To minimize ΩA. we its derivative with respect A zero:
02
=0 0 7
Σ 2B (BAX: - X =0,
0A
which simplifies to:
N Σ
BTB - AXX,)X = 0.
Since BA is designed to be low-rank, the matrix B has more rows than columns. As result, provided that no row or column of B is zero, the inverse (BB)) B) exists. By multiplying both sides of the equation from the left by (BT)) we obtain:
N Σ - AXX, =0 =0, A
N
N X = Σ A.
1=1
Finally, solving for A, we get:
A EAXX!
(ExX!)
(
A.1.2 SOLVING FOR FIXING
When the matrix A is fixed, we can solve for B. The objective function Ω becomes:
QB = Σ [BAX B,AX
To minimize Qn, we set its derivative with respect to B equal to zero:
0Bg aB
0
Σ
2(AX,, X A 0.
Expanding the equation, we get:
N Σ /
Σ BAXXXA AT =0,
1
which simplifies to:
N B Σ AXXX = Σ AT
1=1
Finally, solving for B, we obtain:
B B
N T Σ [AXXX[A') (',
17

---

Published as a conference paper at ICLR 2025
A.2 COMBINATION OF VERA AND REGMEAN
Following appendix A.1 we consider the same minimization problem as Equation 10 In the context of VeRA residual module, replace W with W, A,,,A, where Wo denotes the pre-trained weight matrix and A, A represents the learned weight difference. More precisely, B and A are randomly initialized, while As and Ad are learned. The objective function becomes:
Ω = Σ 11W A,BA,A)X; (Wo + BA,,
While every component of LoRA is full matrix, here considering A, and As, which diagonal matrices. In order to enforce this constraint into the previous formulation, we can rewrite them column vectors \ and Ad:
Ω Σ ([ ((^91) @ B) ((*11)) A)) x; (Wo ((116) B) ((1/1.) @ A))
where 1, and 1d are two row vectors of ones with the same columns as B and A, respectively. Please note that in our notation, indices of column vectors are denoted as superscripts, while those of row vectors are represented as subscripts. Expanding this objective, we can write:
W.X, W,X, + ((>*1) @ B) ((1*1a) @ A) x. ((291.) @ B) ((A o A.
Which simplifies the equation to:
Ω= Σ 1((A11)) o ((1 O) ((A/1a) o B) ((Aia)) o A) x
If we compute the partial derivatives with respect to the unknown vectors A and A and set them equal to zero, we obtain an indeterminate system, as similarly discussed in appendix A. The detailed derivation is left to the reader. However, it is still possible to solve for either X² or X when the other is known.
A.2.1 SOLVING FOR Xd WITH FIXED X
Assuming the vector X is fixed, we can solve for Xd. The objective function S becomes:
22a= = Σ (1(11.)) @ B) ((111)) (() A) ((>11) o B) ((1/1)) o A) x|
This can be rearranged as:
N 224 Σ [(((*1))) @ B) [({111)) OA)x, ((2/1a) o A) x.]11²
Since (1*1b) @ B is a multiplicative constant, it can factored out and disregarded, leading to the following simplified objective:
N
Σ
((1(1)) o A) x
To minimize Sly,, we take its derivative with respect to A and set it to zero:
02
N
0
((111)) A O A Σ
((11)) o 4)XX & AAT = 0.
axd
This simplifies to:
N
N
(1111)) o A) Σ x,X = Σ
(11122) o A
18

---

Published as a conference paper at ICLR 2025
We then solve for X follows:
((111a) の A)
Σ O
Σ
1d1a
Σ
o)xx,,
Σxx
1
A
x x111
Σ
(1/1)) |
Σxxx
1
Since 1o1] = d a scalar, we can finally find:
N
N )
Σ (($1a) x)Xx7)
1 dA
1
A.2.2 SOLVING FOR X WITH FIXED A
When the vector 2d is fixed, solve for Ah. The objective function ! becomes:
2r ((( B) ((A*1)) o A) ((A/1)) 3 B) ((^*1)) A) x:
To minimize Ωxxs, we compute its derivative with respect to X and set to zero:
N
02g = 0 2 > (()))) O ()((A @ A)X,X.
0x
1(1)1)) o B)((1))))) A)X.X (A11a) 1 T B1, = 0.
This simplifies to:
((11)) O B Σ
AX.X,
E ((11) o B)AX,X, AT,
where we use A denote ((1*1a) + A) for simplicity. Solving for X we get:
N
(111b) B)
(1)1)) O X
Σ AX.XTA AT
A
111,~
Σ
(()11)) o BAAXXXTA")
AXXXAA
AT
N AXAXXTATT)
11
Σ (0)))) BTAX,XTAT)
Finally, since 1,1] = d a scalar, we can solve for Ah:
Σ ((1L))) BIAX.XTAT) AT) (ΣAXXXTA))
N
-1
3 dB
I
where = ((1912) @ A).
19

---

Published as a conference paper at ICLR 2025
A.3 COMBINATION OF (IA)² AND REGMEAN
(IA)s (Liu et al. 2022) adapts the pretrained model by learning multiplicative weight vectors & that modulate the intermediate activations where is the dimension of the considered activation. Specifically, for each attention layer, these weight vectors are applied to the input keys, values, and output. Notably, can be demonstrated that, for a single linear layer, (IA)a can be expressed as residual module (Porrello et al. 2024]
X) = [Wo + ((e* - - 11) O W x,
where 1 and 1 represent row vectors of ones, respectively corresponding to the dimensions of the output and input of the considered layer. For the sake of simplicity, we deviate from the original formulation, which initializes & as column vector of ones. Instead, we begin with a column vector of zeroes at the beginning of training. This adjustment does not affect the outcome, simultaneously modify our formulation as follows:
+ (WoX) = [Wo + (6 O Wo] X,
Then, following appendix A.1 we consider the same minimization problem as Equation 10 This time, we replace W with Wo + (E*1,) @ Wo. where Wo denotes the pre-trained weight matrix and 1,) @ Wo represents the learned weight difference and eh is zero initialized. The objective function becomes:
N
aax =
ll + 11) @ Wo) - (Wo + (1) @ 3.
Following appendix A.I we can rearrange C2<rA)" as:
a Σ
( ((f+1n) o Wo)
To minimize R(Iass we take derivative with respect to & and set zero:
( - Wol, 0.
Oek
This simplifies to:
(111))) Σ X Σ
((1))) Wo).,x
1=1
Solving for & we get:
N
((1* Wo) Σ (C11) (
Σ X,X,
e*1,
(C1)) Wo)
Σ
XiX!
1
W
e 1 (
Σ X.X]
1
W D
Finally, since 1 TT = h is scalar, we can solve for th:
Σ ((1)))) o.)x,. )
N
N
-1
Σ X.X!
1
hWo
20

---

Published as a conference paper at ICLR 2025
A.4 REGMEAN APPLIED TO CLASSIFICATION HEADS
While each client adapts the pretrained model to the specific task using residual parameter-efficient modules, the classification head must be re-initialized and trained from scratch, Specifically, follow- ing the approach CffCiaa et al. (2022), practice has recently emerged as the de facto standard in providing performance boost across various methodologies.
initialize a distinct classification head for each task. This Learning, consistently
In the context of Inaseeancremettl Learning, all task-specific classification heads must eventually be merged into unified classifier. To address this, aim minimize the RegMean problem, as defined in Equation [0] across the classifiers of different tasks. Rather than considering a separate set of parameters for each client,
introduce weight matrix for each task:
minimize Pets = Σ WXX - W.X.
Notably, in this case, W and W. differ in dimensionality, as the classification heads (W1, W....., } are trained independently. Specifically, while W and each W, share the same number of columns (input dimension), they differ the number of rows (output dimension). The total number of rows of W corresponds to the sum of the rows of all W,, which is equivalent to the total number of classes C; instead, each W, has c rows. This structure enables us to decompose Pat into 7 separate minimization problems:
T
minimize
Σ
P
where
|| (eete(1+)) X, W,X.
Since the tasks are class-disjoint, each (S sequently, the optimal solution for each 2 is to assign W (2r((+1)) W, for each t.
2 operates on distinct rows of W. Con-
B DETAILS ON DATASETS AND METRICS
B. CIFAR-100
For the incremental split of CIFAR-100, we follow the standard partitioning into 10 tasks, each consisting of 10 classes, as commonly employed in numerous Continual Learning studies (Wang the training set, apply bicubic
et al.) 2022b at Smith al. 2023). For data augmentation interpolation to resize each image from 32 x 32 to 224 x 224, followed by random horizontal flipping and normalization. For the test set, each image is first resized to 256 x 256 using bicubic interpolation, followed by center crop to 224 x 224, and a final normalization step.
B.2 IMAGENET-R
For the incremental split of ImageNet-R, we employ the widely used partitioning into 10 tasks, each comprising 20 classes, as is common in various Continual Learning studies (Wang et al. 2022a Smith et 2023 Zhang et al., 2023a). For data augmentation on the training set, we process each 224 x 224 image with a random horizontal flip, followed by normalization. In the test set, we first resize each image 256 x 256 using bicubic interpolation, apply a center crop to 224 x 224. and then perform normalization.
B.3 IMAGENET-A
For the incremental split of ImageNet-A, we partition the dataset into tasks, each containing an equal number of classes. During training, augmentation includes random resized cropping to 224 x 224 with a scale range of (0.05, 1.0) and an aspect ratio range of (3/4, 4/3), followed by random horizontal flipping. Normalization is performed using mean of (0.485, 0.456, 0.406) and a standard deviation of (0.229, 0.224, 0.225). The test set undergoes resizing to 256 256 using bicubic interpolation, followed by center crop 224 224 and the same normalization step.
21

---

Published as a conference paper at ICLR 2025
B.4 EUROSAT
For the incremental split of EuroSAT, we follow a partitioning scheme of tasks, each consisting of 2 classes, consistent with prior Continual Learning studies (Jung et al. [023] Menabue al. 2024).
No additional data augmentation or pre-processing is applied to either the training or test sets.
B.5 CARS-196
For the incremental split of Cars-196, we divide the dataset into 10 tasks, each comprising an equal number of classes, except for the last task, which contains 16 classes. Data augmentation for the training set includes random horizontal flipping, followed by normalization using mean of 0 and standard deviation of 1 for all channels. The test set undergoes only the normalization step.
B.6 CUB-200
For the incremental split of CUB-200, we follow a partitioning into 10 tasks, each containing an equal number of classes. Data augmentation for the training set includes bicubic interpolation resiz- ing to 256 x 256, random cropping to 224 x 224, and random horizontal flipping. Normalization is applied using mean of (0.485, 0.456, 0.406) and standard deviation of (0.229, 0.224, 0.225). For the test set, each image is resized to 256 x 256 using bicubic interpolation, followed by a center crop to 224 x 224 and the same normalization step.
B.7 FINAL AVERAGE ACCURACY (FAA)
We evaluate the performance of all methods using the Final Average Accuracy (FAA), widely adopted metric in the FCIL literature.
FAA quantifies the mean accuracy across all tasks at the conclusion of the incremental training process. Formally, let a' represent the accuracy on the zth task after completing the incremental training. The FAA is defined as:
1 FAA
Σ a'
(11)
T
where T' represents the total number tasks.
C STANDARD DEVIATIONS
The standard deviations for all evaluated approaches, based on three runs, are reported for the in- domain datasets in Table A and for the out-of-domain dataset in Table B
Table A: Standard deviations (reported in FAA) for CIFAR-100, ImaggeNee,mmaeeett..oe and ImageNet-A.
CIFA R-100
ImageNet-B R
ImageNet-A
Distrib. |3
0.5
0.1
0.05
0.5
0.1
0.05
1.0
0.5
0.2
EWC
1.38
2.32
2.67
1.69
1.14
1.14
1.34
0.75
1.38
LwF
1.15
2.38
4.25
1.91
0.93
1.38
0.30
0.27
1.61
FisherAVG
0.40
3.67
3.18
0.77
0.47
0.93
1.59
1.36
1.49
RegMean
0.01
1.78
4.87
1.10
0.85
1.44
0.59
1.10
0.70
CCVR
1.13
2.49
1.39
0.57
0.49
2.29
1.19
0.63
2.26
L2P
1.62
1.49
1.85
1.17
1.61
1.35
0.72
0.81
2.01
CODA-P
1.15
1.96
0.96
0.98
1.95
1.19
0.16
1.63
0.22
FedProto
2.05
0.94
1.96
2.09
2.28
0.72
1.18
0.71
2.17
TARGET
0.79
1.02
1.81
0.33
1.58
1.21
0.65
0.46
1.34
PILORA
0.28
0.98
4.08
0.29
1.01
0.33
0.08
0.31
0.33
LoRM (ours)
0.27
0.20
0.79
0.04
0.16
0.87
0.86
0.75
0.68
22

---

Published as a conference paper at ICLR 2025
Table B: Standard deviations (reported in FAA) for EuroSAT, Cars-196, and CUB-200.
EuroSAT
Cars- 196
CUB-200
Distrib, |
1.0
0.5
0.2
1.0
0.5
0.2
1.0
0.5
0.2
EWC
7.33
5.78
6.60
1.72
0.46
1.00
0.55
0.94
1.68
LwF
3.32
4.58
5.12
1.35
2.81
2.01
2.07
2.15
2.02
FisherAVG
5.43
6.07
3.40
2.10
1.78
0.78
0.26
1.35
2.21
RegMean
3.99
7.21
5.68
0.23
1.11
1.80
1.79
2.63
2.27
CCVR
9.01
7.16
6.26
1.49
0.87
2.08
1.22
1.69
1.73
L2P
2.52
3.49
2.02
0.87
1.94
0.36
2.38
0.78
1.21
CODA-P
4.52
6.27
6.54
1.05
2.24
1.89
0.51
1.52
1.86
FedProto
3.58
8.74
7.96
1.11
0.35
0.87
0.96
0.67
1.99
TARGET
4.12
6.20
5.31
0.93
0.68
1.54
1.17
0.65
2.06
PILoRA
4.45
4.07
4.27
0.21
0.33
0.18
0.51
0.28
0.51
LoRM (ours)
1.75
3.34
6.51
1.07
0.26
0.92
0.86
0.98
0.44
Table C: Standard deviations (reported in FAA) for Out-of-Scope.
Joint
93.72
Distrib. /
1.0
0.5
0.2
CCVR
0.45
0.36
0.92
TIES
2.75
1.80
1.21
RegMean
0.44
1.30
1.46
LoRM (ours)
84.78
82.58
77.67
D HYPERPARAMETERS
For Method-sppecitt hyperparameters that are not explicitly mentioned, follow the settings de- scribed in their original papers.
D. ACRONYMS
List of all the acronyms used:
Ir: learning rate;
Ir learning rate for prototypes;
AKl. Knowledge Distillation Loss multiplier;
r: rank for low rank matrices;
8ep epochs of training and generation for the generator network:
7: decay factor for off off-diagonal elements of the Gram matrices of the backbone (first elc- ment) and of the classifier (second element).
23

---

Published as a conference paper at ICLR 2025
D.2 CIFAR-100
Distrib.
B
0.5
0.1
0.05
EwC
Ir: le-5
Ir: le-5
Ir: 1e-5
LwF
Ir: le-5
Ir: le-5
Ir: le-5
FisherAVG
Ir: le-5
Ir: le-5
Ir: le-5
RegMean
Ir: le-5; y: (0.5, 0.5)
Ir: le-5; (0.5, 0.5)
Ir: le-5; (0.5, 0.5)
CCVR
Ir: le-5
Ir: le-5
Ir: le-5
L2P
Ir: 3e-2
Ir: 3e-2
Ir: 3e-2
CODA-P
Ir: le-3
Ir: le-3
Ir: 1e-3
FedProto
Ir. le-5
Ir: le-5
Ir: le-5
TARGET
Ir. le-5; AKI. 25: Kep
le-5; AKI.: 25; Bey 30
Ir: 1e-5; AKI.: 25; 8ep 30
PILoRA
Ir. 2e-2; prp: le-4
Ir: 2e-2; pr: le-4
Ir: 2e-2; pr le-4
LoRM
Ir: 3e-4; 7: (0, 0.5)
Ir: 1e-4; 16 y: (0, 0.5)
Ir: 5c-4; r. 16 y: (0, 0.5)
D.3 IMAGENET-R
Distrib. |
0.5
0.1
0.05
EwC
Ir: le-5
le-5
Ir: le-5
LwF
Ir: le-5
Ir: le-5
Ir: 3e-5
FisherAVG
Ir: le-5
Ir: le-5
Ir: le-5
RegMean
Ir: le 0.11
Ir: 1e-5; (0.1, 0.1)
Ir: le-5; (0.1, 0.1)
CCVR
Ir: le-5
Ir: le-5
Ir: le-5
L2P
Ir: 3e-2
Ir: 3e-2
Ir: 3e-2
CODA-P
Ir: le-3
Ir: le-3
Ir: le-3
FedProto
Ir: le-5
Ir: le-5
Ir: 3e-5
TARGET
Ir. le-5; AKL: 25: Sep 30
Ir: le-5; AKL: 25; Sep; 30
Ir: le-5; AKL: 25; 8epi 30
PILoRA
Ir. 2c-2; lrpr: le-4
Ir: 2c-2; lrpr: le-4
Ir: 2c-2; Irpr: lc-4
LoRM
Ir: 3e-3; r: 7: (0, 0.5)
Ir: 1e-3: r: 32 7: (0, 0.5)
Ir: le-3; r: 16 7: (0, 0.5)
D.4 IMAGENET-A
Distrib.
B 1.0
0.5
0.2
EwC
Ir: le-5
Ir: le-5
Ir: le-5
LwF
Ir: le-5
le-5
Ir: 3e-5
FisherAVG
Ir: le-5
Ir: le-5
Ir: le-5
RegMean
le-5;
7: (0.1,
le-5: (.11
Ir: le-5; n: (0.1, 0.1)
CCVR
Ir: le-5
Ir: le-5
Ir: le-5
L2P
Ir: 3c-2
Ir: 3e-2
Ir: 3e-1
CODA-P
Ir: le-2
Ir: 1e-2
Ir: le-2
FedProto
Ir: 3e-5
Ir: le-5
Ir: le-5
TARGET
Ir: le-4; AKI. 25; Kep' 30
Ir: le-4; AKI.: 25; Sepi 30
Ir: le-4; ARI.: 25; Sep 30
PILoRA
Ir. 2e-2; Irpr le-4
Ir: 1e-2;
Irpr: le-4
Ir: 2e-2; Irpr: 1e44
LoRM
Ir: 1e-2; r: 4 (0, 0.5)
Ir: 1e 2; (0, 0.5)
Ir: le-2; 2; 4 7: (0, 0.5)
24

---

Published as a conference paper at ICLR 2025
D.5 EUROSAT
Distrib.
B 1.0
0.5
0.2
EwC
Ir: le-5
Ir: le-5
Ir: 1e-5
LwF
Ir: le-5
Ir: le-5
Ir: 3e-5
FisherAVG
Ir: le-5
Ir: le-5
Ir: le-5
RegMean
Ir: le-5; y: 0.1)
Ir: le-5: (0.1, 0.1)
Ir: le-5; (0.1, 0.1)
CCVR
Ir: le-5
Ir: le-5
Ir: le-5
L2P
Ir: 3e-2
Ir: 3e-2
Ir: 3e-2
CODA-P
Ir: le-3
Ir: le-3
Ir: 1e-3
FedProto
Ir. 3e-5
Ir: le-5
Ir: le-5
TARGET
Ir. le-5; >KI: 25; Kep
le-5; AKI.: 25; Bey' 30
Ir: 1e-5; AKI.: 25; 8ep 30
PILoRA
Ir. 2e-2; Ippr: le-4
Ir: 2e-2; lrpt 1e-4
Ir: 2e-2; pr 1e-4
LoRM
Ir: 3e-3; (0, 0.5)
Ir: 3e-3; 7: 0.5)
Ir: 1e-3; r: y: (0, 0.5)
D.6 CARS-196
Distrib. B 1.0
0.5
0.2
EwC
Ir: le-5
le-5
Ir: le-5
LwF
Ir: le-5
Ir: le-5
Ir: 3e-5
FisherAVG
Ir: le-5
Ir: le-5
Ir: le-5
RegMean
Ir: le-5; 0.11
Ir: 1e-5; y: (0.1, 0.1)
Ir: le-5; (0.1, 0.1)
CCVR
Ir: le-5
Ir: le-5
Ir: le-5
L2P
Ir: 3e-2
Ir: 3e-2
Ir: 3e-2
CODA-P
Ir: 3e-2
Ir: 3e-2
Ir: 3e-2
FedProto
Ir: le-5
Ir: le-5
Ir: le-5
TARGET
Ir: le-4; AKL: 25: Sep 30
Ir: le-4; AKL: 25; Sep 30
Ir: le-4; AKL 25; 8epi 30
PILoRA
Ir: 1e-1; Irpr le-4
Ir: 1e-1; Irpr: lc-4
Ir: le-1; Irpr: lc-4
LoRM
Ir: le-2; r: 8 7: (0, 0.5)
Ir: 1e-2; r: 8 7: (0, 0.5)
Ir: 1e-2; r: 4 7: (0, 0.5)
D.7 CUB-200
Distrib. B 1.0
0.5
0.2
EwC
Ir: le-5
Ir: le-5
Ir: le-5
LwF
Ir: le-5
le-5
Ir: 3e-5
FisherAVG
Ir: le-5
Ir: le-5
Ir: le-5
RegMean
le-5; 0.11
le-5: (0.1, 0.1)
Ir: le-5; n: (0.1, 0.1)
CCVR
Ir: le-5
Ir: le-5
Ir: le-5
L2P
Ir: 3c-1
Ir: 3e-1
Ir: 3e-1
CODA-P
Ir: le-3
Ir. le-3
Ir: 1e-3
FedProto
Ir: le-5
Ir: le-5
Ir: le-5
TARGET
Ir: le-4; ÀKI. 25; Kep' 30
Ir: le-4; AKI.: 25; Sep 30
Ir: le-4; ARI.: 25; Sep 30
PILoRA
Ir: 1; Irpr le-4
Ir: 1; Irpri le-4
Ir: 1; Irpr: le-4
LoRM
Ir: 1e-2; r: (0, 0.3)
Ir: 3e-2; 7: (0, 0.3)
Ir: 3e-2; 72 (0, 0.3)
25

---

Published as a conference paper at ICLR 2025
D.8 OUT-OF-SCOPE
Distrib.
1.0
0.5
0.2
RegMean
Ir: 3e-3; y: (0.5, 0.5)
Ir: 3e-3; y: (0.5, 0.5)
Ir: 3e-3; 7: (0.5, 0.5)
CCVR
Ir: 3e-4
Ir: 3e-4
Ir: 1e-3
Ties Mies-Merging
Ir: le-3
Ir: 1e-3
Ir: 1e-3
LoRM
Ir: le-2; r: 2 y/ 0.5)
Ir: le-2; r: 1y (0, 0.5)
Ir: 1e-2; r: y (0, 0.1)
26


Reviews:
[{"review_id": "8SAOOXr5bB", "reviewer": "Reviewer_rSBD", "summary": "The paper presents a novel optimization method for Federated Class-incremental Learning (FCIL). Based on the LoRA approaches, the authors propose a optimization method to estimate the two low-rank matrices effiently. Rather than the previous approaches upon the various types of assumptions, the proposed algorithm employed the alternating optimization methods by optimizing the two low-rank matrices separately by fixing the other one. The optimization methods was evaluated in the scenario of FCIL, which shows the state-of-the-art performance for CIFAR-100 and ImageNet-R dataset even with the various evalutation settings.", "questions": "The validity of the optimization solver should be checked.\nThe lack of evaluation metrics should be supplemented to show the effectiveness of the proposed algorithm.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The target problem and the contribution are well presented in the paper, and the following mathetmatical derivation is also easy to follow. The experimental results verify the effectiveness of the proposed algorithm, and futher ablation studies are well designed to prove the component-wise effectiveness and the mentioned contributions.", "weaknesses": "Even with the above mentioned strengths, I have several concerns about the technical algorithm and evaluations as follows:\n\n- Contribution (Alternative optimization vs. closed-form):\n The authors emphasized the contribution of this paper as the alternating optimization and the closed-form optimization. However, the relationship between the two different optimization methods is not precisely described. As I understood, the closed-form optimization is performed at each iteration for the respective low-rank matices, and the overall optimization solver follows the mechanism of alternating optimization. Since the two optimization approaches are definitely different to each other, the authors should specify the relationship of the two contributions.\n\n- Convergence check for the optimization solver:\n When we design the optimization solver, it is important to check its convergence. When the optimization fails to convergence, the solution can be diverged, which causes serious problems. If the convergence check fails, the strategy to stop the iteration should be given. In the paper, both the convergence check and the stop criterion are not described.\n\n- Validity of alternating optimization:\n The approach of alternating optimization is one of the most common optimization solvers. When we employ the alternating optimization approach by fixing one other of two parameters, it is important to check the relationship between the two parameters. When the two paraemters are not indenpendent to each other, the iteration with separated optimization can cause serious problem where the basic assumption for the two parameter can be broken. To prevent the issue, the conventional alternating optimization employs a penalty term where the relationship between two optimizing parameters can be preserved as the iteration goes. I hope to know that the proposed algorithm considers the relationship between the two low-rank matrices or the two low-rank matrices are independent to each other.\n\n- Additional measurements:\n The quantitative results are present only by the average accuracy for entire classes. However, to evaluate the FCIL method, tthe measurements of he average forgetting and accuracy of all seen classes in the final task are meaningful. To show the effectiveness of the proposed algorithm for the last rounds and the final round, I recommend such additional measurements in the tables.\n\n- Comparison fairness:\n It seems that the most recent study (PILoRA) shows low performance upon ImageNet-R dataset. Of course, the performance can be dramatically dropped in the new dataset, but it shows too large gap between its reported performance with the ImageNet-based dataset. I hope to know the reason why the large gap happens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p9yDDHsxAA", "reviewer": "Reviewer_H9U5", "summary": "This paper studies the model merging, specifically focusing on merging the low-rank adaptation (LoRA) modules. The author proposed to extend the work of RegMean to merge LoRA modules in a closed form, where the challenge lies in the infeasibility of solving the indeterminate system introduced by the LoRA modules. The author further proposes an alternative optimization algorithm, i.e., freezing one of the two matrices, A and B, in the LoRA modules and optimizing each alternatively. The author chose the federal class incremental learning as the case study for the proposed method, conducted extensive experiments to verify the effectiveness of proposed method.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The extension of RegMean for merging the LoRA module is interesting. Given the extensive applicability of the LoRA for parameter-efficient fine-tuning for foundation models, the reviewer believes that the proposed method may benefit a broad range of applications in the future.\n\n2. The proposed alternative optimization procedure for extending RegMean to Low-rank Regression Mean (LoRM) is intuitive and sound. It is promising that the algorithm can converge quickly in practice, as shown in Fig. 2, compared to the methods that achieved the state-of-the-art (SOTA) results previously.\n\n3. The chosen testbed, i.e., the federal class incremental learning (FCIL), is a surging research direction in continual learning with real-world board applications. The author conducted extensive comparisons with different methods and showed the superiority of the LoRM, which vastly improved the SOTA of FCIL.\n\n4. The code has been attached for replication, which will benefit future study.", "weaknesses": "1. Since the proposed method is general for model merging, the reviewer may want to see whether it can achieve good results in different applications, e.g., parameter-efficient fine-tuning and class-incremental learning in computer vision and/or natural language processing.\n\n2. Although the author shows that the LoRM can converge quickly in practice, the reviewer still wonders whether there are any theoretical guarantees for its convergence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LkBhdvzCx0", "reviewer": "Reviewer_BHoB", "summary": "Low-rank Regression Mean (LoRM) is a novel method to merge LoRA (Low-Rank Adaptation) modules in a closed-form solution specifically designed for Federated Class-Incremental Learning (FCIL) by alternatively freezing and merging A and B submatrices of LoRA method.", "questions": "Please refer to the questions in the weakness section", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The method is privacy preserving and enables continual merging for the Image Classification experiments shown.", "weaknesses": "**Inadequate Experimental Diversity** LoRM builds on LoRA, a parameter efficient finetuning method which was introduced to finetune large LLMs and other large, computationally expensive models for diverse and complex tasks. However, LoRM is only tested on the task of image classification and on a single backbone. This counteracts the motivation of using LoRAs (and LoRM is specific to only LoRAs). Experiments with different backbones - LLM, text-image models and diverse tasks like commonsense reasoning, text to image generation - where LoRA modules are regualrly - are necessary to show the efficacy of LoRM, especially since the contributions in the work are backed by only empirical results. LoRM's effectiveness across other federated learning tasks also remains unclear. Testing LoRM in other tasks, such as reinforcement learning or generative tasks, would strengthen its generalizability​.\n\n**Sensitivity to Data Distribution**: LoRM’s performance may degrade with extreme data heterogeneity. The method could benefit from further optimization or regularization to ensure robustness across highly imbalanced data distributions typical in federated settings​.\n\nIt is unclear why the alternating freezing is needed? Why not train another LoRA which combines the previous base model+ LoRA weights? That appears to be the simplest solution to the model merging problem and would avoid problems like highly imbalanced data distributions across different tasks which could lead to load imbalance on particular matrices. Ablation for this would also have been helpful.\n\nComparisons to model merging and lora merging methods like [1, 2] are needed since this method focuses on this very aspect.\n\n[1] Stoica et al. - ZipIt! Merging Models from Different Tasks without Training\n[2] Yadav et. al - TIES-Merging: Resolving Interference When Merging Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTYFdH60sM", "reviewer": "Reviewer_AkSa", "summary": "This paper proposes a solution for federated continual  learning, where information is learned across multiple devices, with each device continually learning sets of disjoint classes. The proposed method proposes a two stage approach. The first stage involves learning LoRA modules in an alternating cycle, where one of the A or B matrices is kept frozen at a time while the other is optimized across individual devices. Once this has been done for all tasks across all devices, the LoRA matrices are aggregated one final time using RegMean to create one model with knowledge from each device across all tasks. The proposed method outperforms presented CIL and FCIL", "questions": "- Replicated results appear to be conflicting with previous works (L2P and TARGET’s relative ranking is switched in the PILoRA paper). Is there a reason for this discrepancy?  \n\n- It is unusual to see L2P outperforming Coda-P, as Coda-P outperforms L2P in the standard CIL setting. What is the reason for this? How do RANPAC (https://arxiv.org/abs/2307.02251) and DualPrompt (https://arxiv.org/abs/2204.04799) compare? Does this inverse relationship between CIL accuracy and FCIL accuracy hold?  \n\n- ImageNet-A, CUB, and Cars196 have been shown to be more challenging datasets for CIL methods, with wider gaps in accuracy when comparing techniques. How does LoRM perform on these datasets?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Adequate novelty is shown by introducing a training scheme that can properly update LoRA modules across multiple clients.  \n\n- LoRM preserves privacy between devices by using Gram matrices \n\n- Ablations clearly show the importance of the proposed components.  \n\n- Provided results show significant improvement over SOTA.  \n\n- Extremely detailed list of parameters for all experiments is provided, greatly aiding replication.  \n\n- Very straightforward organization, the paper flows logically and is easy to read.", "weaknesses": "- The paper mentions  that there is relatively little work in FCIL, a brief diagram visualizing the problem setup would greatly aid in understanding the problem.  \n\n- Figure diagrams, namely Figure 1 requires more detailed captions. - The current SOTA for exemplar free CIL, RANPAC, is missing from the comparison list, this method also uses Gram matrices collecting information across tasks, so a comparison where it is adapted to be a FCIL method would be relevant.  \n\n- The Previous SOTA, PILoRA shows results for both Quantity-based label imbalance and Distribution-based label imbalance, but results in this paper are only shown for Distribution-based.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)