FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic  Synaptic Intelligence
Authors: Yichen Li
Yuying Wang
Haozhao Wang
Yining Qi
Tianzhe Xiao
Ruixuan Li
Venue: ICML 2025 spotlightposter (2025)
Keywords: Federated Learning
Continual Federated Learning
Data Heterogeneity
Primary area: general_machine_learning

Abstract: Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding \textit{knowledge forgetting} of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named \textbf{FedSSI}, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods.

FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence
Yichen Li Yuying Wang² Haozhao Wang'
Yining Qi'
Tianzhe Xiao
Ruixuan Li
Abstract
found applications in diverse domains, including recommen- dation systems (Yang et al., 2020; Li et al., 2024c) and smart healthcare solutions (Xu et al., 2021; Nguyen et al., 2022).
Continual Federated Learning (CFL) allows dis- tributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of pre- vious data. Despite effectiveness, rehearsal comes at cost memory, and it may also violate data privacy. Considering these, we seek to apply regu- larization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, es- pecially synaptic intelligence, can achieve promis- ing results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without re- hearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state- of-the-art methods.
Typically, FL has been studied in a static setting, where the number of training samples does not change over time. However, in realistic FL applications, each client may con- tinuously collect new data and train the local model with streaming tasks, leading performance degradation pre- vious tasks (Yang et al., 2024: Wang et al.. 2024c). This phenomenon, known as catastrophic forgetting (Ganin et al., 2016), poses a significant challenge in the continual learning (CL) paradigm. This challenge is further compounded in FL settings, where the data on one client remains inacces- sible to others, and clients are constrained by memory and computation resources.
To address this issue, researchers have studied continual fed- erated learning (CFL), which enables each client to continu- ously learn from local streaming tasks. FedCIL is proposed in (Qi al., 2023) to learn generative network and recon- struct previous samples for replay, improving the retention of previous information. The authors in (Li et al., 2024a) propose to selectively cache important samples from each task and train the local model with both cached samples and the samples from the current task. The authors in (Dong et al., 2022: 2023) focus on the federated class-increment learning scenario and train a global model by computing additional class-imbalance losses. The study in (Ma et al., 2022a) employs supplementary distilled data on both server and client ends, leveraging knowledge distillation to miti- gate catastrophic forgetting.
1. Introduction
Federated learning (FL) is facilitate the collaborative training of global deep learning model among multiple edge clients while ensuring the privacy of their locally stored data (McMahan et al., 2017; Wang et al., 2023a; Liu et al., 2024). Recently, FL. has garnered significant interest and
Despite effectiveness, most CFL approaches require each client to cache samples locally, a strategy referred to as data rehearsal. Maintaining the local information on the client side is controversial. On the one hand, clients are often limited by storage resources (typically edge devices) and are unable to cache sufficient previous samples for replay. On the other hand, privacy concerns are non-negligible in the FL setting. For instance, some companies will collect user data to update models in a short period, but this data may contain timestamps and need to deleted. Meanwhile, generating synthetic samples for replay using generative models may also pose risk of privacy leakage. As shown in Table 1,
'School of Computer Science and Technology, Huazhong Uni- versity of Science and Technology, Wuhan, China "School of Computer Science and Technology, Soochow University, Suzhou, China. Correspondence to: Ruixuan Li <rxli@ hus...du.cc>>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FedSSI: Rehcarsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence
Table 1. Primary Directions of Progress in CFI.. Analysis of the recent major techniques in the CFI. system with the main contribution. Here, we focus on three common weak points about data rehearsal, computational overhead, and privacy concerns.
Common Weak Points
Reference
Contribution
Data Rehearsal
Computational Overhead
Privacy Concerns
TARGET (Zhang et al., 2023)
Synthetic Exemplar
(Synthetic Sample)
(Distillaaion, Generator)
X
FedCIL (Qi al., 2023)
Generation Alignment
V (Synthetic Sample)
V(ACGAN)
/ (Shared Generator)
Re-Fed (Li et al., 2024a)
Synergism
V (Cached Sample)
X
X
AF-FCL (Wuerkaiii et al., 2023)
Accurate Forgetting
V (Synthetic Sample)
<NF Model)
V (Shared Generator)
SR-FDIL (Li et al., 2024b)
Synergism
V (Cached Sample)
V (GAN Model)
/Shared Discriminator)
GLFC (Dong et al., 2022)
Class-Aware Loss
V (Cached Sample)
v ((Proxy Server)
FOT (Bakman et al., 2023)
Orthogonality
X
v/ Moulti-Heads,Ptueecctogg
/ /TTsk--DD
FedWelT (Yoon el al., 2021)
Network Extension
X
x
v /Task-ID)
CFeD (Ma et al., 2022b)
Two-sides KD
(Distillation Sample)
(Distillation)
X
MFCL. (Bahakniya et al., 2024)
Data-Free KD
V
/Syntheticc Sample)
V (Extra Training)
X
FedET (I.iu et al., 2023)
Pre-training Backhone
V (Cached Sample)
v ((rrasformer)
X
Ours
Regularization
X
X
X
we provide detailed analysis of the latest CFL algorithms and the specific weak points involved. Moreover, existing methods often overlook the data heterogeneity in real-world Fl. scenarios. Simply applying the same techniques across different clients can lead to decreased model performance when dealing with highly heterogeneous distributed data.
and global understandings of the data. The SI algorithm is then employed to mitigate catastrophic forgetting, with each client training the local model using both the new task loss and the PSM. Through extensive experiments on vari- ous datasets and two types of incremental tasks (Class-ll II and Domain-IL), we demonstrate that FedSSI significantly improves model aceuracy compared of-te-of-thh-ar art ap- proaches. Our major contributions are:
In this paper, we aim to enhance the regularization technique for continual federated learning by adhering to the following three principles: (1) free rehearsal, (2) low computational cost, and (3) robustness to data heterogeneity. In response to the first two principles mentioned above, we observe that a considerable number of traditional regularization techniques in the centralized environment can offer a possible solution intuitively. To verify it, we deploy these techniques into the CFL scenario and conduct extensive experiments, which proves that some of the traditional regularization techniques can indeed achieve fairish results, where Synaptic Intelli- gence (SI) (Zenke et al., 2017) is particularly outstanding and greatly outperforms other methods. Despite the advan- tages of these techniques, we find that directly combining them with CL to maintain stable performance under varying data heterogeneity, with SI experiencing the most significant decline. In principle, this is due to surrogate loss, as the main component of SI, being merely calculated based on the local data distribution, causing the local optimization objective to not align with that of the global data distribution in the case of highly heterogeneous data.
We provide an in-depth analysis of regularization- based CFL, identifying that despite existing methods, especially SI, being effective on homogeneous data, they cannot guarantee performance on heterogeneous data distributions which are common in FL.
We propose FedSSI, a simple yet effective method that tailors SI for the CFL. Specifically, FedSSI regulates model updates with both local and global information about data heterogeneity.
We conduct extensive experiments on various datasets and different CFL task scenarios. Experimental results show that our proposed model outperforms state-of-the- art methods by up to 12.47% in terms of final accuracy on different tasks.
2. Background and Related Work
To address this issue, we propose the Synergistic Synaptic Intelligence for the continual Federated Learning (FedSST). Specifically, in FedSSI, each client will calculate the surro- gate loss for previous tasks based not only on information from the local dataset but also on its correlation to the global dataset. We introduce a personalized surrogate model (PSM) for each client, which incorporates global knowledge into local caching so that the surrogate loss reflects both local
Federated Learning. Federated Learning (FL) is a tech- nique designed train shared global model by aggregating models from multiple clients that are trained on their own local private datasets (McMahan et al., 2017: Wang et al., 2023b; 2024b). However, traditional FL algorithms like FedAvg face challenges due to data heterogeneity, where the datasets on clients are Non-IID, resulting in degraded model performance (Jeong et al., 2018h; Wang et al., 2024a).
2

---

FedSSI: Rchearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence
To address the Non-IID issue in FL,, a proximal term is in- troduced in the optimization process in (Li al., 2020) to mitigate the effects of heterogeneous and Non-IID data dis- tribution across participating devices. Another approach, federated distillation (Jeong et al., 2018a), aims to distill the knowledge from multiple local models into the global model by aggregating only the soft predictions. However, these methods primarily address Non-IID static data with spatial heterogeneity, overlooking potential challenges posed by streaming tasks with temporal heterogeneity.
for K' total clients with local streaming data. We now con- sider each client i can only access the local private stream- ing tasks {Th Th.-- T. where Th denotes the t-th
(i) task of the local dataset. Here Th = > k Mk,t), which has No pairs of sample data (1)
Thet E X and cor-
(i) responding label Ve,, E Y We use X and Y to represent the domain space and label space for the t-th task, which has AA classes and Y = -,= Y where Y denotes the total classes of all time. Similarly, we use X= =U==1 X' to denote the total domain space for tasks of all time.
Continual Learning. Continual Learning (CL) is a machine learning technique that allows a model to continuously learn from streaming tasks while retaining knowledge gained from previous tasks (Hsu et al., 2018; van de Ven & Tolias, 2019). This includes task-incremental learning (Dantam al., 2016; Maltoni & Lomonaco, 2018), class-incremental learning (Rebuffi et al., 2017; Yu et al., 2020), and domain- incremental learning (Mirza et al., 2022; Churamani et al., 2021). Existing approaches in CL can be classified into three main categories: replay-based methods (Rebuffi et al., 2017: Liu et al., 2020), regularization- based methods (Jung et al., 2020; Yin et al., 2020), and parameter isolation methods (Long al., 2015; Fernando et al., 2017). Our focus is on the continual federated learning scenario, which combines the principles of federated learning and continual learning.
In this paper, we focus two types of CL scenarios: (1) Task: the main challenge lies in when the sequence of learning tasks arrives, the number of the classes may change, i.c., Y * Y Vt E [n]. (2) Domain- Incremental Task: the main challenge lies in when the se- quence of tasks arrives, the client needs to learn the new task while their domain shifts, i.e., X1 * X VI E [m]. When the t-th task comes, the goal is to train global model w' overall tasks T (E Ek=1 Tx",, which can be formulated as:
K NY ΣΣΣ
1 T
w' = arg min
LCe
WERd ==
Continual Federated Learning. Continual Federated Learning (CFL) aims to address the learning of streaming tasks in each client by emphasizing the adaptation of the global model to new data while retaining knowledge from past data. Despite its significance, CFL has only recently garnered attention, with (Yoon et al., 2021) being pioneer- ing work in this field. Their research focuses on Task-IL, requiring unique task IDs during inference and utilizing separate task-specific masks to enhance personalized perfor- mance. is studied in (Bakman et al., 2023) that projecting the parameters of different tasks onto different orthogonal subspaces prevents new tasks from overwriting previous knowledge, Other studies, such as (Ma et al., 2022b), utilize knowledge distillation at the server and client levels using a surrogate dataset. Recently, (Li et al., 2024b;a) proposed calculating the importance of samples separately for the local and global distributions, selectively saving important samples for retraining to mitigate catastrophic forgetting. Most existing works need cache extra samples with a memory buffer, and methods like (Bakman et al., 2023) and (Yoon et al., 2021) require the task ID during the infer- CFL
where (.n(.) is the output of the global model wo on the sample and LCE(.) is the cross-entropy loss.
Synaptic Intelligence in CFL. Synaptic Intelligence (SI) first introduced by (Zenke et al., 2017) in 2017 to mitigate catastrophic forgetting when neural networks are trained sequentially on multiple tasks. SI achieves this by estimat- ing the importance of cach synaptic weight change during training and penalizing significant changes to weights that are important for previous tasks. This approach allows the model to preserve knowledge from older tasks while still learning new ones. The key idea behind SI is to assign importance value to each synaptic weight based on its contribution to the overall loss of the model. When training on new task, the model is penalized for changing weights with high importance for previous tasks, thereby reducing the risk of forgetting.
To deploy the SI algorithm in CFL (FL+SI), each client will calculate the surrogate loss with the aggregated global model during the local training. Assuming that client k receives the t-th task, the training-modified loss is given by:
ence stage. Our work focuses on regularization-base methods with free rehearsal and low computational cost.
Liotal (wf) = Lnew + CeCour (w²)
= Lnew (wh) + a Σ sh - w/ 11.
3. Preliminaries
Continual Federated Learning. A typical CFL problem can be formalized by collaboratively training global model
where (new(wr) ) is the original loss function used for train- ing the local model with the new task, and fosr (wf() denotes
3

---

FedSSI: Rehcarsal- Free Continual Federated Learning with Synergistic Synaptic Intelligence
FLY
FLIEWC
Fedffug
CIFAR10 Dalasel
Digit10 Dalasel
CIPAR11 Datauet
DigtiD Datanet
57.5
80
70
Text Acturory
92.5
85
70
00
60
87.5
30
40
50
52.5
70-
30-
a=100
2-10
0=0:1 2=100
a=1.0
gm03
Time Step Each
Time Step Each Task
Mecrrgenee by
Heterogene.ty
Figure 1. Performance comparisons of regularization-basse CFL methods on CIFARIO and Digit10 datasets with IID data.
Figure 2. Performance comparisons of aforementioned meth- ods on CIFARIO and 1igit10 datascts with Non-IID data.
the surrogate loss for the previous tasks, w represents the i-th parameter of local model wt in client k and w'-1 represents the optimal weights in the (1-1)-th timestamp, estimated based on its importance for previous tasks. Client I utilizes Di to measure the importance of the i-th param- eter of the model w for the old local data. a is a scaling parameter to trade off previous versus new knowledge. The importance measure 2 is updated after each training iter- ation based on the sensitivity of the loss function to changes in the i-th weight, calculated as:
we then propose novel method that tailors the synaptic intelligence for the CFL scenario, namely, FedSSI.
4.1. Observations
Based on three principles mentioned in Section 1, here we use four notable algorithms: LwF (Li & Hoiem, 2017), EWC (Kirkpatrick et al., 2017), OGD (Farajtabar et al., 2019), and SI (Zenke et al., 2017) to explore the perfor- mance of traditional centralized regularization techniques in the CFL scenario. We simply apply them during the local training process of each client. Each algorithm is combined with the FedAvg (McMahan et al., 2017) framework, and report the final accuracy A(f) after the model completes training the last streaming task across all tasks. More details about experiments can be found in Section 5.1.
(3)
where e > 0 is a constant to avoid division by zero The variable s/,, measures the contribution of the i-th parameter in client & to the change of the loss function Lnew (w(). It is calculated as:
Observation 1. Regularization especially SI can mitigate forgetting in decentralized scenarios when data is IID among clients. As shown in Figure 1, all regularization methods significantly improve test accuracy compared to the baseline, where FL+EWC, FL+OGD, and FL+SI im- prove the performance by approximately 6%~10%% This suggests that regularization techniques in distributed set- tings with multiple clients are still effective, indicating their robustness to the restriction of data isolation. Moreover, it is worth noting that FL with SI performs the best among all regularization methods, indicating great potential to tackle the knowledge-forgetting problem in CFL.
@Lnow(uh) Owkk
(1t Øt
s = J.-
dt.
where z'-1 and t' denote the start and end iteration of the I-th task. By summing the absolute values of the gradients over all training iterations, SI estimates the overall importance of each weight for previous tasks. Then, it is used to penalize significant changes to those weights during training on new tasks, thus reducing catastrophic forgetting.
Observation 2. Regularization fails to mitigate knowl- edge forgetting when data is NOn-IID among clients, with SI declining the most severely. As shown in Figure 2, we observe that as the level of data heterogeneity increases, the performance of all methods significantly declines. Their performance is almost the same as the baseline FedAvg when the data heterogeneity = 0.1, which is common setting in FL. This phenomenon is more pronounced in FL+SI, where SI experiences the greatest performance de- cline among all methods, sometimes even performing worse than other methods (e.g.. a = 1.0 on the Digit10 dataset). Based on the above observations, if the data heterogeneity
4. Methodology
Regularization techniques have been known to be cost- efficient for CL without data rehearsal. In this section, we analyze several typical regularization techniques used in centralized continual learning and then examine their appli- cation in continual federated learning (CFL) with different data heterogeneity settings. Through these examinations, we identify an interesting observation: existing regularization techniques (especially synaptic intelligence) exhibit great advantages in IID settings but fail to hinder knowledge for- getting in Non-III settings. Based on these observations,
4

---

FedSSI: Rehcarsal- Free Continual Federated Learning with Synergistic Synaptic Intelligence
Algorithm 1 FedSSI
ing clients, allowing the surrogate loss of the global model to be calculated locally without training the global model. (2) A control mechanism should be available to adjust the proportion of local and global information. (3) The new module for data heterogeneity will not significantly increase the computational costs or require rehearsal.
Input: T: communication round; K: client number; ny: learning rate; {== distributed dataset with tasks; w: parameter of the model; of: personalized surrogate model in client k for the t-th task; s/,3² contribution of the i-th parameter in client l with t-th task.
Output: W1; w we: target classification model.
Unlike SI algorithms in centralized environments that only consider the data distribution of a single environment, FedSSI introduces Personalized Surrogate Model (PSM) to balance data heterogeneity across clients. Here, the PSM is not used as the target classification model; it is solely employed to calculate the contribution of parameters. Be- fore clients receive new tasks, a PSM will be trained along with the global model on the current local task. Since this is purely local training assisted by an already converged global model, the training of the PSM is very fast (account- ing for only 1/40 of the training cost per task and requiring no communication). We calculate and save the parameter contributions during the local convergence process of the PSM, which can then be locally discarded after its contri- bution has been computed. Then, each client trains on the new task with the local model and parameter contribution scores. Suppose that the client A receives the global model w'-1 before the arrival of the t-th new task, and the clients update PSM o with the current local samples T in s iterations as follows:
1: Initialize the parameter w;
2: Distribute w to all clients;
3: for c== to T do
4: Server randomly selects subset of devices St and send w''1;
5: for each selected client & E St in parallel do
6: Update o in s local iterations with (5);
7: for during the update of v/4 do Calculate the contribution for each parameter
8:
s/,, with the (t- (t-1)-th task by (6); end for
9:
10:
Training the target classification model w/ with the new task with (2)(1);
11:
Send the model w back to the server;
12:
end for
w' + ServerAggregaation (xeess))))
13:
end for
issues encountered by SI in FL can be resolved, then the problem of catastrophic forgetting in CFL could be signifi- cantly alleviated.
M VL (fre.t
o = =
Σ
(oh vki--1
(5)
4.2. FedSSI: Improved Regularization Technique with Synergism for CFL
+ (A) 1-1 1
In this paper, we seek to unlock the potential of SI in FL with Non-IID settings considering its advantage in IID scenarios. Specifically, we argue that simply combining SI with FL by calculating the surrogate loss based on the global model will make its performance constrained by that of the global model. When data heterogeneity increases, the performance of the global model significantly decreases, therefore reduc- ing the SI performance. Considering this, we propose that the surrogate loss in SI should be defined not only based on the importance of the samples in the local dataset but also on their correlation to the global dataset across clients, This approach ensures that the model can be better trained with the parameter contribution from previously seen tasks. In a standard FL scenario, each client can only access its own local model and the global model, which respectively contain local and global information. A straightforward idea is to calculate two parameter contributions using local and global models and then regularize the local training model based on the summed loss. Building upon this idea, the following capabilities should be added: (1) The global model is aggregated from the local models of participat-
where q(A) = (0, 1), and n is the rate to control the step size of the update. The hyper-parameter À adjusts the balance between the local and global information.
To better understand the update process, we can draw an analogy to momentum methods in optimization. Momentum- methods leverage past updates to guide the current update direction (Chan et al., 1996). Similarly, the term (())(t w²-1) acts momentum com- ponent. It incorporates information from the global model w--1 to influence the personalized surrogate model (PSM) of The hyper-parameter À E (0, 1) controls the weight of this momentum component. Denote that a refers to the degree of data heterogeneity and when a has higher value, indicating a trend towards homogeneity in distribu- tion, clients need to focus more on local knowledge. This means that by setting a larger À value, PSM can rely more on local knowledge. Conversely, as À decreases, the em- phasis shifts more towards learning global knowledge with heterogeneous data distribution. Based on this, we can the- oretically judge there exists a positive correlation between
5

---

FedSSI: Rehcarsal- Freee Continual Federated Learning with Synergistic Synaptic Intelligence
and X, which means that the data heterogeneity will be addressed by controlling the À value.
Proposition 1 (Proportion of Global and Local Information.) For all X E (0, X + f()) is non-increasing:
to compute the Then, each client & leverages the PSM o parameter contribution:
(f(en(A) 2 0,
allvi(A) 0012
0.
(8)
0x
07
6k,, S. (
(t),,(t) dt.
Then, for [K], we can get:
(6)
Ōt
lim (a((A) = w. 2-0
Considering 2g accommodates the local and global knowl- edge simultaneously, this refined contribution of. is expected to achieve a better balance of memorizing knowledge in both the previous global model and in the new data. Next, client uses (3) to compute D for each parameter i with */, 1I. ,
Proof. The proof here directly follows the proof in Theorem 3.1 (Hanzely & Richtárik, 2020). As À declines and (()) grows, the Eq. 7) tends to optimize ||0 - - and increase the local empirical training loss f(v),, leading to the con- vergence on the global model. Hence, we can modify the À to adjust the optimization direction of our model ukk thus the dominance of local and global information.
Finally, client & establishes the local loss with (2) and trains the local model wt. The algorithm of FedSSI can be found in A.g.
Discussion about unique challenges. The unique chal- lenges in this paper are more comprehensive yet practical than existing works. In CFL, there are only a handful of works that successfully tackle both catastrophic forgetting and data heterogeneity simultaneously. addition, these ex- isting works often rely on substantial resource expenditures and risk privacy breaches, employing techniques such generative replay, memory buffers, and dynamic networks, primarily traditional centralized methods that overlook the resource constraints of client devices.
Theorem 1 (Convergence of PSM (Li et al., 2024a).) As- suming we converges to the optimal model w with con- vergence rate g(t) for each client k E [K]. such that E Ilwu - w|l?] S g(t) and lime g(t) = 0. There exists a constant C < oo ensuring that the personalized surrogate model va converges to its optimal counterpart iy at a rate proportional to Cg(l).
Here we introduce Theorem proposed by (Li et al., 2024a), which is stated by our proposed personalized surrogate model. Based on it, we ensure the convergence of the PSM and the effectiveness of FedSSI can be proved along by Proposition 1.
In this paper, we have explored the contradiction between the resource constraints of clients and the generally higher resource costs in CL through empirical experiments. Our paper takes different approach, starting by considering the resource limitations of clients in FL and leveraging appropriate technologies. From this foundation, we aim to solve both catastrophic forgetting and data heterogeneity challenges concurrently.
5. Experiments
5.1. Experiment Setup
Datasets. We conduct our experiments with heteroge- neously partitioned datasets across two federated incre- mental learning scenarios using six datasets: (1) Class- Incremental Learning: CIFARIO (Krizhevsky et al., 2009), CO (Krizhevsky et al., 2009), and ImageNet Learning:
4.3. Analytical Understanding of the Personalized Surrogate Model in FedSSI
In this section, we provide both the effectiveness and the convergence of personalized surrogate models. To simplify the notation, here we conduct an analysis on fixed task while the convergence does not depend on the CL setting.
(Le & Yang. 2015); (2) Digit10, Office31 (Saenko al., 2010), and Office-Caltedd 10 (Zhang & Davison, 2020). The Digit dataset contains 10 digit categories in four domains: MNIST (LeCun et al., 2010), EMNIST (Cohen et al., 2017), USPS (Hull, 1994), and SVHN (Netzer et al., 2011). Details of the datasets and data processing can be found in Appendix A.
Definition 1 (Personalized Surrogate Model Formulation.) Denote the objective of personalized informative model eq on client l while f()) is strongly convex as:
{ (f(ez) + (A))
Baseline. For fair comparison with other key works, we follow the same protocols proposed by (McMahan et al., 2017; Rebuffi et al., 2017) to set up FIL tasks. We evalu- ate all methods using two representative FL models: Fe- dAvg (McMahan et al., 2017) and FedProx (Li et al., 2020); two models designed for continual federated learning with- out data rehearsal: FOT (Bakman et al., 2023) and Fed-
Sg (A) := arg min
|106 - =0]
2
(7)
where q(A) = 2x
AE 10,
where ái denotes the global model.
6

---

FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence
Table 2. Performance comparison of various methods in two incremental scenarios.
CIFARIO
CIFAT10
Tiny-Images ImageNet
1o
Office31
Office Caliee-Cllieh-IIe 10
Method
A(f)
A
A())
A
A())
A
A()
A
A(
A
A())
A
FodAvg
36.6821.12
59.17,a0s
27.15
41.36x024
30.16
50.65an.1
68. 12
80.34
48.97
56.29a1.15
55.41aa52
57.612293
FedProx
35.88:082
59.20io:3
27.84.065
44.92.00.4
29.04.0::
49.93,075
68.95.nms
80.26.0m
46.33.n.16
54.03:am
53.90aas
10.04
FL+LwF
38.042033
59.93aa25
31.91
42.56x0.57
34.58a020
12.91a0.9
67.99um.1
80.02aa06
50.70aa.19
57.20sass
17.1
59.75
FL+EWC
La032
60. 19
33.3620.79
43.25a022
15
53.87
69. 25.255
81.54
52.24
57.91sa35
58.69ga50
60.06
FL+OGD
37.55x078
59. 89.88
32.87.00.9
43.56,0.36
35.71un
53.19
68.07.0.33
79.95unas
$1.86.0.37
58.10sasi
58.01a0a1
60.20.an.s
FL+SI
39.32x1.01
60.96xaa77
33.72s0.4
43.82s0.29
35.87an
53.65
69.79aa.55
80.92am.
53.10a1.1
58.28sam7
51. 31.82a
60.277a048
Ro-Fed
38.08
59,02
32.95a0.31
42.50
33.43am
51.98ah32
67.85a1.37
79.85
1
57.4660.4
59.16
60.01
FedCIL 37.9621.68
58.30x1.22
30.88a1.4
31.35a1
50.93aa.4
68.17.085
80.02ahaa
49.15am.2
56.78a0.7
57.80sa74
39.13
GLFC
38.4321.43
60.03at.16
33.17
43.28
32.1
51.79
68. 68.12m0
80.34
48.97
56.2921.15
55.41aa52
57.61
FOT
18.18
61.41
36, 45
43. 51
37.23
54.87
68.54
79.70am11
42a12
17.
60.30
60.76
FedWeIT
37 96
59. 89
05
44.20
34,98am
53.04
69.71
80.91
51, 39
57.83s0m
58.53so.
59, 72.72
FodSSI
42.58
62.65
37.96.0.35
45.28
40.56
56.80
72.09
82.49auu
$5.28
60.05
62.57
62.94
FestAn
FedP
FL+
RL+S
Re-Fed
FedCIL
FedNeIT
FedSS
CIFAR1O Dataset
CIFAR10O Dataset
Digit10 Dataset
Office31 Dataset
80
90-
70
40
Test Acouray
60-
80-
35
80
55
25
70
45
30
40
o=
a=100
om
a=0.1 a=100
1=1.0
a=0.1
a=100
a= Data Heterogineity
Data Heteroganeity
Data Hotemgeneity
Data Heteragenwity
Figure
Performance w.r.t data heterogeneity a for four datasets.
WelT (Yoon et al., 2021); three models designed for con- tinual federated learning with data rehearsal: Re-Fed (Li et al., 2024a), FedCIL (Qi et al., 2023), and GLFC (Dong et al., 2022); and four custom methods combining tradi- tional CL techniques with the FedAvg algorithm: FF+ (Li & Hoiem, 2017), FL+EWC (Kirkpatrick et al., 2017), FL+OGD (Farajtabar et al., 2019), and FL+SI (Zenke et al., 2017). Details of the baselines and data processing can be found in Appendix B.
5.2. Performance Overview
Test Accuracy. Table shows the test accuracy of various methods with data heterogeneity across six datasets. We report both the final accuracy and average accuracy of the global model when all clients finish their training all tasks. FOT outperforms other baselines on three class-IL datasets as the orthogonal projection in FOT exhibits effec- tive training with obvious boundaries between streaming tasks. However, its performance experiences a significant decline with domain datasets, where the classes of each task remain unchanged. All regularization-base methods work and CFL models with data rehearsal fail to achieve ideal results with limited memory buffer. FedSSI achieves the best performance in all cases by up to 12.47% in final ac- curacy. More discussions and results on model performance and communication efficiency are available Appendix D.
Configurations. Unless otherwise mentioned, we employ ResNet18 (He et al., 2016) as the backbone model in all methods and use the Dirichlet distribution a to distribute lo- cal samples, inducing data heterogeneity for all tasks, where a smaller a indicates higher data heterogeneity, For a fair comparison, we set the memory buffer 300 for each client in models designed for continual federated learning with data rehearsal to cache synthetic or previous samples. We report the final accuracy A(f) upon completion of the last streaming task and the average accuracy A across all tasks. All experiments are run on 8 RTX 4090 GPUs and 16 RTX 3090 GPUs Detailed settings and benchmark parameters are illustrated in Table 5 (In Appendix C).
Data Heterogeneity. Figure 3 displays the test accuracy with different levels of data heterogeneity on four datasets. As shown in this figure, all methods improve test accu- racy with the decline in data heterogeneity, and FedSSI consistently achieves a leading and stable improvement in
7

---

FedSSI: Rehcarsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence
Table Test accuracy for FedSSI wir. data heterogeneity a and hyper-parameter À on CIFARIO, CIFARIO0 and Office31..
10.0
100
Dataset
=
==
A 0.8
A 0.2
A 0.5
A 0.8
0.2
0.5
0.8
0.2
A 0.5
0.8
CIFARIO
42.58xa.79
41.22at.03
39.93aus
51.34aass
51.192877
49.23
60.05ao.:
61.01an4
61.632846
61.282243
63.04a0.19
63.92an.s
CIFAR100
28.43a1.6
27.39
26.4
35.81
37.96s8.1
36.01a036
41.28an
42.67
43,45aa.22
44. 44.79a059
45,23an.3
45.67
Office31
49.64a835
38.56wa13
46.96s
54.00
54.60am
55.28
54.37
64.
64.88
65. 19am
65.76
67.03
67.7% 67.79
68.45
Table 4. Evaluation of various methods in terms of the communication rounds reach the best test accuracy. We report the sum of communication rounds required to achieve the best performance on each task and evaluate with the trade-offs between communication rounds and accuracy. We denote "A" the difference between the accuracy improvement percentage and the round increase percentage of FedSSI and other baselines.
CIFARIO
CIFARI00
Tiny- ImageNet
Digit10
Office31
Office-Caltceh-10
Method
Rounds
A
Rounds
A
Rounds
A
Rounds
Rounds
A
Rounds
A
FedAvg
304
16.09%
839
44.70%
893x298
208
2.35%
1652094
7.43%
132a1.25
4.59%
FedProx
315-1.84
22.17%
852-1.76
42.69%
900-2.88
39.00%
204
5.74%
166+1.66
14.50%
145-0.9
17.46%
FL+LWF
300st.32
10.60%
832a217
23.05% t
897a141
16.29% t
211.1.04
0.60%
162ans
1.63%
134
2.84%
FL+EWC
32421.12
17.32%
810a225
15.27%
90922.16
12.53% t
232,1.81
7.12%
16620.6
1.00%
150gt.32
11.28%
FL4OGD
309s1.65
15.01%
833a234
19.69% f
900a2.10
12.91% f
213x1.19
0.27% f
16421.52
0.50%
148
11.24%
FL+SI
29621.11
5.59%
81621.97
14.78% f
897a2.00
12.07% f
21240.87
2.84%
16521.32
1.35%
149
24.77%
Re-Fed 319:1.23
16.52%%
844:2.15
20.66%%
894:1.92
19.99%
222+1 1.48
4.90%
168s0s8
6.75%
145:0.76
7.14%
FedCII. 323+2.78
18.05%
866+225
30.78%1
903+1.94
29.05%
220+1.42
3.48%1
168-1.07
8.90%
147-0.91
10.97%
GLFC
312s1.47
13.36%
830a2.03
18.30%
901
2
208
2.35%
16520s4
7.43%
132et.25
4.59%
FOT
306+2.26
6.63%*
840+1.35
10.01%t
903+1.00
8.61%
220+0.66
2.91%f
170+asa
19%1
148+1.04
7.14%1
FedWeIT 31221.95
14.73%
870g1.58
14.19%
880s226
13.00%
2172101
0.27%
16S21.79
1.91%f
141
5.48%
FedSSI
304+1.29
/
798-208
906.1.62
225+1.34
174+030
143+0881
performance with different levels of data heterogeneity.
Resource Consumption Table 4 compares the commu- nication efficiency of various methods by measuring the trade-offs between communication rounds and accuracy. Compared to other CFL methods, FedSSI may introduce additional communication rounds but can achieve better per- formance with cost-effective trade with large A value. We observed that FedSSI can achieve significant A value for all datasets except Digit10, indicating that our communi- cation method is efficient and can bring considerable perfor- mance improvements. For Digit a possible reason is that the dataset itself has simple features (one-channel images), and all methods can achieve relatively good accuracy. In such cases, the percentage increase in accuracy by FedSSI is relatively small. However, achieving a high-performance improvement on simple dataset is not easy in itself. In the next version, we will consider using a smaller network model (as learning Digit with CNN is sufficient) for verification. The communication rounds for each task of different datasets are listed Section 5.1.
Hyper-parameter Then, we conduct more research on the setting of hyper-parameter X. In our framework, we mod- ify the À value to adjust the global and local information proportion in PSM. As shown in Table 3, we select three different À values with four different data heterogeneity set- tings and evaluate the final test accuracy three datasets. Experimental results show that the value of À should be chosen accordingly under different data heterogeneity. Nev- ertheless, results exhibit the same trend: as the degree of data heterogeneity increases, FedSSI performs better while decreases as PSM contains more global information. Em- pirically, striking a balance between global and local infor- mation is the key to addressing the data heterogeneity in CFL. Vice versa, a increases, the data distribution on the client side becomes more IID. At this point, the clients require less global information and can rely more on their local information for caching important samples. Although we cannot directly relate a and with a simple formula due the complexity of the problem, even in specialized research on PFL, methods such Gaussian mixture mod- eling are relied upon. we can empirically and theoretically judge there exists positive correlation between and À with Proposition and Table 3.
6. Conclusion
In this paper, we introduced FedSSI, a novel regularization- based method for continual federated learning (CFL.) that mitigates catastrophic forgetting and manages data hetero-
8

---

FedSSI: Reheaaral-Free Continual Federated Learning with Synergistic Synaptic Intelligence
geneity without relying on data rehearsal or excessive com- putational resources. Our extensive experiments show that FedSSI outperforms existing CFL approaches with superior accuracy and stability in resource- cesstrac-constraineed environments.
L. E. Incremental task and motion planning: A constraint- based approach. In Robotics: Science and Systems, 2016.
Dong. J., Wang. L.,, Fang, Z., Sun, G., Xu, S., Wang, X., and Zhu, Q. Federated classinccrmmentl learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10164-10173, 2022.
Acknowledgments
This work is supported by the National Key Re- search and Development Program of China under grant 2024YFC3307900; the National Natural Science Founda- tion of China under grants 62376103, 62302184, 62436003 and 62206102; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Sci- ence and Technology Talent Service Project under grant 2024DIC078; Ant Group through CCF-Ant Research Fund; and Fundamental Research Funds for the Central Universi- ties under grant YCJJ20252319 19. The computation is com- pleted in the HPC Platform of Huazhong University of Sci- ence and Technology.
Dong, J., Li, H., Cong, Y., Sun, G., Zhang, Y., and Van Gool, No one left behind: Real world federated class- incremental learning. IEEE Transactions on Patlern Anal- ysis and Machine Intelligence, 2023.
Farajtabar, M., Azizan, N., Mott, A., and Li, A. Orthogonal gradient descent for continual learning, 2019.
Fernando, C., Banarse, S.S Blundell, C.,, Zwols, Y., Ha, D. R., Rusu, A. A., Pritzel, A., and Wierstra, D. Path- net: Evolution channels gradient descent in super neural networks, ArXiv, abs/1701.087735 2017.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. training of neural networks. The journal of machine learning research, 17(1):096-20300, 2016.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hanzely, F. and Richtárik, P. Federated learning of mixture of global and local models. CoRR, https/2222..051111 2020. URL https: /ttp://aaxivvvii www.iiii...rr.oooooooooooooooeee abb/////////// 2002. 05516 6.
References
Babakniya, S., Fabian, Z., He, C., Soltanolkotabi, M., and Avestimehr, S. A data-free approach to mitigate catas- trophic forgetting in federated class incremental learning for vision tasks. Advances in Neural Information Pro- cessing Systems, 36. 2024.
He, K., Zhang, X., Ren, S., and Sun. J. Deep residual learn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hsu, Y.C.,, Liu, Y-C.,, and Kira, Z. Re-evaluating continual learning scenarios: categorization and case for strong baselines. ArXiv, abs/1810.12488, 2018.
Bakman, Y. F., Yaldiz, D. N., Ezzeldin, Y. H., and Aves- timehr, Federated orthogonal training: Mitigating global catastrophic forgetting continual federated learn ing. arXiv preprint arXiv:2309.01288 2023.
Hull, J. J. A database for handwritten text recognition research. IEEF Transactions on putlern analysis and machine intelligence, (665)5500554 1994.
Chan, L. K., Jegadeesh, N., and Lakonishok, J. Momentum strategies. The journal of Finance, 51(5):168111713, 1996.
Jeong, E., Oh, S., Kim, H., Park, J., Bennis, M., and Kim, S. coanuuuncation-ffficieeeee on-device machine learning: Federated distillation and augmentation under non-iid private data. CoRR, abs/18 1. 1479, 2018a.
Churamani, N., Kara, O., and Gunes, H. Domain- incremental continual learning for mitigating bias in facial expression and action unit recognition. ArXiv, abs/2103.08637, 2021.
Jeong, E., Oh, S., Kim, H., Park, J., Bennis, M., and Kim, S.- L. Communication tenteiiccttoo-efffiientt on-device machine learning: Federated distillation and augmentation under non-iid private data. ArXiv, abs/18111 1. 1479, 2018b.
Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. Em nist: Extending mnist to handwritten letters. 2017 in- ternational joint conference on neural networks (IJCNN), pp. 2921-2928 IEEE, 2017.
Jung, S., Ahn, H., Cha, S., and Moon, T. Continual learn- ing with mode-importance based adaptive group sparse regularization. arXiv: Learning. 2020.
Dantam, N. T., Kingston, Z. K., Chaudhuri, S., and Kavraki,

---

FedSSI: Seheaaral-Free Continual Federated Learning with Synergistic Synaptic Intelligence
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Ttrbska-Barwinskke A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 11((1)33521-33)
Long, M., Cao. Y., Wang, J., and Jordan, M. I. Learn- ing transferable features with deep adaptation networks. ArXiv,
2015.
Ma, Y., Xie, Z., Wang. J., Chen, K., and Shou, L. Con- tinual federated learning based knowledge distilla- tion. In Raedt, L. D. (ed.), Proceedings of the Thirty- First International Joint Conference on Artificial Intel- ligence, IJCAI-22, pp. 2182-2188. International Joint Conferences on Artificial Intelligence Organization, 7 2022a. doi: 10.24963//jcai 20249//3//eeai.2222233.... URL https: / //doi.org/ 21.11...444330 20t.2/4433/311 pttpeaaa...222232 2022/303, Main Track.
2017.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7,7);3, 2015.
LeCun, Y., Cortes, C.. and Burges, C. Mnist handwritten digit database, 2010.
Ma, Y., Xie, Z., Wang, J., Chen, K., and Shou, L. Continual federated learning based on knowledge distillation. In IJCAI, pp. 2182-2188, 2022b.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
Maltoni, D. and Lomonaco, Continuous learning in single-incremental- scenarios. Neural networks : the official journal of the International Neural Network Soci- ety, 116:56-73, 2018.
Li, Y. Li, Q., Wang, H., Li, R., Zhong, W., and Zhang, G. Towards efficient replay in federated incremental learning. The Thirty-Fiith IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR 2024), Seattle, USA, June 17-21, 2024a.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and learning deep
y Arcas, B. A. Communication-efii networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017.
Li, Y., Xu, W., Qi, Y., Wang, H., Li, R., and Guo, S. Sr- fdil: Synergistic replay for federated domain-incrementa. learning. IEEE Transactions on Parallel and Distributed Systems, 35(11):1879-1890, 2024b. doi: 10.1109/TPDS 2024.3436874.
Mirza, M. J., Masana, M., Possegger, H., and Bischof, H. An efficient domain-inccremental learning approach drive in all weather conditions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3001-3011, 2022.
Li, and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2944 2017.
Netzer, Y., Wang, T. Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. 2011.
Li, Z., Long, G., and Zhou, T. Federated recommendation with additive personalization, 2024c. URL https:// arxiv. org/ abs / 2201 09109.
Nguyen, D. C., Pham, Q.-., Pathirana, P. N., Ding, M., Seneviratne, A., Lin, Z., Dobre, O., and Hwang, W.-J. Federated learning for smart healthcare: A survey. ACM 33,
Liu, B., Lv, N., Guo, Y., and Li, Recent advances on federated learning: A systematic survey. Neu- rocomputing, 597:128019, 2024. ISSN 0925-2312.
Computing Surveys
Qi, D., Zhao, H., and Li, S. Better generative re- play for continual federated learning. arXiv preprint arXiv:2302. 2th.:2002.1001,, 2023.
doi: https//bbiooggoo..ooo//ummmmmmmm com/ URL wcwence/eaaaeocce /
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Liu, C., Qu, X., Wang, J., and Xiao, J. Fedet: communication-effice federated class-nncremental learning framework based on enhanced transformer. arXiv preprint arXiv:2306.1 15347, 2023.
Saenko, K., Kulis, B., Fritz, M., and Darrell, T. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: 11th European Conference on Com- puter Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings. Part IV 11, pp. 213-226 Springer, 2010.
Liu, X., Wu, C., Menta, M., Herranz, L.,, Raducanu, B., Bagdanov, A. D., Jui, S., and de Weijer, J. v. Generative feature replay for class-incremental learning. In Proceed- ings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 226-227, 2020.
10

---

FedSSI: Rehcarsal-Frie FFeee Continual Federated Learning with Synergistic Synaptic Intelligence
van de Ven, G. M. and Tolias, A. S. Three scenarios for continual learning. ArXiv, abs/1904.07755 2019.
Yu, L., Twardowski, B., Liu, X., Herranz, L.,, Wang, K., Cheng. Y., Jui, S., and van de Weijer, J. Semantic drift compensation for class-incremental learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6980-6989, 2020.
Wang, H., Li, Y., Xu, W., Li, R., Zhan, Y., and Zeng. Z. Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 20412-20421, 2023a.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence.. In International conference on machine learning, pp. 3987-399799 PMLR, 2017.
Wang, H., Xu, H., Li, Y., Xu, Y., Li, R., and Zhang, T. Fed- eda: Federated learning with cross-rounds divergence- aware aggregation. The Twelfth International Confer- ence on Learning Representations, 2023b.
Zhang, J., Chen, C., Zhuang, W., and Lyu, L. Target: Fed- erated class-continual learning via exemplar-free distil- lation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4782-4793, 2023.
Wang, H., Jia, Y., Zhang, M., Hu, Q. Ren, H., Sun, P., Wen, Y., and Zhang. T. Feddse: Distribution aware sub- model extraction for federated learning over resource- constrained devices. In Proceedings of the ACM on Web Conference 2024, pp. 2902-29029999 2913, 2024a.
Zhang, Y. and Davison, D. Impact of imagenet model selection on domain adaptation, 2020.
Wang, H., Zheng, P., Han, X., Xu, W., Li, R., and Zhang, T. Fednlr: Federated learning with neuron wuise--wise/le learning rates. In Proceedings of the 30th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, pp- 3069-3080, 2024b.
Wang, Q.. Liu, B., and Li, Y. Traceable federated continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12872- 12881, 2024c.
Wuerkaixi, A., Cui, S., Zhang, J., Yan, K., Han, B., Niu, G., Fang, L., Zhang, C., and Sugiyama, M. Accurate forgetting for heterogeneous federated continual learning. In The Twelfth International Conference on Learning Representations, 2023.
Xu, J., Glicksberg, B. S., Su, C., Walker, P., Bian, J., and Wang, F. Federated learning for healthcare informat- ics. Journal of Healthcare Informatics Research, 5:1-19, 2021.
Yang, L.., Tan, B., Zheng, V. W., Chen, K., and Yang, Q- Federated recommendation systems. Federated Learning: Privacy and Incentive, pp. 225-239. 2020.
Yang. X., Yu, H., Gao, X., Wang. H., Zhang, J., and Li, T. Federated continual learning via knowledge fusion: A survey. IEEE Transactions Knowledge and Data Engineering, 2024.
Yin, D., Farajtabar, M., and Li, A. Sola: Continual learning with second-order loss approximation. ArXiv, abs/2006.10974, 2020.
Yoon, J., Jeong, W., Lee, G., Yang, E., and Hwang. S. J. Federated continual learning with weighted inter-client transfer. International Conference on Machine Learn- ing, pp. 22073-12086 PMLR, 2021.

---

FedSSI: Rehcarsal- -Feee Continual Federated Learning with Synergistic Synaptic Intelligence
A. Datasets
Clase-ancremental Task Dataset: New classes are incrementally introduced over time. The dataset starts with subset of classes, and new classes are added in subsequent stages, allowing models to learn and adapt an increasing number of classes.
(I) CIFARIO: A dataset with 10 object classes, including various common objects, animals, and vehicles. It consists of 50,000 training images and 10,000 test images.
(2) CIFARI00: Similar to CIFARIO, but with 100 fine-grained object classes. It has 50,000 training images and 10,000 test images.
(3) Nit:- A subset of the ImageNet dataset with 200 object classes. It contains 100,000 training images, 10,000 validation images, and 10,000 test images.
Domaiin-nccremenal Task Dataset: New domains introduced gradually. The dataset initially contains samples from specific domain, and new domains are introduced at later stages, enabling models to adapt and generalize to new unseen domains.
(1) Digir10: Digit- 10 dataset contains 10 digit categories in four domains: MNIST(LeCun et al., 2010), EMNIST al., 2017), USPS(Hull, 1994), SVHN(Netzer et al., 201 1). Each dataset is digit image classification dataset of 10 classes in specific domain, such as handwriting style.
MNIST: A dataset of handwritten digits with training set of 60,000 examples and a test set of 10,000 examples.
EMNIST: An extended version of MNIST that includes handwritten characters (letters and digits) with training set of 240,000 examples and a test set of 40,000 examples.
USPS: The United States Postal Service dataset consists of handwritten digits with training set of 7,291 examples and test set of 2,007 examples.
SVHN: The Street View House Numbers dataset contains images of house numbers captured from Google Street View, with a training set of 73,257 examples and a test set of 26,032 examples.
(2) Office31: A dataset with images from three different domains: Amazon, Webcam, and DSLR. It consists 31 object categories, with each domain having around 4,100 images.
(3) Office-Calteche A dataset with images from four different domains: Amazon, Caltech, Webcam, and DSLR. It consists of 10 object categories, with each domain having around 2,500 images.
B. Baselines
Representative FL methods in CFL:
FedAvg: It is representative federated learning model that aggregates client parameters in each communication. It is simple yet effective model for federated learning.
FedProx: It is also representative federated learning model, which is better at tackling heterogeneity in federated networks than FedAvg.
Traditional Regularization techniques in CFL:
FL+LwF;
This method integrates Federated Learning with Learning without Forgetting (LwF). LwF helps mitigate catastrophic forgetting by retaining knowledge from previous tasks while learning new ones. In our implementation, we use FedAvg to aggregate parameters and incorporate LwF to preserve previous knowledge during training.
FL+EWC:
This method integrates Federated Learning with Elastic Weight Consolidation (EWC). EWC addresses catastrophic forgetting by imposing a quadratic penalty on the changes to parameters that are important for previously learned tasks, thereby preserving the knowledge acquired from previous tasks while enabling the model to learn new information efficiently.
12

---

FedSSI: Rchcaaral-Free Continual Federated Learning with Synergistic Synaptic Intelligence
FL+OGD:
This method combines Federated Learning with Orthogonal Gradient Descent (OGD). OGD mitigates catastrophic forgetting by projecting the gradient updates orthogonally the subspace spanned by the gradients of previous tasks, thus preserving knowledge of previously learned tasks while allowing new information to be integrated effectively.
FL+SI: This method integrates Federated Learning with Synaptic Intelligence (SI). SI computes the importance of each parameter in a similar manner to EWC but uses a different mechanism for accumulating importance measures over time. We adapt the FedAvg algorithm to include SI, maintaining a balance between learning new tasks and retaining old knowledge.
Methods designed for CFL:
This method deploys personalized, informative model at each client to assign data with importance scores. It is prior to cache samples with higher importance scores for replay to alleviate catastrophic forgetting.
Re-Fed:
This approach employs the ACGAN backbone to generate synthetic samples to consolidate the global model and align sample features in the output layer with knowledge distillation.
FedCIL:
GLFC: This approach addresses the federated laaseinnrremetal learning and trains a global model by computing additional class-imbalance losses. proxy server introduced reconstruct samples help clients select the best old models.
This method uses orthogonal projection technology project different parameters into different spaces to achieve isolation of task parameters. In addition, this method proposes secure parameter aggregation method based projection. In the model inference stage, the method requires assuming that the boundaries of the task are known. We modify it to automated inference of the model.
FOT:
This method divides parameters into task-specific parameters and shared parameters. A multi-head model based on known task boundaries is used in the original method. To ensure fair comparison, we modify it to automate the inference of the model.
Well:
C. Configurations
Table 5. Experimental Details. Settings of different datasets in the experiments section.
Attributes
CIFARIO
CIFARI00
Tiny-ImageNet
Digit10
Office31
Office-Caltech-10
Task size
178MB
178MB
435MB
480M
88M
58M
Image number
60K
60K
120K
110K
4.6k
2.5k
Image Size
3x32x32
3x32x32
3x64x64
Lx28x28
3x300x300
3x300x300
Task number
=
10
n=
=
=
Task Scenario
Class-Il.
Class-Il.
Class-11.
Domain-ll.
Domain-1l.
Domain-1l.
Batch Size
64
64
8 =128
64
32
32
ACC metrics
Top-1
Top-1
Top-10
Top-1
Top-1
Top-1
Learning Rate
= 0.01
0.01
= 0.001
= 0.001
1=0011
1= 01.01
Data heterogeneity
0.1
L.0
10.0
0.1
a= 1.0
1.0
Client numbers
C
C=20
C=20
C=15
C=10
C=8
Local training epoch
E 20
=
E = 20
E = 20
=0
E = 15
Client selection ratio
0.4
=.5.5
0.6
==4.4
k=04
==5
Communication Round
T 80
T= 100
T= 100
T= 60
T=
T
13

---

FedSSI: Rehcarsal-Frie Continual Federated Learning with Synergistic Synaptic Intelligence
Table Training resource overhead for aforementioned methods CIFARIO. Here, we indicate the time cost and provide the main cause for the additional overhead.
Method
Time Cost
Main Cause
FedAvg
3.13h
FL+LwF
4.46h
Knowledge Distillation
FL+EWC
5.75h
Fish Information Matrix
FL+OGD
7.09h
Orthogonal Projection
FedSSI
3.75h
Surrogate Loss
Fedris
FedC&
FadWe
CIARR1
CFARLOO
85-
60
60
55
Test Accunso
50
50-
40
30
25
20
Number Tanks
Number
Figure Performance w.r.t number incremental tasks n two class-incremental datasets
D. Additional Results
In this section, first provide the experiments about training time cost and quantitative analysis. Then, we conduct experiments scalability and bandwidth constraints to validate the effectiveness of our method. Finally, we provide more details about the experiment results of each task. We record the test accuracy of the global model at the training stage of each task and the communication rounds required to achieve the corresponding performance.
D.1. Detailed Results of Time Cost and Quantitative Analysis
Table 6 lists the training time of the four methods mentioned above. Although most traditional regularization algorithms are not suitable for resource-constrained edge devices in the CFL scenario due to their computational efficiency overhead, we also observe that few methods that could potentially be deployed in CFL, such as SI, have a relatively low time overhead, achieving significant performance improvements with less than 20% increase in training time compared to fine-tuning (FedAvg).
Figure 4 shows the qualitative analysis of the number of incremental tasks three to these curves, we can easily observe that our model performs better than other baselines across all tasks, with varying numbers of incremental tasks. It demonstrates that FedSSI enables clients to learn new incremental classes better than other methods.
datasets. According
D.2. Detailed Results of Scalability and Bandwidth Constraints
Table 7 shows the results of test accuracy on scalability and bandwidth constraints. We highlight the best result in bold. To verify scalability, we conducted experiments with 100 clients and a client selection rate of 10% (a = 10.0). To investigate the impact of bandwidth on our method, we reduced the client selection rate by half and decreased the total number of communication rounds.
D.3. Detailed Results of Test Accuracy
Table 8, 9, 10, 11 and 12 show the results of test accuracy on each incremental task in the Ace (Accuray) line. Here we measure average accuracy over all tasks on each client the Ace line and highlight the best test accuracy in bold.
14

---

FedSSI: Rehcarsal- Free Continual Federated Learning with Synergistic Synaptic Intelligence
Table 7. Performance comparison of various methods with scalahility and bandwidth constraints.
Scalability
Bandwidth
CIFARIO
Digit10
CIFARIO
Digit10
Method
A(f)
A
A(f)
A(f)
A
A(f)
A
FedAvg
18.67
45.80
55.91
70.37
29.02
53.89
63.06
78.27
FL+EWC
19.93
46.38
56.82
70.40
28.86
54.52
62.92
78.80
Re-Fed
19.44
44.08
54.91
66.24
28.82
52.84
61.53
76.69
FOT
21.26
47.02
56.06
69.69
31.31
55.56
62.41
76.32
FedSSI
23.61
47.14
59.35
71.27
32.95
56.12
64.91
79.10
D.4. Detailed Results of Communication Round.
Table 8, 9, 10, 11 and 12 show the detailed results of communication round each incremental task in the CoR (Communication Round) line and highlight the results of the fewest number of communication rounds in underline.
Table 8. Performance comparisons of various methods on CIFAR10 with 5 incremental tasks.
CIFARIO
Method
Target
2
6
10
Avg
Acc
88.43
70.23
55.00
45.53
36.68
59.17
FedAvg
CoR
52
63
68
60
61
60.8
Ace
88.43
69.62
56.50
45.58
35.88
59.20
FedProx
CoR
53
66
69
62
65
63
Acc
88,43
70.98
56.
46.12
38.04
59.93
FL+LwF
CoR
52
64
67
59
58
60.0
Acc
88.43
71.32
56.60
46.30
38.31
60.19
FL+EWC
CoR
52
70
73
64.8
Acc
88.43
71.10
56.20
46.14
37.55
59.88
FL+OGD
CoR
52
66
68
62
61.8
Acc
88,43
71.72
57.40
47.95
39.32
60.96
FL+SI
CoR
52
64
67
59
54
59.2
Acc
86.94
69.31
55.12
45.63
38.08
59.02
Re-Fed
CoR
59
68
65
63.80
Acc
85.73
68.44
54.60
44.77
37.96
58.30
FedCII.
CoR
60
68
70
62
63
64.6
Acc
87.16
70.42
57.27
46.89
38.43
60.03
GLFC
CoR
57
66
68
60
61
62.4
Acc
87.53
72.70
58.35
48.28
40.18
61.41
FOT
CoR
55
65
68
60
58
61.2
Acc
87.92
71.15
56.25
46.15
37.96
59.89
FedWeIT
CoR
56
66
67
62
62.4
Acc
88.29
72.58
59.43
50.35
42.58
62.65
FedSSI
CoR
54
65
69
59
57
60.8
15

---

FedSSI: Rehcarsal-Fre -Feee Continual Federated Learning with Synergistic Synaptic Intelligence
Table Performance comparisons of various methods on CIFARIO0 with 10 incremental tasks.
CIFAR100
Method
Target
10
20
30
40
50
60
70
80
90
100
Avg
Acc
69.12
55.48
49.32
42.75
38.56
35,21
33.10
31.23
29.67
27.15
41.36
FedAvg
CoR
76
79
82
84
85
86
87
88
89
83
83.9
Acc
68.62
54.90
48.95
42.45
38.34
35.10
33.00
31.12
28.88
27.84
40.92
FedProx
CoR
81
80
83
85
86
87
88
89
90
83
85.2
Acc
69.12
56.01
50.32
44.75
39.89
36.54
34.12
32.34
30.67
31.91
42.56
FL+LwF
CoR
76
78
80
82
83
84
85
86
87
87
83.2
Acc
69.12
56.34
50.98
45.21
40.45
37.01
34.58
32.98
32.47
33.36
43,25
FL+EWC
CoR
76
78
81
83
84
85
86
87
88
77
81
Acc
69.12
56.45
51.12
45.78
41.12
37.45
35.00
34.57
32.13
32.87
43.56
FL+OGD
CoR
76
77
80
82
83
84
85
86
87
89
83.3
Acc
69.12
56.78
51.67
46.12
41.54
38.12
35.67
33.29
32.17
33.72
43.82
FL+SI
CoR
76
78
80
82
83
84
85
86
87
75
81.6
Acc
69.27
55.31
50.61
43.22
39.54
36.78
33.50
32.19
31.64
32.95
42.50
Re-Fed
CoR
82
80
82
84
83
86
85
87
87
88
84.4
Acc
68.49
56.06
49.76
43.98
38.82
36.93
33.28
32.30
31.07
30.88
42.16
FedCII.
CoR
85
83
85
85
85
87
88
89
91
88
86.6
Acc
68.98
56.67
50.22
44.09
39.21
37.43
35.16
34.53
33.34
33.17
43.28
GLFC
CoR
79
78
80
84
83
84
85
86
85
86
83
Ace
68.56
55.67
50.78
45.56
40.98
36.20
34.40
31.98
31.12
36.15
43.14
FOT
CoR
79
78
81
82
84
85
86
87
88
90
84
Acc
69.46
57.01
50.45
45.45
41.12
37.56
36.45
35.06
33.62
35.84
44.20
FedWeIT
CoR
88
86
87
88
89
90
90
90
90
83
Acc
69.12
57.34
52.12
47.01
43.12
40.01
37.45
35.23
33.45
37.96
45.28
FedSSI
CoR
76
75
78
80
81
82
83
84
85
74
79.8
16

---

FedSSI: Rehcarsal-Fre -Feee Continual Federated Learning with Synergistic Synaptic Intelligence
Table 10. Performance comparisons of various methods on Tiny-Imagen with 10 incremental tasks.
Siny-ImageNe
Method
Target
20
40
60
80
100
120
140
160
180
200
Avg
Acc
78.65
64.23
65.86
55.11
48.78
45,22
43.75
38.93
35.81
30.16
50.65
FedAvg
CoR
95
87
92
84
91
89
86
88
94
87
89.3
Acc
78.15
63.56
65.12
54.78
47.89
44.67
42.98
38.11
34.97
29.04
49.93
FedProx
CoR
97
88
93
85
90
88
87
89
93
90
90.0
Acc
78.65
65.12
67.45
56.84
50.78
48.12
47.23
41.89
38.45
34.58
52.91
FL+LwF
CoR
95
89
94
86
92
90
89
90
91
81
89.7
Acc
78.65
66.15
68.78
57.12
51.64
49.12
47.85
43.12
40.12
36.15
53.87
FL+EWC
CoR
95
88
93
87
91
90
88
91
92
94
90.9
Acc
78.65
65.78
67.92
56.89
50.12
48.78
46.67
41.45
39.89
35.71
53.19
FL+OGD
CoR
95
89
94
86
92
90
89
90
91
84
90.0
Acc
78.65
66.12
68.45
57.34
51.12
49.18
47.89
41.82
40.04
35.87
53.65
FL+SI
CoR
95
89
94
87
92
90
88
89
90
83
89.7
Acc
78.54
65.27
66.39
55.21
49.36
47.45
45.83
40.32
38.03
33.43
51.98
Re-Fed
CoR
96
88
93
87
92
88
86
88
91
85
89.4
Acc
78.91
64.87
65.79
54.91
48.83
45.75
44.66
38.42
35.79
31.35
50.93
FedCII.
CoR
96
89
92
86
91
90
89
90
91
89
90.3
Acc
78.66
65.55
66.94
55.07
50.22
46.78
45.32
39.46
37.75
32.11
51.79
GLFC
CoR
96
88
94
87
91
89
87
89
90
90
90.1
Ace
78.84
66.45
69.08
58.89
53.07
51.06
48.15
43.78
42.32
37.23
54.87
FOT
CoR
96
88
93
86
91
89
88
90
92
91
90.3
Acc
78.51
65.78
67.85
57.34
48.78
48.34
47.45
41.45
39.78
34.98
53.04
FedWeIT
CoR
96
88
93
87
92
85
87
90
91
72
88.0
Acc
78.65
68.08
71.12
60.34
53.78
54.34
50.89
45.45
44.78
40.56
56.80
FedSSI
CoR
95
89
95
87
92
90
88
91
92
87
90.6
17

---

FedSSI: Rehcarsal-Fre FFeee Continual Federated Learning with Synergistic Synaptic Intelligence
Table Performance comparisons various methods on Digit with domains.
Digit10
Method
Target
MNIST
EMNIST
USPS
SVHN
Avg
Acc
94.17
81.93
77.13
68.12
80.34
FedAvg
CoR
58
52.6
49.9
47
51.9
Acc
94.03
81.34
76.72
68.95
80.26
FedProx
CoR
56
51.6
49.5
47
51.0
Acc
94.17
81.58
76.35
67.99
80.02
FL+LwF
CoR
58
53.2
51.1
49
52.8
Acc
94.17
82.73
80.02
69.25
81.54
FL+EWC
CoR
60.4
58.0
56
58.1
Acc
94.17
80.40
77.17
68.07
79.95
FL+OGD
CoR
58
53.4
51.5
50
53.2
Acc
94.17
81.86
77.84
69.79
80.92
FL+SI
CoR
58
53.5
51.4
49
53.0
Acc
93.47
80.92
77.15
67.85
79.85
Re-Fed
CoR
59
54.9
53.8
54
55.4
Acc
93.66
81.31
76.93
68.17
80.02
FedCIL
CoR
56
56.3
54.7
53
55.0
Acc
94.17
81.93
77.13
68.12
80.34
GLFC
CoR
58
52.6
49.9
47
51.9
Acc
93.04
80.25
76.98
68.54
79.70
FOT
CoR
60
55.9
53.8
51
55.2
Acc
93.97
82.15
77.81
69.71
80.91
FedWeIT
CoR
54
55.6
53.9
53
54.1
Acc
94.17
83.90
79.79
72.09
82.49
FedSSI
CoR
58
57.9
55.6
53
56.1
18

---

FedSSI: Rehcarsal-Fre -Feee Continual Federated Learning with Synergistic Synaptic Intelligence
Table Performance comparisons of various methods on Office-31 with domains Office-Caltech- 10 domains.
Office31
Pffice-Caltech110
Method
Target
Amazon
DIsr
Webcam
Avg
Amazon
Caltech
Disr
Wehcam
Avg
Acc
65.46
54.45
48.97
56.29
69.75
57.23
48.07
55.41
57.61
FedAvg
CoR
56
51
58
55.0
36
24
38
34
33.0
Acc
63.15
52.60
46.33
54.03
68.04
56.78
45.70
53.90
56.10
FedProx
CoR
57
54
55
55.3
36
35
37
37
36.2
Acc
65.46
55.45
50.70
57.2
69.75
60.45
51.69
57.11
59.75
FL+LwF
CoR
56
52
54
54.0
36
32
34
32
33.5
Acc
65.46
56.02
52.24
57.91
69.75
60.78
51.01
58.69
60.06
FL+EWC
CoR
56
52
58
55.3
36
38
39
37
37.5
Acc
65.46
56.99
51.86
58.1
69.75
60.12
52.90
58.01
60..2
FL+OGD
CoR
56
55
54.7
36
39
37.0
Acc
65.46
56.29
53.10
58.28
69.75
61.01
58.50
51.82
60.27
FL+SI
CoR
56
53
56
55,0
36
38
36
39
37.2
Acc
65.26
57.02
50.11
57.46
69.82
59.74
51.30
59.16
60.01
Re-Fed
CoR
56
53
59
56
36
36
37
36
36.25
Acc
64.85
56.33
49.15
56.78
69.08
58.83
50.82
57.80
59.13
FedCIL
CoR
55
56
57
56
35
37
38
37
36.75
Acc
65.46
54.45
48.97
56.29
69.75
57.23
48.07
55.41
57.61
GLFC
CoR
56
51
58
55.0
36
24
38
34
33,0
Acc
64.76
54.62
49.12
56.17
69.45
61,45
51.82
60.30
60.76
FOT
CoR
57
58
56.7
36
37
39
36
37.0
Acc
64.98
57.02
51.49
57.83
69.14
60.22
50.99
58.53
59.72
FedWeIT
CoR
53
54
58
55.0
34
36
37
34
35.2
Acc
65.46
59.40
55.28
60.05
69.75
64.34
55.12
62.57
62.94
FedSSI
CoR
56
60
58
58.0
36
35
38
34
35.8
19


Reviews:
[{"review_id": "Da6mC0QOwm", "reviewer": "Reviewer_uLqB", "summary": "The paper introduces FedSSI, a regularization-based continual federated learning (CFL) method designed to address catastrophic forgetting and data heterogeneity without requiring data rehearsal or heavy computational overhead. It identifies limitations in applying traditional regularization techniques like Synaptic Intelligence (SI) to heterogeneous data in federated learning scenarios. To overcome this, FedSSI proposes a Personalized Surrogate Model (PSM) that leverages both local and global information to calculate a surrogate loss tailored to client-specific data heterogeneity effectively. Experiments conducted across multiple benchmarks—including CIFAR10, CIFAR100, Tiny-ImageNet, Digit10, Office31, and Office-Caltech-10—demonstrate that FedSSI significantly outperforms existing methods, achieving accuracy improvements of up to 11.52% across different scenarios.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1&Q4.  Concerns about selection of the hyperparameter $\\lambda$ and its theoretical bound**\n\n**R1:** Thank you for this valuable comment. We conducted relative experiments in Table 3. In Table 3, $\\alpha$ refers to the degree of data heterogeneity, while $\\lambda$ is a control coefficient in the training process of PSM. In Proposition 1, we show that adjusting $\\lambda$ can control whether the proportion of knowledge in PSM leans towards the local distribution or the global distribution (i.e., it is related to $\\alpha$). When $\\alpha$ has a higher value, indicating a trend towards homogeneity in distribution, clients need to focus more on local knowledge. This means that by setting a larger $\\lambda$ value, PSM can rely more on local knowledge. \n\nAlthough we cannot directly relate $\\alpha$ and $\\lambda$ with a simple formula due to the complexity of the problem, even in specialized research on personalized federated learning (PFL), strategies such as **Gaussian mixture modeling** are relied upon. Another possible approach can be adapted from **APFL** [1], where the optimal weights are empirically determined through iterative gradient descent during optimization. However, this process may introduce additional model parameters and computational overhead.\n\nThis specific study goes beyond the resource-efficient CFL that we focus on in this manuscript, so we did not develop new mechanisms for this issue. In this paper, we can empirically and theoretically judge that there exists a positive correlation between $\\alpha$ and $\\lambda$, which is supported by Proposition 1 and extensive experiments conducted in Table 3. We will consider this issue in our future research work.\n\n[1] Deng Y, Kamani M M, Mahdavi M. Adaptive personalized federated learning[J]. arXiv preprint arXiv:2003.13461, 2020.\n\n> **Q2. Concerns about the computational overhead brought by PSM.**\n\n**R2:** Thank you for raising this concern.  The computational overhead of PSM is negligible compared to the overall CFL training process. The computational overhead of the PSM module scales proportionally with the complexity of learning tasks. For edge devices with limited computational capacity, their CFL tasks are typically simpler, thus the PSM’s computational demands scale down correspondingly. Specifically, the PSM will be trained along with the global model on the current local task. Since this is purely local training assisted by an already converged global model, the training of the PSM is very fast (accounting for only **1/40** of the training cost per task and requiring no communication). We calculate and save the parameter contributions during the local convergence process of the PSM, which can then be locally **discarded** after its contribution has been computed. Then, each client trains on the new task with the local model and parameter contribution scores. **We analyze this issue in Line 232 in our submitted manuscript.**\n\n> **Q3. Concerns about the scalability of FedSSI.**\n\n**R3:** Thank you for your helpful comment. We apologize that we are unable to simulate thousands of clients to conduct scalability experiments due to hardware limitations. But we validated the scalability of FedSSI on 100 clients, which is also a common scale experiment in FL's work. We conducted further experiments by increasing the number of clients to 100 while reducing the client selection rate to 10%. We performed some related experiments on CIFAR10 and Digit10 ($\\alpha=10.0$), with the results as follows:\n\n|             |  Metric   | FedAvg | FL+EWC | Re-Fed |  FOT  |  FedSSI   |\n| :---------: | :-------: | :----: | :----: | :----: | :---: | :-------: |\n| **CIFAR10** |  $A(f)$   | 18.67  | 19.93  | 19.44  | 21.26 | **23.61** |\n|             | $\\bar A$  |  45.8  | 46.38  | 44.08  | 47.02 | **47.14** |\n| **Digit10** | $ A(f) $  | 55.91  | 56.82  | 54.91  | 56.06 | **59.35** |\n|             | $ \\bar A$ | 70.37  |  70.4  | 66.24  | 69.69 | **71.27** |\n\nSince the dataset needs to be divided into different numbers of tasks, an excessive number of clients can lead to a very small number of samples per client, making model training difficult. However, **FedSSI still maintains a leading position.**", "claims_and_evidence": "The claims made by the authors, particularly the effectiveness of FedSSI in handling data heterogeneity and mitigating catastrophic forgetting, are supported by substantial experimental evidence. Experiments cover diverse datasets, clearly demonstrating performance gains over baseline methods. The claim that FedSSI addresses limitations of traditional regularization methods (e.g., SI) is convincingly supported by experiments illustrating the superior performance of FedSSI under various levels of data heterogeneity.", "methods_and_evaluation_criteria": "The evaluation criteria and methods proposed (such as Class-Incremental and Domain-Incremental learning tasks, along with different data heterogeneity levels using Dirichlet distribution) are appropriate for validating the method in realistic CFL scenarios. The benchmarks and comparison baselines used in the evaluation make sense and cover a comprehensive range of existing approaches in the literature.", "theoretical_claims": "The paper includes theoretical discussions about the convergence and effectiveness of the Personalized Surrogate Model (PSM). The authors provide a theoretical analysis of the personalized surrogate model's convergence. Proposition 1 and Theorem 1 are theoretically sound and based on prior established results. No specific proof issues or errors were identified upon review.", "experimental_designs_or_analyses": "The experimental designs are thorough, clearly defined, and valid for the CFL tasks explored. The authors have conducted comprehensive experiments across multiple datasets and scenarios, including consideration of data heterogeneity levels, and the analyses provided (e.g., comparing test accuracy and communication efficiency) are sound and convincing.", "relation_to_broader_scientific_literature": "The paper situates itself clearly within the broader literature on Continual Federated Learning and builds explicitly on Synaptic Intelligence (SI), extending this traditional continual learning technique to federated learning settings. It also positions itself relative to recent works addressing catastrophic forgetting in federated scenarios (FedWeIT, FOT, FedCIL, etc.), clearly articulating its contributions against existing CFL approaches.", "essential_references_not_discussed": "This paper reviewed relevant references.", "other_strengths_and_weaknesses": "Strengths\n-------------\n\n1. The paper addresses a challenging and important problem in CFL: catastrophic forgetting without rehearsal.\n\n2. FedSSI extends synaptic intelligence for federated learning scenarios and heterogeneous data distributions.\n\n3. Extensive experimental validation convincingly supports the efficacy of the proposed method.\n\nWeakness\n-------------\n\n1. The personalized surrogate model introduces an additional local model update step, and while computational overhead is claimed to be minimal, practical implications of this overhead on low-resource edge devices might require further clarification.\n\n2. The balance hyper-parameter (λ) requires careful tuning; however, the paper doesn't provide a fully automatic or adaptive mechanism for setting it dynamically in real-world deployments.", "other_comments_or_suggestions": "The paper is well written. No such major suggestions on writing.", "questions_for_authors": "**1. Hyperparameter Selection** How sensitive is FedSSI to the choice of λ, particularly under realistic scenarios where data distribution shifts might be unpredictable? Could an adaptive strategy for tuning λ be implemented practically? By clarifying this question would help to assess the practical usability of FedSSI in dynamic scenarios.\n\n**2. Computational Overhead** While FedSSI is designed to be computationally efficient, but computing surrogate model in resource-limited edge devices will cause an computational overhead? \n\n**3. Client Participation** How does FedSSI's performance scale with increasing numbers of clients? Have you evaluated its robustness or convergence speed in highly scaled FL scenarios (hundreds or thousands of clients with partial participation)?\n\n**4. Theoretical Bound on λ** Is there a theoretical guideline or bound for selecting the optimal λ based on measurable properties of client data distributions (non-IID)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IZJD1eZUut", "reviewer": "Reviewer_hxAj", "summary": "The paper introduces a continual federated learning method, FedSSI, aimed to mitigate catastrophic forgetting without rehearsal. FedSSI employs the personalized surrogate model to strike a balance between global and local knowledge during the training process. Experimental results show that FedSSI can outperform other baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for this professional review. The critical comments have been addressed carefully, and responses have been given one by one.\n\n> **Q1. Minor typos in our manuscript.**\n\n**R1:** Thank you very much for this helpful comment.  We are sorry for the wrong spelling in Table 2 and will correct it. We will carefully polish our manuscript to further improve the presentation. In Eq.(1), we formulate the overall optimization objective of CFL. $w^t$ denotes the converged global model, and the superscript $t$ represents the number of tasks here.\n\n> **Q2. Insufficient description of experimental settings.**\n\n**R2:** Thank you for this valuable comment. We will further provide the relevant experimental details in the supplementary: we assign different numbers of tasks to various datasets. Using CIFAR-10 as an example, we set five tasks for class-incremental tasks, each covering two classes without data overlap. For domain-incremental tasks, each domain represents one task. \n\n> **Q3. Concerns about the LLM foundation for FedSSI.**\n\n**R3:** Thanks for raising this concern. LLMs have gained significant attention for their strong performance on conventional tasks, but their high computational and communication overheads prevent deployment on edge devices. Current CFL methods, including FedSSI, mainly focus on training with traditional architectures (e.g., CNN, ResNet), and we will later consider the LLM foundation for CFL in our future research. Moreover, calculating the contribution for each parameter incurs negligible computational overhead due to the PSM module.  The PSM will be trained along with the global model on the current local task. Since this is purely local training assisted by an already converged global model, the training of the PSM is very fast (accounting for only **1/40** of the training cost per task and requiring no communication). We calculate and save the parameter contributions during the local convergence process of the PSM, which can then be locally **discarded** after its contribution has been computed. Then, each client trains on the new task with the local model and parameter contribution scores.", "claims_and_evidence": "The claims in the paper are clear and supported by convincing evidence.", "methods_and_evaluation_criteria": "The proposed method sounds technical and the evaluation is sufficient with various settings and advanced baselines.", "theoretical_claims": "The theoretical analysis is explicit and easy to understand.", "experimental_designs_or_analyses": "The experimental designs and statistical analyses are rigorous and valid.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "All essential references are included.", "other_strengths_and_weaknesses": "Pros:\n-\tIt is meaningful to explore the training cost in CFL, where edge devices are often equipped with portable but weak hardware. \n-\tThe paper is well-organized and easy to read.\n-\tThis paper innovatively introduces the PSM and successfully addresses the issue of data heterogeneity in CFL at a low cost.\n-\tThe authors have conducted extensive experimental validations across multiple datasets and CFL scenarios, demonstrating the effectiveness of FedSSI.\n\nCons:\n-\tThe hyperparameter $\\lambda$ is based on data heterogeneity. An adaptive adjustment strategy can be discussed to enhance the robustness of FedSSI.\n-\tThe authors can provide the framework of FedSSI for readers who are not so familiar with CFL. It will help enhance the understanding of the technique and CFL settings.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "For edge devices with limited resources, each device may employ models with different architectures. In such a situation, can FedSSI still maintain its advantage?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "19DS6a3r6h", "reviewer": "Reviewer_j1LD", "summary": "This paper focuses on the continual federated learning and systematically analyzes the resource consumption of existing works. The authors propose a resource-friendly method based on the SI algorithm, FedSSI, which balances local and global knowledge. Extensive experiments and analytical understanding have been done to verify the effectiveness.\n\n## update after rebuttal\nAfter reviewing all the author rebuttal and discussions, given the quality of this paper and their thorough rebuttal, I would like to vote for the acceptance of this paper. The authors have solved my concerns regarding experimental settings and paper details.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for providing us with positive comments. In the following, we give detailed responses to each review. \n\n> **Q1. Concerns about the selection of the hyperparameter $\\lambda$** \n\n**R1:** Thanks a lot for raising this concern. In Table 3, $\\alpha$ refers to the degree of data heterogeneity, while $\\lambda$ is a control coefficient in the training process of PSM. In Proposition 1, we show that adjusting $\\lambda$ can control whether the proportion of knowledge in PSM leans towards the local or global distribution (i.e., it is related to $\\alpha$). When $\\alpha$ has a higher value, indicating a trend towards homogeneity in distribution, clients need to focus more on local knowledge. This means that by setting a larger $\\lambda$ value, PSM can rely more on local knowledge. Although we cannot directly relate $\\alpha$ and $\\lambda$ with a simple formula due to the complexity of the problem, even in specialized research on personalized federated learning (PFL), methods such as **Gaussian mixture modeling** are relied upon. This specific study goes beyond the resource-efficient CFL that we focus on in this manuscript, so we did not develop new mechanisms for this issue. In this paper, we can empirically and theoretically judge that there exists a positive correlation between $\\alpha$ and $\\lambda$, which is supported by Proposition 1 and extensive experiments conducted in Table 3. We will consider this issue in our future research work.\n\n> **Q2. Concerns about framework of FedSSI and more details about CFL**\n\n**R2:** Thank you for this helpful comment. We agree that a framework diagram could improve accessibility for readers less familiar with CFL. However, the regularization-based methods in FedSSI are inherently theoretical, making it challenging to explicitly visualize their nuanced mechanisms (e.g., PSM step or SI module) within a high-level framework. To address this, we have included **Algorithm 1**, which details the iterative steps of FedSSI. We appreciate your suggestion and will consider providing the framework in our final version. Moreover, we will further provide the relevant experimental details in the supplementary: we assign different numbers of tasks to various datasets. Using CIFAR-10 as an example, we set five tasks for class-incremental tasks, each covering two classes without data overlap. For domain-incremental tasks, each domain represents one task. \n\n> **Q3. Concerns about FedSSI's adaptability to heterogeneous models**\n\n**R3:**  Thank you for raising this concern. FedSSI still can work under model heterogeneity, but this heterogeneity will introduce novel challenges to CFL systems that have never been addressed. FedSSI’s core mechanism utilizes a PSM Module to address data heterogeneity and employs SI for continual learning. Notably, the SI algorithm is architecture-agnostic, as it operates by quantifying parameter contributions during gradient updates. Similarly, the PSM module—an initial copy of the local model—is independent of other clients’ model architectures. However, model heterogeneity exacerbates system heterogeneity due to divergent feature representation spaces across architectures. As CFL is an emerging research area, existing studies have yet to address model heterogeneity systematically. The reviewer’s suggestion highlights a promising research direction we will prioritize in future work.", "claims_and_evidence": "All claims are well supported by evidence.", "methods_and_evaluation_criteria": "Studying the lightweight federated continual learning is interesting and is crucial for the practical deployment in real-world applications. The proposed FedSSI is simple yet effective with abundant experiments and analysis.", "theoretical_claims": "The analytical understanding is easy to read and solid.", "experimental_designs_or_analyses": "The paper presents a comprehensive evaluation with sufficient baselines across various datasets and scenarios.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "References are sufficient.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is well-organized and easy to read.\n2.\tThe experiments are adequate and sufficient.\n3.\tThe research top is interesting and may contribute to the practical applications.\n4.\tThe proposed method is easy to follow and seems promising.\n\nWeaknesses:\n\n1.\tAlthough this article demonstrates a high level of quality, there are still minor typos in its presentation. The Eq. (1) is not rigorous. To the left of the equal sign, should be w, and authors should unify the notation of the model w. In Table 2, the spelling of “CIFAI100” is wrong.\n2.\tThe authors should provide more details about the data partition and task setting. It is better to expand this, especially for readers who may not be familiar with continual federated learning.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "While FedSSI calculates the contribution of each parameter during the training process, will its performance and cost be affected by the number of model parameters? Nowadays, LLM is a very hot topic. Can this method still work based on LLM?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cD8RkvrEG7", "reviewer": "Reviewer_bU17", "summary": "This paper introduces FedSSI, a novel regularization algorithm for continual federated learning that addresses the challenges of knowledge forgetting and data heterogeneity without replay. FedSSI can empirically and theoretically reduce computational overhead and outperform state-of-the-art methods．\n\n## update after rebuttal\nMy concerns are mainly addressed during rebuttal. Thus, I will keep my positive rating.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable comments. In the following, we give point-by-point responses to each comment.\n\n> **Q1.  Concerns about the storage overhead caused by PSM.**\n\n**R1:** Thank you for this constructive suggestion. The PSM will be trained along with the global model on the current local task. Since this is purely local training assisted by an already converged global model, the training of the PSM is very fast (accounting for only **1/40** of the training cost per task and requiring no communication). We calculate and save the parameter contributions during the local convergence process of the PSM, which can then be locally **discarded** after its contribution has been computed. Then, each client trains on the new task with the local model and parameter contribution scores.\n\nThe model for an FL task is practically not as large as edge clients are mostly resource-limited, and the memory demand by PSM is relatively similar to existing methods like FedProx, as it manipulates over an extra model that has the same size as the federated model.\n\n**Possible Strategy:** If an FL task has large models and limited memory left on the edge, a prerequisite can be reasonably assumed satisfied: transmission capacity (probably after optimization) is sufficient for the system. In such a case, we can record the PSM model in the server, which is only downloaded and updated locally to compute parameter contributions and then uploaded to the server again.\n\n> **Q2. Combination of Gernerative AI and FedSSI.**\n\n**R2:**  Thank you for raising this concern. Generative AI has emerged as a prominent research area in recent years, as it leverages the generalization capabilities of LLMs to enhance task performance and drive productivity improvements significantly. However, even in centralized learning paradigms, continual learning for Generative AI remains hindered by multiple challenges. Unlike conventional backbone models that primarily face catastrophic forgetting, Generative AI systems more frequently encounter difficulties acquiring new knowledge. Additionally, the current deployment of Generative AI models through quantization compression on small edge devices complicates continual learning implementation under resource-constrained conditions. While the continual federated learning has not yet published studies specifically addressing Generative AI and FedSSI is designed for traditional CFL tasks, the growing deployment of edge devices equipped with Generative AI capabilities and their potential to collect real-world data from novel scenarios underscores the urgent need for such research.\n\n> **Q3. Dependence on SI Algorithm.**\n\n**R3:**  Thank you for raising this concern. Although the FedSSI algorithm uses PSM to improve the SI algorithm, the selection of the SI algorithm itself is not random. In our empirical experiments, we analyzed a vast number of existing CFL methods and techniques, as well as traditional CL techniques. The SI algorithm was chosen as the appropriate one among them, and it has been widely recognized for its efficiency and feasibility in most scenarios. We believe that this assumption is **acceptable**, similar to how many FL studies are based on CNN and ResNet series networks without considering the feasibility of techniques on a single Linear network.", "claims_and_evidence": "The claims in this paper are well-supported by clear and compelling evidence.", "methods_and_evaluation_criteria": "FedSSI is significant in addressing the current challenges in CFL by saving resources and alleviating heterogeneity.", "theoretical_claims": "The theoretical claims in FedSSI are supported by clear proofs.", "experimental_designs_or_analyses": "The paper presents a comprehensive evaluation with sufficient baselines across various datasets and scenarios.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "References are high-quality and sufficient.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow, with a clear motivation and a thorough discussion of the limitations of previous work.\n2. This paper is highly commendable for its pioneering exploration of CFL from the perspective of resource constraints. The research provides valuable insights and inspiration for future work.\n3. The proposed method FedSSI is well-motivated and technically sound.\n4. The experiment design is reasonable and comprehensive, and the analysis of results is thorough and exhaustive.\n\nWeaknesses:\n1. Although the proposed method avoids data rehearsal, introducing the PSM could add storage overhead that poses challenges for edge devices. The authors are supposed to discuss the storage costs or provide strategies to mitigate this issue.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Generative AI is an emerging topic. Only classification tasks have been studied and evaluated in the current paper. Can FedSSI generalize to generation tasks well, e.g., using diffusion models?\n2. The proposed method depends on SI. If SI has inherent limitations or does not perform well under certain conditions, how to ensure the performance of FedSSI?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Thank you for submitting your work to ICML 2025, and for your efforts in the rebuttal phase to clarify the reviewers' concerns. Since all the reviewers affirmed the novelty of the authors' work, I therefore recommend acceptance of the paper.
Decision: Accept (spotlight poster)