CAN: Leveraging Clients As Navigators for Generative Replay in Federated Continual Learning
Authors: Xuankun Rong
Jianshu Zhang
Kun He
Mang Ye
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Continual Learning
Primary area: social_aspects->privacy
TL;DR: A client-centric generative replay approach in federated continual learning.

Abstract: Generative replay (GR) has been extensively validated in continual learning as a mechanism to synthesize data and replay past knowledge to mitigate forgetting. 
By leveraging synthetic rather than real data for the replay, GR has been adopted in some federated continual learning (FCL) approaches to ensure the privacy of client-side data. 
While existing GR-based FCL approaches have introduced improvements, none of their enhancements specifically take into account the unique characteristics of federated learning settings. 
Beyond privacy constraints, what other fundamental aspects of federated learning should be explored in the context of FCL?
In this work, we explore the potential benefits that come from emphasizing the role of clients throughout the process.
We begin by highlighting two key observations: (a) Client Expertise Superiority, where clients, rather than the server, act as domain experts, and (b) Client Forgetting Variance, where heterogeneous data distributions across clients lead to varying levels of forgetting.
Building on these insights, we propose CAN (Clients As Navigators), highlighting the pivotal role of clients in both data synthesis and data replay. 
Extensive evaluations demonstrate that this client-centric approach achieves state-of-the-art performance. Notably, it requires a smaller buffer size, reducing storage overhead and enhancing computational efficiency.

CAN: Leveraging Clients As Navigators for Generative Replay in Federated Continual Learning
Xuankun Rong"
Jianshu Zhang *2 Kun He
Mang Ye
Abstract
a) Client Expertise Superiority
Dog
Logits
Generative replay (GR) has been extensively val- idated in continual learning as a mechanism to synthesize data and replay past knowledge to mit- igate forgetting. By leveraging synthetic rather than real data for the replay, GR has been adopted in some federated continual learning (FCL) ap- proaches to ensure the privacy of client-side data. While existing GR- CR-based FCL approaches have introduced improvements, none of their enhance- ments specifically take into account the unique characteristics of federated learning settings. Be- yond privacy constraints, what other fundamental aspects of federated learning should be explored in the context of FCL? 2 In this work, we explore the potential benefits that come from emphasizing the role of clients throughout the process. We begin by highlighting two key observations: (a) Client Expertise Superiority, where clients, rather than the server, act as domain experts, and (b) Client Forgetting Variance, where heterogeneous data distributions across clients lead varying levels of forgetting. Building on these insights, we pro- pose CAN (Clients As Navigators), highlighting the pivotal role of clients in both data synthesis and data replay. Extensive evaluations demon- strate that this client-centric approach achieves state-of-the-art performance. Notably, it requires smaller buffer size, reducing storage overhead and enhancing computational efficiency.
Fox
Server
Label
Logits
Fox
Chient
Lebel
b) Client Forgetting Variance
Acc
After
localizeed training
distribule
Client
Client
Client
Figure 1. a) Client Expertise Superiority (Sec. 3.1): Clients may possess accurate knowledge of specific classes compared the server, highlighting the need to utilize client expertise during synthetic data generation. h) Client Forgetting Variance (Sec. 3.2): The non-lID nature of client data leads to diverse forgetting pat- terns across clients, emphasizing the need for adaptive replay strategies tailored to each client.
1. Introduction
Federated Learning (FL) (Konečný, 2016; McMahan et al., 2017; Yang et al., 2019; Li et al., 2020a: Huang et al., 2024: 2023a) learning paradigm that utilizes distributed data across clients rather than relying centralized datasets. Clients train models locally on their own data, which are then aggregated into global model on the server (Li et al., 2020b; Karimireddy et al., 2020; Fallah et al.. 2020; Pi et al., 2023; Xie et al., 2022). However, one key challenge in this paradigm is that data distributions can change over time (Liu al., 2024; Fang et al., 2023: Huang al., 2023b; Gao al., 2022a: Jiang et al., 2022). For example, user preferences in mobile apps might change (Ma et al., 2022), sensor data collected by 1oT devices could vary with envi- ronmental conditions (Nguyen et al., 2021), or healthcare records may evolve as new medical trends emerge (Xu et al., 2021; Wang et al., 2023). These evolving distributions can lead catastrophic forgetting, a challenge that is particu-
'Equal contribution National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan Uni- versity. Wuhan, China "School of Cyber Science and Engineering, Wuhan University, Wuhan, China. Correspondence to: Mang Ye <yemang@ whu.edu.en..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CAN: Clients As Navigators
larly pronounced in the distributed nature of FL.
bined with newly received data for clients' local training. However, unlike previous methods, CAN takes client- centric approach, where clients navigate the generation of synthetic data and guide targeted replay.
To address forgetting, recent studies (Dong et al., 2022; Ma et al., 2022; Dai et al., 2024; Yang et al., 2024; Usmanova et al., 2021) have introduced Federated Continual Learning (FCL), adapting traditional Continual Learning (CL) meth- ods to the FL setting (Huszár, 2018; Aljundi et al., 2019; Zhao et al., 2020; Zhou et al., 2023). To address privacy concerns in FL, most approaches naturally adopt generative replay (Zhang et al., 2023; Tran et al., 2024; Mei et al., 2024; Liang et al., 2025), which leverages generator to synthesize data for replay, effectively mitigating forgetting. However, we observed that current approaches often make compromises to meet strict privacy constraints. This raises an important question: Beyond privacy constraints, what other fundamental aspects of federated learning should be explored in the context of FCL?
Specifically, our approach can be divided into two stages. The first stage, Expert-Driven Data Synthesis, focuses on generating high-quality synthetic data. For the first time, we position clients as experts, leveraging their specialized knowledge to guide the generator in producing accurate synthetic data. Additionally, to prevent synthetic data from overemphasizing only the most prominent features, we en- courage the generator to capture more subtle characteris- sees-those discernible to domain experts but often over- looked by less specialized models. This allows the synthetic data to be both finely detailed and comprehensive. The second stage is Adaptive Replay. It starts with measuring the unique forgetting patterns of each client, influenced by the varying degrees of interference between newly acquired knowledge and prior knowledge. This process enables us to create a distinct forgetting profile for each client. Next, we adjust the buffer's data composition according to each client's forgetting profile, tailoring the allocation of data within the buffer. By differentiating the buffer across clients, we enable a more targeted and effective replay process. Our contributions can be summarized as follows:
To fill this gap in the current FCL field, we shift our focus to crucial feature of FL: Clients (Yang et al., 2023; Shan- mugarasa et al., 2023; Fu et al., 2023; Fang & Ye, 2022: Yu et al., 2024b; Ma et al., 2025). By exploring the unique char- acteristics that clients can contribute, we aim to help FCL better leverage the distinctive features of the FL scenario. As illustrated in Fig. I, explore two key observations that have been largely overlooked in FCL approaches: 1 Client Expertise Superiority: Due to the non-lII nature of client data in FL settings (Zhao et al., 2018; Konečný et al., 2016; Huang et al., 2022; Ye et al., 2023), some clients naturally become domain experts in specific categories, es- pecially as the degree of non-IID increases (as visualized in Fig. 2). Although these clients may not perform well as the server in a general sense, they often possess more precise and specialized knowledge in particular areas. This makes them especially valuable for guiding the generation of high- quality synthetic data in those domains. However, current approaches focus solely on the server model's looge missing the opportunity to leverage this epient-specific exper- tise. @ Client Forgetting Variance: Due clients' unique local data distributions (Mendieta et al., 2022; Singhal et al., 2021; Yu et al., 2024a), they exhibit varying forgetting pat- terns. Despite this, existing methods use a uniform replay buffer, where each client receives an identical buffer with an even distribution of synthetic data samples across classes. This uniform allocation fails meet the unique needs of each client, who may benefit more from tailored data replay strategy mitigate forgetting.
We identify that existing FCL methods fall short in fully leveraging the unique characteristics of FL. For the first time, we introduce a client-centric perspective and propose two key observations that highlight the necessity of positioning clients as navigators in FCL.
We propose CAN that leverages the unique role of clients in FL. By positioning clients as navigators to guide the generation of synthetic data and adaptive replay tailored to each client's needs, CAN enables more efficient and targeted knowledge retention.
Through extensive experiments on CIFARIO0, Tinylm- agenet, and Imagenet100, we achieve sate-of-the-as (SOTA) performance, with ablation studies validating the effectiveness of our client-centric approach.
2. Related Work
Continual Learning. Methods continual learning can be broadly categorized into three approaches (De Lange et al., 2021; Wang et al., 2024a). Parameter isolation meth- ods (Yoon et al., 2017), such as Progressive Neural Net- works (PNN (Rusu et al., 2016)) and context-dependent gating (XdG (Masse et al., 2018)), segregate parameters to minimize task interference but face scalability issues as knowledge expands. Regularization approaches, including Elastic Weight Consolidation (EWC (Huszár, 2018)) and
Based on the two observations above, we propose CAN (Clients As Navigators), a more native FCL approach that leverages the unique role of clients. Our method follows the basic generative replay framework by first utilizing the teacher model's knowledge to train a generator for synthe- sizing data (Zhang et al., 2023; Tran al., 2024; Shin al., 2017; Wang et al., 2024b). The synthetic data is then com-
2

---

CAN: Clients As Navigators
Non-lID
80
R
50
50
30
50
10
11
16
18
19
10
15
16
17
10
19
Classes
Classes
Non-I 0.5
Non-IID
F3
50
92 30
92
30
83
10
oz
76
10
15
16
Classen
Figure 2. This visualization shows accuracy across different classes under IID and Non-IID (0.1. 0.5, 1.0) settings. The server (S) and clients (C1 to CS) are compared, with the highest accuracy for each class highlighted in bold. Certain clients outperform the server on specific classes, demonstrating their potential class-specific experts. This observation supports the strategy of leveraging client expertise to improve performance on data synthesis. Refer to Sec. 3.1 for details.
Learning Without Forgetting (LwF (Li & Hoiem, 2017)), protect previously acquired knowledge by limiting parame- ter updates. However, their effectiveness depends heavily on accurately identifying important parameters. Data replay methods (Shin et al., 2017; Zhai et al., 2019; Masarczyk & Tautkute, 2020), like Cognitive Replay (CORE (Zhang et al., 2024)), Experience Replay (ER (Rolnick et al., 2019)) and Incremental Classifier and Representation Learning (iCaRL (Rebuffi et al., 2017)). use a replay buffer to re- view historical data.
3. Observations
Clients play a crucial role in FL scenarios, which brings two defining characteristies: 1) the privacy of client-side data, and 2) the non-IlI) (uneven) distribution of data across clients. In transitioning from CL to FCL, existing methods have mainly focused on ensuring data privacy by keeping local data within clients. However, they often overlook the non-lll) distribution of client data. In this section, we shift our focus to this underutilized feature, discussing the potential benefits it can bring to FCL.
Federated Continual Learning. Federated Continual Learning (FCL) (Bui et al., 2018; Dong et al., 2022; Ma al., 2022; Yoon et al., 2021; Gao et al., 2024; Yu al., 2024c) extends Continual Learning (CL.) by integrating Fed- erated Learning (FL) principles (Grammenos et al., 2020; Benmalek et al., 2022: Li et al., 2021), allowing global mod- els to adapt to evolving local data while preserving prior knowledge and ensuring privacy. Existing approaches can be grouped into several categories: 1) regularization-ban methods like FedCury (Shoham et al.. 2019), which use penalty terms to preserve critical parameters; 2) knowledge distillation methods such as CFeD (Ma et al., 2022) and GLFC (Dong et al., 2022), which transfer essential knowl- edge between tasks to mitigate forgetting: 3) exemplar-bart based methods like FedWelT (Yoon et al., 2021), which retain key samples for rehearsal during new task training; 4) generative replay-based methods, including TARGET (Zhang et al., 2023), LANDER (Tran et al., 2024), and DDDR (Liang et al., 2025; Mei et al., 2024), primarily build upon stan- dard generative replay, with most improvements focusing integrating pretrained models, such as CLIP and diffusion models, to enhance performance. In contrast, we re-examine the advantages that clients can offer in FL scenarios and propose a more Fl.-native approach.
3.1. Client Expertise Superiority.
As illustrated in Fig. 2, under an IID data distribution, both clients and the server achieve comparable performance, with each class exhibiting relatively high accuracy. However, in practical FL scenarios, client data distributions often vary significantly, leading to inherently non-IID settings. We observe that as the level of non-IID increases, clients en- counter increasing number of low-accuracy instances. At the same time, certain clients also achieve notably higher accuracy on specific classes. This trend is more pronounced in the non-lID (0.1) setting, where strong polarization effect emerges-individun individual clients demonstrate widely vary- ing performance across different classes, with some signifi- cantly outperforming the server on specific categories.
Leveraging these perffrrin clients as domain ex- perts for certain classes offers a more effective strategy than relying solely on server knowledge. By incorporating their superior knowledge, we can enhance the generation of synthetic data, ensuring a better alignment with real- world FL scenarios. Furthermore, during server-side aggregation, these expert client models can be integrated without ex- posing local data, thereby maintaining privacy. Notably, this approach incurs minimal storage overhead, as it only
3

---

CAN: Clients As Navigators
A) Server-side: Expert-Driven Driven Data Synthesis
B) Client-side: Adaptive Replay
Synthetic Data Initialization
Expertise Knowledge Mapping
Forgetting Pattern Profiling
y
g
X 1ym
C
Num
Deww
Zy o(y)
Aggregant
M(y y By
Task
Real
Synthetic
reallocate buffer space based on forgetting pattern
X syn
Expertise-Guided Refinement
Adaptive Buffer Training
Expertise Navigation
Feature Gap Amplification
Joint Loss Function
adaptive buffer
syy
gap
= 2) Leaar 2 gup
Num
Dillw
Truining Data for Client Task
II
Xyys's space
Figure 3. Overview of CAN, consisting of main stages: Expert-Driven Data Synthesis (Sec. 4.2) and Adaptive Replay (Sec. 4.3). A), we start with initializing synthetic data. Then we identify the most suitable client experts to navigate the data generation. The process is further refined using joint loss that ensures both accurate and detailed data. B), the replay process begins with analyzing how data affects each client's 's retention of old knowledge. This profiling informs Adaptive Buffer Training, where we personalize the internal data distribution of replay buffers based on the unique forgetting pattems each client, enabling more tailored and effective replay.
introduces an additional inference step into the existing aggregation process.
ditionally, we personalize each client's 's replay buffer based on their forgetting patterns, enhancing replay effectiveness. The framework of CAN illustrated in Fig.
3.2. Client Forgetting Variance.
4.2. Expert-Driven Data Synthesis
In FCL scenarios, the degree of interference with previously learned knowledge varies significantly across clients due to their diverse data distributions. Each client's unique lo- cal data introduces different challenges in balancing and old knowledge, leading to varying degrees of conffict. For instance, some clients may encounter frequent overlaps between new and previously seen data, causing a higher risk of interference, while others, with less overlap, might experience distinct but subtle shifts that still lead to forget- ting. This variability naturally results in different levels of interference and forgetting among clients.
4.2.1. SYNTHETIC DATA INITIALIZATION
Due to privacy concerns, the real data owned by clients inaccessible on the server side. To overcome this limita- tion, we introduce a generator, denoted as G, produce synthetic data Xsya that aims to approximate the real data X in distribution. This enables data replay using synthetic data, preserving knowledge from previous tasks while addressing privacy concerns. Here, we first adopt sampling function Φ(y) that applics a noise transformation to obtain Zy. Then, the generator & takes noise 2y as input to generate synthetic data Tsyn corresponding that class. The process can be formulated
Consequently, fittssll-fitt-lll replay strategy is insuffi- cient to address these differences. Instead, replay buffers should be adaptive, tailored to each client's specific for- getting patterns. Recognizing and adapting to this client- specific variability is essential for effective FCL.
Xsyn Px (Mayn G(zy) E, + ((v), y }}, (I)
where Px represents the distribution of the real data X, and ((y) is sampling function that generates the input noise Zy for each label y. Additionally, the synthetic data does not visually resemble the real data, but remains unrecognizable and serves as meta data.
4. CAN: Clients As Navigators
diseuss later in Sec. 5.5,
4.1. Overview
In this section, we introduce CAN, native FCL approach that leverages clients' roles in FL to guide replay. We treat clients as experts to train generator that synthesizes fine- grained data, effectively preserving prior knowledge. Ad-
4.2.2. EXPERTISE KNOWLEDGE MAPPING
To improve the quality of Xsya it is essential to have an effective guide for optimizing g. In light of our observa-

---

CAN: Clients As Navigators
tion of Client Expertise Superiority in Sec. 3.1, we stress the importance of leveraging the expertise of the clients. Specifically, for each class y E y' we utilize an expertise map to identify the most suitable expert from clients C. The expertise map M.)) is defined as follows:
data be identifiable by the expert, but it does not guaran- tee that all relevant features are captured. Specifically, the generator tends to focus on the most prominent and easily recognizable features to satisfy the expert, while ignoring the more nuanced and detailed characteristics that the expert is also capable of recognizing. Consequently, the generated data often lack subtle but important features, resulting in re- duced diversity and limiting their representational richness.
M(y) : E ~ cy- = arg max Acc(c, v);
(2)
CEC
here, MM) selects the expert by with the highest accuracy for class y. Once the generator 9 produces the synthetic data (Faya: the selected expert by is then used to guide the subsequent data refinement process.
To address this, we introduce reference model II, trained only on synthetic data that can be understood as a knowl- edge representation under the guidance of expert knowledge. Therefore, IT captures not only the prominent features but also includes some of the more subtle and nuanced charac- teristics that are often overlooked by less specialized models. In contrast, the server model S, while proficient at identify- ing prominent features, lacks the expert's capacity to capture these finer ones. By encouraging discrepancies between the predictions of the server model S and the reference model II on Xayn. we guide the generator G to produce synthetic data that not only captures the prominent features but also the fine- fine-grained details. The features recognized by the reference model II can be represented as set Fn (x), and those recognized by the server model S as a set (1)
4.2.3. EUPERTISE-GUIDER REFINEMENT
In this part, our goal is to refine synthetic data to not only capture the core characteristics of the original data but also encompass diverse and nuanced features. To achieve this, we implement a cooperative refinement process with two key components: Expertise Navigation, which ensures ac- curate and class-specific representation, and Gap-Driven Enhancement, which promotes richer details.
Expertise Navigation. To ensure that the synthetic data accurately represents the characteristics of its corresponding class, we adopt a strategy that treats a well-trained model as an expert. This expert facilitates knowledge transfer and guides the generation of representative synthetic data. However, unlike previous approaches that rely solely on the server the expert, our method takes full advantage of the FL setting by positioning clients as experts. This allows to harness the unique strengths of different clients, who may have a deeper understanding of particular data distributions compared to the server.
Our objective is to let synthetic data capture the features recognized by II but missed by S, which can be understood as the set difference FT(()) \ ((( in discrete mathemat- ics theory. To achieve this, we aim to minimize the KL divergence (Van Erven & Harremos, 2014) between the predictions of II and S on the test set, focusing only on cases where II makes correct predictions. The process is formalized as:
Lapp (e [Tn (LL(f(())) Il [s(x))), (4)
We first use the expertise map M(-) to select the most suitable expert for each class, ensuring more accurate and targeted refinement. The selected expert ey then generates predicted label for the synthetic data Zsyn corresponding to class y. Given that this expert is highly proficient in recognizing its associated class, the predicted label serves as a reliable proxy for the ground truth, denoted as Vgt-
where B(a.y))- denotes the expectation over X Here, (u( and fz( represent the predicted label by IT and S, respectively. The indicator function In equals 1 if II's pre- diction matches the true label y, and 0 otherwise, ensuring that discrepancies are encouraged only when II correctly identifies the features.
Joint Loss Function. To comprchensively optimize the generator G, we define joint loss function that combines both Lusv and Egpp. The navigation loss Caw aligns the syn- thetic data with the expert's guidance to accurately represent the target class, while the gap-driven loss Cgap promotes di- versity by encouraging the capture of more nuanced details. The overall objective for optimizing G:
When the synthetic data accurately captures the characteris- tics of class y, the expert model e, should be able classify it correctly. Therefore, we compute a cross cross-entropy loss be- tween the predicted class Y/tt from the expert and the target class y, formulated as follows:
L = - llg p(ug- = y ssy))
(3)
La = (1 (1- A)- Lany + A Ceap
(5)
through Luwv, the generator G can be optimized to generate synthetic data that effectively represents the target class.
here, A is hyperparameter balances between the two loss terms, ensuring that the synthetic data is both representative and diverse. Further discussion of different À can be found in Appendix A.3.2.
Feature Gap Amplification. It is important to note that the loss function Lun in Eq. (3) only requires the synthetic
5

---

CAN: Clients As Navigators
4.3. Adaptive Replay
where a and apre balance learning from new data De and retaining past knowledge via buffered data Be- The specific selection of a is detailed in Appendix A.3.4.
A common approach to mitigate catastrophic forgetting in continual learning is to mix previously data, representing old knowledge, with newly incoming data during training. However, in FL. settings, distributed clients have highly het- erogeneous local data, contributing the Client Forgeting Variance phenomenon discussed in Sec. 3.2. This makes naive replay without considering the unique role of clients unlikely to yield optimal results. To improve the effective- ness of replay in the FL scenario, we propose adaptive re- play strategy: 1) Forgetting Pattern Profiling, which profiles the unique forgetting patterns of each client; 2) Adaptive Buffer Training, which reallocate personalized buffer for each client during replay.
5. Experiments
5.1. Experimental Setup
Settings and Metries. We evaluate CAN in both IID and diverse non-III scenarios using two key metries: Average Accuracy (AA) and Average Forgetting Score (F), detailed in Appendix A.I. Additionally, we also examine the impact of different replay buffer sizes to assess the efficiency of our approach under resource constraints. Furthermore, to provide a comprehensive evaluation, we analyze accuracy trajectory across sequential tasks and retention of initial task accuracy, offering deeper insights into the model's capacity to maintain performance throughout training.
Forgetting Pattern Profiling. Initially, a replay buffer pre-allocated to all clients, with synthetic data evenly distributed across all previous tasks T:
Baselines. We compare CAN with several methods: Vanilla, FedEWC, FedLwF, FedWelT, TARGET, and LANDER. Vanilla sequentially finetunes each new class, FedEWC (Kirkpatrick et al., 2017) mitigates forgetting by penalizing changes important parameters using elastic weight consolidation (EWC). FedLwF (Li & Hoiem, 2017) applies knowledge distillation to retain past knowledge. Fed- WeIT (Yoon et al., 2021) preserves crucial weights in fed- erated settings. TARGET (Zhang et al., 2023) generates synthetic data for replay using the server model's knowl- edge, avoiding real data access. LANDER (Tran al., 2024), an extension of TARGET, introduces feature anchors to improve synthetic data clustering. Additionally, we com- pare CAN with DDDR (Liang et al., 2025), which leverages a pretrained diffusion model (see Appendix B).
B == (x ( D*) 1 T, 1
B [T]
(6)
where |- denotes the size of a set. Each client then trains on the replay buffer B alongside the new local data 'D for r iterations. The choice of r controls the balance between efficient training and capturing the impact of the new task, as detailed in Appendix A.3.3.
After training, we evaluate the accuracy of each client on each old task denoted Acc to determine the forgetting pattern for each client. The forgetting weight w², for client on old task t is calculated as follows:
max(AAc tET,
where is small predefined threshold to prevent numerical instability when the accuracy is close zero. A larger we indicates greater forgetting, suggesting that more data is needed for client c'sreplay on task l.
5.2. Main Results on CIFAR100
Tab. compares various methods on CIFAR 100 (Krizhevsky & Hinton, 2009) across IID and various non-IID settings, where our method consistently outperforms others across all settings. As expected, Vanilla sequentially fine-tunes new tasks, leading to severe forgetting. Among non-replay meth- ods like FedEWC (Kirkpatrick et al., 2017), FedWeIT (Yoon et al., 2021), and FedLwF (Li & Hoiem, 2017). forgetting is somewhat alleviated, but these approaches struggle un- der non-lID conditions due to the absence of synthetic data replay. Replay- methods like TARGET (Zhang et al., 2023) and LANDER (Tran et al., 2024), which serve as our main baselines, use synthetic data replay. TARGET relies server-side knowledge with larger buffer but exhibits high forgetting in non-llI) settings. LANDER introduces feature anchors to improve clustering, but it struggles to match our method's 's performance at the same buffer size. In fact, CAN outperforms LANDER even with smaller buffers, highlighting CAN's efficiency in terms of computational
Adaptive Buffer Training. Next, we implement a simple adaptive strategy to allocate more buffer space to tasks with higher forgetting weights. The adaptive replay buffer B, for client e is defined as:
B. = (X* )t) 1
[B], (8)
EVer w
where wi represents the forgetting weight for task L on client and |x is the size of the buffer allocated to task t after adaptation. Following this, each client e trains on their personalized buffer B. alongside their new local data D.. Finally, the loss of the client after task defined as:
L. anew [CE] + af Es. KKL].
6

---

CAN: Clients As Navigators
Table Performance on CIFARIO0 for 5-task and 10-task settings under IID and various non-lID scenarios. For TARGET, we replicate it using buffer size of 2,560 while following its default settings. Our main comparison is with LANDER, exploring different buffer sizes:
1,250 1,000, 750 and 500. The results demonstrate that CAN consistently outperforms LANDER at equivalent buffer sizes and achieves comparable performance even with smaller buffers, highlighting its efficiency and effectiveness in maintaining high accuracy (AA and reducing forgetting score (F), Further analysis can be found in Sec. 5.2.
5 Tasks
10 Tasks
Method
IID
NII(()
(O.5
(0.1
IID
NII((1
(O.5(
(0.1
(1) F (1)
(1)) F (
(t) F (W)
(
J (L)
(
J (L)
(1)
F (1)
A (T) F F()
(
F (L)
Vanila
16.12 78.12
16,33
77.59
15.49
74.95
15.69
72.73
7.83
75.89
8.45
74.89
7.64
71.52
8.04
65.27
FedEWC
16.51
71.12
16.06
68.02
16.86
62.40
17.69
65.84
8.01
65.06
8.84
62.14
8.04
65.23
7.61
58.79
Fed
28.45
52.12
28.56
49.84
24.57
45.96
20.39
43.18
19.68
45.96
15.45
48.54
Fe
30.61
45.32
42.71 30.94
27.59
41.25
35.95
29.04
23.27
37.71
21.16
41.03
17.98
45.23
18.39
43.92
TARGET
36.31
32.23
34.89
34.48
39.23 33.33
28.32
38.23
24.76
35.45
22.85
38.25
20.71
42.23
19.25
45.23
LANDER
45.92
37.23
44.51
39.47
41.63
39.23
37.94
38.18
31.29
46.02
31.07
50.67
26.93
50.34
21.95
42.65
CAN
48.16
33.26
34.61 45.93
36.14 42.16
32.21 39.88
32.52 34.41
32.59
32.93
28.06
33.42
23.60
26.11
LANDER
43.99
40.71
43.46
41.70
40.85
41.49
37.65
38.89
30.25
47.83
28.82
53.64
25.46
52.65
20.37
44.26
CAN
35.01 46.16
27.80 44.82
37.56 42.35
38.91
35.10
33.43 33.96
31.52
34.18
27.32
28.72
22.09
31.75
LANDER
44.72 41.56
40.90
45.94
38.30
45.70
37.06
41.17
29.96
52.46
27.39
55.93
23.23
53.18
19.86
48.54
CAN
32.42 46.15
43.15
40.22
41.56
30.44
37.14
37.32
33.71
36.67
30.66
36.54
25.76
30.05
21.31
32.90
LANDER
36.99
51.87
36.41
52.74
35.63
49.80
33.34
46.59
27.00
56.70
23.86
46.71
22.16
55.83
19.00
52.60
CAN
29.51 44.92
43.64
34.34
41.04
34.65
35.38
43.63
31.83
37.34
27.13
42.75
24.99
32.87
21.07
33,91
NIID(1.))
NIID(0.5)
NIID(0.1)
Accuracy 196)
26
28
24
14
24
22
NEAT
10
TedueT
18
CAN
CNN
16
2500
5000
10000
26
25
5000 Buffer Size
10000
25:00
5000
10000
26
2500
5000 Buffer Size
10600
Buffer Size
Butfer Size
IID
NIDD(1.0)
NII((O.5)
NIID(0.1)
10
18
18-
18-
Forgetting 5008
34-
JAN44
32
Forgetting
Forgetting
28
28
26
26
24
24
14
72-
22-
2500
5000
1000e
20
25.00
5000
13h00
25.00
5000
13600
20
2500
5000
100.00
Buffer Size
Buffer Size
Buffer Size
Buffer Size
Figure Results on Tiny TinyImageNet across 5 tasks under various data settings (IID, NIID 1.0, NIDD 0.5, and NIID 0.1). The first row shows accuracy (f), while the second row displays the forgetting
(L). We compare CAN with LANDER different buffer sizes (2,500, 5,000, and 10,000 samples) and include FedLwF as a bascline (the dashed line). CAN consistently outperforms other methods, demonstrating superior accuracy and lower forgetting. Please see Sec. 5.3 for detailed discussion.
costs. This demonstrates CAN's efficiency, effectively lever- aging client-specific expertise and adaptive buffer allocation to maintain robust performance.
tions. Notably, we observe a "V" phenomenon with CAN: as buffer size increases, accuracy initially dips before rising, while forgetting follows an inverted "V" pattern, suggesting trade-off between data quality and quantity. For LANDER, larger buffers compensate for lower data quality, whereas CAN's high-quality synthetic data benefits less from simply adding more samples. This suggests future improvements should focus on further enhancing data quality to better leverage larger buffers and optimize FCL performance in federated settings.
5.3. Results on More Complicated Settings
As shown in Fig. 4, CAN consistently achieves otate-of- the-art performance on TinylmageNet (Le Yang, 2015),
particularly non-III settings. It outperforms both LAN- DER (Tran et al., 2024) and FedLwWF (Li & Hoiem, 2017), demonstrating robustness in handling diverse data distribu-

---

CAN: Clients As Navigators
Table Average accuracy trajectory measured after the com- pletion of each task on ImageNeet10 over a sequence of 5 tasks. CAN consistently outperforms LANDER across all tasks and non- IID settings, demonstrating its effectiveness in maintaining higher accuracy throughout the sequence. Please refer to Sec. 5.3.
45
w/o (E+A)
w/o
w/o E
purs
Accuracy
40
35
Settings
Method
Task 2
Task 3
Task 4
Task 5
Avg
30-
LANDER
40.40
31.30
23.92
17.48
28.28
NIID(0.5)
CAN (Ours)
50.15
40.87
32.70
26.52
37.56
IID
NII((1.0)
NII(O..5)
NIDDD.11)
LANDER
26.20
15.90
10.75
7.02
14.97
1)
CAN (Ours)
40.90
31,00
23.30
17.50
28.18
Figure 5. Ablation study exploring the importance of client exper- tise in data synthesis (E) and adaptive replay (A) across different settings, as discussed in Sec. 5.4, highlighting their critical roles.
Table 3. Retention of final accuracy on the first task after the completion all tasks on CIFARIO0. (#) indicate the difference compared to CAN with 500 buffer size. CAN consistently outper- forms LANDER across all buffer sizes, even when LANDER uses
twice much data. Please refer to Sec. 5.3 for further analysis.
Method
IID
NIIDLD))
NIID(0.5)
NIID(0.1)
LANDER 5500
18.20 (2020
14.65 (19.40)
14.55 (11000)
15.75 (6.0))
LANDER *750
24.65 (1177))
21.30 (12.75)
20.35
(11220)
20.20 (1:55)
Figure 6. Visualization of the synthetic data generated by CAN. The synthetic data represents features in the form of meta data, abstracting class distributions without resembling real samples, ensuring privacy preservation. Details refer to Sec. 5.5.
LANDER #1000
29.70 (3.0))
26.95 67710)
23.25 63.30)
21.80 (.005)
LANDER #1250
33.05 (3335
28.20
24.35 (4:20)
22.25
CAN *500
38.40
34.05
30.55
21.75
To further evaluate the robustness of our method, we investigate its accuracy tra- jectory the more challenging ImageNet1 (Russakovsky et al., 2015) dataset over a sequence of five consecutive tasks. As presented in Tab. 2, CAN consistently outperforms LAN- DER (Tran et al., 2024) across all tasks, exhibiting a sub- stantial improvement in average accuracy. This performance gap becomes even more pronounced under the more chal- lenging NIID(O.1) 1) setting, where CAN achieves an average accuracy of 28.18%, significantly exceeding LANDER's s 14.97%. These results highlight the effectiveness of our client-centric approach in maintaining strong performance across diverse learning conditions.
Accuracy Trajectory on ImageNet.
uniform buffer allocation across tasks. These findings high- light two critical factors for replay-based FCL: 1) the quality of synthetic data and 2) the effectiveness of buffer allocation. Notably, the removal of adaptive replay leads to the most substantial accuracy drop, underscoring the necessity of tai- lored replay strategics to mitigate client-specific forgetting patterns. Furthermore, the exclusion of client expertise also degrades performance, demonstrating its essential role in enhancing synthetic data quality.
5.5. Privacy Analysis of CAN
CAN builds upon the generative replay FCL framework, similar to prior works like TARGET (Zhang et al., 2023) and LANDER (Tran et al., 2024). It leverages client models on the server side for knowledge transfer, synthesizing high- quality representations without direct access to original data and without introducing additional privacy concerns. As shown in Fig. 6, the synthetic data exhibits distinct visual differences from real data, serving as abstract representa- tions of class distributions. Throughout the entire process, we ensure that each client's 's private data remains local and inaccessible to others. The synthetic data distributed for replay serves only as meta-level representations, effectively capturing features while remaining free from any identifi- able real-data characteristics, thereby safeguarding privacy.
Retention of Initial Task Accuracy. Tab. 3 presents the final accuracy the initial task, offering a targeted eval- uation of how well FCL methods retain early knowledge. Unlike the average accuracy across all tasks, which may obscure the forgetting patterns of individual tasks, this mea- sure directly reflects a model's ' ability to preserve knowledge. CAN consistently outperforms LANDER, even when using smaller buffer sizes. Notably, CAN achieves superior reten- tion of initial task accuracy even as LANDER utilizes buffer sizes up to twice as large, underscoring CAN's efficiency and effectiveness in mitigating forgetting.
5.4. Ablation Studies
6. Conclusion
As shown in Fig. 5, we analyze the impact of removing key components from our method. The variant that ex- cludes both client expertise and adaptive replay exhibits a significant performance decline. Specifically, without client expertise. data generation depends solely on server knowledge, while the absence of adaptive replay results in
In this paper, we introduce CAN, a client-centric approach that enhances synthetic data quality and mitigates forget- ting through adaptive replay. Unlike prior methods, CAN tailors replay strategies to client-specific needs, improving knowledge retention in FCL. Experiments show that CAN
8

---

CAN: Clients As Navigators
outperforms existing baselines, even with smaller buffers, highlighting the effectiveness of client-aware strategies in FCL and paving the way for future research.
Fu, L., Zhang, H., Gao, G., Zhang, M., and Liu, X. Client selection in federated learning: Principles, challenges, and opportunities. IEEE IoTJ, 2023.
Gao, L., Fu, H., Li, L., Chen, Y., Xu, M., and Xu, C.-Z. Fedde: Federated learning with non-iid data via local drift decoupling and correction. CVPR, pp. 10112- 10121, 2022a.
Acknowledgement
This work is supported by National Natural Science Founda- tion of China under Grant (62361 166629, 62176188), and the National Key Research and Development Program of China (2024YFC3308400) The supercomputing system at the Supercompuring Center of Wuhan University supported the numerical calculations in this paper.
Gao, Q. Zhao, C., Ghanem, B., and Zhang. R-dfcil: Relation-guided representation learning for data-free class incremental learning. In European Conference on Computer Vision, pp. 423 439. Springer, 2022b.
Impact Statement
Gao, X., Yang, X., Yu, H., Kang, Y., and Li, T. Fedprok: Trustworthy federated class-incremental learning via pro- totypical feature knowledge transfer. In CVPR, pp. 4205- 4214, 2024.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Grammenos, A., Mendoza Smith, R., Crowcroft, J., and
Mascolo, C. Federated principal component analysis. NeurIPS, pp. 6453-6464, 2020.
References
Aljundi, R., Kelchtermans, K., and Tuytelaars, T. Task-free continual learning. In CVPR, pp. 11254-11263, 2019.
Huang, W., Ye, M., and Du, B. Learn from others and be
yourself in heterogeneous federated learning. In CVPR, 2022.
Benmalek, M., Benrekia, M. A., and Challal, Y. Security of federated learning: Attacks, defensive mechanisms, and challenges. RIA, pp. 49-59, 2022.
Huang, W., Ye, M., Shi, Z., and Du, B. Generalizable heterogeneous federated cross-correlation and instance similarity learning. TPAMI, 2023a.
Bui, T. D., Nguyen, C. V., Swaroop, S., and Turner, R. E. Partitioned variational inference: unified framework encompassing federated and continual learning, arXiv preprint ar 181 I. /206, 2018
Huang, W. Ye, M., Shi, Z., Li, H., and Du, B. Rethinking federated learning with domain shift: A prototype view. In CVPR, 2023b.
Dai, S., Sohn, J.-y.. Chen, Y., Alam, S., Balakrishnan, R., Banerjee, S., Himayat, N., and Lee, K. Buffer-based gradient projection for continual federated learning. arXiv preprint arXiv:2409.0 5585, 2024.
Huang, W., Ye, M., Shi, Z., Wan, G., Li, H., Du, B., and Yang, Q. A federated learning for generalization, robust- ness, fairness: A survey and benchmark. IEEE TPAM1, 2024.
De Lange. M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A contin- ual learning survey: Defying forgetting in classification tasks. IEEE TPAMI, pp. 3366-3385, 2021.
Huszár, F. Note the quadratic penalties in elastic weight consolidation. PNAS, pp. F2496-FE2497 2018.
Jiang, M., Wang, 7., and Dou, Q. Harmofl: Harmonizing local and global drifts in federated learning on heteroge- neous medical images. In AAAI, pp. 1087-1095, 2022.
Dong. J.. Wang, L., Fang, Z., Sun, G., Xu, S., Wang, X., and Zhu, Q. Federated class- dass-incremental learning. In CVPR, pp. 10164-10173, 2022.
Karimireddy, P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pp. 5132- 5143. PMLR, 2020.
Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning: A meta-learning approach. arXiv preprint arXiv:2002.07948, 2020.
Fang, X. and Ye, M. Robust federated learning with noisy and heterogeneous clients. In CVPR, 2022.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., et al. Overcoming catastrophic forgetting in neural networks. PNAS, pp. 3521-3526, 2017.
Fang, X., Ye, M., and Yang, X. Robust heterogeneous federated learning under data corruption. In ICCV, pp. 5020-5030, 2023.

---

CAN: Clients As Navigators
Konečný, J. Federated learning: Strategies for im- proving communication efficiency. arXiv preprint arXiv::610.05492, 2016.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Areas, B. A. Communication-effccientt learning of deep networks from decentralized data. AISTATS, pp. 1273- 1282. PMLR, 2017.
Konečný, J., McMahan, H. B., Ramage, D., and Richtárik, Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXtv:1610.02527,, 2016.
Mei, Y., Yuan, L., Han, D.-J., Chan, S., Brinton, G., and Lan, Using diffusion models as generative replay in continual federated learning-whe will happen? arXiv preprint 1..xi::2111.611111 2024.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Mendieta, M., Yang, T., Wang, P., Lee, M., Ding, Z., and Chen, C. Local learning matters: Rethinking data hetero- geneity in federated learning. In CVPR. pp. 8397-8406, 2022.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 23IN, pp. 2015.
Nguyen, D. C.,, Ding, M., Pathirana, N., Seneviratne, A., Li, J., and Poor, H. V. Federated learning for internet of things: comprehensive survey. IEEE CST, pp. 1622- 1658, 2021.
Li, Q. Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., and He, B. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE TKDE, pp. 3347-3366 2021.
Pi, R., Zhang, W. Xie, Y., Gao, J., Wang, X., Kim, S., and Chen, Q. Dynafed: Tackling client data heterogeneity with global dynamics. CVPR, pp. 12177-12186, 2023.
Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Feder- ated learning: Challenges, methods, and future directions. IEEE SPM, pp. 50-60, 2020a.
Rebuffi, S.-A., Kolesmikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In CVPR, pp. 2001-2011 2017.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proc. Mach. Learn. Syst., 2:429-450, 2020b.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. NeurlPS, 32, 2019.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE TPAMI, pp. 2935-2947, 2017.
Liang, J., Zhong, J., Gu, H., Lu, Z., Tang, X., Dai, G., Huang, S., Fan, L., and Yang, Q. Diffusion-driven data replay: A novel approach to combat forgetting in feder- ated class continual learning. European Conference on Computer Vision, pp. 303-319. Springer, 2025.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, C., and Fei-Fei, L. Imagenet large scale visual recognition challenge. IJCV, pp. 21 1-222, 2015.
Liu, Y., Kang, Y., Zou, T., Pu, Y., He, Y., Ye, X., Ouyang, Y., Zhang, Y.-Q., and Yang, Q. Vertical federated learning: Concepts, advances, and challenges. IEEE TKDE, pp. 3615-3634, 2024.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. arXiv preprint arXiv:1606.0m 1XXv:1600004671. 2016.
Ma, Y., Xie, Z., Wang, J., Chen, K., and Shou, L. Continual federated learning based on knowledge distillation. In IJCAI, pp. 2182-2188, 2022.
Shanmugarasa, Y., Paik, H.-y., Kanhere, S. S., and Zhu, L. A systematic review of federated learning from clients' perspective: challenges and solutions. Artif. Intell. Rev., pp. 1773-1827, 2023.
Ma, Y., Dai, W., Huang, W., and Chen, J. Geometric knowledge-guidd localized global distribution alignment for federated learning. In CVPR, 2025.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. NewrIPS, 30, 2017.
Masarczyk, W. and Tautkute, I. Reducing catastrophic for- getting with learning on synthetic data. In CVPR, pp. 252-253, 2020.
Shoham, N., Avidor, T., Keren, A., Israel, N., Benditkis, D., Mor-Yosef, I.,, and Zeitak, I. Overcoming forgetting in federated learning on non-iid data. arXiv preprint 1rXXh:1900007766 2019.
Masse, N. Y., Grant, G. D., and Freedman, J. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. PNAS, pp. E10467-E10475, 2018.
Singhal, K., Sidahmed, H., Garrett, Z., Wu, S., Rush, J., and Prakash, Federated reconstruction: Partially local federated learning. NewrlPS, 34:1 12:11220-1122, 2021.
10

---

CAN: Clients As Navigators
Tran, M.-T., Le, T., Le, X.-M., Harandi, M., and Phung. D. Text-enhanced data-free approach for federated class- incremental learning. In CVPR, pp. 23870-23880, 2024.
Yu, H., Yang. X., Gao, X., Feng, Y., Wang, H., Kang, Y., and Li, T. Overcoming spatial-temporal catastrophic forgetting for federated class-incremental learning. In ACMMM, pp. 2280- 2024a.
Usmanova, A., Portet, F., Lalanda, P., and Vega, G. A distillation-bass approach integrating continual learn- ing and federated learning for pervasive services. arXiv preprint Xii::2009004997 2021.
Yu, H., Yang, X., Gao, X., Kang, Y., Wang, H., Zhang, J., and Li, T. Personalized federated continual learning via multi-granulaate prompt. In KDD, pp. 4023-4034 2024b.
Van Erven, T. and Harremos, P. Rényi divergence and kullback-leibler divergence. IEEE TIT, pp. 3797-3820, 2014.
Yu, H., Yang, X., Zhang, L., Gu, H., Li, T., Fan, L., and Yang. Q. Addressing spatial-temporal data heterogeneity in federated continual learning via tail anchor. arXiv preprint 2r.iv:2412..4 18355, 2024c.
Wang, L., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE TPAMI, 2024a.
Zhai, M.. Chen, L., Tung, F., He, J., Nawhal, M., and Mori, G. Lifelong gan: Continual learning for conditional im- age generation. In CVPR, pp.
Wang. N., Deng, Y., Feng, W. Yin. J. and Ng, S.- K. Data-free federated class incremental learning with diffusion biffus generative memory. arXiv preprint arXiv:2405. 1.0000..77455 7745,, 2024b.
2019.
Zhang, J., Chen, C., Zhuang, W., and Lyu, L. Target: Fed- erated class-continual learning via exemplar-fi distilla- tion. In CVPR, pp. 4782-4793, 2023.
Wang, W., Li, X., Qiu, X., Zhang, X., Brusic, V., and Zhao, A privacy preserving framework for federated learning in smart healthcare systems. IPM, pp. 103167, 2023.
Zhang, J., Fu, Y., Peng, Z., Yao, D., and He, K. Core: Mitigating catastrophic forgetting continual learning through cognitive replay. arXiv preprint arXiv:2402.00 348, 2024.
Xie, Y., Zhang, W., Pi, R., Wu, F., Chen, Q. Xie, X., and Kim, S. Optimizing server-side aggregation for robust federated learning via subspace training. arXiv preprint arXiv:2211.05554, 2022.
Zhao, B., Xiao, X., Gan, G., Zhang, B., and Xia, S.-T. Main- taining discrimination and fairness in class incremental learning. In CVPR, pp. 13208-1333 13217, 2020.
Xu, J., Glicksberg, B. S., Su, C., Walker, P., Bian, J., and Wang, F. Federated learning for healthcare informatics. HIR, pp. 1-19, 2021.
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. Federated learning with non-iid data. arXiv preprint arXiv: 000.100000000, 2018
Yang, Q. Liu, Y., Chen, T., and Tong, Y. Federated machine learning: Concept and applications. TIST, pp. 1-19, 2019.
Zhou, D.-.,, Cai, Z.-W., Ye. H.-J., Zhan, D.-C., and Liu,
Z. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338 2023.
Yang, X., Huang, W., and Ye, M. Dynamic personalized federated learning with adaptive differential privacy. In NeurlPS, 2023.
Yang, X., Yu, H., Gao, X., Wang, H., Zhang, J., and Li, T. Federated continual learning via knowledge fusion: A survey. IEEE TKDE, 2024.
Ye. M., Fang. X., Du. B., Yuen, C., and Tao, D. Hetero- geneous federated learning: State-of-the-arr and research challenges. CSUR, 2023.
Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1700 1708.000000 51547, 2017.
Yoon, J., Jeong, W. Lee, G., Yang, E., and Hwang. S. Federated continual learning with weighted inter-client transfer. /CML., pp. 12073-12086. PMLR, 2021.

---

CAN: Clients As Navigators
A. Experiment Details
A.1. Evaluation Metrics
We employ two metrics to assess the performance of our method in the federated continual learning setting.
Average Accuracy (A) This metric evaluates the model's classification performance on the after completing the final task, reflecting its ability to learn tasks while retaining previously acquired knowledge. It is computed as:
A = Drl
I(ύ - y)
(10)
(r.
where Dr represents the test dataset of the overall tasks, and I(-) is the indicator function that returns 1 if the predicted label matches the ground truth y, and O otherwise.
Average Forgetting Score (F) This metric quantifies the extent of knowledge degradation across previously learned tasks due to the introduction new tasks. We first compute the maximum accuracy achieved for each task throughout training and compare with the final accuracy after training all tasks. The forgetting rate for task t defined as:
F = max ALe Ar,
(11)
15t
where Ar, is the accuracy of task after training on task i, and Ar, is the final accuracy on task after training on all T tasks. The Average Forgetting Rate (JF) then computed as:
T-1 F
T-1 Σ F 1
(12)
A.2. Generator Architecture and Optimization
To ensure fair comparison under consistent conditions, we adopt the same generator architecture as LANDER (Tran et al., 2024) (as shown in Tab. 4 and Tab. 5). To optimize the generator, Adam optimizer is employed with learning rate of 2e-3, ensuring stable convergence. The synthetic batch size is adjusted based on dataset complexity and hardware constraints, set to 256 for CIFAR- 100 and Tiny- ImageNe,, and 128 for ImageNet100 to maintain GPU memory usage below 24GB (NVIDIA RTX 4090). This design ensures that the generator remains scalable and efficient across diverse tasks, contributing the robustness of our method in challenging settings.
Table 4, Generator Architecture for CIFAR- 100 and TinylmageNee.
Layers
Output
Input
256
Linear, BatchNormiD, Reshape
128 x h h/4 X w/4
SpectralNorm (Conv (3 x 3)), BatchNorm22, LeakyReLU
128 h/4 X w/4
UpSample (2x)
128 x h/2 x w/2
SpectralNorm (Conv (3 x 3)), BatchNorn22, LeakyReLU
x h/2 x w/2
UpSample (2x)
64 x h X
SpectralNorm (Conv (3 x 3)), Sigmoid, BatchNorm21)
x x 1
12

---

CAN: Clients As Navigators
Table 5. Generator Architecture for ImageNet100.
Layers
Output
Input
256
Linear, BatchNormID, Reshape
128 x h/16 x w/16
SpectralNorm (Conv (3 3)), BatchNorm2D, LeakyReLL
128 x h/16 16 x w/16
UpSample (2x)
128 h/8 w/8
SpectralNorm (Conv (3 3)), BatchNorm2D, LeakyReI.U
128 x h/8 x w/8
UpSample (2x)
128 x h/4 x w/4
SpectralNorm (Conv (3 3)), BatchNorm22, LeakyReLU
64 x h/4 x w/4
UpSample (2x)
64 x h/2 x w/2
SpectralNorm (Conv (3 3)), BatchNorm22, LeakyRel.U
x h/2 x w/2
UpSample (2x)
64 X h X
SpectralNorm (Conv (3 x 3)), Sigmoid, BatchNorm2D
3 x h x w
A.3. Hyperparameter Choice
A.3.1. DEFAULT HYPERPARAMETER
To clarify the experimental setup, we provide the key hyperparameter configurations the table below. These settings were consistently applied across all experiments unless otherwise specified.
Table 6. Training parameters.
Hyperparameter
Value
Communication Rounds
100
Clients Numbers
5
Local Epochs
Local Batch Size
128
Synthesis Batch Size
256
A.3.2. BALANCING Crav AND Egop
We analyze the effect of varying in Eq. (5) balance Lusv and Lepp- As shown in Tab. the results indicate that placing greater emphasis on Legap (higher values) leads improved performance. This trend suggests that while prominent features relatively easy to capture through Laav. the finer, more nuanced details-althougi! harder learn-are also crucial in real-world scenarios and thus warrant additional focus. The stronger weighting on Egap enables the generator to refine and diversify synthetic data by capturing these subtle characteristics. This finding further supports the importance of introducing the gap loss Cgsp to achieve richer and more comprehensive data.
Table Effect of varying the value on accuracy, as defined in the joint loss function E in Eq. (5). Increasing À (placing more emphasis legu)) leads to better performance.
0.33
0.5
0.67
Ace (AA)
41.01
42.03
43.25
A.3.3. ITERATIONS DURING PROFILING STAGE
In Tab. 8, we find that setting 7 to 20 consistently yields either the best or second-best accuracy across different buffer sizes, making an effective choice for balancing efficient training and capturing the impact of new tasks. Additionally, we observe that the accuracy improves when the loss stabilizes around 150 during training. For simplicity and consistency, we use 20 as the basis for measuring forgetting patterns our experiments.
13

---

CAN: Clients As Navigators
Table Effect of varying the number of iterations during the profiling stage across different buffer sizes, highlighting the best and second-bes performance.
10
15
20
25
Size
Acc (AA)
37.12
37.62
41.44
42.33
41.22
500
Loss (L)
162.75
160.76
152.33
147.97
146.80
Acc (AA)
41.53
40.86
41.56
42.36
42.58
750
Loss (C)
166.93
162.92
161.03
155.80
151.87
Acc (AA)
41.78
42.06
42.35
42.27
41.96
1000
Loss (L) 160.31
154.29
153.03
150.35
147.54
Acc (AA) 40.78
41.23
41,46
42.34
40.02
1250
Loss (L)
161.77
155.32
151.81
149.45
147.17
A.3.4, PARAMETER SELECTION ON LOCAL TRAINING
Following Gao et al. (Gao et al., 2022b), we adopt an adaptive strategy for setting the scaling factors Snew and afres which balances the learning of new and previously encountered knowledge. As the ratio of previous classes new classes increases, the difficulty of preserving past knowledge also grows. To address this, the scaling factors defined as:
1 1/16
new =
Occur:
Apre = KōOxpre:
(13)
8
where
1D9
+ 1). 8
| D
log2
(14)
2
and |Y*| represents the number of classes in task t. Here, anew and Opre are the base scaling factors.
This formulation ensures dynamic adjustment of the importance weights, mitigating the challenges posed by an increasing number of prior classes. The logarithmic function in 6 provides a smooth scaling mechanism, while the square-root term in ő prevents excessive emphasis earlier tasks. For more details, we refer readers to Gao al. (Gao al., 2022b).
B. Further Comparison and analysis
Generative replay is key technique in FCL to mitigate catastrophic forgetting. Two prominent approaches to replay-based methods are:
Diffusion-Basse Generative Replay: Diffusion models have recently gained attention due to their ability to generate high-quality synthetic samples for replay. These models iteratively refine noise into structured data distributions, enabling data-free continual learning.
Data-Free Knowledge Distillation (DFKD): Our approach leverages data-free knowledge distillation, which bypasses explicit data synthesis by directly distilling knowledge from previous models into new tasks. This eliminates the need for storing synthetic samples while still retaining knowledge through logit-based or feature-based distillation.
B.I. Performance Analyze
In this section, we compare CAN with two state-of-the-at methods that utilize pretrained models for generative replay: DDDR (Liang et al., 2025), which leverages a pretrained diffusion model, and LANDER (Tran et al., 2024), which incorporates pretrained language models. Unlike these approaches, CAN does not rely on any external pretrained models but instead fully exploits the role of clients achieve effective generative replay.
14

---

CAN: Clients As Navigators
Table Performance comparison CIFAR-100 with DDDR and LANDER across different buffer sizes and data heterogeneity levels.
Evaluated with Average Accuracy (A).
Buffer Size
Methods
IID
NIID(1.)
NIID(0.5)
NIID(0.1)
DDDR 44.21
41.82
40.06
36.93
500
LANDER
36,99(7222)
(6.41
35.63(443)
33.34(-3.59)
CAN
(4.92(H0.1))
43.64(+1.82)
41.04
35.38(-1.55)
DDDR 45.30
44.10
41.43
37.80
750
LANDER 41.56(-3.74)
40.90(-320)
38.30(-3.13)
37.06(-0.74)
CAN (6.15
43.15
(1.56(+0.13)
37.14(.066)
DDDR 46.81
44.51
41.11
38.55
1000
LANDER (3.99((2822)
43.46(-1.05)
(0.85
(7.65(.0.90)
CAN 46.16(-0.65)
44.82(40.31)
(2.35(+1124)
38.91(40.35)
DDDR 46.50
45,20
43.04
39.06
1250
LANDER 45.92(-0.58) (4.51(-0.69)
41.63(1.41)
37.94(1.12)
CAN 48.16(+1.66)
42.16 45.93
39.88
As shown in Tab. 9, DDDR achieves strong performance due to the superior generative capabilities of pretrained diffusion models, which enhance replay effectiveness. However, our proposed method, CAN, matches or even surpasses DDDR across various settings- all without relying on any pretrained models. Specifically, CAN consistently outperforms LANDER and achieves results comparable to DDDR, even under the most challenging NIID(0.1) scenario. This demonstrates that the strategic of client-side knowledge alone is sufficient to bridge the performance gap between diffusion- based and traditional DFKD approaches, eliminating the need for computationally expensive pretrained models.
These results highlight the effectiveness of client-aware generative replay, showing that using pretrained models is not a prerequisite for strong performance in federated continual learning. Instead, CAN effectively leverages local client expertise to enhance knowledge retention and mitigate forgetting, achieving state-of-the results with significantly lower computational overhead.
B.2. Cost Analysis
We analyze both the computational overhead (synthesis and local replay costs) and the storage requirements (buffer size and model storage) of CAN, comparing it with DDDR and LANDER.
B.2.1. COMPUTATIONAL COST
Synthesis Overhead. DDDR relies a pretrained diffusion model, which requires multiple iterative steps to produce high-quality synthetic samples. This multi-step process significantly increases inference time and computational complexity. In contrast, both CAN and LANDER are DFKD-based approaches that generate synthetic samples in a single step, making them substantially more efficient in terms of synthesis time and computational cost.
Local Replay Overhead. Local replay introduces additional computation during training. By maintaining a smaller replay buffer, CAN reduces the number of samples involved in this phase, resulting shorter replay times and lower computational load. In comparison, LANDER and DDDR utilize larger buffers, prolonging the replay phase and increasing the overall training cost. Hence, replaying fewer samples allows CAN accelerate local training and enhance efficiency in federated continual learning.
GPU Memory Consumption. DDDR must load a large pretrained diffusion model, which consumes considerable GPU memory due to its high parameter count and multi-step generation process. LANDER, reliant on pretrained language models, also incurs a sizable memory footprint. In contrast, CAN operates without any pretrained models, reducing GPU memory consumption and making a practical option for resource- limited federated settings.
15

---

CAN: Clients As Navigators
B.2.2. STORAGE COST
Buffer Size Requirements. CAN achieves competitive performance with substantially smaller buffer (see Table 9), reducing storage needs while maintaining high accuracy. LANDER struggles retain knowledge effectively due to lower-quality generated data and the absence of an adaptive buffer strategy, forcing to rely a larger replay buffer and incurring higher storage costs, yet still underperforming relative to CAN. Although DDDR benefits from the strong generative capabilities of pretrained diffusion model, it lacks an adaptive replay strategy. As a result, despite producing high-quality synthetic samples, its performance remains on par with or slightly below CAN in most scenarios, illustrating that superior data quality alone is insufficient without client-adaptive replay mechanism.
Pretrained Model Storage Overhead. DDDR demands substantial storage for a pretrained diffusion model, which can range from hundreds of megabytes several gigabytes, depending on the architecture. LANDER also incurs notable storage costs due to its reliance pretrained language models, which can occupy multiple gigabytes. CAN eliminates these overheads entirely by avoiding external pretrained models, drastically reducing storage requirements and enhancing scalability.
16


Reviews:
[{"review_id": "OuscjLqZTR", "reviewer": "Reviewer_aG9E", "summary": "This paper examines Federated Continual Learning, specifically exploring how client expertise aids in generating replay data and adjusting replay buffer sizes based on the forgetting variance among clients. The CAN approach first identifies accurate expert clients, utilizing their predictions to train the generator. By promoting discrepancies between the server model (S) and reference model (Π), it extracts fine-grained features. Additionally, the replay buffer is dynamically managed during training to minimize further forgetting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer aG9E:\n\nWe greatly value the time and expertise you invested in reviewing our submission. Your feedback has been instrumental in helping us improve the clarity of our work. We address your comments in detail below.\n\n### Response to Weaknesses\n\n> **W1:** The paper mentions that CAN is similar to TARGET and LANDER. However, as far as I know, both methods fall under the category of Data-Free Knowledge Distillation (DFKD). The authors should provide a more detailed distinction and explanation regarding how their approach differs from these methods. \n\n**Our Response:** We appreciate your point regarding the similarity between CAN and existing Data-Free Knowledge Distillation (DFKD) methods. While all three methods adopt a generative replay framework, CAN introduces two key innovations that fundamentally distinguish it from TARGET and LANDER:\n\n1. **Client Expertise Superiority**: Unlike TARGET and LANDER, which rely solely on the global server model for guiding data synthesis, CAN explicitly identifies and leverages *client-side experts* to guide generation. Specifically, our *expertise map* selects the best-performing client for each class, whose model is then used to supervise the generator through an *Expert Navigation Loss*. This enables class-specific fine-grained guidance that cannot be captured by a centralized server alone.\n2. **Client Forgetting Variance**: Prior methods use uniform replay buffers across clients, overlooking the heterogeneous forgetting behavior induced by non-IID data. In contrast, CAN profiles each client’s unique forgetting patterns and adapts the buffer allocation accordingly, improving both performance and efficiency.\n\n> **W2:** The explanation of the Synthetic Data Initialization section is unclear, which raises concerns. The authors should provide a more detailed and precise clarification of this part.\n\n**Our Response:**  Thank you for pointing this out. We agree that the Synthetic Data Initialization section could benefit from more detailed clarification. Below we provide a clearer description of the process.\n\nThe initialization begins by sampling class-conditional latent codes $ z_y \\sim \\mathcal{N}(0, I) $, where $ y $ denotes the target class. These latent codes are passed through the generator $ G $ to obtain synthetic images $ x_{\\text{syn}} = G(z_y) $. We then use the server model to perform inference on each $ x_{\\text{syn}} $, obtaining predicted class probabilities. To ensure that the generated samples are aligned with their intended class labels, we compute an entropy-based loss between the prediction and the target label $ y $. This loss encourages the generator to produce more discriminative and class-consistent samples. Once the initial set of synthetic data is generated, we train the reference model $ \\Pi $ from scratch on this data. The purpose of $ \\Pi $ is to capture fine-grained features potentially missed by the server model. We then compare the predictions of $ \\Pi $ and the server on synthetic samples and apply a KL divergence loss (described as $ \\mathcal{L}_{\\text{gap}} $) to encourage the generator to produce samples that expose the discrepancy between the two models. This results in more informative and transferable data for replay. We will revise Section 4.2.1 in the final version to clearly explain this.\n\n> **W3:** The origin of the Reference Model is not clearly explained. Does it require training from scratch? What is the computational overhead? How many training iterations are needed, and how are the training parameters set? These aspects should be addressed in more detail.\n\n**Our Response:** Thank you for the helpful comment! Please allow us to explain it. The purpose of the reference model $ \\Pi $ is to guide the generator to produce more diverse samples. $ \\Pi $  is trained from scratch on synthetic data and shares the same architecture as the server model (a ResNet-based classifier). This helps encourage the generation of informative, boundary-sensitive synthetic data, rather than overfitting to overly confident or less representative examples. We train it for 40 epochs using SGD with a learning rate of 0.1, momentum 0.9, and weight decay 0.0002. In terms of computational overhead, $ \\Pi $ is only used during the training phase to guide the generator and is lightweight and isolated from client participation. It introduces less than 5% additional training time, and does not affect inference or deployment, thus incurring minimal cost in practice. These implementation details will be included in the appendix.\n\n### Response to Suggestions\n\n> **S1:** The **Expert-Driven Data Synthesis** section lacks a clear focus, making it difficult to grasp the key points. The authors need to improve the logical flow of this section.\n\n**Our Response:** Thanks for your valuable suggestions to improve the presentation of our paper! We will revise the *Expert-Driven Data Synthesis* section to improve its clarity and logical flow in the final version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The author provides multiple formulas to explain their method, and the theoretical claims are correct.", "experimental_designs_or_analyses": "The experimental results are comprehensive, with the author conducting various experiments under different settings to ensure thorough evaluation.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n1. The idea of the method is intuitive and easy to follow.\n2. The authors provide a clear motivation and a well-structured discussion of related work.\n3. The experiments conducted across different settings effectively demonstrate the strategy’s effectiveness and generalizability.\n\nWeaknesses\n1. The paper mentions that CAN is similar to TARGET [1] and LANDER [2]. However, as far as I know, both methods fall under the category of Data-Free Knowledge Distillation (DFKD). The authors should provide a more detailed distinction and explanation regarding how their approach differs from these methods.\n2. The explanation of the Synthetic Data Initialization section is unclear, which raises concerns. The authors should provide a more detailed and precise clarification of this part.\n3. The origin of the Reference Model is not clearly explained. Does it require training from scratch? What is the computational overhead? How many training iterations are needed, and how are the training parameters set? These aspects should be addressed in more detail.\n[1] Zhang, J., Chen, C., Zhuang, W., and Lyu, L. Target: Federated class-continual learning via exemplar-free distillation. In CVPR, pp. 4782–4793, 2023.\n[2] Tran, M.-T., Le, T., Le, X.-M., Harandi, M., and Phung, D. Text-enhanced data-free approach for federated class-incremental learning. In CVPR, pp. 23870–23880, 2024.", "other_comments_or_suggestions": "The **Expert-Driven Data Synthesis** section lacks a clear focus, making it difficult to grasp the key points. The authors need to improve the logical flow of this section.", "questions_for_authors": "Please refer to the Strengths and Weaknesses section for specific details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3LbO9d2Zg", "reviewer": "Reviewer_czny", "summary": "The paper focusing on Federated Continual Learning (FCL) scenario, highlights two key observations: Client Expertise Superiority and Client Forgetting Variance. The study shifts attention from the server to the client and proposes using the unique capabilities of client-side knowledge to improve Generative Replay. This new perspective explores how Federated Learning can reduce forgetting in FCL tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer czny:\n\nWe are truly grateful for your insightful review and the constructive feedback provided. Your suggestions helped us identify areas for clarification, and we respond to each point below with careful consideration.\n\n### Response to Weaknesses\n\n> **W1:** Could more experiments be conducted under additional buffer sizes, such as 2560 and 5120? I’d like to see whether CAN can still maintain the same level of performance under larger buffer size conditions.\n\n**Our Response:** Thanks for your valuable insights!  Following your advice, we conducted additional experiments under larger buffer sizes (2560 and 5120). As shown in the table below, CAN consistently outperforms LANDER across all settings, including under larger buffer budgets. Notably, even when the buffer is doubled (5120), CAN maintains a clear performance advantage under both IID and non-IID scenarios. These results further confirm the scalability and robustness of our method across different buffer regimes.\n\n|           | LANDER(2560) | CAN(2560) | LANDER(5120) | CAN(5120) |\n| :-------: | :----------: | :-------: | :----------: | :-------: |\n|    IID    |    47.97     | **49.36** |    49.19     | **51.81** |\n| NIID(1.0) |    46.33     | **48.31** |    47.82     | **50.86** |\n| NIID(0.5) |     43.3     | **46.3**  |    45.43     | **47.72** |\n| NIID(0.1) |    40.01     | **40.98** |    42.14     | **43.72** |\n\n> **W2:** The paper does not mention the number of clients in experiments, which is a critical factor in federated learning. Could experiments be conducted in scenarios with a larger number of clients?\n\n**Our Response:** We appreciate the suggestion. In our main experiments, we used 5 clients by default. To evaluate scalability under more realistic federated settings, we conducted additional experiments with 10 and 15 clients. As shown in the table above, CAN consistently outperforms LANDER across all client counts, especially under non-IID scenarios. These results demonstrate that CAN maintains its effectiveness even as the number of clients increases, confirming its robustness in more complex and distributed environments.\n\n|           | LANDER (10) | CAN (10)  | LANDER(15) |  CAN(15)  |\n| :-------: | :---------: | :-------: | :--------: | :-------: |\n|    IID    |    45.53    | **46.39** |   45.32    | **45.68** |\n| NIID(1.0) |    43.00    | **44.64** |   41.79    | **42.65** |\n| NIID(0.5) |    41.53    | **42.81** |   40.02    | **40.25** |\n| NIID(0.1) |    27.89    | **28.86** |   22.73    | **24.28** |\n\n> **W3:** The \"V phenomenon\" mentioned in Sec. 5.3 suggests a trade-off between buffer size and data quality. The authors should discuss how to optimize this balance in practice.\n\n**Our Response:** We appreciate the reviewer’s insightful observation. This behavior arises from the interplay between buffer size and adaptive allocation efficiency. When the buffer is small, the system allocates limited replay slots more selectively to the most forgotten tasks, resulting in efficient knowledge retention. As the buffer increases, this selective effect weakens, and additional samples contribute less to mitigating forgetting, leading to a temporary dip in performance. In practice, we suggest starting from a moderate buffer budget and gradually increasing it while monitoring key metrics such as retention accuracy on early tasks and per-task replay effectiveness. If performance plateaus or degrades, it may indicate that further buffer increase offers diminishing returns. Such observations can guide the choice of buffer size under memory or communication constraints.\n\n### Response to Suggestions\n\n> **S1:** Typo: Table 3 should refer to Sec. 5.4, not Sec. 5.3.\n\n**Our Response:** Thank you for pointing this out. We acknowledge the typo and will revise the reference from Sec. 5.3 to Sec. 5.4 in the final version of the paper.\n\n### Response to Questions\n\n> **Q1:** I was wondering that which dataset was used for the experiments in Table 3? It doesn’t seem to be mentioned in the paper. No other questions.\n\n**Our Response:** Thanks for pointing this out! The experiments presented in Table 3 were conducted on the CIFAR-100 dataset. This table reports the final accuracy on the first task after the completion of all tasks, which serves as an important metric to evaluate catastrophic forgetting. We will clarify it in the final version of the paper.", "claims_and_evidence": "Yes, the claims made in this paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, The proposed method “CAN” is practical and easy to follow, make sense for the problem or application at hand.", "theoretical_claims": "This paper doesn’t include theoretical claims, just with necessary formulas.", "experimental_designs_or_analyses": "The experimental design is well-structured, and the tests were conducted on three widely used datasets: CIFAR100, TinyImageNet, and ImageNet100. However, the study falls short in including experiments with broader generalizability，see details in Weaknesses.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and clearly organized, making it easy to understand.\n2. Comparative experiments demonstrate that the proposed method achieves notable effectiveness in terms of computational cost and reduction of forgetting rates.\n3. The proposed method does not compromise the privacy-preserving capabilities of the federated learning paradigm.\nWeaknesses\nPersonally, I don’t see any big problems with the method described in the paper. What I’m really curious about are some of the experimental details.\n1. Could more experiments be conducted under additional buffer sizes, such as 2560 and 5120? I’d like to see whether CAN can still maintain the same level of performance under larger buffer size conditions.\n2. The paper does not mention the number of clients in experiments, which is a critical factor in federated learning. Could experiments be conducted in scenarios with a larger number of clients?\n3. The \"V phenomenon\" mentioned in Sec. 5.3 suggests a trade-off between buffer size and data quality. The authors should discuss how to optimize this balance in practice.", "other_comments_or_suggestions": "Typo: Table 3 should refer to Sec. 5.4, not Sec. 5.3.", "questions_for_authors": "I was wondering that which dataset was used for the experiments in Table 3? It doesn’t seem to be mentioned in the paper. No other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8mj0fVZvcw", "reviewer": "Reviewer_z1gv", "summary": "CAN addresses the challenges imposed by non-IID data in Federated Continual Learning (FCL) and introduces a novel approach that leverages the specialized expertise of individual clients. By employing Expert-Driven Data Synthesis, CAN enhances the quality and representativeness of the generated data, ensuring effective retention of client-specific knowledge. Moreover, the integration of Adaptive Replay optimizes the utilization of synthesized data, improving memory efficiency and model stability over time. Collectively, these components enable CAN to improve continual learning performance within the Federated Learning framework, effectively addressing key issues in knowledge retention and adaptation across heterogeneous clients.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer z1gv:\n\nThank you for your encouraging remarks and the critical questions you posed. We have reflected thoroughly on your feedback and provide our detailed responses below to further clarify our method and contributions.\n\n### Response to Weaknesses\n\n> **W1**: The reference model Π is described as being trained on synthetic data; however, its architecture and training parameters (e.g., number of epochs, optimizer details) are not provided, which complicates a complete understanding of this section.\n\n**Our Response:** Thank you for the suggestion. A similar question regarding the reference model $ \\Pi $ was raised by Reviewer aG9E. To avoid redundancy, we have provided a detailed response under that review, including training configuration, architecture, and computational overhead. We kindly refer you to our response to Reviewer aG9E for the complete explanation.\n\n> **W2:** Given that Federated Learning emphasizes communication efficiency, quantifying aspects such as the number of communication rounds and the computational cost associated with generative replay would enhance the study’s practical relevance. The paper currently provides only a descriptive account without experimental evidence to substantiate these claims.\n\n**Our Response:** Thank you for the suggestion! All experiments in our paper are conducted under a consistent setting of 100 communication rounds. To support our claims on computational efficiency, we provide a detailed comparison with DDDR (a diffusion-based method) and LANDER. As shown in the table below, CAN reduces overall computation cost by over 75% compared to DDDR (97 vs. 406 minutes). This highlights the advantage of avoiding multi-step diffusion generation, which is computationally intensive and less practical in federated settings. Compared to LANDER, CAN incurs the same client-side training cost (57 minutes), ensuring fairness in edge device usage. Although CAN introduces slightly more server-side cost due to expert-guided processing, we argue that this overhead is minor and acceptable, especially given that most computation in FL occurs on the client side.\n\n|        | Local Training | Image Generation | Overall |\n| :----: | :------------: | :--------------: | :-----: |\n|  DDDR  |      186       |       220        |   406   |\n| LANDER |       57       |        28        |   85    |\n|  CAN   |       57       |        40        |   97    |\n\n### Response to Suggestions\n\n> **S1:** It could be worth exploring how to apply such methods in Diffusion-based approaches within FCL.\n\n**Our Response:** Thank you for the thoughtful suggestion. Integrating Diffusion-based approaches into the FCL setting is indeed a valuable direction. Compared to GANs, diffusion models are known to generate higher-quality and more diverse samples, which could further improve replay effectiveness in non-IID settings. However, applying diffusion in a federated and continual learning context also brings new challenges, such as communication efficiency, and privacy preservation. We believe this is a promising extension and plan to explore how our expert-guided replay strategy could be adapted to this setting in future work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The methods are logically structured, and the proposed approach effectively mitigates the problem of catastrophic forgetting in federated continual learning.", "theoretical_claims": "No theroretical claims.", "experimental_designs_or_analyses": "Experiments are sound，extensive experiments across three datasets (CIFAR100, TinyImagenet, ImageNet100) with varying non-IID degrees.", "relation_to_broader_scientific_literature": "Builds on FCL works like TARGET and LANDER but uniquely integrates client expertise.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Pros:\n1. The article is thoroughly composed, featuring well-organized figures and tables that effectively support the main findings.\n 2. The proposed approach is particularly compelling, as leveraging client-specific expertise offers a novel perspective and valuable insights for the research community. \n3. Experimental results demonstrate that CAN achieves state-of-the-art (SOTA) performance across various datasets and task configurations. \n\nCons:\n1. The reference model Π is described as being trained on synthetic data; however, its architecture and training parameters (e.g., number of epochs, optimizer details) are not provided, which complicates a complete understanding of this section.\n2. Given that Federated Learning emphasizes communication efficiency, quantifying aspects such as the number of communication rounds and the computational cost associated with generative replay would enhance the study’s practical relevance. The paper currently provides only a descriptive account without experimental evidence to substantiate these claims.", "other_comments_or_suggestions": "It could be worth exploring how to apply such methods in Diffusion-based approaches within FCL.", "questions_for_authors": "Please refer to Strengths And Weaknesses. No extra questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mktwbVeDCl", "reviewer": "Reviewer_TT97", "summary": "This paper introduces Clients as Navigators (CAN) method, which tackling catastrophic forgetting in FCL tasks due to heterogeneous client data. CAN introduces a novel Generative Replay strategy that different from existing methods by selecting teaching clients for generator training and using an adaptive data distillation process to combat forgetting, dynamically adjusting to the client’s needs. The author responses match with my expectation ## update after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TT97:\n\nWe appreciate your engagement with our work and the thoughtful observations you made. We aim to address your concerns in our detailed responses below, hoping to provide clarity and demonstrate the effectiveness of our proposed approach.\n\n### Response to Weaknesses\n\n> **W1:** The challenges associated with FCL mentioned in the article are very interesting. To address these challenges, the authors proposed CAN, which has been shown to perform well through extensive experimentation. However, the improvements in methodology that CAN seems to be not substantial compared to the baseline LANDER. I am quite curious about the specific methodological improvements.\n\n**Our Response:** Thank you for your interest in the methodological contributions of CAN. A similar concern regarding the distinction between CAN and prior approaches such as LANDER was raised by Reviewer aG9E. To avoid redundancy, we have provided a detailed explanation in our response to Reviewer aG9E, outlining the key innovations of CAN in terms of client-side expert supervision and adaptive buffer replay based on forgetting patterns. We kindly refer you to that response for the complete clarification.\n\n> **W2:** Although the author has already provided the settings for some of the hyperparameters in the experiments, I would like to know about the settings for other important parameters, such as the number of communication rounds, the number of clients, and so on.\n\n**Our Response:** Thank you for your suggestion. To clarify the experimental setup, we provide the key hyperparameter configurations in the table below. These settings were consistently applied across all experiments unless otherwise specified.\n\n|   Hyperparameters    | Value |\n| :------------------: | :---: |\n| Communication Rounds |  100  |\n|   Clients Numbers    |   5   |\n|     Local Epochs     |   2   |\n|   Local Batch Size   |  128  |\n| Synthesis Batch Size |  256  |\n\n\n\n### Response to Questions\n\n> **Q1:** I am curious whether the buffer size affects the allocation algorithm mentioned in the article. When the buffer size is small enough, is there a possibility that the accuracy of some labels is too low to allow for proper allocation?\n\n**Our Response:** Thank you for raising this important question. We agree that in scenarios with small buffer sizes, the accuracy on certain labels or tasks might be very low, which could potentially lead to unstable or skewed buffer allocations. To mitigate this issue, we introduce a predefined lower-bound threshold $ \\epsilon $ in the calculation of the forgetting weight (Equation 7). Specifically, we define:\n$$\nw_c^t = \\frac{1}{\\max(\\text{Acc}_c^t, \\epsilon)}\n$$\nThis ensures that when the classification accuracy $ \\text{Acc}_c^t $ is too low, the forgetting weight is capped to avoid extreme values. By preventing division by near-zero accuracies, this threshold stabilizes the allocation algorithm and ensures a reasonable distribution of buffer space even under small-buffer regimes. The value of $ \\epsilon $ is empirically chosen (set to 0.1 in our experiments) and does not require tuning across datasets.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes. The methods are well evaluated.", "theoretical_claims": "No specific theoretical claims are made.", "experimental_designs_or_analyses": "Yes, the experimental design is comprehensive, and the authors effectively validate the method’s effectiveness through extensive experiments.", "relation_to_broader_scientific_literature": "The client-centric approach proposed by the authors offers valuable insights for the Federated Continual Learning (FCL) field, as previous research has rarely explored this perspective.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n1. This paper is well-motivated and provides a thorough analysis of why client's expertise knowledge is critical. The work has potential value across multiple related fields.\n2. The paper's figure is color-coordinated and well-constructed, making it very easy for me to understand the content.\n3. The experiments are thorough, and the experimental results sufficiently demonstrate the effectiveness of the method.\n\n\nWeaknesses\n1.\tThe challenges associated with FCL mentioned in the article are very interesting. To address these challenges, the authors proposed CAN, which has been shown to perform well through extensive experimentation. However, the improvements in methodology that CAN seems to be not substantial compared to the baseline LANDER. I am quite curious about the specific methodological improvements.\n2.\tAlthough the author has already provided the settings for some of the hyperparameters in the experiments, I would like to know about the settings for other important parameters, such as the number of communication rounds, the number of clients, and so on.", "other_comments_or_suggestions": "None", "questions_for_authors": "I am curious whether the buffer size affects the allocation algorithm mentioned in the article. When the buffer size is small enough, is there a possibility that the accuracy of some labels is too low to allow for proper allocation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper discussed Generative Replay in Federated Continual Learning. After some discussion, the overall verdict was that this submission could be accepted in ICML-2025 while the concerns of reviewers should be addressed in the final version.
Decision: Accept (poster)