Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting
Authors: Milad Khademi Nori
IL MIN KIM
Guanghui Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Class-Incremental Learning
Federated Learning
Global Forgetting
Local Forgetting.
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: A mathematical framework for federated class-incremental learning and a hybrid approach to overcome local and global forgetting.

Abstract: Federated Class-Incremental Learning (FCIL) refers to a scenario where a dynamically changing number of clients collaboratively learn an ever-increasing number of incoming tasks. FCIL is known to suffer from local forgetting due to class imbalance at each client and global forgetting due to class imbalance across clients. We develop a mathematical framework for FCIL that formulates local and global forgetting. Then, we propose an approach called Hybrid Rehearsal (HR), which utilizes latent exemplars and data-free techniques to address local and global forgetting, respectively. HR employs a customized autoencoder designed for both data classification and the generation of synthetic data. To determine the embeddings of new tasks for all clients in the latent space of the encoder, the server uses the Lennard-Jones Potential formulations. Meanwhile, at the clients, the decoder decodes the stored low-dimensional latent space exemplars back to the high-dimensional input space, used to address local forgetting. To overcome global forgetting, the decoder generates synthetic data. Furthermore, our mathematical framework proves that our proposed approach HR can, in principle, tackle the two local and global forgetting challenges. In practice, extensive experiments demonstrate that while preserving privacy, our proposed approach outperforms the state-of-the-art baselines on multiple FCIL benchmarks with low compute and memory footprints.

Published as a conference paper at ICLR 2025
FEDERATED CLASS-INCREMENTAL LEARNING: A HY- BRID APPROACH USING LATENT EXEMPLARS AND DATA- DATA-FREE TECHNIQUES TO ADDRESS LOCAL AND
GLOBAL FORGETTING
Milad Khademi Nori'
II-Min Kim
Department of Computer Science Toronto Metropolitan University Toronto, Ontario, Canada mkn@torontomaa
Electrical and Computer Engineering
Queen's University
Kingston, Ontario, Canada
Almin kim8 queensu. ca
Guanghui Wang Department of Computer Science Toronto Metropolitan University Toronto, Ontario, Canada wanges@tor oront omu. ca
ABSTRACT
Federated Iaass-Incremenall Learning (FCIL) refers to a scenario where a dy- namically changing number of clients collaboratively learn an ever-increasing number of incoming tasks. FCIL is known to suffer from local forgetting due to class imbalance each client and global forgetting due class imbalance across clients. We develop mathematical framework for FCIL that formulates local and global forgetting. Then, we propose an approach called Hybrid Rehearsal (HR), which utilizes latent exemplars and data-free free techniques to address local and global forgetting, respectively. HR employs a customized autoencoder designed for both data classification and the generation of synthetic data. To determine the embeddings of new tasks for clients the latent space of the encoder, the server uses the Lennard-Jonse Potential formulations. Meanwhile, at the clients, the decoder decodes the stored didemmimniinnll latent space exemplars back to the high-dimensional input space, used address local forgetting. To overcome global forgetting, the decoder generates synthetic data. Furthermore, our mathematical framework proves that our proposed approach HR can, in principle, tackle the two local and global forgetting challenges. In practice, extensive experiments demonstrate that while preserving privacy, our proposed approach outperforms the state-or of-the-art baselines on multiple FCIL benchmarks with low compute and memory footprints.
INTRODUCTION
Federated Learning (FL) is a distributed machine learning solution that enables clients to collab- oratively optimize models without compromising their data privacy (Konečný et al. 2016). By exchanging model weights or gradients rather than centrally aggregating data, FL enhances data privacy while benefiting from the diversity of data boost model robustness (MeMahan et 2017). However, as the field of FL advances, new challenges arise, particularly in scenarios where the number of classes be learned increases over time (Yoon al., 2021). This scenario, known as Federated Tlass-Increment incoming classes and the inherent class imbalance at both local (client-level) and global (system-wine wide) scales (Dong et al. 2022):
Learning (FCIL), presents two issues because of the dynamic nature of
MMKN's personal email is mi 1 lackaademi nari Egma com.

---

Published as a conference paper at ICLR 2025
One of the two issues in FCIL is local forgetring (Dong al. 2022) which occurs due to class imbalance within each client's dataset. This class imbalance arises because in FCIL, at a given time, clients possess data for only subset of the classes, leading biased learning and forgetting of previously learned classes. Concurrently, global forgetting (Dong et al. 2022 the second issue, arises because of class imbalance across different clients.
Furthermore, any proposed approach for FCIL not only needs to address the two aforementioned issues but also account for the hardware limitations (memory) often faced by clients. Specifically, must cope with the incoming flow of tasks. These requirements set FCIL apart from the traditional federated learning scenario, underscoring the demand for tailored approaches (Babakniya et al., 2023 2024).
The current approaches for FCIL fall into two categories: data- based, also known as exemplar- based (Dong et al.) (2022) and data-free (Babakniya et al., 2023 2024). Data-based approaches in- volve storing several exemplars for each class in the memory so that when the clients train their mod- els new set of classes, they can use previous samples and form mini-batch that statistically rep- resents all the previous classes to mitigate forgetting. However, this approach may be impractical in FCIL due to memory constraints and presents privacy concerns. To remedy that, data-free approaches have recently gained popularity (Zhang al., 2022, Gao al., 2022 Shi rely on a generative model to produce synthetic data. However, data-free approaches may not be as performant as the data-based ones (Dong et al., 2022).
8
BB BB
Mayia
Client 1
Client
Figure I: Local and global forgetting occur due to class imbal- ance at both the local (client-level) and global (system-wide) scales. Whereas local forgetting refers to the combination of sub-optimal losses for the previous tasks and intra-client task confusion, global forgetting involves inter-client task confusion.
2023 /Zhang et al. 2023). These approaches usually
To harness the advantages of both data-based and data-free approaches, we present Hybrid Replay (HR), a unified approach with low memory overhead and high performance. Our contributions in this work are as follows:
We develop a mathematical framework for FCIL to prove that our proposed approach HR can, in principle, tackle the two local and global forgetting challenges.
HR unifies two approaches: (i) HR incorporates our efficient version of exemplar replay based on our customized autoencoder. Our efficient exemplar replay inflicts an order of magnitude less memory overhead than vanilla exemplar-based approaches. (ii) HR includes our version of an autoencoder bbase data-free approach with almost no memory overhead. Our data-based and data-free approaches distinct from those in the literature, utilize a new strategy and a novel autoencoder to implement these approaches.
The backbone of HR, as mentioned above, our customized autoencoder that is used for both data classification and the generation of synthetic data. To determine the embeddings of new tasks for all clients the latent space of the encoder, the server uses the Lennard- Jones Potential formulations (Jones) 1924). Meanwhile, at the clients, to overcome local forgetting, the decoder decodes the stored low-dimensional latent space exemplars back to the high-dimensional input space, used to address local forgetting. To overcome global forgetting, the decoder generates synthetic data in a data-fre free manner.
Extensive experiments demonstrate that while preserving privacy, our proposed approach outperforms the state-of-the-- baselines on multiple FCIL benchmarks with low compute and memory footprints.
RELATED WORK
Federated Learning (FL) can collaboratively train robust models by utilizing the diverse local data of many clients while promising an enhanced level of privacy (Yurochkin et al., 2019: Wang et al. 2020: Li et al. 2021 Yang et al. 2021). In FL., the server aggregates and averages the weights of
2

---

Published as a conference paper at ICLR 2025
the local models of different clients (McMahan et al.
2017). Follow-up works focus mitigating
the drift problem of weights at different clients (Shoham et al., 2019: Li et al. 2020). or aim to
minimize the communication overhead (Chen et al. 2019 Zhang al.
2020b ch. Fast convergence
(Karimireddy et al. 2020) and personalization (Fallah et al.
2020; Lange et al. 2020) have also heen
studied. Unsupervised FL. has shown promising results (Peng al.
2019 Zhang et al. 2020d Dong
2
it a 2021; Liu et al., 202 Chu et al. 2022; Liu ct al.) 2022). However, current FL approaches
cannot work in scenarios where the number of classes grows constantly. This is because the clients
have memory limitations and cannot accumulate all the previous classes (Hamer et al., 2020: Lyu &
Chen 202T).
Plass-nerementaa Learning (CIL) is a scenario where the model has to visit stream of classes
(Zhang et al., 2020a| Khademi Nori & Kim 2025), , learn them incrementally (Ahn et al. 2021), and discriminate them all without forgetting earlier classes (Kim & Choil (02T). Regularization is one of the earliest attempts to mitigate catastrophic forgetting (Kirkpatrick et al.) 2017). Knowledge
distillation (Li & Hoiem, 2017) has been shown effective in retaining the knowledge of previous classes (Shmelkov et al.] 2017). p. Generative models have been adopted to produce synthetic data to be interleaved with the new classes to mitigate biased learning and forgetting (Shin et al., [017]]
Wu et 2018). It has been frequently noted that class imbalance is the most important obstacle in CIL (Douillard et al. 2020 Liu et al., 2020; Rebuffi et al., 2017]] Wu et al. 2019). Bias-correction
(Liu et 2021bb Yan et al. 2021) and knowledge distillation (Hu et al. 2002;; (Simon et al.] (021)
have been shown effective in overcoming class imbalance. Hybrid replay approaches integrate
model-based replay (generative modeling) and exemplar- based replay. In HR, instead of storing raw
exemplars, latent space features are stored, which require orders of magnitude less memory (Hayes
et al. 2020 Van de Ven et al. 2020: Wang et al. 2021 Tong et al. 2022, Zhou et al., 2022). Table compares our proposed work HR and other hybrid baselines such as Remind (Hayes et al.. 2020p.
Remind+ (Wang et al. 2021), and i-CTRL (Tong et al. 2022).
Federated Class-Incremental I Learning (FCIL) a scenario where multiple distributed clients collaboratively and incrementally learn new classes over time without sharing their data, facing both challenges of local and global forgetting due to imbalanced (non-IID) data distributions (Yoon) et al. 2021 Dong et al.) 2022). FCIL approaches fall into two categories of (i) exemplar-based approaches and (ii) data-free approaches. (i) As an exemplar of exemplar-base based, Global-Local Forgetting Compensation (GLFC) model integrates class-aware gradient compensation and class- semantic relation distillation to mitigate local forgetting. while a proxy server and a prototype gradient-based communication mechanism tackle global forgetting (Dong al., 2022).
(ii) Data-free approaches. Relation guided Representation Learning for Date-Free Learning (R-DFCIL.) employ relational knowledge distillation to maintain compatibility between new and previous class representations (Gao et al. [2022). The Federated Fine-Tuning with Generators (FedFTG) employs knowledge distillation and hard sample mining optimize the global model and address data heterogeneity across clients (Zhang et al. 2022). TARGET (federated class-continudl learning via exemplar- free distillation) uses a pre-trained global model for knowledge transfer and a generator to simulate data distribution, without needing additional datasets or storing private data (Zhang et al. 2023). In (Babakniya et al., 2023), a server-trained generative model synthesizes past data samples from discriminators and then trains generators that are given to clients, enabling clients to locally mitigate catastrophic forgetting without compromising privacy.
The Local-Global Anti-forgetting (LGA) model uses aategory-balancee gradient-adaptive com- pensation loss and semantic distillation to balance learning among local clients, while a proxy server manages global forgetting through sulf-supervised prototype augmentation (Dong et al. 2023). FedSpace adapts to asynchronous and varied task sequences of clients using prototype- based learning. representation loss, fractal pre-training, and a modified aggregation policy (Shenaj et al. 2023). Shi & Yel (2023) introduces a prototype reminiscence mechanism that dynamically reshapes old class feature distributions by integrating previous prototypes with new class features, in tandem with an augmented asymmetric knowledge aggregation. An improved version of (Babakniya et al. 2023) has been proposed by [Baakkniya et al. (2024).
3

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of our approach with other hybrid approaches Remind (Hayes al. 2020
Remind+ Wang et al. 202Tp, and i-CTRL (Tong et al. 2022).
Classtication Approach
Quantization
Learning Scenario
Space Rep- resentation
Architectural Simplicity
HR
Classtficttion occurs
Incorporating quan- tization within Offine learning.
Uses a structured la- space via the Lennard-Jones Po- tential
Our work showcases clean and minimalistic designs, depicted in Fig. _
within the latent space the cacoder, loveraging Euclidean distance.
architecture would be straightlowardd furtber
improving perfornance.
Remind
Classifiation occurs alter ihe decodinn prucess, ploying a cross-entropy as depicted in Fig. of Remind.
Applies quantization to the latent exemplars hance compessioon
Online learning.
Uses an unstructured latent space for CNNs.
Fig. 2 of Remind shows ad hoc and unnecessar ily complex archirecture, complicating implementa- tion and scalahility.
Remind+
Classilcation occurs al- decoding with entropy loss function.
Incorporates quanlization feature compression.
Online learning.
Relies on an unstruc- tured latent space autoencoder.
Figs. and of Remind+ demonstrate ad hoc and overly complex designs, hindering scalability.
i-CTRL Classilication within the latent space of the encoder, levernging Eaclidean distance.
Incorporating quan- tizution within their architecture would be straightforward, further improving performance.
Offinne learning.
Uses structured la- tent space via Linear Discriminative Rep- resentation.
Has clean and minimal- istic design, depicted of i-TTT...
FCIL PROBLEM FORMULATION
In traditional machine learning settings (supervised learning), models trained using fixed dataset, where the class distribution is known in advance and the training data are accessible during the model's training phase. The objective is typically to minimize loss function that measures the discrepancy between the predicted labels and the true labels:
To = -xxx - ([So(z), ((o(()) v)p(a, v) dx dy,
where and y represent the space of inputs and labels, respectively. fo denotes the model parame- terized by 0, f is the loss function, and p(æ, y) is the joint distribution of inputs and labels. When ), the label set, has N classes, the loss function in Eq. can be specifically written as:
N Σ
(ff((z).))p(;; y) do dy. with Uij = J [xx(1s)
la N
N-1
- I=1
Uajo
j=it1
Eq. 2 reflects the need to evaluate the model's performance across all possible pairs of class labels, ensuring that each pair considered in the loss calculation.
When we introduce the concept of tasks in CIL, kth task T4 encompasses specific subset of classes from V. The model must adapt its parameters minimize the loss across successive tasks while maintaining robust performance of classes V& y. Assuming all tasks contain the same number of classes, denoted by N, the cumulative loss function across tasks can be modeled as:
previous tasks. Each task Tj involves learning designated set
K K
fex {(fo(x). y) dx dy. (3)
1o
Ui with
Win.ji
N
Here, K represents the total number of tasks, and the uniform number classes per task.
For task Tk+1 that includes N classes, the cumulative loss function after incorporating this task can be recursively updated from the previous tasks as follows:
N
1 (k+1)
((6)
+
Usa Je+t
(4)
NZ
j=1
1=1
where (8) represents the cumulative loss function up task k. The term N N
U1441
accounts for the interactions within the task TR+1, and the term Σk M EE=1
4

---

Published as a conference paper at ICLR 2025
multiplied by 2, captures the bidirectional interactions between all classes of the task and each class of the previously learned tasks. The normalization factor N ensures uniform weighting of class interactions across all tasks.
In FCIL, the model evolution takes advantage of collaborative contributions from different clients, each maintaining its local dataset. As new tasks and their associated classes arrive at different clients, each client updates its model to accommodate new knowledge while ensuring coherence with the collective knowledge maintained across the federated network.
The recursive update formula for the global loss function, as formulated in Eq. 5 encapsulates this
dynamic. Here, Ig ((++1) represents the global loss after incorporating task (A: 1). This is computed
by aggregating updates from each client pair (m, n),, accounting for both their individual and mutual contributions:
11++11
m.).m.m. + Alo Arbh+t)).m.))
(5)
where
N m.
Al mk+1)),,,
mm Wik
(6)
N2
Eqs. 5 and ootline the critical balance required in FCIL: the preservation of knowledge gained
in the past, represented by Ig I(M),,,.. This preservation crucial for maintaining system stability (stability-plastiiity dilemma) and is typically achieved through mechanisms such as regularization, knowledge distillation, and replay.
Meanwhile, in Eq. o there are two loss terms, the first loss term has to do with learning the new task and encompasses all the interactions between classes in the new task, and the second loss term, which is very important, accounts for the loss terms between the new task and all the previous tasks. Failing to minimize the second loss would result in intra-client task confusion, where the model fails to differentiate between tasks within the same client. There is also an- other task confusion implicit in Eq. 5 that we call inter-client task confusion addressing the loss term between different tasks different clients. Failure to minimize inter-client task con- fusion will render the model incapable of distin- guishing between different classes of different clients. Thus, overall, there are four challenges that need to be considered: (i) preserving prior knowledge, (ii) effectively integrating new tasks, (iii) mitigating intra-cliens task confusion, and (iv) resolving inter-client task confusion. In the literature (i) and (iii) together are called local forgetting while (iv) is called global forgetting (Dong et al. [2022). In the next section, we will present our proposed approach and discuss how our proposed approach tackles these challenges.
D2
ab
Fhane
Figure 2: Our HR approach, except for Task con- sists of two key phases for handling tasks in FCIL. For Task 1, the autoencoder is trained, and the com- pact latent representations M1 are stored. For sub- sequent tasks in the Model Update Phase, com- pact representations from previous tasks M1 are decoded, interleaved with the new task's 's data Dr, and used to update the model. In the Memory Update Phase, with the updated model, compact representations for both the old tasks and the new task are computed and stored Mi
PROPOSED APPROACH: HYBRID REPLAY
Our proposed approach (shown in Fig. Hybrid Replay (HR), is based on specialized autoencoder that serves two primary functions: data classification and synthetic data generation. This customized autoencoder comprises
encoder and decoder operating as follows. The encoder f(x) IR" - R maps the input data x E IR" to a low-dimensional latent representation z E Rm The decoder g(z) m. R then reconstructs the input data from this latent representation.
While traditional Variational Autoencoders (VAE) (Kingma & Welling 2013) focus primarily on generalization and generating images, HR adapts the VAE framework balance modeling data
5

---

Published as a conference paper at ICLR 2025
distributions with enhanced class-specific clustering in the latent space. Accordingly, the modified loss function incorporates the standard VAE loss terms with an additional loss term designed to cluster samples around their class centroids:
L â. 2) ELBO ă, z) + (z,
K
N
(L(O((()))p(()) +
Fi
Pi.
(7)
=1
where ELBO represents the Evidence Lower Bound, KL denotes the Kullback-Leible divergence, I1-112 indicates the norm, Pi, represents the centroid embedding for the i-th task and j-th class, and À is hyperparameter.
To determine the embeddings new task centroids for all clients in the encoder's latent space, the server utilizes the Lennard-Jones Potential formulations (Jones) 1924). Specifically, the total potential energy V of the system, which represents the interactions between all pairs of class centroid embeddings, calculated as follows:
K,N Σ
12
4e N=
Σ
(8)
Piş
Piy
where lpv,, - -P; || the Fuclidean distance between the centroid embeddings. The parameters and a determine the depth of the potential well and the equilibrium distance, respectively.
To update the centroid embeddings, the server iteratively adjusts each p:, using the gradient of the potential energy, optimized with learning rate 1). The update can be mathematically expressed as:
Pi, - Pi,
Σ P.jj
24s
(pi, - -Pi,).
(9)
|pe,
Pi, 13
|P:. Pi, Pi, 11
Leonard-Jones formulations laid out above determine the global alignment centroids for new classes in new tasks. As described in lines 6-8 of Algorithm where clients compute the unaligned emheddings for new classes and send them to the server. Then the server aligns these centroids and sends them back the clients. Clients use these aligned centroids train on new tasks, ensuring minimal overlap with existing classes globally.
Memory update routine at the clients follows the well-known fixed memory procedure (Rebuffi
et al. (017)) (as opposed to growing memory design (Masana et al., 2020 where the number of exemplars of the classes decreases new classes arrive (Dong et al. 2022). To not cause clutter we refrain from reciting those well- well-known routines.
Synthetic data generation for replay operates through dual-path mecha- nism, as outlined in Algorithm [ For each client, if exemplars of partic- ular class from previous tasks exist in the client's 's memory M they are passed through the decoder g( to regenerate the corresponding sam- ples (Line 6 of Algorithm 2) This ensures faithful reproduction of pre- viously encountered data, preventing 10:
Algorithm HR Algorithm for FCIL at the Server
Inputs: R: total clients, K: tasks, clients per round, T.: local epochs, Q: communication rounds
2: Output: Optimized autoencoder 0jx
3: Initialize: O, at the server
4: for / = 1 K do
Select clients randomly from
6: Receive unaligned centroids (Pr,) from clients for task
7: Align centroids (Ph,) for task f via Eq. 9
8: Broadcast aligned centroids {Px, } for task h to clients
9:
for + = to f2 do
for each client from selected clients in parallel do
local forgetting. If exemplars are ab- sent, the centroid embedding pf, of that class is perturbed with Gaussian 13:
Update O,, at client using Algorithm end for
12:
Receive 0% from clients after communication round w
noise (0, o²) (Line 8 of Algorithm 14:
0, + Σ'= 0ī FedAvg (McMahan el al 2017
2h and then decoded to generate syn-
15:
Broadcast O, to clients
thetic samples. This method enables
16: end for
the client to generate synthetic data
17:
end for
for tasks classes it has not encountered directly, effectively preventing global forgetting.

---

Published as a conference paper at ICLR 2025
Classification in the model is performed by mapping imput a through the encoder f(0", az) directly. The predicted class for a is determined by argmin,,i.,l f x) Pi,ll where || denotes the Euclidean distance. This metric identifies the class whose centroid pi from client c is nearest to the encoded representation æ.
Local and Global Forgetting (Dong et al. 2022) has been detailed in the previous section in our mathematical framework where in Eqs. and 6 we identified four primary challenges in FCIL: (i) preserving prior knowledge, (ii) effectively integrating new tasks, (iii) mitigating intra-client task confusion, and (iv) resolving inter-client task confusion. In the literature, the challenges of preserving prior knowledge and mitigating intra-client task confusion are collectively referred to as local forgetting, while resolving inter-client task confusion is termed global forgetting (Dong et al. 2022). In the following, we discuss how Algorithm 1 and Algorithm 2 overcome local and global forgetting:
During the learning of task h, local forgetting addressed through a dual strategy: replaying previously seen data (Lines 7 and of Algorithm ( and applying knowledge distillation (Lines 19, 20, and 21 of Algorithm[2) (2) to retain knowledge from earlier tasks. In contrast, global forgetting handled differently. Losses related to task h across different clients, expressed in Eqs. [5 and shown Fig. []] cannot be minimized during the same session due to the unavailability of the relevant data or its synthetic representation at the other clients. Instead, this minimization is deferred to the (h + 1) session, where clients receive the class centroid embeddings (pi,) J (Line 8 of Algorithm T]. These embeddings enable clients to generate synthetic data representing tasks from other clients, helping the optimization of cross-client task losses and mitigating global forgetting.
5 EXPERIMENTS
Following the simulation settings outlined by Dong et al. (2022).
Algorithm 2 HR Algorithm for FCIL at Clients
I: Input: {T;, T2y..., }. {f(()) g(u))) memory Mo
our experimental evaluations were
2: Output: 6%. memories Mxx and (Di,)
conducted across five benchmarks:
3: for h = 1 to K do
CIFAR-100 100 (10/10/50/5 and
4: Initialize dataset D2 TT
ImageNet-Subsse (10/20/100 100/20/100/10 and 100//0).
20/5/50/5),
5: for cach client each previous task < h, each class j do
if memory Mi, exists then
and TinylmageNet 300/3300// where (A/B/C/D) indicates the simulation configuration, with A denoting the number of tasks, B. classes per task, C, number of clients, and D. the number of active clients per round. The inclusion of both short and long task sequences for CIFAR- 100 and ImageNet-Subset allows us to assess the robustness
7:
Dr. Dr. g(0n,
Use memory data
to synthesize samples
8:
else
9:
Dr Dr U g( Pf, +N(00,
Synthesize data samples from centroids with Gaussian noise end if end for
10:
11:
12:
Copy f(0.-1) and 9(0,-1) to f(O)), g(O))
13: Calculate unaligned (Pa,) = (f(((((n-1)
14: Transmit unaligned {Ph, to the server
of different baselines across varying
15:
for 1 to 9 do
For each communication round
16:
e to E do
b For each epoch
task lengths (Masana al. 2020).
17:
for b b= 1 to B do
b> For each minibatch
We utilized a ResNet-18 architecture
18:
Minimize the following loss:
for the diseriminatorrencooder /(). as
19:
L(Dr., 0 f(0., Dx)), (O.. Da))++
referenced in (Babakniya et al. 2023
( D.) f(O., Dx)
2024). and four-layer CNN for the generator/decoder 9(). The data for each task is distributed among clients using Latent Dirichlet Allocation
21:
l[g 1, (0,
Da)) ((0)))00... e f(O,, Dx))
22:
Using an optimizer to obtain f(0)), g(0,)
23:
end for end for
24:
(LDA) with the parameter a = 1. Clients utilize an SGD optimizer for local model training. The reported results represent the averages from ten random initializations accompanied by SEMs.
end for
26:
Update memory: M, + (On..g(O (nn.)))
Re-encode using new encoder and old decoder outputs Remove
27:
28:
Mr. t random(Th) > Randomly sample from the current
task's 's data to populate the memory
29: end for
7

---

Published as a conference paper at ICLR 2025
Table 2: HR versus exemplar-based and exemplar-free baselines. Accuracies and SEMs for 10 runs.
Belhods<Benchmarks
C10AR-100
Subsel
TinyImageNed
FCIL Configuration
10/10/50/2 50/5
20/5/50/5
10/20/100/10
20/10/100/10
10/5/300/30
Exemplar-based
Approaches (20 exemplars per class)
iCaRL Rehulfi al. 2017
50.25 +0.53
47.83 +0.49
46.05 =0.37
42.71
+0.32
47.34 +0040
BIC
WWr
2013)
54.38 +0.49
51.03 10.43
48.83 =0.41
45.02
1:036
48.79 40.43
P DDun ard ét 2020
58.97 1..47
5-.009 1:0.40
51.38
10.46
18.91 1:0.4
53.12 1:047
DDE
Hu 2021 + C..RL
56.46 +0.50
54.12 +0.48
52.01
+0.49
48.31
+0.40
53.06
+0041
GeolD
60.73 +0.57
56.98 +0.54
49.89
=0.40
47.11
+0.35
50.70
+0.41
SS-11.
Ahn et al
202
52.97 +0.52
50.23 +0.47
45.49 20.43
41.49
+0.39
45.08 +0.32
GLFC
Dong 0r
2022
61.83 1:059
57.09 1:0.5
53.83
19.06
1:0.4
54.23
Data-Free Approaches
FedFTG
Zhung
D029
46.86 +:049
43.01 +0.43
42.89 +0.31
39.08 +0.27
43.23 +0039
FedSpace
(Shena et
202A
45.73 00.51
41.57 40.45
40.69
20.24
38.28 +0.31
42.71 20.32
TARGET
Zhang
48.38 40
42.79 10.39
41.04
39.92 10.34
45.13
MFC1.
Bahukmya el al
2124
50.03 +0.33
44.88 +0.30
43.71 =0.38
40.04 +0.36
46.71
+0.47
Hybrid Approaches (200 Intent exemplars per class)
REMIND Hayes et al.
3020
62.29 +0.52
58.22
+0.42
53.80
=0.47
49.51 +0.45
54.79 +: 60060
REMIND wangg
63.47 10.56
59.61 10.44
54.79
50.23 4:049
55.92 1:0.7
iCTRL. Tong m. 2022
63.31 +0.49
58.93 +0.40
54.55 =0.38
49.92 +0.36
55.23
+:0.5
Our Proposed Hybrid Approach
(200 latent exemplars per class) and Ablation Study
HR
65.84 +0.57
60.48
+0.47
56 48 =0.38
52.35 +0031
57.86 +0059
HR-mini w 10x les memory
60.42 10.49
56.81 10.44
53.04
48.74 +0.29
54.02 10.53
HR w/o Latent Exemplurs
52.27 +0.32
45.18 +0
47.85 +0.31
43.62 +0.27
48.95 +:049
w Perfect Exemplars
66.53 +0.45
+0.40
57.65 =0.32
54.46 4:027
59.25 #0.51
HR w/o KD
64.07 +0.52
58.13 10.41
55.71
10.34
50.38
1:0.29
54.80
1:055
HR w/o Global Replay
51.76 4:0.4
11.38
1:0.23
16.27 00.25
41.09
1:0.21
46,95
4:0.42
HRv RFA
65.73 +0.60
60.50
40.48
56.44
+0.39
52.35
+0.33
57.81
+:0.3
We benchmark HR against a diverse set of baselines that can be categorized into three approaches: data-based, data-free, and hybrid approaches. Following Dong al. (2022) we include iCaRL
(Rebulfi et al. 2017). BiC (Wu et al. 2019). PODNet (Douillard et al. 2022). DDE (Hu et al.
2021 +iCaL. GeoDL (Simon et al.) 2021) + iCaRL, SS-IL (Ahn et al. 2021). and GLFC (Dong et al.
2022) as the data-based approaches. For data-free approaches, we include FedFTG ZZang
jtt al.
2022. , FedSpace (Shenaj al. 2023). TARGET (Zhang et al. 2023). and MFCL (Babakniya
et al.
2024).
And finally, for hybrid approaches, REMIND (Hayes et al., 2020), REMIND+ (Wang
ct al.
202T
and i-CTRI (Tong et al. 2022) are considered.
Results from Table 2 show that exemplar- based approaches generally outperform model-based (data-free) approaches, particularly in scenarios involving longer task sequences. This is because exemplar-based methods can directly use stored exemplars to mitigate forgetting, ensuring more stable and consistent performance across tasks. However, these approaches often come with significant memory costs, which can be impractical for FCIL applications. In contrast, hybrid approaches not only match but often exceed the performance of exemplar-based methods. Their advantage lies in the ability to store compressed or encoded exemplars instead of raw data, enabling up to 10 times more exemplar storage within the same memory budget.
In Table our proposed HR approach, even without latent exemplars (denoted HR-mini w 10x less memory), surpasses existing data-free methods by relying solely on synthetic data generation to address forgetting (Line 9 of Algorithm ( When HR operates in its default mode, using latent exemplars (Line 7 of Algorithm 2). it significantly outperforms exemplar- based baselines, not only in short task sequences but also in longer ones. We attribute this superior performance to HR's strong mathematical foundations leading to well-justified approach, which ensures effective mitigation of both local and global forgetting.
Ablation results are provided in the lowest section of Table where the first scenario labeled HR- mini w 10x less memory reports the performance of our approach under a significantly reduced memory budget. This scenario demonstrates the potential of HR to perform on par with exemplar- based approaches, despite using 10 times less memory. HR w/o Latent Exemplars explores a truly model-based and data-free variant of our approach. This model-based variant of HR outperforms the data-free baselines considerably, proving the robustness of HR even without relying on stored latent exemplars.
8

---

Published as a conference paper at ICLR 2025
HR Perfect Exemplars investigates the potential performance drop caused by the compression and storage of exemplars in the latent space. Interestingly, the results show that the performance difference between the storage of perfect exemplars and latent/encoded exemplars is negligible. This aligns with recent studies in (Van de Ven et al. 2020), suggesting that, for overcoming forgetting (as opposed to learning), the image quality for replay is not major factor. Thus, even compressed latent exemplars are sufficient for preventing forgetting. HR w/o KD examines the impact of removing the knowledge distillation (KD) regularization Ioss function (Lines 20 and 21 of Algorithm 2 Without KD, we observe noticeable performance degradation, particularly in long task sequences, primarily due to increased weight drift, which aligns with findings from studies conducted by RRbuff al.) (2017); Masana et al. (022)). This reinforces the importance of regu larization in mitigating forgetting and striking the right balance in the stability-plastiiig dilemma.
Another critical aspect of our approach is global replay, which ensures that clients have access to representations of tasks from other clients (Line of Algorithm 2). HR w/o Global Replay evaluates the consequences of removing this feature. The results reveal a severe performance drop, which can be attributed to the onset of global forgetting. As expected, without global replay, the system struggles to retain knowledge of tasks learned by other clients, leading significant degradation in overall performance. Lastly, in the case labeled HR w RFA, we replaced the Lennard- with the Repulsive Force Algorithm (RFA) (Nazmitdinov al. 2017) for class embedding alignment. The results indicate that RFA performs comparably to the Lennard-J formulation (Tones) 1924).
potential
Privacy-Pefformance Trade-off. Fig. B (first row) illustrates the trade-off between privacy and performance across different replay approaches. In exemplar-base methods, performance improves consistently as more memory becomes available to store additional exemplars. By contrast, model- based approaches lack this flexibility, as their performance remains constant regardless of memory size. However, model-based methods offer significant privacy advantage, as they do not store exact data samples.
Hybrid replay strikes an effective balance between these two approaches: it achieves high performance with low memory consumption while retaining the ability to improve further by storing more exemplars. This can be seen in Fig. 3 (first row) where hybrid replay approaches of HR (ours), REMIND+, and i-CTRL outperform, respectively. In terms of privacy, hybrid replay positions itself between model-based and exemplar-based methods, better than exemplars-base approaches but worse than model-based approaches.
Figure 3] (second row) illustrates the forgetting trends across three FCIL configurations. Hybrid replay approaches, particularly our approach, exhibit the least forgetting. This advantage arises from the ability of hybrid methods to store a greater diversity of exemplars within the same memory budget. The increased memory diversity significantly enhances performance and mitigates forgetting, making hybrid replay a highly effective solution.
3
Table compares the performance of our hybrid replay with three other exemplar- based replay baselines. It can be seen that for the same given memory and compute budget HR yields greater performance.
6 CONCLUSION
In the FCIL literature, two main categories of approaches have been explored to address local and global forgetting: exemplar-based (data-based) and model-based approaches. In this work, we introduced Hybrid Replay (HR), a novel approach that leverages customized autoencoder for both discrimination and synthetic data generation. By combining latent exemplar replay with synthetic data generation, HR effectively mitigates both local and global forgetting. Furthermore, we presented a mathematical framework to formalize the challenges of local and global forgetting and demonstrated how HR addresses these issues in principle. Extensive experiments across diverse basclines, benchmarks, and ablation studies confirmed the effectiveness of the proposed approach.
Limitations of HR and potential directions for future work. Since, our proposed FCIL approach, named HR, heavily relies on data generation based on latent exemplars and class centroids, requires careful consideration of the decoder's 's design to strike the right balance between realism and privacy. If the decoder lacks sufficient representational capacity, it may fail to generate realistic images necessary for effective replay. Conversely, if the decoder is overly complex, it could inadvertently
9

---

Published as a conference paper at ICLR 2025
Hemory Sice impoctt for OFAR 19011510550))
Nomory Size Imsoct for ImogeNet 0005e111002000000000
Nemory Size imppcct for Teyylmgge 10000000000
55-
57-
GuFD TAB
CCTRL
30
10-
120
14
120
120
Exemotan
Forpetting Trand CIFAR 19011510/5055
Forgetting
Subcc11020300100
30-
87-
75-
50
45-
Toals
Figure 3: In the first row, the table reports the impact of the memory size on the final accuracy for the three FCIL configurations. In the second row, the forgetting trends are shown for HR and various baselines. Hybrid replay consistently outperforms the model-based or buta-based approaches.
Table 3: Performances for a given memory and compute budgets for two FCIL configurations.
Memory
Strategies
Benchmarks
Exemplars
# Epochs
WC'iime
Performance
Decoder
Exemplar
CIFAR 100 10/ 10/50/5)
150 (latent)
1.4M
4.6M
682min
65.43 =0.93
ImageNet-Subset 10 22/1100/10)
190 (latent)
1.8M
40.54M
70
974min
56.09 #00.64
GooDL
ImageNet-Subsee
CIFAR- 10/ 10/50/3)
220//00/000 (100/0001000 10)
20 (raw)
20 (raw)
42. 2.
6M
1146min
703min
60.12 1:091
49. 23 1:0.62
CIFAR- (00 10/ 10/5075)
20 (raw)
6M
720min
52.81 1:074
SS-IL
Immgg/Nat-Sbsee(10000///1
20
42.31M
1096min
CIFAR- 100( 10/11//5//3)
20 (raw)
6M
788min
61.70 4:0.79
GLFC
ImageNet Subset (10/2000000000000/
20 (raw)
42.34M
1259min
53.83 4:055
memorize training samples, raising potential privacy concerns. This trade-off is particularly important because, for replay purposes, unlike for learning, generating high-quality samples is not necessary and a larger decoder would inflict a higher computational load during training (but not inference because the classification is only done via the encoder). Future work could explore alternative decoder architectures that optimize this balance, tailoring the trade-off between computational efficiency, replay performance, and privacy to the requirements of the target application.
ACKNOWLEDGMENT
This research was partly funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Rogers Cybersecure Catalyst Fellowship Program.
REFERENCES
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss- il: Separated softmax for incremental learning. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 844-853, 2021.
Sara Babakniya, Zalan Fabian, Chaoyang He, Mahdi Soltanolkotabi. and Salman Avestimehr. Don't memorize; mimic the past: Federated class incremental learning without episodic memory. arXiv:2307. 30700 2023.
10

---

Published as a conference paper at ICLR 2025
Sara Babakniya, Zalan Fabian, Chaoyang He, Mahdi Soltanolkotabi, and Salman Avestimehr, A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks. Advances in Neural Information Processing Systems, 36, 2024.
Yang Chen, Xiaoyan Sun, and Yaochu Jin. Communication--fffcieentt federated deep learning with layerwise asynchronous model update and temporally weighted aggregation. IEEE transactions on neural networks and learning systems, 31(10)42229443
2019.
Tong Chu, Yahao Liu, Jinhong Deng, Wen Li, and Lixin Duan. Denoised maximum classifier dis- crepancy for source-free unsupervised domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 472-480, 2022.
Jiahua Dong, Yang Cong, Gan Sun, Zhen Fang. and Zhengming Ding. Where and how to transfer: Knowledge aggregation-induced transferability perception for unsupervised domain adaptation.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(3):1664-1681, 2021.
Jiahua Dong, Lixu Wang. Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10164-10173, 2022.
Jiahua Dong, Hongliu Li, Yang Cong, Gan Sun, Yulun Zhang, and Luc Van Gool. No left behind: Real-world federated class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Inteligence, 2023.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16, pp. 86-102. Springer, 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostie meta-leamiin approach. Advances neural information processing systems, 33:3557-3566 2020.
Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfcil: Relation-guided representation learning for data-free class incremental learning. European Conference on Computer Vision, pp- 423-439, Springer, 2022.
Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication-effiienn algorithm for federated learning. International Conference on Machine Learning, pp. 3973-3983. PMLR, 2020.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In European conference on computer vision, pp- 466-483. Springer, 2020.
Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in dass-incremennal learning. Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pp. 3957-3966, 2021.
John Edward Jones. On the determination of molecular fields.-ii. from the equation of state of gas. Proceedings of the Royal Society of London. Series A, Containing Pupers of a Mathematical and Physical Character, 106(738):463 4777 1924.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference machine learning, pp. 1132-5143, PMLR, 2020.
Milad Khademi Nori and I1-Min Kim. Task confusion and catastrophic forgetting in dass-incremental learning: mathematical framework for discriminative and generative modelings. Advances in Neural Information Processing Systems, 011477778-77700 2025.
Jong- Ycong Kim and Dong- Wan Choi. Split-and- bridge: Adaptable class incremental learning within a single neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8137-8145. 2021.
11

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 2013.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114
(13):3521-3526, 2017.
Jakub Konečný, Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:/610.05492 2016.
Matthias De Lange, Xu Jia, Sarah Parisot, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Unsupervised model personalization while preserving privacy and scalability: An open problem. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp- 14463-14472, 2020.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, Proceedings of Machine learning and systems, 2:429-450, 2020.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv:2102.07623, 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2944 2017.
Yahao Liu, Jinhong Deng, Xinchen Gao, Wen Li, and Lixin Duan. Bapa-net: Boundary adaptation and prototype alignment for cross-domain semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8801-8811 2021a.
Yahao Liu, Jinhong Deng, Jiale Tao, Tong Chu, Lixin Duan, and Wen Li. Undoing the damage of label shift for cross-domain semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7042-7052, 2022.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition, pp. 12245-12254, 2020.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In Proceedings of the IEEE/CVF conference Computer Vision and Puttern Recognition, pp. 2544-2553, 2021b.
Lingjuan Lyu and Chen Chen. A novel attribute reconstruction attack in federated learning. arXiv:2108.00 069100 2021.
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: Survey and performance evaluation on image classifica- tion. arXiv:2010.15277, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficiee learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
RG Nazmitdinov, A Puente, M Cerkaski, and M Pons. Self-organization of charged particles in circular geometry. Physical Review E, 95(4):042603, 2017.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. arXiv:1911.02054, 2019.
Sylvestre-AIvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
12

---

Published as a conference paper at ICLR 2025
Donald Shenaj, Marco Toldo, Alberto Rigon, and Pietro Zanuttigh. Asynchronous federated continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5055-5063, 2023.
Wuxuan Shi and Mang Ye. Prototype reminiscence and augmented asymmetrie knowledge aggrega- tion for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 1772-1781, 2023.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE international conference on computer vision, pp. 3400-3409, 2017.
Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor- and Itai Zeitak. Overcoming forgetting in federated learning on non-iid data. arXiv:1910.07796, 2019.
Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On learning the geodesic path for incremental learning. In Proceedings of IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 1591-1600, 2021.
Shengbang Tong, Xili Dai, Ziyang Wu, Mingyang Li, Brent Yi, and Yi Ma. Incremental learning of structured memory via closed-loop transcription. 2rXiv:2202.05411,, 2022.
Gido M Van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):4069, 2020.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging, arXiv:2002.06440 2020.
Kai Wang. Joost van de Weijer. and Luis Herranz. Acae-remind for online continual learning with compressed feature replay. Pattern Recognition Letters, 150:122-129, 2021.
Chenshen Wu, Luis Herranz, Xialei Liu, Joost Van De Weijer, Bogdan Raducanu, et Memory replay gans: Learning to generate new categories without forgetting. Advances in neural information processing systems, 31, 2018.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proc. of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3014-3023, 2021.
Haibo Yang. Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv:2101. 11003, 2021.
Jachong Yoon, Wonyong Jeong, Giwoong Lec, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference Machine Learning, pp. 12073-12086. PMLR, 2021.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametrie federated learning of neural networks. In International conference on machine learning, pp. 7252-7261. PMLR, 2019.
Duzhen Zhang, Xiuyi Chen, Shuang Xu, and Bo Xu. Knowledge aware emotion recognition in textual conversations via multi-task incremental transformer. Proceedings of the 28th International Conference Computational Linguistics, pp. 4429-4440 2020a.
Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lyu. Target: Federated class-continual learning via exemplar-free distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 4782-4793, 2023.
13

---

Published as a conference paper at ICLR 2025
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10174-10183, 2022.
Xinbang Zhang, Jianlong Chang, Yiwen Guo, Gaofeng Meng, Shiming Xiang, Zhouchen Lin, and Chunhong Pan. Data: Differentiable architecture approximation with distribution guided sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):2905-2920) 2020b.
Xinbang Zhang. Zehao Huang, Naiyan Wang, Shiming Xiang, and Chunhong Pan. You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions Puttern Analysis and Machine Intelligence, 43(9):2891-299) 2020c.
Yiyang Zhang. Feng Liu, Zhen Fang, Bo Yuan, Guangquan Zhang, and Jie Lu. Clarinet: A one-step ap- proach towards audget-friendly unsupervised domain adaptation. arXiv preprint arXiv22007.16612 2020d.
Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient caass-incremental learning. arXiv:2205. / 13218, 2022.
14


Reviews:
[{"review_id": "wIxx9pI1xo", "reviewer": "Reviewer_Wd4J", "summary": "This paper categorizes current work in federated class-incremental learning into exemplar-based and data-free approaches, noting that exemplar-based methods face memory constraints and potential privacy risks, while current data-free methods suffer from efficiency issues. The authors propose an HR mathematical framework to address both local and global forgetting. HR leverages a new autoencoder and Jones Potential formulations to generate synthetic data with minimal memory overhead, aimed at mitigating the forgetting problem.", "questions": "1.\tIn Equation (3), what do the subscripts $i$ and $j$ represent? Additionally, in Equation (4), what does $j_l$ signify?\n2.\tWhen clients train on the same task, they each have data from different classes. How is the classifier configured in this scenario? Is it set up in a task-incremental mode or a class-incremental mode?\n3.\tThe authors should clearly explain how the proposed method achieves sample replay through Equations (10) and (11).", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe authors accurately summarize the limitations of current exemplar-based and data-free approaches.\n2.\tExperimental results indicate that the proposed method achieves lower computational and memory overhead compared to several optimal baseline methods.", "weaknesses": "1.\tThe motivation of this study appears somewhat outdated, as the results in the literature (e.g., [1]) indicate that method has already effectively addressed the issue of class imbalance within clients. \n2.\tThe authors should clearly outline the specific problem their work addresses. For example, in Figure 1, it is necessary to further clarify the mechanisms causing local forgetting and global forgetting, with separate, detailed explanations for each. \n3.\tThe study lacks essential comparative methods.\n4.\tThe study lacks visualized experimental results, such as accuracy on all old tasks after completing each task, and is missing essential forgetting metrics, such as Backward Transfer (BWT).\nReference：\n[1]\tYavuz Faruk Bakman, Duygu Nur Yaldiz, Yahya H. Ezzeldin, Salman Avestimehr. Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning. ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZFM480bBrk", "reviewer": "Reviewer_bkZ1", "summary": "This paper addresses Federated Class-Incremental Learning by focusing on two types of forgetting: local forgetting (at the client level) and global forgetting (between clients). The authors propose a hybrid approach, called Hybrid Replay (HR), which combines data-based and data-free methods to mitigate these issues. They introduce a mathematical formulation to formalize the forgetting problem and the presented approach HR. The approach uses autoencoders for synthetic sample generation and latent exemplars. Comparisons against other data-free and data-based approaches demonstrate that HR achieves better performance results.", "questions": "* The paper mentions that exemplar-based methods are memory-intensive. Could the authors provide an estimate of this memory cost, along with a comparison of memory usage between HR and other exemplar-based methods? An example or comparison with HR could help clarify this point.\n* While latent exemplars may save memory, the process of forwarding these exemplars through the decoder for sample generation could incur computational costs. It would be helpful if the authors discuss or quantify these costs when addressing the efficiency of their approach.\n* What metric is used for evaluation? Motivation of paper is mainly local and global forgetting but there is not any result or evaluation for forgetting.The table results do not clearly specify whether they represent incremental accuracy or the accuracy of the last task. Besides, higher incremental accuracy does not directly indicate lower forgetting. \n* The authors state that local and global forgetting are caused by class imbalances at both the local and global levels. Do they have any scenarios or relevant results that illustrate this point better?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper addresses an important problem in FCIL by combining data-based and data-free approaches to overcome local and global forgetting.\n* The hybrid approach integrates both latent exemplars and synthetic data generation, which are efficiently used to mitigate forgetting and results show that HR works better.\n* The mathematical formulation provided to describe these forgetting issues offers theoretical foundation.\n* Ablation studies in the paper contribute valuable insights into different aspects of HR and improve the interpretability of the approach.", "weaknesses": "Major Concerns:\n* The methodology, particularly the role of the autoencoder in addressing local forgetting, is not fully clear and can be explained better. For instance, while the paper states that the autoencoder helps address local forgetting, the specific details of how this is achieved are somewhat vague. The paper would benefit from a more detailed, step-by-step breakdown of how the autoencoder is employed for both local and global forgetting.\n* The paper lacks a clear visual representation of the HR approach. Including a diagram of the proposed method could significantly enhance understanding, especially as the provided Figure 1 only illustrates the problem without outlining the proposed solution. I believe such visual representations make papers to understand much better.\n* The results mention comparisons with a \"Hybrid Approach,\" but there’s little discussion on how HR stands out from other hybrid methods, such as  REMIND+ What makes HR approach unique when compared to other Hybrid Approach ? Clarifying these distinctions would strengthen the contribution of HR to the field.\n* The conclusion lacks discussion on the limitations of HR and potential directions for future work. Addressing this would provide a perspective on the approach’s implications and its broader applicability.\n* The method heavily relies on data generation based on latent exemplars and class centroids, which raises concerns since we don't have a direct control of generated data and Variational Autoencoders (VAEs) are known to be suboptimal for high-quality synthetic data generation. Over time, this could degrade the quality of latent features and ultimately impact classification performance.\n* The paper frequently references the Lennard-Jones formulation, but it doesn’t provide enough explanation about its purpose or why it’s important for the proposed method.\n\nMinor Concerns:\n* In Section 4, line 232, the acronym \"AHR\" is introduced without prior definition.\n* The caption for Table 1 could be made more descriptive to make the table more self-explanatory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wu83kulnCB", "reviewer": "Reviewer_PuPw", "summary": "This proposes a mathematical framework to demonstrate the global/local forgetting of FCIL and propose the Hybrid Replay (HR) to addressed these issues.", "questions": "1. What is the data partition of FCIL? Are the classes in different tasks disjoint? In the traditional CIL, categories in different tasks are disjoint. From the setting of ImageNet-Subset (10/20/100/10, 20/10/100/10) and Tiny-ImageNet (10/5/300/30), if A denotes the number of tasks, B denotes the classes per task and the classes in different tasks are disjoint, the total numbers of classes will be 200 for ImageNet-Subset and 50 for Tiny-ImageNet. However, the total numbers of categories in these two datasets are 100 and 200 respectively. Could you explain more about this?\n2。 Why does each client need to train from task 1? From Algorithm 1, the client only needs to update $theta_{h}$, but in the line 3 of Algorithm 2, the algorithm begins h=1.\n3. The practical FL systems may have stragglers. It is interesting to know whether the proposed HR algorithm can deal with the issue of stragglers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper establishes the revolution of training loss functions within the scope of both training tasks and clients, demonstrating the two challenges of global and local forgetting.\n2. A novel replay mechanism with centroids of each category is presented.", "weaknesses": "1. The presentation is not very clear. For example, exemplars in HR and other exemplar-based methods seem different but are used interchangeably in this paper. Also, I cannot get the indication of global/local forgetting in Figure 1.\n2. The mathematical formulation of FCIL and the proposed approach are not linked closely. Can you provide more information that how you establish the method based on the framework, especially for the global forgetting? It seems that the HR benefits from the class centroid embeddings and use them to address the global forgetting, is there any further analysis?\n3. Experiments are not sufficient. The results are limited to the LDA setting with alpha=1. Extended empirical results under different skewness (e.g., alpha=0.1 or more) should be included.\n4. Analysis of memory footprint should be included, e.g., the number of parameters need to be stored and transferred during the communication.\n5. Error in literature review. The Prototype reminiscence and augmented asymmetric knowledge aggregation [1] only addresses the CIL and it is placed within the FCIL methods.\n\n[1] Wuxuan Shi and Mang Ye. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1772–1781, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M6oaGIianJ", "reviewer": "Reviewer_WDPN", "summary": "The paper presents an approach named Hybrid Rehearsal (HR) for Federated Class-Incremental Learning (FCIL), addressing the challenges of local and global forgetting due to class imbalance. HR employs a customized autoencoder for both classifying data and generating synthetic data, leveraging latent exemplars to tackle local forgetting and synthetic data generation to overcome global forgetting. The paper's contributions include a mathematical framework to formalize forgetting in FCIL, a novel autoencoder design that balances class-specific clustering and data distribution modeling, and extensive experiments demonstrating HR's effectiveness over existing methods with low computational and memory costs.", "questions": "see the questions in the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper introduces Hybrid Rehearsal (HR), which combines the benefits of data-based (exemplar-based) and data-free approaches. This hybrid approach leverages latent exemplars for local forgetting and data-free techniques for global forgetting, providing a comprehensive solution to the forgetting problems in FCIL.\n2. The authors develop a mathematical framework to formalize the challenges of local and global forgetting in FCIL. This framework not only aids in understanding the underlying problems but also provides a theoretical basis for the proposed solutions.\n3. The paper provides extensive experimental evaluations across multiple benchmarks and compares the proposed approach with state-of-the-art baselines, demonstrating the effectiveness of HR.\n4.The paper is well-organized and most related works are properly cited.", "weaknesses": "1. The paper mentions using a customized autoencoder to leverage features for replay. I'm curious about what would happen if the encoder itself experiences forgetting? Additionally, since the stored features are fixed, but the encoder is continuously updated, how is this distribution inconsistency handled? 2. The paper mentions that the client receives class centroid embeddings ${p_ij}$ (line 8 of Algorithm 1). These embeddings enable the client to generate synthetic data representing tasks from other clients. However, if the received class centroid embeddings {pij} are for classes that the client has not seen, how can synthetic data be generated, and could this be detrimental to the client's learning? 3. How can the bias problem caused by multiple clients each learning a subset of categories be resolved when uploading the global model for model training and merging?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)