Overcoming Spatial-Temporal Catastrophic Forgetting for Federated Class-Incremental Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 19:01:43

Abstract: This paper delves into federated class-incremental learning (FCiL), where new classes appear continually or even privately to local clients. 
However, existing FCiL methods suffer from the problem of spatial-temporal catastrophic forgetting, i.e., forgetting the previously learned knowledge over time and the client-specific information owned by different clients. 
Additionally, private class and knowledge heterogeneity amongst local clients further exacerbate spatial-temporal forgetting, making FCiL challenging to apply.
To address these issues, we propose Federated Class-specific Binary Classifier (FedCBC), an innovative approach to transferring and fusing knowledge across both temporal and spatial perspectives.
FedCBC consists of two novel components: (1) continual personalization that distills previous knowledge from a global model to multiple local models, and (2) selective knowledge fusion that enhances knowledge integration of the same class from divergent clients and shares private knowledge with other clients. 
Extensive experiments using three newly-formulated metrics (termed GA, KRS, and KRT) demonstrate the effectiveness of the proposed approach.

Main paper: 
 introduction: 
 Federated Continual Learning (FCL) is a novel yet non-trivial research topic that bridges Federated Learning (FL) and Continual Learning (CL), aiming at building a federated model to collaboratively learn a sequence (possibly never-ending) of tasks. Most existing FCL methods aim to integrate CL techniques into FL to enhance the practicality of FL [3, 11, 34, 36] . However, these methods suffer from a fundamental challenge, namely, spatial-temporal catastrophic forgetting. We will expatiate this challenge shortly.In practice, the data collected by each client is usually different and heterogeneous [38] . For instance, a federated system of worldwide zoos would tackle different types of data. A desert wildlife monitoring station could have data about camels, and an oceanic monitoring station might have data about whales. We use the term "private class" to describe the classes that can only be accessed by one client. For us humans, we can gain indirect experience from others, acquiring knowledge that we haven't personally experienced. After learning the appearance of a tiger from the teacher, a child can easily identify it, even though he has never seen it before. It is seemingly easy for humans to adopt knowledge from others. However, it is extremely hard for a deep learning model to fuse heterogeneous knowledge from different clients training on different datasets, leading to severe spatial-temporal catastrophic forgetting [13, 27, 33] .What's more, knowledge acquired from others may conflict with one's existing knowledge [1] . Such a conflict will also happen in FCL. Human knowledge exchange occurs through communication. However, in FCL, knowledge exchange is accomplished through the aggregation of local models and the distribution of the global model. The knowledge of a model is often represented by parameters. Learning on heterogeneous data results in heterogeneous models. The more different the data, the greater the parameter divergence. Aggregating heterogeneous local models can lead to critical parameters for certain local tasks being overwritten, causing spatial catastrophic forgetting, i.e., poor performance on local test sets [18, 19, 38] . When the aggregated global model is distributed to clients and used as a base model to continually learn new tasks, the important parameters of the previous task have been further rewritten [22] , known as temporal catastrophic forgetting. Spatial-temporal catastrophic forgetting impedes traditional FL or CL techniques for FCL. The presence of private classes intensifies the heterogeneity among clients, making the fusion of local knowledge challenging and exacerbating spatial-temporal catastrophic forgetting [7, 39] . Existing research in FCL completely ignores the potential impacts of private classes. Moreover, they fail to solve spatial-temporal catastrophic forgetting, a fundamental challenge in FCL.To thoroughly investigate the spatial-temporal catastrophic forgetting in FCiL under extremely imbalanced class distributions, we considered an real-world application setting, as illustrated in Fig. 1 . In this problem, each client needs to continually tackle its training data, and the classes of these data also increase over time. Each client encounters different classes, including both public classes and private classes. Clients upload their direct experience to the server and receive indirect experience from other clients. In this way, clients continually gain the capability to recognize classes they have never observed before but are observed by other clients. The FCiL has two main objectives: the first is to ensure that local models do not forget the knowledge of old classes while learning new ones, which refers to overcoming temporal catastrophic forgetting. The second is to enable the global model to recognize all the classes encountered across all clients, addressing the problem of spatial catastrophic forgetting. To the best knowledge, we are the first to deal with such a problem.To address these issues, in this work, we propose a novel framework called Federated Class-specific Binary Classifier (FedCBC), which effectively overcomes spatial-temporal catastrophic forgetting in FCiL. Generally, we introduce the concept of anomaly detection [32] for classification as a promising strategy. To be more specific, we construct a class-specific binary classifier for each class, rather than the conventional deep neural network approach. Moreover, our approach allows for the fusion of relevant knowledge while excluding conflicting knowledge. On the server side, we realize selective knowledge fusion, enhancing the generalization performance of the global model and mitigating spatial and temporal catastrophic forgetting. On the client side, we utilize the global model to generate previous data and add to the new task's dataset, thereby overcoming temporal catastrophic forgetting. Additionally, we employ the global model as a teacher model to perform knowledge distillation on the local model. Compared to several recent baseline methods, our approach achieves state-of-the-art performance in terms of average accuracy on various benchmark datasets. Moreover, we design three new metrics to evaluate the performance of models in this setting. The contributions of this paper are summarized as follows:• We define a fundamental challenge in FCL, referred to as spatialtemporal catastrophic forgetting. In addition, we introduce a novel scenario, in which every client is required to perform classincremental learning, and each client possesses private classes that are exclusively accessible to them, with no data from these classes ever being available to others.• We propose a novel framework called FedCBC to address both spatial and temporal catastrophic forgetting. Moreover, employing variational autoencoders helps prevent the leakage of raw data, ensuring privacy and security. To our knowledge, the framework we designed exhibits state-of-the-art average accuracy performance in this problem domain. • We design three new evaluation metrics in terms of global accuracy, spatial knowledge retention, and temporal knowledge retention to measure the degree of heterogeneous knowledge fusion and the level of spatial-temporal knowledge forgetting. Experimental results on three datasets show the superior performance of the proposed approach against baseline methods. 
related work 2.1 federated class incremental learning: 
 Federated Class-Incremental Learning (FCiL) is a newly emerging research area, which focuses on overcoming catastrophic forgetting of previous tasks and data heterogeneity among clients jointly [12, 23, 37] . FedLwF [31] addresses catastrophic forgetting by distilling the knowledge of past local models to current local models, and addresses non-iid by distilling the general knowledge of global models to local models. GLFC [5] designs a class-aware gradient compensation loss to correct the imbalanced gradient propagation of old classes and a class-semantic relation distillation loss to keep inter-class relations consistent across tasks, and selects the best global model iteratively for preserving old knowledge with a proxy server. FedReconnaissance [11] treats the FCiL problem as maintaining the knowledge of the superset of classes observed by all clients and proposes to solve it with a prototypical network. AFCL [29] performs a prototype aggregation and a modified federated averaging aggregation on the server to overcome forgetting and client drift jointly. However, existing methods overlook a crucial challenge in FCL, called spatial-temporal catastrophic forgetting. Additionally, they equally disregard the challenges posed by aggregating heterogeneous models when each client possesses unique private classes. Furthermore, these methods [5, 36] rely on storing a portion of old samples, thereby compromising privacy-preserving protocols.Variational auto-encoders refer to a class of generative models that aim to learn the probabilistic mapping between the data space and the representation latent space [14] . A typical VAE consists of two main components: an encoder and a decoder. The encoder maps input data into a probabilistic distribution in the latent space, while the decoder, on the other hand, maps samples from the latent space back to the data space [25] . The training objective of VAE is to minimize the reconstruction error while regularizing the latent space to follow a specific prior distribution. Instead of directly learning the conditional distribution 𝑝 (𝑦|x), a VAE-based generative classifier learns the joint distribution 𝑝 (x, 𝑦), which is factored as 𝑝 (x|𝑦)𝑝 (𝑦), and to classify the samples via Bayes' rule [17] . In general, the application of VAE in federated learning is still limited, and existing works mainly focus on mitigating the cross-model covariate shift to address non-iid issues or detecting Byzantine attacks. VIRTUAL [2] uses a hierarchical Bayesian network on both the client and server side, transfers posterior within the FL system, and performs interference with variational methods. FedDNA [6] decouples gradient parameters and statistical parameters to reduce the divergence between the global model and local models. FREPD [8] uses VAE to compute the reconstruction error of local updates to detect and defend against malicious attacks. ss In this paper, we innovatively combine VAEs with binary classifiers, utilizing the reconstruction loss of samples to serve as a class-specific binary classifier to mitigate the spatial forgetting of the global model, i.e., non-IID issues, and achieve continual personalization of local models. Continual personalization aims to ensure that, in the iterative federated training process, the clients do not underfit their private classes, even if private class samples are only accessed by the client itself and are non-dominant in quantity. 
problem statement 3.1 federated class-incremental learning: 
 In traditional FL [20, 35] , there are 𝑎 clients A = {𝐴 1 , . . . , 𝐴 𝑎 } and one central server 𝑆. And each client {𝐴 𝑖 , 1 ≤ 𝑖 ≤ 𝑎} only has access to its own data D 𝑖 due to privacy concerns. Basically, one communication round should contain three steps: 1) Server 𝑆 distributes the initial model or the global model from the last round to clients, 2) Client 𝐴 𝑖 would use its private data D 𝑖 to train its local model 𝑀 𝑖 based on the model from the server, and 3) Server collects local models {𝜃 1 , . . . , 𝜃 𝑎 } then aggregates them to update the global model. The performance of the final global model should be very close to the performance of a centralized trained model [24] .We now extend the traditional FL to the class-imbalanced FCiL. C 𝑖 = {C 1 𝑖 ∪C 2 𝑖 ∪. . . , ∪C 𝑛 𝑖 𝑖 }. • At task 𝑟 , the global model 𝜃 𝑟 -1 𝑔 can distinguish C 𝑟 -1C 𝑟 𝑔 = {C 𝑟 -1 𝑔 ∪ C 𝑟 1 ∪ C 𝑟 2 ∪ . . . ∪ C 𝑟 𝑐 }.The goal of this setting is to end up with a global model that has assimilated all the tasks' knowledge acquired by individual clients, avoiding temporal catastrophic forgetting from the incremental local task progression and spatial catastrophic forgetting from aggregating heterogeneous local models of distinct clients (see Fig. 1 ).Catastrophic Forgetting is a fundamental challenge in CL, which mainly refers to a phenomenon that a model would forget the knowledge learned on old tasks when it is training on new tasks [4] . The reason for catastrophic forgetting is that the well-learned network parameters on the old tasks are overwritten during training on the new tasks [9] . In real-world applications, data is often collected gradually and a pre-trained model would continually train on the newly collected data for the new task requirements [28] .In the FCL setting, catastrophic forgetting exists as well. The assumption of static datasets in conventional FL is impractical. In a real-world scenario, data arrives at clients consecutively in the form of task streams, causing temporal catastrophic forgetting. When coming to the "aggregation" stage, the central server collects local models and aggregates them into one global model. After that, the server distributes the global model back to clients. Local models are trained with different training data. Aggregating them leads to the overwritten of certain task-specific crucial parameters, consequently causing a decline in the performance of the global model on local-specific tasks. Adopting the global model consolidated such conflict knowledge exacerbates the temporal catastrophic forgetting of each client itself previous tasks, especially for the non-overlapped classes, i.e., private classes.In a nutshell, temporal catastrophic forgetting is caused by the unavailability of data in time. Spatial catastrophic forgetting is caused by the inaccessibility of data in space. In FCiL, clients also need to preserve the knowledge learned from previous tasks and learn new knowledge on newly arrived tasks. On the other hand, the server should achieve a selective knowledge fusion to maximize the retention of local knowledge from different clients, especially for the knowledge of those private classes. 
proposed method: fedcbc: 
 In this section, we present the proposed method, i.e., Federated Class-specific Binary Classifier (FedCBC), to overcome spatialtemporal catastrophic forgetting, class privacy, and knowledge heterogeneity in FCiL. We firstly introduce binary classifiers for image classification instead of the traditional discriminative classifier in FCL. Specifically, on the client side, we construct a Class-specific Binary Classifier (see Section 4.1) for each class to determine whether a sample belongs to that class based on the reconstruction loss of variational auto-encoder. Subsequently, due to this unique network structure, it becomes easier to achieve Selective Knowledge Fusion (see Section 4.2) at the server, avoiding spatial catastrophic forgetting. It also enables the knowledge of private classes to be shared seamlessly between the server and clients. Finally, after receiving the more generalized global model from the server, the clients proceed to perform Continual Personalization (see Section 4.3) locally. This adaptation of the global model to local data distributions helps prevent temporal catastrophic forgetting.The overall framework of the proposed method is shown in Fig. 2 and the algorithm is summarized in Algorithm 1.When learning on a new task, the parameters of the network trained on previous tasks are overwritten, which results in temporal catastrophic forgetting. Furthermore, the overwritten parameters also bring about an overfitting towards new classes. In a vanilla class-incremental setting, the norms of weight vectors in the fullconnected layer for new classes tend to be larger, leading to network mispredictions of samples from previous classes as new ones.Based on the above analysis, it can be concluded that the key to mitigating forgetting is to prevent interference among class knowledge. To address this issue, we propose to use class-specific VAEs to memorize the specialized knowledge of each class separately. We start by training class-specific VAEs for each class on each client, and preserve the class-wise knowledge in them. When we compute the reconstruction loss with the original samples (features), they become a kind of class-specific binary classifier. When encountering a new class, all we need is to train another classifier for the new class, leaving the rest for previous classes unchanged. Training Stage: A VAE model consists of two parts: an encoder 𝑞 𝜙 and a decoder 𝑝 𝜃 . The encoder 𝑞 𝜙 maps the input 𝑥 to a posterior distribution 𝑞 𝜙 (𝑧|𝑥), and the decoder 𝑝 𝜃 is used to reconstruct the input sample 𝑥 from the latent variable 𝑧. Moreover, the prior distribution 𝑝 𝑝𝑟𝑖𝑜𝑟 (𝑧) is typically assumed to follow a standard normal distribution during the training process of a VAE, which is defined as:EQUATIONEQUATION)EQUATIONwhere 𝜇 𝜃 is the output of the decoder.The VAE models are trained by optimizing two parts. The first is about the Kullback-Leibler divergence between 𝑞 𝜙 (𝑧|𝑥) and N (0, 1). The second part is the reconstruction loss. Formally, the lower bound (or ELBO) is formulated as follows When it comes to the prediction stage, the test sample 𝑥 would go through all the VAEs and generate 𝑣 samples, where 𝑣 represents the number of VAEs. And the one that has the smallest reconstruction loss is the prediction class. In summary, the classification is done using: ŷ𝑥 = arg minL ELBO (𝜽, 𝝓; 𝒙) = 𝐸 𝑞 𝝓 (𝒛 |𝒙 ) log 𝑝 𝜽 (𝒙, 𝒛) 𝑞 𝝓 (𝒛 | 𝒙) = 𝐸 𝑞 𝝓 (𝒛 |𝒙 ) [log 𝑝 𝜽 (𝒙 | 𝒛)] -𝐷 𝐾𝐿 𝑞 𝝓 (𝒛 | 𝒙)∥𝑝 prior (𝒛) .EQUATIONwhere C denotes the entire class set and 𝑥 ′ 𝑦 is the reconstruct sample from the VAE of class 𝑦. In order to mitigate temporal-spatial forgetting, we introduced selective knowledge fusion on the server side. The main idea is to merge knowledge about the same class from different clients, along with the integration of past knowledge. Since the class-wise knowledge is stored separately in different VAEs, the selective knowledge fusion process is straightforward by simply consolidating the lists of key-value pairs uploaded from clients. Specifically, the server will group the model key-value pairs collected from various clients based on their class names. That is, the VAEs of the same class while from different clients are grouped together and fused separately. Such an approach by preventing the merging of unrelated knowledge is useful to avoid spatial catastrophic forgetting, especially when the data distribution is extremely Non-IID. Furthermore, if there is already a global model of the same class from the previous round, it can be also included in the group for selective knowledge integration. It is still helpful for alleviating temporal catastrophic forgetting.In the FCiL setting, the client's dataset consists of two types of data: samples from public classes and samples from private classes. Private classes refer to those classes that only the respective client has access to throughout the entire training process. Aggregation enables clients to acquire knowledge about private classes from other clients, granting them the capability to identify classes they have never encountered before. This approach avoids direct data sharing and prevents privacy.Public Class: Public class implies that multiple clients possess training data for this class and upload related models. Therefore, within this group, there will be multiple class-specific VAEs, including the global model of the last round if it exists. In such groups, we feed Gaussian noise data sampled from a normal distribution into the decoder part of the VAE to generate pseudo-samples. While these pseudo-samples belong to the same class, each VAE generates samples with its own unique local characteristics, much like coffee beans from different origins.Subsequently, these pseudo-samples are used as a training set for the next distillation step to generate a more generalized global model. First, we initialize a new VAE as the global model of this class, denoted as 𝑀 𝑔 . For the decoder part of 𝑀 𝑔 , the traditional MSE loss is used to train its reconstruction ability. And for the encoder part, there's something different about the training process. We consider the other local VAEs as teacher models, while the global VAE is the student model. The encoder of 𝑀 𝑔 maps input 𝑥 to a posterior distribution𝑞 𝑔 𝜙 (𝑧|𝑥) = N (𝜇 𝑔 , 𝜎 2 𝑔 ).Training the encoder part involves reducing both the KL divergence between N (𝜇 𝑔 , 𝜎 2 𝑔 ) and N (0, 1) and the KL divergence between N (𝜇 𝑔 , 𝜎 2 𝑔 ) and the average posterior distributions of other VAEs N ( μ, σ2 ). The final loss function of this stage is formulated as:EQUATIONwhere 𝛼 and 𝛽 are the hyperparameters. Finally, the generalized global VAE is obtained. Private Class: Private class means that there is only one local VAE specific to that class within the group. A naive method for private classes is to use this model directly as the global model. However, due to privacy concerns, this method is not feasible. Therefore, we follow a similar approach for handling public classes, where the local models and the previous round's global model still rehearsal samples. Afterward, these pseudo-samples are used as the training set to train and distillate the global model. Once the global model for a private class is trained, it will be distributed to the participating clients along with the models for other public classes. This way, other clients gain the capability to identify classes they have never encountered before, achieving knowledge sharing.Following the selective knowledge fusion on the server side, all global VAEs are distributed to the clients that just participated in the aggregation process. For the classes they are familiar with, clients will possess a more generalized VAE. Simultaneously, for classes that they have not encountered themselves but others have, clients will also have the capability to recognize them, as they have received knowledge about these unknown classes from others.Although the global model would be more generalized, the performance on the local test set could still be worse than existing local models [26] . Moreover, when learning new data about the old classes, some knowledge may still be forgotten because of concept drift. Therefore, based on the idea of personalized FL [30] , the global model would only be used as a teacher model on the client side. On the one hand, the global model rehearsals previous knowledge samples and integrates them into newly collected data, thereby curbing temporal catastrophic forgetting at the data level. On the other hand, employing knowledge distillation limits the output of local models, mitigating temporal catastrophic forgetting in terms of model output. 
experiments 5.1 experiment setup: 
 Datasets. To evaluate the performance of our method, we use three datasets: MNIST [16] , CIFAR-10 [15] and CIFAR-100 [15] in our experiments. In our setup, the federated system consists of three clients and one central server, and each client possesses a sequence of five unique tasks. Initially, we divide the data for each class into three parts using a Dirichlet distribution to ensure that there is no data overlap between clients. For MNIST and CIFAR-10, each client exclusively owns two classes that were only accessible to itself and not accessible to others. Therefore, all clients could only access four classes. The data for each client's tasks is randomly sampled from these six classes, with three classes chosen for each task. For CIFAR-100, we allow each client to have 25 private classes, resulting in 25 common classes. Each task consists of 10 classes sampled from both private and common classes, with no class overlap between tasks.Baselines. To have a comprehensive evaluation, we compared our method with representative existing FCiL methods. The compared methods include: (1) FedAvg [24] , a standard approach for FL.(2) FedAvg+EWC, integrating a regularization-based approach of continual learning to the standard FL framework. (3) FedProx [21] , a well-known FL method aiming to address statistical heterogeneity. (4) GLFC [5] , a newest and famous FCiL method using multiple complex components. (5) FedSpace [29] , an asynchronous FCiL method. For baseline algorithms, we employed ResNet-18 [10] as the backbone network.Implementation. The code of each method is implemented in PyTorch. For the baseline algorithms, the employed classification network is ResNet-18. We conducted experiments on each dataset using three different random seeds (42, 1999, 2002) and averaged the results. We set the number of global epochs as 5 and the number of local epochs as 50. For CIFAR-10 and CIFAR-100, we utilized 5% of the data of each class for pretraining the feature extractor. The whole training process is performed sequentially on an NVIDIA GPU RTX-3090. Our code is now anonymously hosted at: https://anonymous.4open.science/r/FedAE-CDE7/.Since spatial-temporal catastrophic forgetting is a novel challenge that we first introduced, lacking measurements, we have designed three different metrics to assess heterogeneous knowledge integration, temporal knowledge retention and spatial knowledge retention. All three metrics are designed based on accuracy.Global accuracy. Specifically, it is the accuracy of the global model testing on the test set of all classes. This metric is used to measure the degree of heterogeneous knowledge fusion. Since each client has some private classes, testing the aggregated global model on a test set containing all classes can detect whether these unique knowledge aspects have been preserved. For example, if only one client's model was trained on Apple images, and after aggregation, the global model still performs well on an Apple test set, it indicates that it has retained the unique knowledge about apples without being overwritten. In short, it is used to evaluate the ability of the global model to recognize all the classes encountered across all clients.Temporal knowledge retention. We use 𝐾𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 𝑅𝑒𝑡𝑒𝑛𝑡𝑖𝑜𝑛 as measurement of forgetting. Temporal knowledge retention is designed to measure the extent to which local models retain knowledge of old tasks as they learn on the task sequence. 𝐴𝑐𝑐 From the results, we can see our method achieves 90.01% on MNIST, 67.74% on CIFAR-10, and 29.35% on CIFAR-100, showing the state-of-the-art performance of fusing heterogeneous local models.In the following section (i.e., Sec. 5.4), we will evaluate each method using new metrics, i.e., temporal knowledge retention and spatial knowledge retention, to evaluate the resistance of spatialtemporal catastrophic forgetting.To validate the effectiveness of our proposed method in mitigating both spatial and temporal catastrophic forgetting, we conducted experiments along with baselines to test the preservation of spatialtemporal knowledge. Additionally, we performed ablation experiments on our method. Fig. 3 shows the spatial knowledge retention of all methods on three datasets. In Fig. 3b and Fig. 3c , we notice that after aggregation on the server, the global model performs even better than the local models. It indicates our method is robust to spatial catastrophic forgetting. However, the spatial knowledge retention drops shapely when we remove the selective knowledge fusion on the server side (Ours-w/oSKF). It indicates this strategy helps. Fig. 4 shows the temporal knowledge retention of all methods, which indicates the ability to migrate catastrophic forgetting in time. When we removed the continual personalization on the local side, the temporal knowledge retention would drop to around 90% on the MNIST and CIFAR10. It is mainly determined by the model's architecture and is not heavily influenced by the mechanisms. Furthermore, we set up one VAE module for each class. However, due to the redundancy of neural networks, data of multiple classes can be included in a single VAE module for classification, further reducing communication overhead and storage space.Although the storage space required by our method grows linearly with the number of classes when facing a large number of class-labeled data, the increase in parameter count is tolerable compared to its superior performance in overcoming catastrophic forgetting in both spatial and temporal aspects. Figure 5 : As 𝛼 increases, the quality of replayed fake samples improves, enhancing the accuracy of the method but simultaneously reducing the security of privacy. Fig. 5 shows the accuracy of our method as 𝛼 in Equ. 7varies. On one hand, 𝛼 controls the quality of generated pseudo-samples by regulating the weight of the MSE loss. On the other hand, 1 -𝛼 controls the weight of the KL divergence between the true latent distribution and the standard normal distribution. In other words, higher 𝛼 values indicate higher quality of generated pseudo-samples and lower privacy protection. Conversely, lower 𝛼 values signify a more distorted shift in the latent distribution, resulting in stronger privacy protection. Figure 3 : Spatial knowledge retention (Eq. 9). Note that, 'Ours-w/oSKF' refers to our method without selective knowledge fusion on the server side. 'Ours-w/oCP' refers to our method without continual personalization on the client side. Fig. 6 shows the visualization of the generated pseudo-samples on MNIST when 𝛼 = 10 -3 and 𝛼 = 10 0 . Clearly, when 𝛼 = 10 -3 , the generated pseudo-samples are very blurry, resulting in the lowest accuracy for FedCBC. However, when 𝛼 = 10 0 , although the generated pseudo-samples are clear and the accuracy is satisfactory, privacy protection is not controlled. FedCBC can adjust the trade-off between privacy security and performance by changing 𝛼 
conclusion: 
 Federated Clsss-Incremental Learning (FCiL) is a novel yet nontrivial research topic. This paper investigated a new and real-word setting problem, where new classes appear continually to each client and some classes are private to certain clients. Therefore, class privacy emerging on certain clients and knowledge heterogeneity coming from different clients are two basic challenges for this problem. In addition, we discussed a significant challenge, referred to as Spatial-Temporal Catastrophic Forgetting.To address these challenges, we proposed a Federated Classspecific Binary Classifier (FedCBC) approach. To evaluate the performance of FedCBC and its ability to resist spatial-temporal catastrophic forgetting, we designed three new metrics to measure the ability to fuse heterogeneous knowledge and the preservation of temporal and spatial knowledge. Experimental results on three datasets showed that the proposed approach outperformed the existing baseline methods markedly. 
 
 Appendix:  


Decision: This paper focuses on federated class-incremental learning and points out the spatial-temporal catastrophic forgetting including forgetting previous knowledge and forgetting local knowledge. The author introduces the Federated Class-specific Binary Classifier with continual personalisation and selective knowledge fusion. The authors also suggest evaluation strategies for the efficacy of the proposed methods, of which the results reveal the superiority of their proposal. All the reviewers agree that the paper has some novelty and the experimental results are sound. However, it requires additional training time and more efforts should be paid to explain the concept of spatial-temporal catastrophic forgetting. Overall, I tend to accept the paper, while I hope the authors could pay more efforts to address aforementioned issues in their revision.

Reviews: review: paper_summary: paper_summary: This paper proposes FedCBC to overcome Spatial-Temporal Catastrophic Forgetting on class-incremental learning.
strength_weakness: strength_weakness: 1. The methods are intuitive and well-motivated.
2. The figures are good.
3. Experiments are extensive and solid.
limitations: limitations: 1. Comparsions are not sufficient. More SOTA Federated methods should be included, not just FedProx.
2. The concept of Spatial-Temporal Catastrophic Forgetting have not been sufficiently analyzed, remaining on the words description. Reference:
Lee G, Jeong M, Shin Y, et al. Preservation of the global knowledge by not-true distillation in federated learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 38461-38474.
Chen J, Zhu J, Zheng Q. Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 8697-8706.
paper_summary: paper_summary: This manuscript proposes a federated class-specific binary classifier framework, named FedCBC, to address the issue of spatial-temporal catastrophic forgetting in federated class-incremental learning (FCiL). FedCBC transfers and fuses knowledge across both temporal and spatial perspectives through continual personalization and selective knowledge fusion. Experiments have been conducted on three datasets.
strength_weakness: strength_weakness: 1. **Novelty and Importance:** This manuscript is the first study to address the spatial-temporal catastrophic forgetting problem in FCiL, which is crucial and fundamental. In this scenario, each client possesses private classes, with no data from these classes being available to other clients.
2. **Innovative Framework:** The proposed framework, FedCBC, is presented clearly and contributes several new ideas. The main ideas of FedCBC are creative and distinctive.
3. **Comprehensive Experiments:** Many experiments have been conducted to validate the effectiveness of the proposed method.
limitations: limitations: 1. **Baselines:** The chosen baselines are not strong enough. FedSpace is published in a related workshop. Additionally, only GLFC is specifically designed for federated class-incremental learning. Please provide more comparison results with other previous works.
2. **System Complexity:** The use of variational autoencoders (VAEs) may increase the system's complexity and computational overhead. Thus, the training time of FedCBC compared to other baseline methods needs to be included.
3. **Privacy Analysis:** The paper does not mention the theoretical analysis of privacy protection provided by VAEs, despite claims that they help prevent the leakage of raw data.
paper_summary: paper_summary: This paper focuses on federated class-incremental learning and points out the spatial-temporal catastrophic forgetting including forgetting previous knowledge and forgetting local knowledge.  The author introduces the Federated Class-specific Binary Classifier and demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The paper is well-written and easy to understand. The problems are clear with the respective solutions.
limitations: limitations: There are several limitations.

First, the author introduces the class-specific VAEs for each class on each client. However, it brings additional computation costs for each client and requires additional communication costs in the federation. I urge the authors to discuss the relative computation and communication cost with the relative methods.

Second, I recommend adding the experiments with more task-splitting settings. For example, CIFAR-100  with 10/20 lass-
incremental tasks for each client. Furthermore, I wonder whether to conduct the experiments with various feature shifts. For example, digits setting, different client from the MNIST, SVHN domains to evaluate the robustness of VAE.
paper_summary: paper_summary: This paper addresses the challenge of Spatial-Temporal Catastrophic Forgetting in Federated Class-Incremental Learning, a critical concern for maintaining continuous learning without losing previously gained knowledge in distributed systems. To counter this issue, the paper introduces the Federated Class-specific Binary Classifier (FedCBC), a ground-breaking framework designed to mitigate forgetting through two key innovations: (1) continual personalization, which distils and transfers knowledge from a global model to local models, and (2) selective knowledge fusion, which integrates and shares class-specific insights among diverse clients. The efficacy of FedCBC is evaluated using three newly developed metrics: GA, KRS, and KRT. Experimental results demonstrate that FedCBC not only improves learning accuracy but also reduces communication costs compared to five other existing methods.
strength_weakness: strength_weakness: This paper includes two novelties: The introduction of the FedCBC framework as a novel solution to the issue of Spatial-Temporal Catastrophic Forgetting in Federated Class-Incremental Learning is a primary novelty. The introduction of three new metrics (GA, KRS, KRT) for evaluating the performance of federated learning models is another strength, which provide a broader and perhaps more precise evaluation framework tailored to the specific challenges addressed by the paper.

The approach replaces the traditional discriminative classifier with binary classifiers, each specific to a class, which evaluate if a sample belongs to a class based on the reconstruction loss from a variational auto-encoder. This setup simplifies the Selective Knowledge Fusion process on the server side, effectively preventing spatial catastrophic forgetting and facilitating the sharing of knowledge about private classes between the server and clients. Following the server's distribution of a more generalized global model, clients customize this model locally through Continual Personalization to adapt to local data variations, thereby mitigating temporal catastrophic forgetting.

The paper makes extensive experiments, which compare FedCBC against five existing methods using three datasets, lend significant credibility. This comparative analysis underscores FedCBC's improvements in terms of accuracy and communication costs.

In summary, the paper’s introduction of the FedCBC framework is notable for its innovative approach to a significant problem in federated learning. The theoretical underpinnings are sound, and the detailed experimental evaluation provides a strong justification for the proposed methods. The clarity of the presentation enhances its impact, making it a valuable contribution to the field.
limitations: limitations: In section 1, the explanation of the causes of spatial-temporal catastrophic forgetting is somewhat dispersed and could benefit from being more logical and clearer. For example, Line 101-104 and line 130 need more specific explanation and evidence to help reader understand well.

In section 3.2, comparing to FL and FCL, the analysis of what leads to spatial-temporal catastrophic forgetting in FCiL is inadequate.

In section 5.3 and 5.4, There are inconsistencies of components between table 1-3 and figure 3-4, which affect the credibility and persuasiveness of experimental data and results. AFCL is not in the table but in the figure. FedSpace is in the table but not in the figure. If this is correct, it should be stated clearly at the beginning of 5.3 and 5.4.

Other minor errors: Line 131 ‘an’ should be ‘a’, line 658 ‘spatial’ may be ‘temporal’.