Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator

Authors: Thinh Nguyen, Khoa D Doan, Binh T. Nguyen, Danh Le-Phuoc, Kok-Seng Wong
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: federated learning, continual learning, federated continual learning, generative model

Abstract:
Federated Class-Incremental Learning (FCIL) increasingly becomes essential in the decentralized setting, where it enables multiple participants to collaboratively train a global model to perform well on a sequence of tasks without sharing their private data. In FCIL, conventional Federated Learning algorithms such as FedAvg often suffer from catastrophic forgetting, resulting in significant performance declines on earlier tasks. Recent works based on generative models produce synthetic images to help mitigate this issue across all classes. However, these approaches' testing accuracy in previous classes is still much lower than recent classes, i.e., having better plasticity than stability. To overcome these issues, this paper presents Federated Global Twin Generator (FedGTG), an FCIL framework that exploits generative-model training on the global side without accessing client data. Specifically, the server trains a data generator and a feature generator to create two types of information from all seen classes. Then, it sends the synthetic data to the client. The clients then use feature-direction-controlling losses to make the local models retain knowledge and learn new tasks well. We extensively analyze the robustness of FedGTG on natural images and its ability to converge to flat local minima and achieve better predicting confidence (calibration). Experimental results on CIFAR-10, CIFAR-100, and tiny-ImageNet demonstrate the improvements in accuracy and forgetting measures of FedGTG as well as the robustness of domain shifts compared to previous frameworks.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730173849519

thread_type:
official_review

<thread_object>
creator: reviewer_asls

<content>
summary:
Federated Learning (FL) is a privacy-preserving machine learning approach, but it faces challenges related to resource limitations and data heterogeneity, especially as client data distributions evolve over time. Federated Class-Incremental Learning (FCIL) combines FL and Class-Incremental Learning (CIL), enabling models to continuously learn new knowledge from distributed data sources without forgetting prior knowledge. However, existing methods such as TARGET and MFCL, which use data generation to balance knowledge retention for new and old tasks, still encounter forgetting issues. To address this, this paper proposes the Federated Global Twin Generator (FedGTG) framework, which trains and shares data and feature generators on the server side, using direction-control loss to enhance stability and plasticity on the client side. Experiments demonstrate that FedGTG outperforms existing methods in terms of accuracy and forgetting rate.

strengths:
1. This paper enhances current GAN methods for Federated Incremental Learning by integrating both a feature GAN and a data GAN, with the feature GAN employing regularization to achieve improved performance and stability across incremental tasks.  
2. Extensive experiments across multiple datasets demonstrate the robustness and validity of the proposed model, showcasing its effectiveness in various scenarios.  
3. Unlike prior GAN-based methods, which often suffer from catastrophic forgetting, this work introduces innovative mechanisms that significantly mitigate such issues, representing a valuable contribution to the field.

weaknesses:
1. The experiments in the paper lack critical details, including the ratio of generated data used in each training cycle, the proportion within individual batches, and the hyperparameters for each loss function, which should ideally be explored through ablation studies. Additionally, the class-incremental setup is not clearly explained, leaving me unsatisfied with the experimental rigor.

2. The datasets used throughout the paper are relatively simple; I recommend conducting further experiments on high-resolution datasets such as ImageNet to strengthen the evaluation.

3. The contributions of the paper are limited. For instance, incorporating a feature GAN as a regularization term is not substantially different from traditional regularization methods in incremental learning.

4. It is unclear whether the GAN trained after each task is independently trained or incrementally trained based on the previous task. If trained independently, this approach would impose excessive storage and communication costs; if incrementally trained, the GAN itself could suffer from catastrophic forgetting. These aspects present limitations in the proposed method.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730183782746

thread_type:
official_review

<thread_object>
creator: reviewer_nvr3

<content>
summary:
- This paper focuses on federated continual learning, federated class-incremental learning to be more specific. It proposes a new method of using two generator to generate synthetic data and feature. The synthetic data and feature are sent to the client for local training. In local training, this work proposes multiple losses to improve the training process.

strengths:
- The propose method of using an additional generator is simple and intuitive, while it is not straightforward to make it work. It is technically sound that the authors propose additional loss functions to stablize the training. 
- The paper is generally well-written and easy to follow.
- Extensive experiments are conducted through 3 datasets with detailed ablation studies. Sufficient exisiting works are compared. The proposed method achieves much strong performance than the compared counterparts.

weaknesses:
- Figure 1 is not easy to  understand. It would be useful if the authors could provide more explanations or making it more intuitive.
- The abstract claims that “these approaches’ testing accuracy in previous classes is still much lower than recent classes”, but it seems the paper does not provide experiments showing that the proposed method addresses this issue.
- It seems that only a single backbone is used in the evaluatiion. 
- Typo in line 83, should be “methods”.

questions:
- Does this work focus on cross-silo or cross-device FL? What is the number of clients evaluated?
- What is the meaning of “stability-plasticity” ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730451848699

thread_type:
official_review

<thread_object>
creator: reviewer_1trm

<content>
summary:
This paper introduces Federated Global Twin Generator (FedGTG), and proposes a framework leveraging generative models on the global side, sending synthetic data to clients for retaining knowledge and learning new tasks effectively. FedGTG shows enhanced accuracy, reduced forgetting, and robustness against domain shifts on CIFAR-10, CIFAR-100, and tiny-ImageNet.

strengths:
1. The paper is overall well-written in terms of writing.
2. The introduction part clearly especify the problem about generators.
3. The experiments are showing nontrivial improvements over compared techniques.

weaknesses:
1. Not clear what the biggest contributions are. Is it the two generator structure? If so, the novelty does not seem very high. The paper also introduces a direction-controlling objective, which should also stand out, but not very clearly in the paper (e.g., Figure 1).
2. Need to include training time or cost in the comparison. It seems that training two generators are quite costing, and it already needs more parameter during the training.
3. Is the method an exemplar-free technique? If so, the literature reviewer is not very complete. Quite a few new works on exemplar-free CIL are missing in the continual learning section, such as the analytic continual learning branch [1-3], and prototypes-based CIL.

[1] "ACIL: Analytic class-incremental learning with absolute memorization and privacy protection." Advances in Neural Information Processing Systems 35 (2022): 11602-11614.

[2] "GKEAL: Gaussian kernel embedded analytic learning for few-shot class incremental task." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] "DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.

questions:
see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517965082

thread_type:
official_review

<thread_object>
creator: reviewer_6tck

<content>
summary:
This paper introduce FedGTG (Federated Global Twin Generator), a novel FCIL framework that leverages generative models on the server side without accessing client data. FedGTG trains a data generator and a feature generator to produce synthetic representations of all seen classes, which are then shared with clients to help balance knowledge retention (stability) and learning new tasks (plasticity). Through extensive experiments on CIFAR-10, CIFAR-100, and tiny-ImageNet, FedGTG shows significant improvements in both accuracy and resistance to forgetting

strengths:
1. The paper writing is clear and easy to follow.
2. The proposed FedGTG absorbs the advantage of many other algorithms, leading to a strong framework for FCIL.
3. The experimental results are good. The proposed FedGTG outperforms all baselines on each benchmark and each incremental task. And the ablation study shows that each loss term is crucial and  has a large impact on the final results.

weaknesses:
1. The proposed lacks enough novelty. All loss functions used in FedGTG are proposed by other papers.
2. The framework contains too many hyperparameters, as each loss term needs one. It seems the framework is hard to be finetuned to the optimal result, as there are too many combinations of choice of hyperparameters. And the paper lacks the experiments and discussion of these hyperparameters.
3. More baselines should be compared. Except for generative-based methods, authors should compare FedGTG with other SOTA FCIL algorithms.

questions:
Why do you design the model architecture of data and feature generation as you shown in the paper? Are there any key ideas of why using these architectures?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732515170641

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We want to thank you the reviewers for providing constructive comments and valuable questions, helping us to improve our paper's quality. In our rebuttal, we have included new experiments in response to all of the reviewers' questions and recommendations. We hope all reviewers to provide any further feedback they may have. We promise to respond to any further inquiries as soon as possible. Lastly, we respectfully ask for the re-evaluation the score assigned to our paper. We really appreciate your help.

### Main changes

1. We added training time comparison between FedGTG and other methods. The training time of FedGTG is comparable with MFCL but achieved better performance, as shown in Table 1, Figure 4-6.
2. We conducted the experiment on different architectures, including ResNet34 and ResNet50 on CIFAR-10, CIFAR-100, tiny-ImageNet and HealthMNIST, as well as on additional regularized-based algorithms, which are FedWeIT and FedEWC. Experimental results have shown that our method can retain old knowledge and learn new tasks efficiently, while the two prior works fail to do so.
3. We conducted additional dataset which is SuperImageNet to test the effectiveness of FedGTG under a more challenging scenario.
4. We addressed hypeparameter selection to help users fine-tune their work easier.
5. We address minor typos in the manuscripts.

### References

[1] Babakniya, Sara, et al. "A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks." Advances in Neural Information Processing Systems 36 (2024).

[2] Yin, Hongxu, et al. "Dreaming to distill: Data-free knowledge transfer via deepinversion." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[3] Smith, James, et al. "Always be dreaming: A new approach for data-free class-incremental learning." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[4] Yoon, Jaehong, et al. "Federated continual learning with weighted inter-client transfer." International Conference on Machine Learning. PMLR, 2021.

[5] Zhang, Jie, et al. "Target: Federated class-continual learning via exemplar-free distillation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces FedGTG (Federated Global Twin Generator), a framework that leverages generative models on the server side without accessing client data. The main idea is quite simple, using two generators to generate synthetic data and features, which are sent to the client for local training. One common concern shared by multiple reviewers is the novelty of the paper. In particular, as pointed out by reviewers, the main loss function design for local training boils down to a combination of existing losses in the literature, and it lacks principled justification on why one should do so. The authors' responses answered some of the clarification questions from the reviewers but the key limitation still remains. The reviewers have reached a consensus of not accepting the paper.