Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting

Authors: Milad Khademi Nori, IL MIN KIM, Guanghui Wang
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Class-Incremental Learning, Federated Learning, Global Forgetting, Local Forgetting.

Abstract:
Federated Class-Incremental Learning (FCIL) refers to a scenario where a dynamically changing number of clients collaboratively learn an ever-increasing number of incoming tasks. FCIL is known to suffer from local forgetting due to class imbalance at each client and global forgetting due to class imbalance across clients. We develop a mathematical framework for FCIL that formulates local and global forgetting. Then, we propose an approach called Hybrid Rehearsal (HR), which utilizes latent exemplars and data-free techniques to address local and global forgetting, respectively. HR employs a customized autoencoder designed for both data classification and the generation of synthetic data. To determine the embeddings of new tasks for all clients in the latent space of the encoder, the server uses the Lennard-Jones Potential formulations. Meanwhile, at the clients, the decoder decodes the stored low-dimensional latent space exemplars back to the high-dimensional input space, used to address local forgetting. To overcome global forgetting, the decoder generates synthetic data. Furthermore, our mathematical framework proves that our proposed approach HR can, in principle, tackle the two local and global forgetting challenges. In practice, extensive experiments demonstrate that while preserving privacy, our proposed approach outperforms the state-of-the-art baselines on multiple FCIL benchmarks with low compute and memory footprints.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730294917392

thread_type:
official_review

<thread_object>
creator: reviewer_wdpn

<content>
summary:
The paper presents an approach named Hybrid Rehearsal (HR) for Federated Class-Incremental Learning (FCIL), addressing the challenges of local and global forgetting due to class imbalance. HR employs a customized autoencoder for both classifying data and generating synthetic data, leveraging latent exemplars to tackle local forgetting and synthetic data generation to overcome global forgetting. The paper's contributions include a mathematical framework to formalize forgetting in FCIL, a novel autoencoder design that balances class-specific clustering and data distribution modeling, and extensive experiments demonstrating HR's effectiveness over existing methods with low computational and memory costs.

strengths:
1.The paper introduces Hybrid Rehearsal (HR), which combines the benefits of data-based (exemplar-based) and data-free approaches. This hybrid approach leverages latent exemplars for local forgetting and data-free techniques for global forgetting, providing a comprehensive solution to the forgetting problems in FCIL.
2. The authors develop a mathematical framework to formalize the challenges of local and global forgetting in FCIL. This framework not only aids in understanding the underlying problems but also provides a theoretical basis for the proposed solutions.
3. The paper provides extensive experimental evaluations across multiple benchmarks and compares the proposed approach with state-of-the-art baselines, demonstrating the effectiveness of HR.
4.The paper is well-organized and most related works are properly cited.

weaknesses:
1. The paper mentions using a customized autoencoder to leverage features for replay. I'm curious about what would happen if the encoder itself experiences forgetting? Additionally, since the stored features are fixed, but the encoder is continuously updated, how is this distribution inconsistency handled? 2. The paper mentions that the client receives class centroid embeddings ${p_ij}$ (line 8 of Algorithm 1). These embeddings enable the client to generate synthetic data representing tasks from other clients. However, if the received class centroid embeddings {pij} are for classes that the client has not seen, how can synthetic data be generated, and could this be detrimental to the client's learning? 3. How can the bias problem caused by multiple clients each learning a subset of categories be resolved when uploading the global model for model training and merging?

questions:
see the questions in the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730449855767

thread_type:
official_review

<thread_object>
creator: reviewer_pupw

<content>
summary:
This proposes a mathematical framework to demonstrate the global/local forgetting of FCIL and propose the Hybrid Replay (HR) to addressed these issues.

strengths:
1. This paper establishes the revolution of training loss functions within the scope of both training tasks and clients, demonstrating the two challenges of global and local forgetting.
2. A novel replay mechanism with centroids of each category is presented.

weaknesses:
1. The presentation is not very clear. For example, exemplars in HR and other exemplar-based methods seem different but are used interchangeably in this paper. Also, I cannot get the indication of global/local forgetting in Figure 1.
2. The mathematical formulation of FCIL and the proposed approach are not linked closely. Can you provide more information that how you establish the method based on the framework, especially for the global forgetting? It seems that the HR benefits from the class centroid embeddings and use them to address the global forgetting, is there any further analysis?
3. Experiments are not sufficient. The results are limited to the LDA setting with alpha=1. Extended empirical results under different skewness (e.g., alpha=0.1 or more) should be included.
4. Analysis of memory footprint should be included, e.g., the number of parameters need to be stored and transferred during the communication.
5. Error in literature review. The Prototype reminiscence and augmented asymmetric knowledge aggregation [1] only addresses the CIL and it is placed within the FCIL methods.

[1] Wuxuan Shi and Mang Ye. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1772–1781, 2023.

questions:
1. What is the data partition of FCIL? Are the classes in different tasks disjoint? In the traditional CIL, categories in different tasks are disjoint. From the setting of ImageNet-Subset (10/20/100/10, 20/10/100/10) and Tiny-ImageNet (10/5/300/30), if A denotes the number of tasks, B denotes the classes per task and the classes in different tasks are disjoint, the total numbers of classes will be 200 for ImageNet-Subset and 50 for Tiny-ImageNet. However, the total numbers of categories in these two datasets are 100 and 200 respectively. Could you explain more about this?
2。 Why does each client need to train from task 1? From Algorithm 1, the client only needs to update $theta_{h}$, but in the line 3 of Algorithm 2, the algorithm begins h=1.
3. The practical FL systems may have stragglers. It is interesting to know whether the proposed HR algorithm can deal with the issue of stragglers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730752694734

thread_type:
official_review

<thread_object>
creator: reviewer_bkz1

<content>
summary:
This paper addresses Federated Class-Incremental Learning by focusing on two types of forgetting: local forgetting (at the client level) and global forgetting (between clients). The authors propose a hybrid approach, called Hybrid Replay (HR), which combines data-based and data-free methods to mitigate these issues. They introduce a mathematical formulation to formalize the forgetting problem and the presented approach HR. The approach uses autoencoders for synthetic sample generation and latent exemplars. Comparisons against other data-free and data-based approaches demonstrate that HR achieves better performance results.

strengths:
* The paper addresses an important problem in FCIL by combining data-based and data-free approaches to overcome local and global forgetting.
* The hybrid approach integrates both latent exemplars and synthetic data generation, which are efficiently used to mitigate forgetting and results show that HR works better.
* The mathematical formulation provided to describe these forgetting issues offers theoretical foundation.
* Ablation studies in the paper contribute valuable insights into different aspects of HR and improve the interpretability of the approach.

weaknesses:
Major Concerns:
* The methodology, particularly the role of the autoencoder in addressing local forgetting, is not fully clear and can be explained better. For instance, while the paper states that the autoencoder helps address local forgetting, the specific details of how this is achieved are somewhat vague. The paper would benefit from a more detailed, step-by-step breakdown of how the autoencoder is employed for both local and global forgetting.
* The paper lacks a clear visual representation of the HR approach. Including a diagram of the proposed method could significantly enhance understanding, especially as the provided Figure 1 only illustrates the problem without outlining the proposed solution. I believe such visual representations make papers to understand much better.
* The results mention comparisons with a "Hybrid Approach," but there’s little discussion on how HR stands out from other hybrid methods, such as  REMIND+ What makes HR approach unique when compared to other Hybrid Approach ? Clarifying these distinctions would strengthen the contribution of HR to the field.
* The conclusion lacks discussion on the limitations of HR and potential directions for future work. Addressing this would provide a perspective on the approach’s implications and its broader applicability.
* The method heavily relies on data generation based on latent exemplars and class centroids, which raises concerns since we don't have a direct control of generated data and Variational Autoencoders (VAEs) are known to be suboptimal for high-quality synthetic data generation. Over time, this could degrade the quality of latent features and ultimately impact classification performance.
* The paper frequently references the Lennard-Jones formulation, but it doesn’t provide enough explanation about its purpose or why it’s important for the proposed method.

Minor Concerns:
* In Section 4, line 232, the acronym "AHR" is introduced without prior definition.
* The caption for Table 1 could be made more descriptive to make the table more self-explanatory.

questions:
* The paper mentions that exemplar-based methods are memory-intensive. Could the authors provide an estimate of this memory cost, along with a comparison of memory usage between HR and other exemplar-based methods? An example or comparison with HR could help clarify this point.
* While latent exemplars may save memory, the process of forwarding these exemplars through the decoder for sample generation could incur computational costs. It would be helpful if the authors discuss or quantify these costs when addressing the efficiency of their approach.
* What metric is used for evaluation? Motivation of paper is mainly local and global forgetting but there is not any result or evaluation for forgetting.The table results do not clearly specify whether they represent incremental accuracy or the accuracy of the last task. Besides, higher incremental accuracy does not directly indicate lower forgetting. 
* The authors state that local and global forgetting are caused by class imbalances at both the local and global levels. Do they have any scenarios or relevant results that illustrate this point better?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730767412066

thread_type:
official_review

<thread_object>
creator: reviewer_wd4j

<content>
summary:
This paper categorizes current work in federated class-incremental learning into exemplar-based and data-free approaches, noting that exemplar-based methods face memory constraints and potential privacy risks, while current data-free methods suffer from efficiency issues. The authors propose an HR mathematical framework to address both local and global forgetting. HR leverages a new autoencoder and Jones Potential formulations to generate synthetic data with minimal memory overhead, aimed at mitigating the forgetting problem.

strengths:
1.	The authors accurately summarize the limitations of current exemplar-based and data-free approaches.
2.	Experimental results indicate that the proposed method achieves lower computational and memory overhead compared to several optimal baseline methods.

weaknesses:
1.	The motivation of this study appears somewhat outdated, as the results in the literature (e.g., [1]) indicate that method has already effectively addressed the issue of class imbalance within clients. 
2.	The authors should clearly outline the specific problem their work addresses. For example, in Figure 1, it is necessary to further clarify the mechanisms causing local forgetting and global forgetting, with separate, detailed explanations for each. 
3.	The study lacks essential comparative methods.
4.	The study lacks visualized experimental results, such as accuracy on all old tasks after completing each task, and is missing essential forgetting metrics, such as Backward Transfer (BWT).
Reference：
[1]	Yavuz Faruk Bakman, Duygu Nur Yaldiz, Yahya H. Ezzeldin, Salman Avestimehr. Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning. ICLR 2024

questions:
1.	In Equation (3), what do the subscripts $i$ and $j$ represent? Additionally, in Equation (4), what does $j_l$ signify?
2.	When clients train on the same task, they each have data from different classes. How is the classifier configured in this scenario? Is it set up in a task-incremental mode or a class-incremental mode?
3.	The authors should clearly explain how the proposed method achieves sample replay through Equations (10) and (11).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732783746725

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Paper Uploaded

comment:
Dear reviewers,

A revised paper addressing your concerns has been uploaded.

Thanks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733197831352

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal & Revision Summary

comment:
# **Rebuttal & Revision Summary**

## **1. Overview**

We sincerely thank the reviewers for their thoughtful feedback, constructive suggestions, & engagement throughout the review process. Their insights have significantly enhanced the quality & clarity of our work.

We are pleased to report that our responses & revisions have addressed the reviewers' concerns, leading to an increase in scores from **6553 (average 4.75)** to **6665 (average 5.75)**. This reflects the growing confidence of the reviewers in the strength & contributions of our paper.

Below, we provide a summary of the discussions, emphasizing the **points of strength of the original paper**, the **requested revisions**, & the **enhancements included in the revised paper**.

---

## **2. Points of Strength of the Original Paper**

The reviewers acknowledged the following strengths of our work:

- **Novelty of Approach:**
  - Our **mathematical framework** was praised for setting a **solid theoretical foundation** & formalizing the challenges of local & global forgetting in federated class-incremental learning (FCIL).
  - The **proposed hybrid replay mechanism** was commended for its ability to **enhance memory efficiency & performance** while offering **the flexibility to provide multiple degrees of privacy**.

- **Experimental Rigor:**
   - The reviewers appreciated the **extensive ablation studies** provided in the paper, highlighting their clarity & the valuable insights they offer into the contributions of individual components within the hybrid replay mechanism.

- **Clarity of Presentation & Contributions:**
  - The reviewers acknowledged that the presentation of our work is well-organized, with properly cited related studies, effectively clarifying how our proposed mechanism compares to & builds upon prior work.

---

## **3. Revisions & Enhancements**

Below, we systematically categorize the reviewers' feedback and suggestions, followed by the corresponding revisions we have made to address each point.

| **Category**                     | **Revisions Requested by Reviewers**                                                                                       | **Enhancements in the Revised Paper**                                                                                                         |
|----------------------------------|---------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| **Impact of the Memory Size on Performance**   | - Inclusion of figures to demonstrate the impact of memory size on the performances of different mechanisms **(Reviewer PuPw)**.  | - Included three figures for three benchmarks. **See Fig. 3 (top row)**.   |
| **Learning-Forgetting Dynamics** | - Requested visualizations of accuracy trends for all baselines as they learn new tasks **(Reviewer bkZ1)**.  | - **See Fig. 3 (bottom row)**. |
| **Class Imbalance Scenarios & the memory footprint**    | - Additional experiments simulating imbalanced datasets **(Reviewer bkZ1)**.                              | - Conducted additional experiments on class imbalance scenarios with varying skewness conditions (e.g., alpha = 0.1 & alpha = 0.5). **See Table 3 & Fig. 3**. |
| **Visualization Improvements**          | - Inclusion of a detailed figure depicting the architecture of our approach **(Reviewer Wd4J)**.                                              | - Included a figure demonstrating how latent exemplars and the decoder network help to overcome forgetting. **See Fig. 2**.                       |
| **Clarifications of Novelty**    | - More precise explanations of how the hybrid replay mechanism compares with other hybrid replay mechanisms in the literature **(Reviewer bkZ1)**. | - Included a table that compared our hybrid mechanism with three other hybrid mechanisms. **Refer to Table 1**. |
| **Leonard-Jones Formulations**    | - More precise explanations on the role of Leonard-Jones formulations **(Reviewer WDPN)**. | - Added a paragraph discussing how these formulations help the server to determine the positions of new classes globally. **Refer to the paragraph starting from line 284**. |

---

## **4. Conclusion**

We are hopeful that our revisions have addressed the reviewers' concerns & significantly improved the quality of our paper. The increase in scores from **6553 to 6665** reflects the reviewers' recognition of the strength of the contributions of our work after discussion. We kindly ask the AC & SAC to consider these updates during their final decision-making process. However, **regardless of your final decision**, we are forever grateful to the reviewers for their thoughtful engagement & valuable suggestions, which have significantly strengthened our work. Thanks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary

The paper tackles incremental learning in federated learning, where dynamically changing clients and tasks, along with imbalanced data, lead to local and global forgetting. It proposes a mathematical framework leveraging latent exemplars and data-free techniques to address these challenges. A VAE-inspired autoencoder, with an encoder for embedding new tasks in latent space and a decoder for generating synthetic data, forms the core of the approach. Theoretical guarantees and extensive experiments demonstrate the method’s effectiveness in mitigating forgetting and handling imbalanced data distributions.

Strengths
1. The authors provide an accurate and thorough summary of the limitations in current exemplar-based and data-free approaches, making the motivation for their method clear.
2. The use of a novel autoencoder, and the approach of storing low-dimensional examples and decoding them into high-dimensional information, is an innovative contribution in methodology.
3. Extensive experimental results and ablation studies demonstrate that, despite the combinational nature of the problem and method, each factor in the combination contributes significantly to the final performance.

Weaknesses
1. Lacking studies on the theoretical properties of the proposed mathematical formulations - which is the main contribution of the paper. 
2. While the paper aims to leverage both data-based and data-free approaches, it may inherit the disadvantages of both. The paper needs to demonstrate that their method effectively mitigates these drawbacks, which has not been adequately shown.
3. The paper addresses a complex problem involving both local and global forgetting. However, the interconnections between these sub-problems are unclear. The techniques proposed are similarly combinational rather than cohesive.

Reasons for decision

With two reviewers rating the paper as 6, one as 5, and one remaining at the initial score of 6, the consensus leans towards moderate acceptance. Given the improvements made, particularly in empirical results, and the partially addressed concerns, I recommend the paper for borderline acceptance (or a poster). 

However, the authors are encouraged to further clarify the significance of their contribution and enhance the clarity of their technical rationale in future iterations.