Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the data heterogeneity problem in federated learning, by drawing an analogy with the forgetting problem in continual learning. The authors find that the global model forgets the knowledge from previous rounds, when aggregating clients' locally trained models that optimize toward the local data distribution. They propose a novel and simple algorithm to mitigate the forgetting of knowledge. Specifically, the authors add to the training loss a term that penalizes the discrepancy of the predicted class distribution between the local model and the global model, where the discrepancy considers all but the ground-truth class. The authors perform experiments to thoroughly compare with state-of-the-art federated learning methods and demonstrate the preservation of global knowledge in their method.
strengths_and_weaknesses: 
The proposed method is simple and elegant. It appears to work quite well. The authors study the heterogeneity problem in federated learning by drawing inspiration from continual learning and knowledge distillation. They successfully develop a technique that can cope with data heterogeneity. The idea is intuitive and it could be easily used by practitioners.

The authors spend a significant amount of effort to study the problem of forgetting. They define a forgetting measure, illustrate the forgetting behavior, justify suppressing gradient diversity as a way to mitigate forgetting, propose a novel loss term for training, and demonstrate its effectiveness in preserving the global knowledge. All in all, the work is solid and well done.

The paper is generally well written, but it needs thorough proofreading. Many obvious grammatical errors are spotted all over the paper.

A few comments for improvement:

- Line 123. It is a bit unclear what "... (the) accuracy varies on $p(D)$ and $\tilde{p}(D)$" means. The authors may want to elaborate.

- Line 146. It is unclear, in the first pass of reading, why the local gradient $\nabla f^k$ is computed in the said way. This may have something to do with the loss function (10) that is proposed only later. The authors may want to give some motivation before Proposition 1. 
questions: The authors may respond to the comments raised in the "Strengths And Weaknesses" section.

limitations: The authors are aware that privacy is an important subject in federated learning and state that their approach does not compromise data privacy. The authors also alert that bias in the global model, if any, may cause a similar tendency in the local models.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper suggests that forgetting could be the bottleneck of global convergence and then proposes FedLSD to handle the forgetting problem by using Not-True Distillation in the local training stage. Unlike traditional distillation methods in Federated Learning, FedLSD does not require additional auxiliary datasets because the distillation works on the clients. To preserve global knowledge on out-local distribution, the distillation only considers the not-true classes. Some experiments are conducted to show its effectiveness.
strengths_and_weaknesses: **Strengths**
1. Knowledge forgetting in the global model is a key issue. This paper makes a lot of analysis on forgetting in federated learning.
2. The main strength of the algorithm is that there is no need to use additional auxiliary data during distillation.
3. This paper is well written and easy to read.

**Weaknesses**
1. The knowledge distillation used on the clients is straightforward, although the author expands it to not-true distillation, the novelty is still limited.
2. In the experiment section, direct distillation on clients should also be used as a baseline; otherwise, one cannot see how much impact the proposed Not-True Distillation brings to the results.
3. Figure 1 is less informative. Any method based on regularization can be expressed as Figure 1(c), so it is hard to see the characteristics of the proposed method.
4. According to previous observations and experience, distillation on the client side is very likely to limit the model's learning from local data on the clients. Although this paper only uses the Not-True classes for distillation, such a problem may still exist to restrict the learning of new knowledge.

questions: See Weaknesses 2 and 4.
limitations: 1. The novelty of the paper is limited.
2. Distillation on the client side may limit the model's learning from local data on the clients
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to mitigate catastrophic forgetting in federated learning, which focuses on improving the performance of the local model by teaching the knowledge of the global model to the local model through an improved knowledge distillation method. According to the experimental results presented in the paper, the method possesses a good performance.
strengths_and_weaknesses: **Strengths**
1. The experimental setup of this paper is relatively adequate, taking into account scenarios with different federal settings
2. The experimental approach is simple and achieves good results
3. The method is well analyzed

**Weaknesses**
1. lack of comparison. The following paper is similar in approach to the method proposed in this paper, and also performs knowledge distillation locally
    - Mendieta M, Yang T, Wang P, et al. Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 8397-8406.

questions: 1. For the MOON method, the results in the table seem to be not as good as the performance of the original paper. the MOON method is more sensitive to parameters, and it is suggested that more comparisons of different hyper parameters can be put in the appendix

2. In fact, I am curious why discarding not-true has a positive impact, can you describe it simply?
limitations: See the weaknesses and questions
review_title: null

Meta-review: metareview: This paper studies the data heterogeneity problem in federated learning. It is well-written, with some novel ideas and good efforts on analyzing the problem of forgetting and experimental studies. We hope the authors can revise the paper carefully per the reviewers' suggestion and add some new experimental results during rebuttal into the final version.
confidence: Less certain
Decision: Accept