Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors study an important problem in federated learning, i.e., personalized federated learning, where the goal is to balance the training of the local model of each client and the global model shared by all the clients. In particular, the authors do not follow previous works on model fine-tuning in each local client and size-based weighting in global aggregation, but propose to address the balancing challenge from a statistical uncertainty perspective. Specifically, the authors design a novel solution called Self-FL, which uses uncertainty-driven approaches for the training of the local models and the global model. Empirical results on some well-known datasets show the effectiveness of the proposed Self-FL.
strengths_and_weaknesses: Strengths:

1 The authors address an important challenge in personalized federated learning, i.e., balancing of the training of each local model and the global model.

2 The authors propose a novel perspective for the studied problem and design some novel uncertainty-driven approaches.

Weakness:

1 The authors may include more discussions and quantitative analysis about the communication cost.

questions: 1 Can the proposed approach be applied to other tasks, e.g., classification/recognition with some auxiliary data such as knowledge graph, federated recommendation?
limitations: The authors do not include sufficient discussions about the communication and computational cost, which are suggested to be included.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper is on personalized federated learning, where the critical challenge is in balance of local and global model tuning. More specially, the goal of global model tuning and local personalized model tuning are not fully aligned.  Toward this problem, the authors develop a self-aware personalized FL method to balance the personal model and global model update based on the inter-client and intra-client uncertainty quantification. The paper recognized and clearly defined critical research questions. To answer the questions, the authors formulate  personalization from a  hierarchical model-based perspective and propose Self-FL, an active personalized FL solution that guides local training and global aggregation via inter- and intra-client uncertainty quantification. The experimental results confirm the effectiveness of the proposed method from task performance and training stableness. 
strengths_and_weaknesses: The authors clearly defined the challenging questions and work on the fundamental research questions. The proposed methods are derived  in a principal way with theory. The performance is promising and experiments are pretty comprehensive. The personalization naturally puts more requirements on evaluation and analysis due to involving performance evaluation across clients. I include my suggestions in the following sections and encourage authors to provide more analysis to uncover personalization challenges in term of data dimension. 
questions: One of the challenges in personalization is how to achieve good performance while clients do not have enough data and prevent overfitting.  Could authors provide some intuitions to explain how the model handle the case when the labeled data is limited on some clients?
limitations: The authors propose to characterize the performance by weighted accuracy. Even though this is a good way to show the performance, the detailed performance distribution is also suggested. For example, a violin plot could depict the distribution of performance. 
Another suggestion is to uncover the relationship between performance and other factors including data size and class distribution on client sides.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors present a self aware personalized FL system which uses Bayesian hierarchical modeling to balance training of local versus global models. This helps to quantify inter and intra client uncertainty quantification. The authors present both theoretical justification and empirical analysis of the proposed technique - Self-FL.
strengths_and_weaknesses: Strengths
- The authors identify an important problem -- that of personalizing federated learning problems
- The paper is well written
- A reasonably well written Related Work and Experimental Studies section

Weakness
- The problem of tuning local versus global models has been studied extensively in literature and therefore the novelty of the proposed approach is questionable. 
questions: - The motivation for using Bayesian hierarchical modeling for personalization needs to be better explained. 

-  How exactly is personalization quantified and how is the notion of "personalization" different from learning parallel algorithms where local models are updated by information from global models?

Minor comments
Section 2.1. to bake prior knowledge -> to take prior knowledge 


limitations: - The use of local versus global training  is extensively used in literature. It is unclear how this idea is different from the notion of "personalization" introduced here. Infact, the basic premise seems to be the construction of a generative model at the client side and update its parameters using a global counterpart (assuming a generative model for global training).
review_title: null

Meta-review: metareview: In this submission, the authors study personalized federated learning and propose a self-aware personalized method to address the balancing challenge from the perspective of uncertainty quantification. This problem is interesting and important (as pointed out by b7BG and rRk4), and the proposed method is derived in a principal way from theory (as pointed out by jvSV) and useful for applications. I recommend accepting this submission.

The authors can include some discussion about the communication and computation cost (as suggested by b7BG), and add an ablation study to show the effect of other factors (as suggested by jvSV), to make this submission better.
Hope that the suggestions from all the reviewers and the discussion between reviewers and authors can make this submission a better on.


confidence: Less certain
Decision: Accept