Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces Bayesian paradigm to personalized FL, allowing for both personalization and cheap uncertainty quantification. Specifically, this paper introduces the fixed part (hyperparameter \phi) to capture the common representation of all local data and the random part via a common prior modeling data heterogeneity. The authors derive the non-asymptotic convergence guarantees for their federated stochastic optimization algorithm.
strengths_and_weaknesses: Merits:

1) The paper tackles an important problem in federated learning, which allows uncertainty quantification for personalized FL.
2) FedPop reduced the memory costs by introducing the mixed-effects instead of high-dimensional hyperparameter.
3) This paper provides convergence guarantees for the proposed algorithm.

Cons:
1) The novelties of proposed algorithm are unclear to me. It seems the difference between Bayesian FL and FedPop lies in the design of probabilistic hyperparameter. FedPop use two hyperparameters instead of a high-dimensional hyperparameter, including a common prior, to reduce storage costs.
2) The algorithm uses many independence assumptions, for example the independence on \phi and \beta at line 114 and the independence between local likelihood functions below line 117.
3) The authors may introduce more discussions on the computational complexities and communication burdens. 
questions: Please see the weaknesses part.
limitations: As I can see, this paper has no potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel method for personalized FL having uncertainty quantification by using the population modeling paradigm where the client's models involve fixed common population parameters and random effects. From their empirical results, their method is robust to client drift, practical for inference, and uncertainty quantification under mild computational and memory overheads. Their method FedPop performs personalization and cheaply works for quantifying the uncertainty in an on-device manner.
strengths_and_weaknesses: * Strength
- Their claim is supported by both theoretical and empirical results. 
- The authors point out a major problem in related field.

* Weakness
-  the set-up of experiments and results reported in this paper are still questionable. (Datasets, Architectures, FL settings such as participation ratio, data heterogeneity settings with Dirichlet distribution)
- The empirical evidence for Theorem 1 may be needed. 




------updated after rebuttal---------

Thank you for your detailed response for my raised concerns. still I think this work needs more comparisons with related FL approaches based on bayesian approaches, but most concerns are solved. So,  I decided to increase my score from 4 to 5.
questions: Q1) During the training dynamics, how and why can FedPop is robust to client drift, practical for inference on new clients?

Q2) Is your theorecial result empirically guaranteed after the FL training? how does the trained model behavior have?

Q3) Do your methods have robustness even in I.I.D. settings?
limitations: Despite their significant empirical results, there remain limitations: 1) More other methods should be conducted to verify its robustness in applications to advanced Personalized FL methods. 2) ablation studies on each factor of FedPop such as warm start strategy, gradient estimator etc... as well as hyperparamter settings of other regularization methods such as weight decay, augmentation...
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new algorithm FedPop for personalized federated learning by modeling it as a mixed-effects probabilistic problem. Specifically, a client’s local distribution depends on a fixed-effect part (constant for all clients) and a random-effects part (captures the heterogeneity).  FedPop is developed by maximizing a posteriori probability via stochastic approximation. The algorithm works as follows: in each round, the participating clients will use MCMC to generate samples of the random-effects random variable, use them to compute approximate gradients, and send the updates to the server. A convergence bound is established for the proposed algorithm under appropriate assumptions. The authors empirically show that the proposed algorithm outperforms related works over a synthetic dataset, and two versions of CIFAR datasets (CIFAR-10 and CIFAR-100).
strengths_and_weaknesses: Strength:

- The paper is well-written and easy to read. 

- Modeling the personalized federated learning problem using the mix-effects paradigm seems new in the literature. The proposed algorithm allows each client to learn a personalized model and at the same time perform uncertainty quantification.

- In addition to the empirical comparisons, the authors provide theoretical guarantees and establish convergence bounds for the proposed algorithm.

Weakness:

- My main concern is that while the paper claims that the proposed algorithm is designed for the cross-device FL setting, it is unclear to me that FedSOUK (Algorithm 1) is indeed suitable for real-world cross-device applications. Cross-device settings typically involve a very large population (e.g., hundreds of millions mobile devices), and it is common that a device only participates at most once during the training process [1]. As a result, cross-device setting usually needs *stateless* algorithms rather than *stateful* algorithms. The main difference is whether the algorithm relies on clients maintaining local states from one round to another (see section 5.1 of [2] for why the stateful algorithm performs poorly in the cross-device setting). The proposed algorithm FedSOUK (Algorithm 1) is a stateful algorithm, because the participating clients warm-start MCMC from the $Z$ value from the previous round. In the real-world cross-device setting where most clients participate only once in training, this means that warm-starting is impossible, will this constraint affect the performance of the proposed algorithm? In that case, will each client need to draw a lot of $Z$ samples locally (how do we set $M$ in practice)? All the experiments shown in the paper are using datasets with a small number of clients (b=100), which is a scale of typical *cross-silo* federated settings [1]. It is thus not convincing to me that the proposed algorithm can perform similarly well in the real-world cross-device setting with millions of model devices and really low sampling rate per round (where most clients participate at most once in the entire training process). 

- Other comments/questions:
  - FedAvg + fine-tuning (FT) a simple, effective, and stateless personalization algorithm (see, e.g., [3][4]) used in real-world cross-device FL applications (e.g., [5]). It is better to include that in Table 2 for a more comprehensive comparison (similar to Table 1 in [6]). On image datasets like CIFAR, fine-tuning only the last layer is usually better than fine-tuning the entire model. Besides, use the last layer as the personalization layer is also consistent with the proposed algorithm FedSOUL. Therefore, it is more convincing to add the baseline experimental results of FedAvg+FT with only fine-tuning the last layer to Table 2 (note: both the fine-tuning learning rate and number of epochs are hyperparameters that should be tuned from data).
 
  - The posterior distribution $p(\phi, \beta | D)$ equation (Line 117) does not contain $p(D)$ on the right-hand side, why is that?

  - In Algorithm 1, the step of server “Send {\beta_{k+1}, \phi_{k+1}} to clients belonging to A_{k+1}” happens at the end of every round, why does server not broadcasting the values to the participating clients at the beginning of every round? 

[1] Kairouz et al., "Advances and open problems in federated learning." Foundations and Trends in Machine Learning,  (2021): 1-210.

[2] Reddi et al., “Adaptive federated optimization.”, ICLR, 2021.

[3] Wu et al., “Motley: Benchmarking Heterogeneity and Personalization in Federated Learning”, arXiv: 2206.09262, 2022.

[4] Matsuda et al., “An Empirical Study of Personalized Federated Learning”, arXiv:2206.13190, 2022.

[5] Wang et al., "Federated evaluation of on-device personalization." arXiv preprint arXiv:1910.10252, 2019.

[6] Collins et al. "Exploiting shared representations for personalized federated learning." ICML, 2021.

-------------------------update after the author rebuttal-------------------------------
The reviewers showed additional experimental (and promising) results in the cross-device setting: https://github.com/anonneurips1/rebuttal_neurips/tree/main
One limitation of the proposed method is that it requires each client to maintain $M$ samples each with dim($z$), where $M$ is around 50 in the cross-device setting. If dim($z$) is similar to dim($\phi$) (which is the case for the StackOverflow experiments), this means that the memory cost is 50 times that of other personalized algorithms such as FedAvg with fine-tuning. 

I decided to increase my score from 5 to 6 because: 1) the memory&computation limitation is common for other Bayesian FL algorithms; 2) the proposed algorithm seem a strong baseline when the model size is small (or when the client's local resources are not constrained, such as in the cross-silo setting).
questions: See my detailed comments above for questions and suggestions, especially my concerns on application in the large-scale cross-device setting and the missing baseline FedAvg+FT with only fine-tuning the last layer (note: both the fine-tuning learning rate and number of epochs are hyperparameters that should be properly tuned from data).
limitations: The authors pointed out the limitations and future work in Section 6.
review_title: null

Meta-review: metareview: The reviewers agree that the paper makes significant progress on uncertainty quantification for personalized federated learning. This is clearly a fundamental problem. The new approach is shown to have theoretical guarantees. The paper is easy to read. Based on the above, I recommend acceptance. Meanwhile, please carefully revise the paper and incorporate new experimental results in the final version according to the reviews.
confidence: Certain
Decision: Accept