Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel hierachical Bayesian approach to FL, with variational inference. The idea is to use two levels of random variables, the higher level $\phi$ which is shared among clients, and lower level $\theta_i$'s for each client $i$. This allows flexibility of conditional independence and personalization. There are two main methods: Normal-inverse-Wishart (NIW) and Mixture Model (Mix.). The methods are supported by convergence and generalization proofs, as well as experiments on two datasets. 

--post rebuttal--
Thanks to the authors for the careful response to my questions. I have updated my score accordingly. After reading the response, I think the paper indeed improves, but there are still some weaknesses to address. It seems that on the theoretical side, the only main novelty is the generalization bound, as the authors admit. However, after checking the proof, I found this generalization bound shares a lot of similarities with Bai et al. 2020. Moreover, in the main paper, Theorem 4.2 needs a lot of discussions, such as the meaning of each term (e.g. $r_n$, $\epsilon_n$), the implication of this theorem, and comparison with existing results (e.g. Bai et al). On the algorithmic side, since hierarchical Bayesian is proposed before, what is the main novelty of this paper? Is it simply an application for personalized FL? Finally, on the experimental side, after adding more baselines, the improvement over baselines is not significant. For instance, for MNIST the improvement is less than 1%. Even for CIFAR-100, for $\tau = 10$ the improvement is much less than $\tau = 1$. This casts doubt on the practicability of the proposed algorithms. 
strength_and_weaknesses: Strengths:
1) The Bayesian approach to FL is interesting and seems suitable for personalization. Using the hierachical approach and variational inference, the authors propose two different choices of hierachical models, that seem to be work well in practice.
2) The proposed Bayesian FL algorithms are supported by convergence analysis and generalization error bound.
3) The method is tested on CIFAR-100 and CIFAR-C-100 and compared with baseline algorithms, with noticeable improvement.

Weaknesses:
Although this hierachical Bayesian approach seems interesting, and it is supported by some theoretical analysis as well as experimental comparison, I feel this paper is currently not ready for publication due to several reasons:

1. The motivation is not very clear and some claims are not well-supported. For example:
 1) Sec 1, paragraph 1: "FedAvg and FedProx are well-known to suffer convergence issues". Any reference? What are the assumptions? Why would personalization resolve the convergence issues? 
2) Sec 1, paragraph 2: why would Bayesian perspective be good for personalization? Is the current method complete or principled? Despite this being a bit subjective, I think at least the choices of NIW and Mix. are not principled and ad hoc.
3) What is the meaning of the higher level random variable $\phi$? Is it like some prior or shared knowledge? More intuitive explanation is needed.
4) Figure 1: what are $y^p$ and $x^p$? 
5) Sec 2.1: reference of ELBO and the original ref of block coordinate optimization are needed.
6) Sec 2.2: the notation $p^p$ seems a bit weird as the two $p$'s have different meanings.
7) eq.10: what does the approximation sign mean? In which cases does the equality hold?
8) Sec 3: how the two models are proposed does not seem straightforward. Why are NIW and Mix. good choices and how are they proposed?
2. Convergence analysis. In Appendix D the authors added 3 assumptions. In Assumption 1, the definitions of "locally convex" are not quite clear. Assumptions 2 and 3 have some overlap regarding the Lipschitzness and the radius of the domain needs to be justified.
1) The convergence of the proposed methods has $O(1/\sqrt{T})$ convergence rate. This is at the same level as FedAvg (see e.g. [1]). Therefore the proposed method has no advantage over FedAvg which the authors claim to have some convergence issues.
3. The generalization error bound is not well explained. For example, what is $\epsilon_n$? What does $\lambda_i^*$ mean? Why do you use Hellinger distance? How is the error bound compared to (Mohri et al 2019)?
4. The empirical evalution can be improved. The dataset is mainly CIFAR-100 and its variant, and only 4 baselines are compared. FedAvg and FedProx are pretty similar and more Bayesian approaches should be compared, such as FedPA, FedBE, FedEM, and PredictiveBayes [2].

[1] Adaptive federated optimization. Reddi et al, ICLR 2021. 
[2] Robust One Round Federated Learning with Predictive Space Bayesian Inference, Hasan et al, https://arxiv.org/pdf/2206.09526v1.pdf.
clarity,_quality,_novelty_and_reproducibility: At the current stage the paper is not clearly written and the quality cannot be easily judged. I believe there is some novelty in this work. About the reproducibility, the authors attached the code but it cannot be run. There is no package imported and the data collection process is not clear. Therefore, the reproducibility of this paper cannot be checked.
summary_of_the_review: In summary, this paper proposes hierachical Bayesian FL using variational inference which looks interesting. However, the paper largely suffers from presentation, and it is not ready for acceptance till it's more clearly presented.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper uses Bayesian paradigm for personalized Federated Learning. 
Specifically, this paper suggests a two-level hierarchical model for data $D_{1:N}$ distributed on $N$ clients of the form:
$$
p(D_{1:N} | \theta_{1:N}) = \prod_{i=1}^N p(D_i | \theta_i) \quad , \quad p(\theta_{1:N} | \phi ) = \prod_{i=1}^N p(\theta_i[\phi) .
$$
Here, $\theta_{1:N}$ represent individual client parameters which are linked trough the globally shared variable  $\phi$.
Taking some prior for $\phi$, the posterior distribution is
$$
p(\phi, \theta_{1:N}| D_{1:N} ) \propto p(\phi) \prod_{i=1}^N p (\theta_{i} | \phi) p(D_i | \theta_i).
$$
This distribution is intractable and the authors use variational inference to approximate it by
$$
q(\phi,\theta_{1:N} | L_{0:N}) = q(\phi | L_0) \prod_{i=1}^N q(\theta_i|L_i),
$$
where $L_{0:N}$ is the parameters which are learnt maximizing the ELBO
$$
\mathcal{L} = \sum_{i=1}^N \mathcal{L}_i  + \mathcal{L}_0 ,
$$
where $\mathcal{L}_i$ depends on $L_i$ and $D_i$. 

Therefore, $\mathcal{L}$ can be optimized using a block-coordinate optimization scheme. At each step, and given $L_0$, each client optimizes $\mathcal{L}_i$ and sends its update to a central server which in turn optimizes $\mathcal{L}_0$ to update the global parameter $\phi$ for fixed $L\_{1:N}$. 

Based on their inference, the authors also suggest how to make personalization for an unseen new client. 
In addition, they derive convergence bounds for their FL optimization algorithm and obtain generalization bounds for their variational approximations as the number of observations goes to $\infty$. Finally, numerical experiments compare the method proposed in the paper with some FL baselines. 


strength_and_weaknesses: Strength:
- The paper tackles the important problem of personalization in federated learning using a hierarchical Bayesian approach.
- The approach using variational inference to infer local and global parameters is interesting.
- The paper provides theoretical guarantees on the convergence of the method.

Weakness:
- Practical implementation of the method could be better presented as well as its computational complexity. 
Moreover, the paper would benefit from a better description of how the proposed solution addresses the specific problems of FL, in particular communication constrains and privacy. In the current version, it seems to me that it only focuses on personalization. 
- The method does not address the problem of partial communication which is common in FL.
clarity,_quality,_novelty_and_reproducibility: - The paper is in some parts hard to read. For example, in the section on global prediction and personalization,  the authors suggest to use a lot of approximations that they should be more discussed. The theoretical results are barely commented in the current version. Finally,  while I appreciated the illustration of the method on practical examples in Section 3, I think that the authors could only focus on a particular example and give all the details they postpone to the supplement. Indeed, without these complements, it was very difficult to me to catch the main ideas.

- Related work and the literature should be more discussed. From what I saw, the discussion is for the moment limited to Bayesian approaches. 

- The use of hierarchical Bayesian model for FL has already been proposed in https://arxiv.org/abs/2206.03611 
I wonder how the method of the paper under review compares with the one developed in this paper. 
summary_of_the_review: The paper is interesting but should be more specific on how it addresses FL problems. In addition, comparisons with other personalization methods should be added.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hierarchical Bayesian approach to Federated Learning. The proposed Bayesian model makes the block-coordinate descent solution becomes a distributed algorithm. In addition, the paper also derives convergence analysis that shows the block-coordinate FL algorithm converges to a local optimum of the objective at the rate of $O(1/\sqrt{t})$, which is the same as the regular SGD; and a generalization error analysis that shows the test error is asymptotically optimal. The effectiveness of this approach is demonstrated empirically on two benchmark datasets (CIFAR-100 and CIFAR-C-100). The proposed approach outperforms the best work in terms of both global prediction and personalization. 
strength_and_weaknesses: # Strength
1. The paper is well structured. The paper first introduced a general Bayesian FL framework and derived the ELBO objective optimization function with block-coordinate optimization solution. Then, the paper presents two concreate models (NIW and Gaussian Mixture), which illustrate the hierarchical Bayesian framework very well.
2. The paper presents principled hierarchical Bayesian approach, in which client’s individual parameters are governed by a common prior. Compare with other Bayesian FL approaches, the proposed method is fully Bayesian and do not need ad-hoc heuristics nor strong assumptions.
3. The effectiveness of the proposed Bayesian hierarchical FL is empirically tested on benchmark datasets (outperforms the best by large margins most of the time) in both global and personalization prediction.
4. The paper provides strong theoretical analysis of the convergence and the generalization error of the proposed block-coordinate FL algorithm.    

# Weakness 
1. The discussion with related work needs to include other Bayesian treatment for personalized FL, e.g. [1]
2. The empirical evaluation is only on two benchmark datasets, considering including more relevant datasets in [2] 

References: 
[1] Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized
federated learning with gaussian processes. In Advances in Neural Information Processing
Systems 34: Annual Conference on Neural Information Processing Systems, pages 8392–8406,
2021.
[2] Chen, D., Gao, D., Kuang, W., Li, Y., & Ding, B. (2022). pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning. arXiv preprint arXiv:2206.03655.

clarity,_quality,_novelty_and_reproducibility: The paper is well written. Although Bayesian hierarchical modeling is a standard probabilistic modeling for group/individual data, the application to Federated Learning is quite novel.
summary_of_the_review: I would like to recommend the paper for acceptance based on the following reasons:
1) The paper is clearly written and well organized.  
2) The proposed method by the paper is well motivated and technically correct. 
3) The effectiveness of the paper is demonstrated both empirically and theoretically.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel hierarchical Bayesian approach to federated learning, and derive ELBO objective function using variational inference techniques.  Then the block-coordinate descent algorithm is devised, which fit well in the federated learning regime. Theoretical result on the convergence of the algorithm is obtained. Empirical studies demonstrated the superior performance of the proposed method over existing ones.
strength_and_weaknesses: Strength:
1.	The idea of employing Bayesian approach and variational inference techniques for federated learning is of novelty. 
2.	Theoretical analysis shows the convergence of the proposed algorithm is guaranteed. 
3.	The develop framework can tackle the problem of distribution disparity among the local learner. 
Weaknesses:
There are some studies that deal with the heterogeneous distribution in domain generalization and adaption, which is not covered by the analysis of related works; the empirical studies also only compare the proposed algorithm with standard FL methods. The conclusions will be more convincing if more comparison is made with some approaches that deal with the heterogeneous distribution.

clarity,_quality,_novelty_and_reproducibility: The code to reproduce the numerical results is currently unavailable.
summary_of_the_review: This paper develops a novel federated learning method with hierarchical Bayesian approach, and it can cope with the heterogeneous local distributions. Sound theoretical results on the convergence and the generalization performance are attained. Empirical studies are relatively weak, because only simple dataset, cifar100, is used and the compared algorithms are some standard FL approaches.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a hierarchical Bayesian approach to Federated Learning. The proposed Bayesian model makes the block-coordinate descent solution becomes a distributed algorithm. In addition, the paper also derives convergence analysis that shows the block-coordinate FL algorithm converges to a local optimum of the objective at the rate of O(1/sqrt{t}), which is the same as the regular SGD; and a generalization error analysis that shows the test error is asymptotically optimal. The effectiveness of this approach is demonstrated empirically on two benchmark datasets (CIFAR-100 and CIFAR-C-100). The proposed approach outperforms the best work in terms of both global prediction and personalization.

The paper tackles the important problem of personalization in federated learning using a hierarchical Bayesian approach. The Bayesian approach to FL is interesting. Using the hierachical approach and variational inference, the authors propose two different choices of hierachical models and they work well in practice. The proposed Bayesian FL algorithms are supported by convergence analysis and generalization error bound. 

However, practical implementation of the method could be better presented as well as its computational complexity. The paper would benefit from a better description of how the proposed solution addresses the specific problems of FL, in particular communication constrains and privacy. In the current version, it only focuses on personalization. The method does not address the problem of partial communication which is common in FL. Moreover, there are some studies that deal with the heterogeneous distribution in domain generalization and adaption, which is not covered by the analysis of related works. The empirical studies only compare the proposed algorithm with standard FL methods. The conclusions will be more convincing if more comparison is made with some approaches that deal with the heterogeneous distribution.
justification_for_why_not_higher_score: However, practical implementation of the method could be better presented as well as its computational complexity. The paper would benefit from a better description of how the proposed solution addresses the specific problems of FL, in particular communication constrains and privacy. In the current version, it only focuses on personalization. The method does not address the problem of partial communication which is common in FL. Moreover, there are some studies that deal with the heterogeneous distribution in domain generalization and adaption, which is not covered by the analysis of related works. The empirical studies only compare the proposed algorithm with standard FL methods. The conclusions will be more convincing if more comparison is made with some approaches that deal with the heterogeneous distribution.
Decision: Reject