Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes algorithms that search for suitable personalized models in a client-server type federated learning setup. The algorithms are inspired by the classical theory of parametric Bayesian risk minimization. In the personalized parameter estimation regime, the authors assume two population distributions: Gaussian and Bernoulli. Under Gaussian distribution, they report that in case the parent population becomes degenerate (i.e., variation tending to nullity), the global average turns out to be the ‘best’ estimator. Moreover, the posterior personalized mean estimator in this setup also turns out to be optimal in general. If the parent population follows a Bernoulli law, having sufficient observations from local sub-populations suggests against collaborating. The following ‘personalization’ algorithms utilize different prior distributions and regularization schemes to ensure client privacy.
strength_and_weaknesses: Strength:

The language of the article is lucid, and the presentation is also of good quality. The discussion leading up to the theoretical analyses and the algorithms is precise. I find the statistical analysis rigorous and very well represented. Prior works and relevant references are well-placed throughout the paper. 

Weakness/Issues:

The authors have altered the standard structure of the article, as guided by ICLR instructions. The abstract should not be full page-wide. This is a violation of the code and gives them an undue advantage over others. 

The current article looks incomplete, lacking a ‘Conclusion’ section. Also, sufficient discussion regarding limitations and future work is missing.

I suggest the authors present accompanying codes maintaining anonymity. 

It would be very helpful if the problem statement is presented more precisely in the introduction. The authors provide a lot of references to prior work. However, amidst such a crowd, the motivation somehow fades. 

As I have acknowledged, the discussion is quite rigorous. However, the is significant room for improvement when it comes to organization. 

The empirical results seem insufficient, and I suggest the authors put more datasets to test if feasible.

clarity,_quality,_novelty_and_reproducibility: The paper presents an overall well investigated research work. To enhance clarity the authors need to answer the following questions:

Questions:

What does it mean statistically to “estimate i through the help of the server.”?

In the section ‘Personalized Estimation’, should it be “client i outputs an estimate i of i” instead?

Does the prior distribution necessarily need to have a density in general? How ‘realistic’ are the special case assumptions of Gaussian and Bernoulli as global population distributions?

In the statement of Theorem 2., should it be “[-r,r]d” instead? 

Does the notation Ɛ mean expectation [In Theorem 2, 3, etc.]? If so, kindly maintain any one symbol throughout. 
The authors also use the same notation for a different purpose in Section B.2.1.

Shouldn’t we have the sum also over the quantity on the right-hand side of equation (10) [Section 3.1]?

The quantity it may penalize the heterogeneity, but does not denote the variance. The authors should call it something else instead [Section 3.2, last paragraph].

Is the strict imposition of the value ‘1’ necessary in the definition of ‘neighborhood’ [Section A.3], since there is a clear possibility to generalize the result even if two datasets D and D' differ at multiple terms?

In Theorem 2, the upper bound on MSE  (5) loses its worth in a higher-dimensional setup. Can the authors talk about any remedy to the same?

summary_of_the_review: The paper may be considered for acceptance provided the authors address the above listed concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors develop a statistical framework for personalized federated learning, in which they study different choices of population distribution. They provide theoretical bounds and novel algorithms for private personalized estimation, design, and conduct privacy analysis of new private personalized learning algorithms. 
strength_and_weaknesses: Strengths:
* The paper is theoretically grounded. Authors provide theoretical results for different population models.
* In contrast to many competitors, the paper considers privacy aspects of the problem and provides new algorithms with provable privacy guarantees

Weaknesses:
* "Based on (Xi, Agg(q1, . . . , qm)), client i outputs an estimate \hat{\theta_i} of \theta. " Should be the estimate of \theta_i ? Otherwise not clear what is \theta as it is not defined before.
* It seems that there is a missing opportunity to compare with another class of personalized models. Namely [1], when there is a split of shared and personalized parameters. Moreover, this model could be considered as a particular case of your statistical framework, when (for Gaussian population model) \sigma_\theta = [0,\infty] where 0 corresponds to the shared (common) part of parameters, and \sigma_x = 0. It is worth mentioning the paper in related work and comparing the results.
* The choice of datasets for experiments seems a bit low-key. I suggest considering CIFAR100.

Minor issues:
* page 6 first appearance of FedAvg — no link
*"... given 6 election data we did 1-fold cross-validation. " - what does 1-fold cross-validation mean? Did not you mean 6-fold cross-validation?


[1] Collins L. et al. Exploiting shared representations for personalized federated learning //International Conference on Machine Learning. – PMLR, 2021. – С. 2089-2099.

clarity,_quality,_novelty_and_reproducibility: The ideas raised in the paper are original, and the paper is well-written and easy to read. 
summary_of_the_review: I am generally positive about the paper ideas, but there are some issues that are worth correcting (see above).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors developed a new Bayesian framework for personalized federated learning and derived theoretical bounds and novel algorithms for private personalized estimation. AdaPeD was proposed to use information divergence constraints along with adaptive weighting of local models and population models. AdaMix was proposed to adaptively weigh multiple global models and combines them with local data for personalization. Certain privacy analyses of new private personalized learning algorithms were done.
strength_and_weaknesses: I think this paper is strong. It studies an important yet remarkably understudied problem that can build the bridge between Bayesian learning and federated learning. It gives two types of algorithms: AdaPeD and AdaMix. The algorithm and the theoretical results are sound. The algorithm is fairly natural in Bayesian perspective, which is to update the Gaussian posterior based on the prior and likelihood. The AdaPeD uses a knowledge distillation regularization. It shows we can federatedly train the model via MAP well when preserving user-level privacy.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and easy to follow when including the appendix. I feel the paper is more suitable to a journal since a lot of important explanations are set aside in the appendix.
It is better to compare the upper bounds with the best bounds for the method without the privacy guarantee. By such a comparison, we can see the tightness of the present bounds roughly.
summary_of_the_review: The paper is clearly written, but the notations are heavy. The contribution is very technical from the perspective of Bayesian Federated Learning.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a statistical framework for personalized learned and estimation in a federated learning setting. The authors first explicitly state the modeling assumptions of how local model parameters are drawn from a global distribution of parameters. They then develop private and non-private estimation and learning algorithms. 
strength_and_weaknesses: Strengths: 
1) The modeling assumptions of how local model parameters are drawn from a global distribution of parameters seems quite reasonable, albeit a little restrictive. 
2) The proofs seemed correct. Although, I am not a theorist, so I wouldn't state this with a lot of certainty. 
3) The authors seemed to have given a lot of thought to the practical problems in federated learning setting and have tried to tackle a lot of them systematically. 

Weakness: 
1) The paper tries to do too much in my opinion, which makes it quite difficult to follow at times. 
2) The experiments seem sparse and handpicked, particularly for the personalized learning setting. Several questions that came to my mind are: what is the reason for picking m to be 50 and 66? Why didn't the authors try a range of values and reported all the results? 
3) Similarly, for the estimation task, I'd like to see more experiments. This can be done easily with different choices of synthetic parameters. 
4) Add more motivation for each estimation and learning setting. When are these problems relevant? Why should we care about them? This is not very clear from the paper. 
clarity,_quality,_novelty_and_reproducibility: I found the paper quite difficult to read. 
summary_of_the_review: While I think the technical contributions of the paper are quite novel, it is very hard to read. This is in large part because the authors have tried to fit too many things into a single paper. For instance, have the authors considered writing this as 2 separate papers on personalized learning and estimation? Alternatively, it might be better suited as a long-form journal paper (say at JMLR or TMLR).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents privacy-preserving empirical and hierarchical Bayes algorithms. The analysis and development are both fine, and the topic is interesting, at least for me, relative to the standard 'propagate gradients' idea of most federated learning algorithms. The only complaint was that more experiments might be necessary. However, it is my feeling that the paper is good enough as is.

justification_for_why_not_higher_score: The audience for this paper may not be very wide.

justification_for_why_not_lower_score: It addresses an important topic from a well-grounded theoretical perspective, both in terms of estimation and privacy.
Decision: Accept: poster