Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a new perspective on federated learning through the lens of posterior inference. The paper designs a computation- and communication-efficient posterior inference algorithm—federated posterior averaging (FEDPA), which generalizes FedAvg. FEDPA is compared with the strong baselines in Reddi et al. (2020) on realistic FL benchmarks, which achieves state-of-the-art results with respect to multiple metrics of interest. 

Overall, the paper is well written and easy to follow. I tend to accept the paper. The detailed comments follow. 

Pros: 
1. Viewing federated learning through the lens of posterior inference is new. The motivating example in Figure 1 gives a nice explanation why FedPA may outperform FedAvg. 
2. The proposed FedPA algorithm enjoys provable performance guarantee.  
3. The simulations are extensive, which demonstrate the clear advantage of FedPA over FedAvg on multiple benchmark tasks. 

Cons:
1. The algorithm itself is not well explained in the first eight pages. Consider moving some key equations from appendix to main text to explain clientMCMC. 
2. The convergence result is weaker than those on optimization for federated learning. Specifically, no finite-time analysis is provided, and the dependence of the number of local update K on the performance is not clear in the current form. 
3. The new challenge relative to distributed MCMC literature is not well explained. Communication? Privacy? or else. 
review_title: Review on Federated Learning via Posterior Averaging

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is nice and seems technically sound. However, some part must be clarified.
For instance, the discussion of the state-of-the-art must be improved.
My main concerns is the degree of novelty. It is not clear the difference with other works.
See my comments below.

- Regarding Eq. (1): you state that "For example, least squares loss corresponds to likelihood under a
Gaussian model, cross entropy loss corresponds to likelihood under a categorical model, etc. Thus, Eq. 1 corresponds to maximum likelihood estimation (MLE) of the model parameters ." This sentence must be clarified since it is not straightforward to see it from Eq. (1) and it is an important equation and sentence for your work.  Do you mean that F(\theta) is a log-posterior or a  log-likelihood?
Generally,  it is not a mixture of components. Clarify this point.

- Regarding Eq. (3):  the state-of-the-art related to this equation must be improved. For instance, the following relevant contributions must be considered

Lavancier, F., Rochet, P.: A general procedure to combine estimators. preprint arXiv:1401.6371 (2014)

D. Luengo et al, "Efficient linear fusion of partial estimators", Digital Signal Processing, Volume 78, Pages: 265-283, 2018.

Cattivelli, F.S., Sayed, A.H.: Diffusion LMS strategies for distributed estimation. IEEE Transactions on Signal Processing 58(3), 1035–1048 (2010)

Bordley, R.F.: The combination of forecasts: A Bayesian approach. Journal of the Operational Research Society 33(2), 171–174 (1982)

- It is not clear the difference among your strategy and the consensus one and the others in the papers above. Please clarify.

- Regarding parallel MCMC chains, other schemes could be considered such as:

R. Craiu et al. Learn from thy neighbor: Parallel-chain and regional adaptive MCMC. Journal of the American
Statistical Association, 104(448):1454–1466, 2009.

F. Llorente et al, "Parallel Metropolis-Hastings Coupler", IEEE Signal Processing Letters, Volume 26, Number 6, Pages 953-957, 2019.

J.Corander et al. Parallel interacting MCMC for learning of topologies of graphical models. Data Mining and Knowledge Discovery, 17(3):431–456, 2008

- Regarding the Laplace approximations: you still need the knowledge of the MAP  (maxima a-posteriori). You still need to compute/reach these maxima. Please clarify the computational cost of these previous steps for finding these maxima.


review_title: The paper is nice and seems technically sound. The discussion of the state-of-the-art must be improved.

final_score: rating: 7: Good paper, accept
review_content: The authors propose a new method of generating local (client) updates in Federated Learning (FL), where the clients return an adjusted version of their usual local updates to the server. The authors derive this new local update rigorously from the viewpoint of estimating the posterior distribution of the data (under Gaussianity assumptions). They also provide an efficient method for calculating this new update, and show that it outperforms Federated Averaging on several datasets.

Pros:

The new method (FEDPA) is well motivated and is 'as simple as possible, but not simpler' based on the derivation.

FEDPA has standard Federated Averaging (FEDAVG) as a special case, and suggests a family of new methods based on approximations of the covariance matrix, which likely would exhibit bias-variance-tradeoff-esque behaviour.

The authors provide a practical way of calculating the required new quantities efficiently.

Experimental results show FEDPA has superior performance compared to FEDAVG, especially in regimes where client compute is high. 

Cons:

The big O analysis of the dynamic programming method for computing the local updates is very useful, but it would also be good to have empirical results on the additional cost on the client. It seems like the cost should not be too significant, but evidence of this would be very valuable, since the cost of FEDPA is strictly greater than the cost of FEDAVG. 

The addition of more tuning parameters in FL is never ideal, especially with the knowledge that using too small of a burn in time can lead to arbitrarily bad behaviour. However since the positive effects of FEDPA appear very quickly after the burn in ends, this may be less of a concern in practice. 

Would be valuable to include the results of the FEDAVG-1E in the experiments in the main paper, especially since it outperforms both FEDAVG-5E and FEDPA-5E on the StackOverflow LR, which is a surprising and unexpected result. 

In all experiments the FEDAVG locally updated using SGD (likely to maintain the connection and comparison with SGD used in IASG). However for FEDAVG, the optimization procedure CLIENTOPT could be something else, such as adam. It would be valuable to know how this FEDPA compares to FEDAVG when the local optimizer is a more powerful method than SGD, since FEDAVG has the freedom to change the local optimizer. (Of course it is only fair to also empirically compare FEDPA where the CLIENTOPT can be the same method. However since the use of SGD in IASG provides it with certain properties, it is less clear if this substitution can be safely made). 

The authors provide ample justification for their new method and sufficient evidence that it outperforms the existing standard method FEDAVG.  
review_title: A well motivated and effective method for deriving local updates on the client

Meta-review: The reviewers raised a number of concerns which are addressed by the authors. The paper provides an interesting/novel perspective for federated learning (as a posterior inference problem rather than an optimization problem) which can potentially allow for faster and more accurate solutions. 
Decision: Accept (Poster)