Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 1/ Summary

This paper introduces an aggregation mechanism designed for neural networks with batch normalisation layers. This mechanism relies on two parts: probabilistic mixing weights of the loss function and the use of a weighted pool estimator for aggregating the BN variance parameters. The mixing weights are derived from a GMM with variational inference. A convergence result in the *convex* case is provided. Experimental results on 3 image datasets show that this approach yields better results than other standard FL algorithms (FedAvg, FedProx, q-FedSGD, FedMA…) as well as a better resilience to heterogeneity (understood as class imbalance).

2/ Acceptance decision

Despite its seemingly good experimental results, I am in favour of rejecting this paper as correcting its weak points would require major changes.

3/ Supporting arguments

A/ The theoretical results (Sec 4, one page) are irrelevant to the problem tackled by the paper.

Indeed, they are proved in the case of a convex function. However, the proposed method is relevant for neural networks with batch normalisation layers, so more than 1 layer, and therefore these networks yield non-convex loss functions, which do not satisfy the hypothesis of the results. Further, the unbiased gradients assumption (assumption 1) seems dubious in the non-iid case tackled by the authors. Last, but not least, it is difficult to understand if this assumption is valid or not (and the same holds for the results) because the notations used to state the theorem in Eq 11 do not correspond to the notations of the loss function Eq 2. 

B/ Important technical details are omitted, which makes it difficult to understand the whole algorithm, threatens reproducibility, and shades some doubts on the experimental results.

- In section 3.2.3, the authors introduce a GMM for the mixing weights \pi_k while dropping the upper script t for convenience, which is understandable. However, it is not stated how the estimation process of \pi_k^t evolves throughout the different batches. Does one need to do 1 full variational inference at each aggregation step? Can one re-use previous parameters through aggregation rounds? Does one rely on constant \pi_k?
- A related question is the behaviour of the proposed method when reducing communication frequency. Indeed, the algorithm is only exposed in the case of communication after each local gradient computation, which is typically avoided in the edge setting considered by the authors. How is it extended in this case? In particular, how does the estimation of the parameters \pi_k change?
- In a related note, the number of local computations done before any aggregation is omitted in the experimental section. Are all models compared with the same number of local steps?

C/ Writing is not very clear and should be improved. In particular, some notations do not make sense; notations tend to change a lot across the different sections, making it difficult to see a global picture emerge.

4/ Additional comments

- Delving into appendix A.2 which explains the aggregation rule for variance layers, I do not understand the transition from the 3rd equation to the 4th (from top to bottom). Could the authors elaborate more on it?
- Although the approach is « decoupled », the mixing weights used in the aggregation rules (Eq 3) (standard layers) and ( Eq 4) (BN means) are the same. Have the authors tried completely decoupling these mixing weights?
- In Section 3.2.3, if I understood correctly the means \mu_k and \sigma_k are related to the empirical means and variances of the batches in the different clients. Sharing means and variances of each batch could yield to large privacy leaks. This remark goes together with the communication frequency remark above.
- In Section 3.2.3, the authors state that they assume that data are drawn from a Gaussian distribution « without loss of generality ». However, it is well known that e.g. image distributions are not Gaussian.
- In Section 5.1, I do not understand the notation $pr(x) = \mathcal{N}(0, 1)$. How can a probability take negative values?
- How does performance of the different methods vary with respect to the communication frequency? This is an important metric of the training efficiency, and missing from the experiments.
- Given that the experiments have been run multiple times, it would be nice to provide error bars to understand the significance of the different results.
- Some typos:
    - page 4,  $W_{BN}$ instead of $W_{NN}$ at the beginning of Sec 3.2
    - "Form" -> "from" in the introduction
- The presentation of figures 4, 5, 6 make it difficult to read the captions.

review_title: Not good enough

final_score: rating: 3: Clear rejection
review_content: PAPER SUMMARY

This paper presents a new federated learning scheme for neural network that accounts simultaneously for different layer types and different (heterogeneous) local data distributions. In particular, the proposed method differentiates between parameter updates for conv, fc versus bn (batch-norm) layers. 

The main argument for this differentiation is that aggregating parameter gradient for bn layers is vulnerable to high bias when local data are heterogeneously distributed. This leads to a new proposal on aggregating bn parameters directly (instead of using their gradient).

Both parameter gradient and parameter updates are weighted by the same set of probabilistic weights that seem to be associated with mixture weights of a GMM model that generate local training data. These weights are learned via a deep generative model that encode local statistics of local data into a latent space, which seems to be associated with random drawn from a (latent) categorical distribution associated with those mixture weights. 

A theoretical convergence guarantee for the proposed algorithm is also provided.

NOVELTY & SIGNIFICANCE

This paper poses an interesting view on the parameter aggregation of federated learning. I appreciate the authors' perspective that perhaps the aggregation of parameters defining different layer types need to be diversified to fit the nature of the data. 

However, I find the proposed solution and its fundamental idea of modeling such heterogenity somewhat arbitrary and perhaps flawed. To elaborate:

First, it is not at all clear to me why the authors associate the weights that combine parameter gradients (for fc and conv) and parameters (for bn) with the mixture weight that generate local data. This seems like an arbitrary choice to me and there seems to be a flaw here: if a local device receives very little data but its data come from a mixture component with large weight, its gradient will likely be biased (due to the lack of data) but will still dominate others (due to its large mixture weight) -- I would like to hear the authors' thoughts on this.

Second, if I understand correctly, the auto-encoder model devised to learn those mixture weights is supposed to encode the local statistics (i.e., mean & variance estimates of a Gaussian that fit the local training input) to a categorical distribution over label classes. I am not sure if I missed something here but this is a bit strange: by right, the mean and variance of the *input* distribution do not contain any information about its *label* -- how does the embedding manage to generate information that does not exist in the first place? Likewise, how would such *label* information be related to the mixture weights of the *input* distribution via Eq. (10) -- could the authors elaborate further on this?

Third, on a high-level of idea, I am also not fully convinced that the aggregating the bn parameter gradient is vulnerable to high bias. This statement somehow came across as a politically correct statement with no concrete substantiation. It will be better if there is some technical demonstration that explicitly show that for bn layer, the bias of the aggregated gradient is higher than the bias of its aggregated parameters.

Last, I find the theoretical analysis is not at all particular about the inner working of the proposed algorithm. The analysis is on one hand generally applicable to any function that satisfies the stated assumption and, on the other hand, not specific enough to account for the facts that the combination weights are also being updated and that the bn parameters are not updated via gradient aggregation. Given this, I do not think the analysis is applicable to this setting, and it should be evaluated separately in its own setting.   

TECHNICAL SOUNDNESS

I have made high-level check over the main bulk of technical derivation and have not spot any glaring issues. But, as I said above, on the idea level, the three technical contributions here (separating bn aggregation from those of fc and conv, learning the mixture weights from local statistics (barring access to data) and the theoretical analysis) are orthogonal with little connection to one another -- for the first two contributions, there seems to be flaws on the idea level that need further clarification.

EXPERIMENT

The reported results appear positive but it seems the authors only generate those for one single run. As the improvement margin is relatively small, it is important to average results over multiple runs to make sure the improvement is significantly above the deviation margin.

Furthermore, reporting only the final performance gives very little insight regarding how the invented components help avoid accumulating high bias, and also whether the auto-encoding scheme correctly recover the mixture weight -- perhaps this can be shown by evaluating the proposed mechanism on a synthetic dataset where we know the ground-truth. 
review_title: An interesting view on parameter aggregation for federated learning which devises specific parameter aggregation rules to fit the nature of different layer types.  The proposed solution and its fundamental idea are however a bit arbitrary (in many technical choices) and perhaps flawed. Clarifications and further elaborations are needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:
 
The paper proposed FedDEC, a novel approach to conduct model updates aggregation in federated learning. The main motivation of this paper is to decouple the aggregation of normal model weights and statistics in BNs separately such that both data and model heterogeneity can be handled. Theoretical analysis indicates that the proposed FedDEC method enjoys a good convergence guarantee. Extensive experimental results are provided to show that FedDEC enjoys high efficiency and better model accuracy under the non-IID environment compared to the considered baseline methods.

##########################################################################

Reasons for score: 
 
Overall, I think the current manuscript is marginally above the acceptance threshold of the ICLR conference. Studying the effectiveness of the model aggregation process in federated learning is a promising research direction. The proposed approach to conducting the decoupled model aggregation over the model parameters and statistics aggregated in the BN layers is promising. The proposed FedDEC method is justified both theoretically and empirically. Theoretically, the authors show that FedDEC enjoys the same convergence rate as normal SGD. Extensive experimental results over the simulated federated learning environment indicate that FedDEC enjoys good training effectiveness while reaching to better model accuracy. If my concerns on the current manuscript (please see "Cons") are addressed, I will be happy to improve my evaluation score.
 
##########################################################################

Pros: 
 
1. The paper is well written. The research direction on improving the model aggregation in federated learning is promising and practical.
 
2. The formulation and theoretical analysis of the proposed FedDEC method looks promising.
 
3. Extensive experimental results are provided for the image classification tasks under the simulated non-iid environment, which indicates that FedDEC enjoys high effectiveness in improving the model test accuracy under the data heterogeneity.
 

##########################################################################


Cons: 
 
1. A detailed discussion of the algorithmic aspect of FedDEC is missing. Section 3 discusses the formulation of FedDEC, however, it’s not super clear on the exact steps that FedDEC will run e.g. what information will the clients upload to the data center?; what are the exact operations that the data center will conduct? The authors’ are highly encouraged to add an algorithm box to describe the FedDEC algorithm. 
2. From the description in Section 3, it seems FedDEC always assumes the local clients to train only ONE local epoch. If that’s the case, then the communication efficiency of FedDEC can be worse than FedAvg due to higher communication frequency. A more detailed discussion on this can be helpful to understand FedDEC better. 
3. The convergence rate of FedAvg has been explicitly studied in [1]. It would be helpful to make a clearer comparison between the convergence rates of FedDEC and FedAvg. 
4. In Section 3.2.2., the mean and variance in BN are referred to as BN layer parameters. However, in a BN layer, there are two trainable weights (\gamma + \beta [2]) and mean + variance statistics collected over the training batches. It would be helpful to make the notations clearer. 
5. Although language models usually do not have BN layers, it’s still easy to imagine that the FedDEC algorithm can be applied over language tasks. Adding a language processing task can make the current manuscript stronger. 

[1] https://arxiv.org/pdf/1907.02189.pdf

[2] https://arxiv.org/pdf/1502.03167.pdf
 

#########################################################################


Minor Comments: 

1. Typos in the draft: (i) in the abstract “suffer form” —> “suffer from“; (ii) “$W_{NN}$ (parameters of parameters of BN layers)” —> “$W_{BN}$ (parameters of parameters of BN layers)” in Section 3.2. 
2. Missing references: [1-4]. 

[1] https://arxiv.org/pdf/1907.02189.pdf

[2] https://arxiv.org/pdf/1812.01097.pdf

[3] https://arxiv.org/pdf/1908.07873.pdf

[4] https://arxiv.org/pdf/1912.04977.pdf
review_title: A novel and effective model aggregation method for federated learning

final_score: rating: 3: Clear rejection
review_content: In this work, the authors propose new ways of averaging updates received at the server from a subset of clients in a federated scenario. Specifically the authors aim to address issues arising from the non-iid nature of the data that arises in FL and propose to treat BatchNorm parameters differently from other NN parameters. 

Introduction
Reading this paper, I am confused about terminology used by the authors. Specifically, the authors discuss 'data bias' and 'parameter bias'. 
In in the introduction, the authors claim that 'Conventional approaches average gradients uniformly from the clients, which could cause great bias to the real data distribution'. Assuming the authors understand FedAvg to be a conventional approach, the averaging with $|x_k|/|x|$ (Eq. 1) is not uniform but weighted by the local dataset size. Further, the meaning of 'causing bias to the real data distribution' is not clear to me. Unfortunately, the further explanation and Figure 1 don't help me in understanding what is meant. The 'GroundTruth' distribution of labels for cifar10 is approximately uniform. From the context, I understand that the authors try to describe some consequence of a non-i.i.d sampled distribution of data according to labels, but I cannot understand the point they try to make. 
Next, the authors discuss 'parameter bias'. The authors distinguish between BN parameters and other NN parameters. They term the BN parameters 'statistical parameters such as mean and variance'. Generally speaking, BN contains the 'scale and shift' parameters $\gamma$ and $\beta$ (https://arxiv.org/pdf/1502.03167.pdf), which I am assuming the authors reference to. Again, the authors make use of the term 'bias' to say: '[...] bias on the BN layer parameters'. I am not familiar with the notion of 'bias on a parameter' and would like the authors to clarify. Based on Figure 2 I assume they aim to convey that BN parameters in a FL setting converge to different values compared to a centrally trained model. 
In Section 3.1 the authors further make the distinction explicitly between 'gradient parameters', by which they mean weights and biases as opposed to 'statistical parameters' of BN. Since the scale-and-shift parameters of BN are also updated by gradient descent, I am wondering if the authors mean the mean and variance estimate across data-points for feature-maps, which also plays a central part in (federated) BatchNormalization. The authors in their experimental section make no mention of how they form these global estimates for mean and variance in BN, so they omit that crucial detail there.

The authors specifically focus on label-skew as source of non-i.i.d-ness in this work, but never make this limitation explicit. Since the non-i.i.d challenges in FL are not limited to label skew, I believe the authors should make this explicit. 

Related work:
The issues with BN in the federated setting has been described for example in Section 5 of https://arxiv.org/pdf/1910.00189.pdf. There, the authors propose to replace BN with GroupNorm, an approach that has been adopted in several follow-up and recent works in FL with models that originally contain BN. I would encourage the authors to compare their work against this approach, both in the RW section and also in the experimental section.

Method Section.
Notation-wise, I encourage the authors to not use $x$ or $x_k$ to denote a (labeled) dataset, since $x$ is usually reserved for a single data-point with associated label $y$. In Eq. (1) the loss formulation of FL is a bit sloppy since the parameters to optimise for, $W$ do not appear in the RHS of the equation. In FedAvg, we explicitly optimise $min \sum_k |x_k|/|x| L_k(W,x_k)$, where the local parameter estimates $W_k$ appear as intermediate parameters as a consequence of multiple local optimisation steps. 
The authors claim that 'data points available locally could be biased from the overall distribution'. Again, I believe to understand the intended meaning to be the non-iid issue, but I encourage the authors to make their understanding of 'bias' more concrete. 

In FedAvg, the individual clients do not transmit gradients $\nabla L_k ()$ to the server (Section 3.2.1). This is the approach in conventional distributed SGD as employed in a high-speed-connected data-centre for speeding up centralised training. In this centralised setting, the non-iid problem does not exist. Instead, in FL, clients transmit parameters that have been updated through a series of gradient-descent steps. More recent work (https://arxiv.org/abs/2003.00295) makes the role that these transmitted parameters have in an interpretation as a gradient. This distinction is important.

The derivation in Appendix A.2 for show-casing the unbiased-ness of the variance parameter averaging seems wrong. Going from the third to fourth equation makes a mistake and also if the original expectation was equal to the sum of weighted expectations, then simply averaging would actually be the unbiased estimator. The authors here are falsifying their own argument through a derivation mistake. The last equation should only pull the expectation into the sum in the right-most term and you are done.

Section 3.2.3
I like the notion of modelling the datasets as GMM and to infer responsibilities at averaging time. The explanation of the approach is confusing to me, however. 
If I understand it right, then EQ 9 describes a VAE setup per client k. There is no sharing of parameters or latent space between clients. $s_k = [\mu_k,\sigma_k]$ are the mean and standard-deviation across the whole local dataset at a client $k$. The authors propose to encode this single vector $s_k$ into a latent space z_k of dimension $C$. The authors do not explicitly specify the prior p(z_k), but given the constraints on $z_k$, I assume it is meant to be a Dirichlet distribution. From context I could imagine that it has something to do with the per-client label distribution. 
Since each ELBO is per-client and each client has just one data-point $s_k$, I do not understand the need for auto-encoding. Simply infer z given a decoder-model for the single data-point.  The authors mention the use of neural networks, but they do not detail their architecture choices anywhere. It is also unclear to me how $\pi_k$ falls out in equation 10. I imagine it I corresponds to some sort of posterior across all local models' encoding z. Since the latent-spaces across clients k do not share any meaning, I don't see how that can be sensible.  All together, this section is not readable to me. 
I can see the appeal of reweighing updates through a specific formulation pi_k, but since the updates are label-independent (eq. 3,4,5), how does that play into this.

Section 4
My understanding of proofs of this form is somewhat limited. From what I can gather, this proof shows that FedAvg converges if the per-client loss-function is pre-multiplied by a constant factor $\pi_k$. As such, the convergence proof should be analogous to what is presented in e.g. https://arxiv.org/pdf/1907.02189.pdf with the exception of the update in Eq 5. Maybe the other reviewers can comment further on this. 

Section 5.
In the Federated Setting, 20 clients should be considered not enough for experimental validation generally speaking. I appreciate the breath of experiments in terms of models, algorithms and related algorithms. 
Unfortunately, the authors chose to define their own non-iid-split of the datasets. In general, I would appreciate to see comparisons with existing data-set splits in related work on non-iid data, such as for example https://arxiv.org/abs/2003.00295. This would help avoid the bifurcation of the literature. 
The reference to q-FedSGD seems to be wrong.

After reading the experiment section, some serious questions arise:
What is the fraction of selected clients per communication round? What is the number of local epochs per client? Some of the related works seem to suggest performing FedSGD (also the first paragraph in Section 3.2.1 suggests this). If you have single-gradients per device and equal-size data-sets per client, then I don't see how non-iid data-distribution across clients are an issue as this approaches a global mini batch step, where each mini-batch consists of smaller mini-batches, one from each client. The non-i.i.d issue in FL stems from the fact that each client optimises on its own for a sufficiently long time that the resulting progress is destroyed by averaging in parameter space. The authors need to specify their setup here. Concretely, I would want to at least see Cifar10 split into 100 clients, 10 of which are selected at every round. Each client needs to optimise locally for a full epoch on its own dataset of 45000/100 = 450 data-points. 

The authors propose two things: A new averaging approach by computation of $\pi_k$, as well as a new approach to estimating the gradient for the scale-parameter of BN using pooled averaging. These two things need to be studied separately by setting $\pi_k = |x_k|/|x|$ in one scenario. At the moment, my trust into the computation of $\pi_k$ is very low, since the corresponding section is not understandable to me. The pooled averaging approach seems sensible and I am curious to see if it solves the issue of BN in FL. Additionally, the authors need to clarify how the BN-statistics are computed at test time. I would also like to see a comparison of the proposed models with BN and with the state-of-the-art method which is replacing them with GroupNormalization. 

Finally, I would like to thank the authors for the interesting approach to training BN-equiped Neural Networks in a federated setting. This idea seems promising to me. The approach for computing pi_k is not clear to me and I would like the authors to revise it. Please specify the role that knowledge of the label-distribution plays and if the method is applicable when non-iid-ness stems from other sources than label skew. (I propose looking at the FEMNIST dataset for example). 
Many issues with this paper remain and I encourage the authors to overhaul their work. 

I see no issues with the Code of Ethics
review_title: Interesting ideas but many issues remain

Meta-review: The authors’ feedback has not fully addressed the reviewers’ concerns and the reviewers think that the paper is not ready for the publication. The authors should consider the following issues for the future submission:

1) The concern from Reviewer 1: if a local device receives very little data but its data come from a mixture component with large weight, its gradient will likely be biased (due to the lack of data) but will still dominate others (due to its large mixture weight).

2) Numerical experiments are not consistent with theoretical results. The theory is for convex but experiments are with non-convex loss. The response from authors does not resolve this issue. 

3) Notation is confusing and changing throughout. We strongly suggest the authors revise carefully this and make it clear. 

Although the experimental results are potential, we would like the authors to revise it carefully by addressing the reviewers’ concerns and further improve it by considering theoretical results for non-convex in order to submit to the next venues. 

Decision: Reject