Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper develops a modified version of FedAvg by local batch normalization that is tailored for federated learning with non-i.i.d. data. Different from most of existing work that consider the unbalanced labels, this paper uses unbalanced features to motivate the non-i.i.d. federated settings. Specifically, the unbalanced features are captured by the difference in local covariances. 

Merits:
1. The paper tackles an important problem in federated learning, which may have practical impact.
2. There is an effort on quantifying the performance gain of FedBN vs FedAvg.

Cons:
1. The assumption that the feature shift on local datasets is captured by the difference in local covariances needs better justification. Why this is a valid assumption?

2. The comparison is on the convergence to f and f* under FedAvg and FedBN. It is not clear if f and f* have the same learning accuracy or training loss. If not, then the comparison is not fair.

3. The convergence result is not stated in a clear form. In general, exact linear convergence to the optimal solution is not possible for SGD-type algorithm. The condition under which this holds true needs to be explicitly mentioned in Theorem 4.4 and Corollary 4.5. In addition, how the convergence depends on the averaging interval E is not clear. In FedAvg, there is non-negligible error in non-i.i.d. setting that depends on E. What technique you have used to eliminate it? These details are all unclear. 

4. The current analysis is mainly following Dukler et al. (2020). The new technical challenge and difficulty in analyzing the federated version of Dukler et al. (2020) needs to be highlighted. 

5. The simulations are not convincing. On one side, the state-of-the-art baseline FedAdam Reddi et al. (2020) has not been compared; on the other hand, the performance gain relative to FedAvg seems to be marginal. 
review_title: Review of FedBN

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Update following answer:

Thanks for your detailed answer, which confirms me in my initial assessment.

------


1/ Summary of the paper

This paper introduces the use of local batch normalisation layers in order to circumvent data shifts issues in FL, called FedBN.
Building on a simplified model of BN (neural network with 1 hidden layer and BN rescaling) that was previously introduced by [1] to study the impact of BN on convergence, it is proven that the convergence of the proposed FedBN on training data can only be faster than the convergence of FedAvg on the same data points.
Parametric experiments on a simple heterogeneous dataset (built as the union of digit classification tasks) show that the proposed method yields a better performance, and a more stable one, than standard FedAvg.
Experiments on 3 real-world datasets containing heterogeneity to simulate different centers show again that the proposed FedBN method improves upon FedAvg.

2/ Acceptance decision

While ideas related to BN and FL have been floating in the community this year, this paper proposes a novel approach with extensive and convincing numerical results, and interesting theoretical results on the effect of fedBN.
Provided the paper is updated to better acknowledge other very related works, I think it should be accepted as it will be a very valuable contribution to the FL community.


3/ Supporting arguments

A/ Novelty

A very close reference not acknowledged by the paper is [2], which builds on the same ideas of domain adaptation as this paper and proposes to keep local BN weights in the FL setting, called SiloedBN, showing that results are improved and more stable with respect to FedAvg.
However, the proposed FedBN is different from siloedBN [2] in the sense that SiloedBN proposes only to keep local BN statistics, while FedBN keeps local BN layers altogether (trainable and untrainable parameters thereof).
Therefore, although not entirely new, FedBN is still novel.

B/ Theoretical results

Setting aside the drift issues encountered when the number of local updates increases, the paper uses the same formalism as [1] to get a linear convergence bound for FedAvg on the training set, which is controlled by some key constant mu_0 (the higher, the faster the convergence).
For this analysis, the contribution of this paper is to put FedBN under the same framework as [1] and to show that the resulting constant mu_0^* is larger than the constant mu_0: therefore, FedBN’s training error diminishes faster than FedAvg’s.
Although the underlying model is a simplification of the experimental reality, the problem tackled is very complex, so this result is still a significant contribution for the community.

C/ Experimental results

The experimental results are in three parts: 
- An introductory toy example, which helps to better understand the importance of local BN layers in the case of domain shift.
- Benchmark on a simple hand-crafted FL digit recognition dataset built by concatenating different digit datasets with different domain shifts (e.g. MNIST, SVHN…), thereby borrowing from the domain adaptation literature. While I have some remarks to make the results even more complete (cf next section), I think these results help to better understand the behaviour of FedBN and FedAvg under heterogeneity.
- Experimental results on 3 hand-crafted FL datasets with a varying number of centers (4, 6) and different tasks. In almost all cases, FedBN significantly improves the final testing accuracy upon FedAvg and FedProx in almost centers.
These experiments are exhaustive and convincing, and are definitely an asset of this submission.

D/ Writing
Although dense, the paper is well written and easy to follow. There are minor typos.

4/ Additional comments
1. How could FedBN be adapted to transfer a model on a new center from another domain?
2. There is a minor error in the proof of Corollary 4.6, which does not invalidate the results. Indeed, the authors claim that since G^{*, \infty} = diag(G_1^{\infty}, \ldots, G_N^{\infty}), one has, for all i, \lambda_min(G^{*, \infty}) > \lambda_min(G_i^{\infty}). This is false in general: one can only claim that \lambda_min(G^{*, \infty}) \geq \min_i \lambda_min(G_i^{\infty}). Since all the minimal eigenvalues of G_i^{\infty} are lower bounded by the minimal eigenvalues of G^{\infty}, the final result still holds.
3. This is extremely minor, but in Corollary 4.6, I am not sure that in all generality one can have strict inequalities: only >= statements may be available
4. Related to the previous remark, although very interesting, the result of Corollary 4.6 is frustrating as it does not quantify the speed improvement. Even if a closed-form quanitification is out of hand, it may be interesting to produce numerical experiments on synthetic data on this simplified model to quantify the gap between the convergence bounds.
5. In section 5.1, how is computed the testing accuracy reported in figure 4? Is it the mean of the per-center accuracy on the local testing datasets? The training sets are artificially balanced, but how balanced are the testing datasets?
6. In order to better understand the significance of all the numbers reported, it would have been nice to report the performance of a model trained in a pooled-equivalent fashion, i.e. with all data points in a single center.
7. In Fig 4 b), one studies the effect of the local dataset size on the performance, and, not surprisingly, the performance of all methods diminish in this case. Why isn’t FedAvg reported here? In particular, why was the 10% fraction chosen in the other experiments, even if it led to suboptimal results?
8. In Figure 4 c), are the points with different x values related in any case? Or are they coming from independent experiments. The text « we started with including… then, we simultaneously added n clients » is a bit ambiguous.

Typos and other remarks
9. The use of both m and M in Sec 4 can be confusing
10. In Lemma 4.3, assumption 1 should be replaced by assumption 4.1
11. There are 2 typos in theorem 4.4: missing parentheses around Equation 2, and n should be replaced by N.
12. In Figure 5, the colours are hard to parse for colour-blind people.

Refs
——
[1] Dulker, Gu, and Mont\’{u}far, Optimization theory for relu neural networks trained with normalisation layers, in proceedings of ICML 2020
[2] Andreux, Ogier du Terrail, Beguier, and Tramel, Siloed federated Learning for Multi-Centric Histopathology Datasets, in proceedings of MICCAI DCL 2020
review_title: A very good contribution

final_score: rating: 7: Good paper, accept
review_content: This work proposes an extremely simple approach to a well-know problem within Federated Learning: batch normalisation. Indeed, FL usually imply an averaging of different model parameters trained on different distributed devices. But what happen with the running statistics that some training methods have ? Such as batch norm and some optimisers ? Well, this work proposes to address the first issue by introducing a specific batch norm method named Local Batch Normalization (FedBN). Standard BN statistics cannot simply be aggregated due to strong variations potentially occurring when training data across the clients aren't IID distributed (i.e strong shift in the input representation). To alleviate this issue, the authors propose to exclude the BN parameters from the aggregation (i.e. each client has its own BN excluded from FL). 

This paper is self-contained in the sense that the initial problem is well-defined with some theoretical background and visualisations. Then a very simple solution is validated both theoretically and empirically. While the underlining idea is very, very simple, the theoretical and empirical justifications remained unclear until this work. According to this paper, we now are able to motivate the use of local BN in the context of FL. Moreover, this approach has been evaluated (convergence rate, variations w.r.t local epochs, different data distributions, heterogeneity variations, results vs SOTA)  in different non-IID FL tasks (USPS, MNIST, SynthDigits, MNIST-H). Finally, an example on "real-world" data is given to better highlight the importance of the demonstration. The proposed FedBN convincingly outperformed all the other approaches in all the setups. 

However, I found the paper hard to read due to the lack of a clear plan. I think the content should be a bit more structured. 

Some remarks and concerns are listed bellow:

Pros:
+ Good theoretical backup for a very simple idea.
+ Finally validating the use of local BN that was certainly used by many researchers but for "unexplained" reasons.
+ Good results compared to SOTA methods.
+ Real-world datasets are not really common in FL papers. It's great to see one. 

Cons:
- The idea remains a bit "too" simple, can't we build on the top of these findings to further improve the BN ?
- Where are the appendix ?
- I find the organisation of the Sections a bit hard to follow. In particular, from the end of page 3 to the end of page 5 the reader has to follow a single big Section without any segmentation while others sections are 2 paragraphs long. As an example: Is Section 3 really needed ? Can't it be integrated into a revised Section 4 (with proper sub-sections) ?
- As BatchNorm statistics are kept local to each client. What happen at testing time ? Let's say we want to distribute the trained model to a new client that does not have already computed statistics ?

Remarks:

The paper could benefit from some proof-reading. Examples are:
- Colloray 4.6
- In particular, the average model w = (w⇤ +w⇤)/2 and average BN parameters  = (⇤ +⇤)/2 has a high error -> unclear
- Widely known ag- gregation strategy in FL, FedAvg (McMahan et al., 2017) it often su↵ers when data is heterogeneous over local client. -> not correct
- As Fig. 1 shows the local squared loss between is very di↵erent between the two clients -> not correct
- What is ck in Eq 1 ?

review_title: Approaching the important problem of batch normalisation in federated learning

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a minor modification to the existing FL learning framework, which can be easily implemented. The major contribution of this work comes from a few theoretical analyses. The experiments are conducted on several publically accessible data sets with intuitive explanations. 

Pros:
- The FL toy example is quite interesting, and we get a nice starting idea for your overall paper.
- The theoretical analysis seems pretty solid and intuitive. I didn't fully proofread all the details, but the parts that I read look correct to me.
- The experiments reveal certain properties of FedAve and FedBN, most results are convincing and the relative improvement looks pretty predictable based on the explanation.

Cons:
- The motivation on the toy example may not work well on real-world data set, especially since these assumptions may not hold in general. 
- The experiments are oversimplified and lacks a proper explanation for this kind of preprocessing.
- Without any large-scale publically-accessible data set being utilized, I concern a lot about its performance on real applications.

Detailed concerns:
- The necessity of preprocessing all data set into the same setting is not provided. Why only keep 7438 examples for each data set? Does it have something to do with the non-iid setting? Is it possible to keep the data set in their original size? 
- I checked the supplementary as well and didn't find how do you make sure the data set assigned to each client is non-iid.  The claimed covariate shift and concept shift didn't have corresponding experimental settings being explicitly stated, which looks quite confusing to me. Also, please add the comparison with the non-iid scenario with the one proposed in FedAvg.
-  I highly suggest the authors add a few large-scale data set to verify the effectiveness of FedBN, e.g., cifar-10, cifar-100, and imagenet. Because these data sets are more close to the scenario where each client has non-iid data. 
- Each client has 10% data seems a bit too easy especially for some simple data set, e.g., MNIST, and its variants, I highly believe a comprehensive study like the one did in the FedAvg paper by changing K, B, E, and all their combinations, is the correct way to evaluate your proposed framework. Especially, FedBN is extremely easy to implement. Without these comparisons, I believe the experiment is not complete.
- Code is not attached, many details of FedBN remains unclear. The results may perform differently when we implement our own version. 


Minor comments: 
Inconsistent notations: sometimes FedAvg but sometimes using FedAVG.
review_title: a new FL training strategy for non-iid data 

Meta-review: The paper addresses the problem of batch normalization (BN) in federated learning, which is of great interest to the community including practitioners.  The proposed method here simply excludes the BN parameters from the aggregation, and evolves them locally.

As a main contribution, reviewers particularly liked the solid justification of the proposed scheme, both with substantial theory and extensive experiments. Presentation style can be slightly improved, the usage at test time can be clarified more, and some references mentioned by R3 should be added, but this overall does not affect the strong level of contributions present in the work, and the discussion phase with the authors was already constructive.
Decision: Accept (Poster)