Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary:
The paper presents an empirical study of causes of parameter divergence in federated learning. Federated learning is the setting where parameter updates (e.g. gradients) are computed separately on possibly non-IID subsamples of the data and then aggregated by averaging. The paper examines the effects of choice of optimizer, network width and depth, batch normalization, weight decay, and data augmentation on the amount of parameter divergence. Divergence is defined as the average cosine distance between pairs of locally-updated weights, or between locally updated weights and weights trained with IID data. The paper generally concludes that regularization methods like BN and weight decay have an adverse effect in the federated setting, the deepening the network has an adverse effect while widening it might be beneficial, and that adaptive optimizers like Adam can perform poorly if their internal statistics are not aggregated.

I recommend that the paper be rejected. The main shortcoming of the paper is the lack of rigororous statistical analysis to support its conclusions. The paper contains a lot of raw data, but the discussion mainly highlights trends that the authors seem to have observed in the results, without quantifying the relative sizes of effects, how consistent they are across experimental conditions, etc. The writing is also quite unclear, to the point that I often didn't understand exactly what argument was being made.

Details / Questions:
The main problem is the lack of quantitative analysis of the trends the paper identifies. For example, regarding "Effects of Batch Normalization", there seem to be two claims made:
1. Batch normalization makes things worse (somehow) in the federated setting
2. Batch re-normalization still makes things worse, but not as much
How are these effects quantified? How large are they? Do they hold across all datasets, architectures, and optimizers considered? Ideally there would be a table summarizing each experimental manipulation, its effect on performance, whether that effect is significant, etc. Of course this requires some care because the paper is doing an exploratory analysis and there are many hypotheses to test; a good reference is [1].

The paper also relies heavily on parameter divergence as a measure of performance in federated learning, but I see no evidence presented that parameter divergence is predictive of test accuracy (which is presumably what we actually care about). Intuitively I can see how it might be related, but since divergence is basically being used as a proxy for accuracy, it is vital to show convincingly that the two are related. What do we gain by analyzing parameter divergence rather than simply comparing test accuracy?

Regarding the "steep fall phenomenon": The paper seems to present this as an indicator that a manipulation performs poorly in the federated setting. But, isn't it a good thing if parameter divergence goes down? Why does specifically a sudden, sharp decrease in divergence indicate a problem?

Finally, some improvements might be made to the experiment setup. For one, the case of completely-disjoint label sets in different local learners seems extreme to me. Wouldn't at least partial overlap be more common in practice? (This is not my area so I don't know). Experimenting with different degrees of overlap would be useful. As for network architectures, it would be valuable to look at a greater variety of standard architecture styles (e.g. ResNet, Inception, etc). I realize there are some experiments with ResNet, but the focus is mainly on the single-path VGG-like architecture. I do realize this is a lot of experiments to do.

Minor points:
* In the setting described as "IID" in Table 1 is not, the subsampled for each learner are not IID subsamples of the full dataset because they are class-balanced (if I'm understanding correctly)

References:
[1] Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1-30.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper experimentally studies the reasons for the slow convergence of the Federated Averaging algorithm when the data are non-iid distributed between workers in the multiclass-classification case. Paper performs extensive experimental study and observes that the main reasons for failure are connected to (i) the parameter divergence during the local steps, (ii) steep-fall phenomena when parameters on different nodes are getting close fast, and to the (iii) high training loss. 

My score is weak reject. The paper provides extensive but unclear experimental results. Improving presentation would significantly improve the paper. For example, why in experimental and theoretical study different parameter divergence metrics were used, etc (see below), why different networks use different optimizers. 
Moreover, provided experimental comparison might be unfair. The learning rate is constant throughout all of the experiments, depending only on the optimizer, but not on the neural network architecture. This can affect the final results. 

Concerns and questions that should be addressed:
1. The initial learning rates were not tuned properly. It is set to be the same for different neural network topologies, which might significantly affect the results. What did the choice of initial learning rates is based on? 

2. Why the parameter divergence metric in Definition 1 is not the same as in the theoretical study (Appendix B)? What is the intuition behind Definition 1?

3. Why the divergence of parameters is considered only at the last layer? It seems to hide many important interactions in the other layers. 

4. Some important experimental details --- should be added:
   - At which moment the parameter divergence is computed in the plots? Is it computed at the end of the local iterations right before synchronization? 
   - How the training loss was computed in the plots? before or after synchronization? on the local only or the global data?
   - Which batch size was used? 
   - Improve the figure caption to detail the experimental setup. (e.g. in fig 3. the network architecture was mentioned only for one of the figures, include which optimized was used, etc)

5. In experiments on Fig. 2. and Fig.3 (middle) what is the accuracy for IID baseline? Is the observed phenomena connected to the poor network architecture or to the non-iid data? 

6. In table 5 of the appendix, why experiments use Adam optimizer, but not Momentum SGD as in the main paper to compare the performance of ResNet14 and ResNet20?

7. Better re-prase the definition of the steep fall phenomena, now it is not very clear: in the IID setting parameter divergence values are also sometimes reducing sharply; in the network width study parameters divergence doesn’t experience sudden drop. Also, how does this phenomena (and parameter divergence too) connects to the training loss? 

8. Why for different experiments different baseline models are used? (NetA, NetB, NetC)


Other minor comments: 
- Appendix B, first equation on page 13. (d_q)^t -> (d_q)^t_k; The size of gradient \nabla_w [E ...] is different from the size of (d_q)_k. They cannot be added together.
- page 7, last sentence of the first paragraph: what is the accuracy achieved with Batch Renormalization? Why the reason for accuracy gap is “significant parameter divergence”? on fig. 3 “parameter divergence” is smaller than for the baseline.
- Why the name of the section on page 7 is “excessively high training loss of local updates” if later it is stated that it is actually smaller than for the IID case? 
- Defenition 1, line 4: “the then” -> “the”
- section 3: “A pleasant level of parameter divergence can help to improve generalization” -> where was it shown?
- section 4.2, paragraph 2: what is meant by “hyperparametric methods”?
- section 4.2, paragraph 3: “quantitative increase in a layer level” -> not clear what does it mean.
- page 4, effect of optimizers: what do you refer to as “all model parameters”? 
- page 5, last paragraph: Hinton et al... -> (Hinton et al…). Use \citet(\citep) instead of \cite. 
- why Dropout yields bigger parameter divergence if on Fig 2, right it actually helps? 
- Last line of the page 5. Where was this observed? 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors empirically investigate parameter divergence of local updates in federated learning with non-IID data. The authors study the effects of optimizers, network depth/width, and regularization techniques, and provide some observations. In overall, I think this paper study an important problem in federated learning.

However, there are some weakness in this paper:

1. The paper is nearly pure empirical. There is no theoretical analysis supporting the observations proposed in Section 4.1, which weaken the contribution of this paper.

2. This paper only raises some issues in federated learning with non-IID data, and discusses the potential causes. No suggestions or potential solutions is proposed in this paper, which weaken the contribution of this paper.

3. Since this is nearly a pure empirical paper, I hope the authors can make the experiments thorough. However, there are some experiments I expect to see but not yet included in this paper:

    3.1. The authors only studies Nesterov momentum in this paper. However, in practice, it is more common to use Polyak momentum. I hope the authors can also study FL SGD with Polyak momentum in this paper.

    3.2. In this paper, the authors assume that different workers has the same number of local data samples (in Definition 1). However, due to the heterogeneous setting, it is very likely that different workers have different numbers of local data samples, which could be another source of divergence. Furthermore, different numbers of local data samples also results in different numbers of local steps, which may also cause divergence.

    3.3. [1] proposes a regularization mechanism (FedProx) to deal with the heterogeneity. Instead of studying weight decay, it is more reasonable to study the regularization technique proposed by [1].


4. There are some missing details (maybe they are already in the paper but I didn't find them):

    4.1. What is the definition of Adam-A and Adam-WB? And, what are the differences between Adam-A, Adam-WB, and vanilla  Adam? (and also, what is the "A" in NMom-A?)

    4.2. When using Adam in federated learning, how are the variables synchronized? Note that for Adam, there are 3 sets of variables: model parameters, 1st moment, and 2nd moment. Due to the local updates, all the 3 sets of variables are not synchronized. When the authors use Adam in FL, did they only synchronize/average the model parameter and ignore the 1st and 2nd moments, or did they synchronize all the 3 sets of variables?


----------------
Reference

[1]  Li, Tian et al. “Federated Optimization for Heterogeneous Networks.” (2018).
review_title: Official Blind Review #1

Meta-review: This paper studies the problem of federated learning for non-i.i.d. data, and looks at the hyperparameter optimization in this setting. As the reviewers have noted, this is a purely empirical paper. There are certain aspects of the experiments that need further discussion, especially the learning rate selection for different architectures. That said, the submission may not be ready for publication at its current stage.
Decision: Reject