Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new federated learning framework called HeteroFL, which supports the training of different sizes of local models in heterogeneous clients. Clients with higher computation capability can train larger models while clients with less computation capability train smaller models, and all these model architectures belong to the same model class. This approach dramatically benefits clients with limited computation capability and fully exploits their computation power. 

Strengths:
1. The paper is well-motivated. The communication and computation problem does exist in federated learning, which made the proposed approach practical to apply. 
2. The idea is novel enough. As far as I know, no other papers exploit the potential of model heterogeneity in federated learning.
3. The experiment is solid and rigorous enough to support the main idea. The authors use three different models and three corresponding datasets to conduct the experiments. The results show that model heterogeneity is promising, which outperforms state-of-the-art federated learning approaches.
4. The paper is well written. The authors organize the whole paper concisely and comprehensively, which makes the paper easy to read.

Weakness:
1. Although static batch normalization and Masked Cross-Entropy Loss are not the main contributions of this paper, I think it's more persuasive to prove their effectiveness by comparing it with baseline experiments instead of simply applying it.
2. The experiment setting should be elaborated more, such as the hyperparameters.

Also, I have some questions regarding the paper content:
1. Why the parameter size of Standalone, FedAvg, and LG-FedAvg is smaller than many other models in Table 2? Intuitively, Standalone, FedAvg, and LG-FedAvg use complete model architecture, so the parameter size should be the largest. However, it is less than models like a and b in Table 2.
2. How did you choose the shrinkage ratio? Have you compared the results of different shrinkage ratio? And also, Have you tried other methods of shrinking models?
review_title: Well motivated idea, well written

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a framework named HeteroFL to address heterogeneous clients equipped with very different computation and communication capabilities.

The proposed methods were examined on three datasets for image classification and language modelling tasks.

Experimental results show that the proposed method boosts accuracy of the baseline FedAvg.

However, there are various undefined notation and missing details in the paper. Therefore, the paper is not easily readable and requires readers to make assumptions to fill the gaps. Some of the proposed strong claims should be supported and verified either theoretically or experimentally. In addition, experimental analyses should be improved by comparing the proposed method with the other related work.

More detailed comments and suggestions are as follows:
 
- Please use different notation for matrices and sets. For instance, $W$ denotes both a matrix/tensor and a set in the paper, which causes a problem in equations (1)-(3).

- In equation (1), does W_i^p denote a set or a matrix/tensor? 

- What do W^t_g[: d_m, : k_m] and W^{p−1,t+1}_ g \ W^{p,t+1}_g denote?

- Could you please elaborate why "small local models can benefit more from global aggregation by performing less global aggregation for part of larger local model parameters."? That is, what are the benefits, and how do you assure that this approach enables to have these benefits?

- Please explain how you calculate statistics of hidden representations from local data after the model converges more precisely. How do you determine the convergence criteria?

- It is stated that "Local models only upload their statistics for once after optimization is completed." What are these statistics and how are they used when they are uploaded to the server?

- r^{p-1} is a scalar constant. Therefore, Scaler module scales feature activations by a constant. How does this help training?

- Please define \phi() (activation layer) more clearly. Do you refer to a non-linear activation function?

- Please define local capabilities information and L_{1:k} and L_m.

- Please describe details of architectures of the CNN, PreResNet18 and Transformer used in the experiments.

- Please explain what "Standalone" denotes in the tables.

- Please explain how you construct complexity levels in detail. 

- Please explain the masking operation used in Masked Cross-Entropy Loss. More precisely, how do you mask out the output?

- The proposed method follows similar motivation and approaches of split learning and federated-split learning methods. 
Could you please elaborate similarity/difference between your proposed methods and these methods? 
A comparative experimental analysis would be also helpful to explain the novelty over these methods.

Following the rebuttal:

I checked comments of other reviewers and response of authors. Most of my questions were addressed in these responses. Therefore, I improve my rating.
review_title: Promising results, but the paper and the work should be improved.

final_score: rating: 7: Good paper, accept
review_content: This work presents a novel FL algorithm named HeteroFL (the name might sounds weird to some peoples) and 3 different simple methods to improve FL in heterogeneous conditions (i.e. both in term of clients and data partitioning). These tricks are: 1. A revised batchnormalisation; 2. a pre-activity scaling; 3. a masked loss (i.e only consider local classes)  to help with non-IID datasets. All these modifications have been tested on 3 different datasets and 2 different tasks. From the results, we can see that the proposed approach works better. Although, it is not clear from where the benefit comes. 

My biggest concern with this paper lies in the complexity of the problem raised vs the lack of analysis of the proposed solutions. Appart from HeteroFL that is definitely a very nice idea, the small tricks try to address very important concerns related to FL. 1. What do we do with running statistics?; 2. How do we manage highly non-iid partitioning?. Unfortunately, the given solutions appear to be small "tricks" or "fixes" without any real theoretical or empirical insights. In my opinion, each problem should be detailed and discussed in a standalone paper. In this extent, reading this paper is a little bit like: "We present this method (HeteroFL), that works pretty well if we add these little fixes to very important problems". But we have no-idea on how these little fixes actually help HeteroFL (and thus could also help FedAVG, FedPROX etc etc). 

However, it is worth noting that the core idea of this paper: HeteroFL, is a very simple and elegant way to deploy FL on an heterogeneous client set. 

Pros:
+ HeteroFL is a very nice idea to deploy FL with an heterogeneous set of clients, and it seems to work well.
+ While the reasons aren't clear, the 3 proposed methods to stabilise and enhance the training process could help with bigger FL questions.
+ The paper is self-contained.

Cons:
- some claims are wrong: "the clients with the lowest capabilities will not be the bottleneck of FL’s performance" -> Well, according to the results, adding smaller clients (i.e. less powerful) always harms the performance. "The results show that HeteroFL can boost clients’ performance with low computation and communication capabilities by allowing the training of heterogeneous models with larger computation complexities." -> This sentence is unclear and leads to a wrong statement. HeteroFL isn't boosting the performance of small clients. Indeed, we can expect that small clients aren't even able to train the model, so HeteroFL is allowing us to train on such clients, but it is not boosting the performance. According to the results, smaller models = worst performance.
- sBN, scaling and masked loss "seem" to help, but it absolutely unclear to which extent. Also, why are the statistics finally uploaded to the server for inference (isn't this a leak of information ?). The theoretical motivation of the Scaler with the dropout analogy isn't clear at all. I'm pretty sure that it is mostly a matter of re-phrasing. 

Remarks:
- Eq. 1, 2 and 3 are a bit hard to read. Is the "\" symbol used to denote integer divisions ? From a first read, it is not clear how weights are aggregated in intermediate complexity levels.
- The explanation of the effect of the proportionality is really unclear. Therefore, it is very hard to understand what Figure 2 (and the others) are about ... "To demonstrate the effect of proportionality of clients with various computation complexity levels, we interpolate from 10% to 100% with step size 10% of global model proportionality. For example, a − b means to interpolate between a and b models starting from 10% of clients assigned level a and 90% of clients assigned level b to 100% level a clients. " -> This is very hard to understand. What is global model proportionality ?
- Most of the Figures aren't black and white compatible (all the curves are almost impossible to distinguish).
- Section 3 should be splitted in 4 parts. One for each proposed method.




review_title: A new FL algorithm and few tricks to improve heterogeneous FL. 

final_score: rating: 7: Good paper, accept
review_content: UPDATE:
The authors have consistently improved their argumentation over the course of the review and addressed my concerns sufficiently. I have increased my score and recommend acceptance.


ORIGINAL REVIEW:
The authors propose three elements:
A way to approach model heterogeneity across clients with different resource constraints, a 'masking trick' for approaching non-iid-ness and a modification to BatchNormalization in the Federated Learning setting.

In order to receive differently sized models that still allow to be aggregated systematically on a central server, the authors propose to deterministically prune away neurons/feature maps of NNs to effectively create networks of different widths. While a more powerful end-device receives, computes with and updates all feature maps, a less powerful device computes only with a fraction of those feature maps. This idea seems novel and interesting to me and I commend the extensive experimental study. 

For the proposed 'static BN', the authors propose to not worry about running estimates across all clients until convergence of the model. I disagree with both, the proposition that 'static BN' is something new compared to normal BatchNormalization, nor that it inherently solves the issue with BN in the federated setup. Firstly, BN in section 3.1 of the original paper suggests using moving averages only as a means to estimate test-time performance during training. At convergence, the final model requires re-computing statistics on the whole dataset, identical to what is proposed in this work. Secondly, apart from the problem of finding a global statistics estimate at the end of training, the usage of mini-batch statistics during training is used as an alternative to the global estimates since the global estimates are too expensive to compute during training. In centralised training, a random mini-batch represents the global data-set well enough. In non-iid data settings, a random mini-batch from a client does not serve as a good estimate of the global data statistics, only for the statistics of its local data-set. 
I therefore do not understand how the proposed solution of 'sBN does not track running estimates and simply normalize [sic] batch data' addresses the issue that a client-level mini-batch does not represent global statistics. 
A common approach to dealing with BN is to simply replace BN with, for example Group Normalization (https://arxiv.org/pdf/1910.00189.pdf) (Also see Section 5 for a discussion of the issues of BN in FL). 

I am therefore asking the authors to explain the exact differences of sBN to normal BN with respect to re-estimation of global statistics at the end of training and to elaborate on the issue of using mini-batch statistics as an estimate of global data-set statistics. If indeed there is a difference to normal BN, I would like to see explicit experimental results that compare normal BN to sBN and, ideally, to GroupNormalization as a way to circumvent the BN issue. 
Maybe I am misunderstanding in the sense that the 'Masking Trick' also somehow alleviates the non-iid data issues with BN. If that is the case, I would like to see an explicit ablation study that distinguishes between the two. 


With respect to non-iid data and the proposed 'masking trick' the authors cite Zhao et al. stating that the weight divergence mostly occurs in the last classification layer of networks. Inspecting Figure 2 of that paper, this conclusion can be drawn only for one of the three experiments at display. I agree that this is a minor point and the proposed trick sounds reasonable and interesting to me. In oder to see its effectiveness, however, there needs to be an ablation study with and without that trick. I cannot find such an experiment in the paper. Furthermore it should be stated that label skew is just one of the possible sources of non-iid-ness in FL. Lastly, the authors mention that the masking trick 'allows local clients to switch to another subtask simply by changing its mask...'. I have troubles understanding what is implied here. From a client's perspective there is only its local label-distribution. If a client is assumed to be new to the federation of clients, the new client would receive the un-masked global model presumably. In which setting would a client require a new mask (from another client?)

Experimental Evaluation:
The authors present a large range of experiments for different scenarios and levels of heterogeneity between clients.
I do have issues understanding the results precisely though. The authors do not mention what the x-axes in Figure 2 represent. Is the y-axis local or global accuracy? In combination with Tables 1 and 2, I am confused. If Standalone and FedAvg have 633K parameters respectively, how does a 100% model a (first row in Table 1) have 1.6M parameters? Presumably, experiments were conducted with the same full, 100% CNN model architecture. Alternatively, the hyper parameters in Table 4 in the Appendix do not make sense to me. An alternative interpretation would be that these are the amount of parameters communicated until convergence - but then again the hyperparamters are inconclusive and additionally, the space requirements of 100% model a should still be the same as FedAvg. Or does this column describe the amount of communication at 32bit float precision? 
If the authors chose a different architecture for their baselines, then the results are inconclusive. 
I am assuming that 'Standalone' refers to no communication between clients, but that needs to be specified! 
Also I am assuming that 'Local' assigns zero probability on $p(y=c|x)$ for those classes $c$ that are not present on a client during training. Again, this is not explicitly specified. Are the reported results averaged across clients? Are they weighted by the amount of data in the local test-sets?  In the conclusion, the authors state that their method achieves better results with fewer number of communication rounds. I can no-where see a comparison of communication rounds. 

The authors mention two scenarios: Fix and Dynamic and explicitly say 'We annotate Fix for experiments with a fixed assignment of computation complexity levels and Dynamic for local clients uniformly sampling ...". I cannot find this annotation anywhere and am therefore confused which setting the results correspond to. 

I am missing one axis of evaluation: In a heterogeneous (Fixed) setting with, for example, 50% a and 50% level e, how is the average local performance on devices with model a and model e separately. In the text, the authors describe 100% model e achieves 77.09% accuracy and a 50-50 mix with model a achieves 89%. But that does tell me nothing about how much the (weak) clients with model e improved through the increased power in these other 50% devices. In the next sentence the authors claim that 'HeteroFL can boost client's performance with low computation and communication capabilities'. But reporting the average could also allow for the conclusion that only clients with higher compute power and a larger model achieve higher performance. 

Conclusion
The paper proposed three elements, a heterogeneous modelling approach, sBN and the masking trick. Apart from confusion in motivation and explanation, the experimental section requires most attention in my opinion. Things are simply very unclear to me.  Terms, axes and results need to be properly discussed. Furthermore, the effects of HeteroFL, sBN and the masking-trick need to be independently studied, otherwise no conclusion can be drawn on the effectiveness of the individual ideas. I would recommend the authors to re-focus their paper on the heterogeneous training idea alone and leave sBN, which I don't understand at the moment, and the non-iid remedy trough masking to another paper. 

I believe that the idea of dynamically adjusting the model width to the local compute capabilities in the way the authors present it is promising. However I need to be convinced that less powerful clients can meaningfully contribute to the global model and have higher performance compared to training a small-sized global model in the first place. I encourage the authors to revisit their motivation for elements of this work (HeteroFL, sBN and masking), refocus, and fix their experimental discussion. I believe that there is enough merit to this idea and paper to be accepted with major effort during the rebuttal. 
review_title: Good idea, but bad execution and a lot of confusion

Meta-review: The reviewers had a number of concerns which seem to have been addressed by the authors in the discussion phase.  All the reviewers are in favor of accepting the paper. The paper provides an interesting/novel idea for federated learning with heterogenous clients/devices. 
Decision: Accept (Poster)