Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of adding sparsity to local training in federated learning. Under the approach each node uses only the top-K weights for computations in the forward and backward pass of training (and only top-K activations for computations in the backward pass). The authors empirically show that adding sparsity in this way, while speeding up local training, significantly degrades the accuracy when directly used in federated learning. To mitigate this issue the authors propose to combine sparse training with sparse aggregation since their empirical study suggests that the accuracy drop may be due to dilution of weights that are active in only a few nodes. Experiments on CIFAR10 with non-IID splits using ResNet-18 as the model show some gain in accuracy along with a drop in communication cost when using the proposed schemes.
main_review: Strengths:

1. The authors introduce a new problem of adding sparse local training to federated learning. The problem appears well motivated (sparse training can lead to significant power savings at the edge) and challenging (directly adding sparse training to conventional aggregation methods like FedAvg leads to significant accuracy drops).

2. The proposed solutions are good initial approaches backed by the empirical study and do appear to lead to some improvement in accuracy over the naive approach.

Weaknesses:

1. For a paper that relies entirely on empirical evidence for its claims, the empirical study appears a bit limited as it considers only a single dataset (CIFAR10) and a single model architecture (ResNet18). Without seeing results for some other image/text datasets and other model architectures (MLP/LSTM) it is difficult to completely believe the claims especially since the proposed solutions rely heavily on the claims that the positions of the top-K weights do not vary significantly across models and across rounds.

2. The experimental results are also a bit limited. In particular:

a) It is misleading to highlight both max accuracy and max comms savings values since the two correspond to different settings and thus cannot be achieved simultaneously. I would suggest removing the bold font for the max comms savings model and just reporting the comms savings for the different mask ratios. An interesting future direction would be to think of a metric that can combine the two since it is currently unclear if the communication saving (and local computation cost reduction) is significant enough to compensate for the drop in accuracy w.r.t conventional FL (black line in Fig. 1)

b) Since the authors acknowledge that accuracy is expected to first increase and then decrease on increasing mask ratio, it would be worth including results for all mask ratios in the appendix at least to see the trend clearly.

c) The results with i.i.d data partitioning are not presented. Since from Fig 1 it seems like accuracy drops even for the i.i.d case, it is important to show if the proposed schemes alleviate the issue in that case as well.

d) Lastly, following point 1, in my opinion all experiments should be performed with some other datasets and model architectures before the paper can be considered for acceptance.

3. (Minor) There doesn't seem to be any real difference between Top-K-Weights and the Diff on Top-K-weights in that the weight updates will be exactly the same in both cases as far as I can tell. If that is so then there are essentially 2 (and not 3) new schemes being proposed. 






summary_of_the_review: The authors introduce an interesting and important problem but given that the paper relies purely on empirical evidence for its claims the current evaluation is quite limited and not enough for acceptance. I find the claims fairly intuitive, however, and so would be willing to increase my score if more empirical evidence (in line with my comments above) is presented to back them.

Comments after rebutal : I appreciate the added experiments which certainly demonstrate that the proposed schemes are able to improve performance across different datasets. I have increased my score to reflect the same.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates sparse training to accelerate on-device federated learning. It begins by taking an off-the-shelf sparse training method (SWAT) and analyzes its limitations. Then, the method proposes three strategies to improve upon SWAT. Experiments are conducted on CIFAR-10 dataset and show some improvements compared to SWAT.
main_review: The paper has following strengths:
1.	Indeed, for more realistic deployment and to move federated learning towards practical systems, it is important to accelerate the on-device training (and not just the inference which is mainly the current practice). So, the problem the authors have considered is important.
2.	The proposed strategies clearly indicate the improvements over existing centralized sparse training technique like SWAT.
3.	Some of the insights shared (like the non-sparse weights staying the same throughout FL) were interesting.

The paper has following weaknesses:
1.	The main weakness of this paper is the experiments section. The results are presented only on CIFAR-10 dataset and do not consider many other datasets from Federated learning benchmarks (e.g., LEAF https://leaf.cmu.edu/). The authors should see relevant works like (FedProx https://arxiv.org/abs/1812.06127) and (FedMAX, https://arxiv.org/abs/2004.03657 (ECML 2020)) for details on different datasets and model types. If the experimental evaluation was comprehensive enough, this would be a very good paper (given the interesting and important problem it addresses).
2.	One other thing (although this is not the main focus of this paper), the authors should provide comparisons between strategies that result in fast convergence (without sparsity) vs. sparse methods? For example, do non-sparse, fast convergence methods (like FedProx, FedMAX, and others) result in small enough number of epochs compared to sparse methods? Can the fast convergence methods be augmented with sparisity ideas successfully without resulting in significant loss of accuracy? Some discussion and possibly performance numbers are needed here.

summary_of_the_review: Even though the reviewer is quite happy about the problem addressed in this paper, it lacks a solid experimental section. I would be willing to increase the score if the authors can present a more comprehensive results section.

Post rebuttal:
I have read the author response and other reviews. Thanks to the authors for a detailed rebuttal and new experiments. It addressed my concerns and so I have increased the rating from 5 to 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes ZeroFL; a method that allows for sparse neural network training at the edge along with reduced upload communication, both of which being important aspects in federated learning.  ZeroFL essentially follows a prior work on sparse neural network training, namely SWAT, and adapts it appropriately for the federated setting. The SWAT method works by setting a target weight sparsity percentage (denoted by $sp$) which is enforced in both the forward and backward pass by a top-k operation on the weights. This allows for using sparse convolutions and thus offering training time speed-ups. The authors note that it is important to use the same set of sparse weights for inference as well, in order to not affect performance. 

ZeroFL then applies this idea in the federated setting by letting each client perform SWAT type of training with a target sparsity of $sp$. Instead of the clients then communicating to the server the exact set of sparse weights obtained at the end of their local training procedure, ZeroFL proposes to add an extra “slack” $r_{mask}$ and communicate $1 - sp + r_{mask}$ percentage of weights (instead of $1 - sp$). The authors motivate this change via the results of an ablation study which showed significantly lower performance when directly applying SWAT to the federated setting. They argue that with the extra $r_{mask}$ weights, a “cleaner” training signal from the clients can be given as these weights are not “corrupted” by the weights of the other clients when averaging at the server. The authors further propose three separate weights of updating the server model, i.e., 1) using the client top-k weights, 2) using the difference between the server and client top-k weights and 3) using the top-k differences between the server and client weights.  

The authors experimentally validate the ZeroFL method on cifar 10 using 100 clients and both iid and non-iid splits.

main_review: This submission is reasonably well-written and easy to follow. The idea is simple to implement and seems to work, at least in this specific setting that the authors considered. This constitutes a nice advantage to ZeroFL. Nevertheless, I believe this submission needs quite a bit of work before I can recommend for acceptance:

- There are comparisons with other relevant baselines missing; e.g., federated dropout (which can be done on the weight level as well) is missing.
- The overall contribution seems incremental compared to the prior work of SWAT. Furthermore, the authors discuss how SWAT also employs activation sparsification but they seem to not use it at all in their experiments. 
- The overall state of the experimental evaluation is a bit weak; there is only one dataset considered where the number of clients is relatively small (100). What happens on datasets with more users for example on EMNIST / FEMNIST? Furthermore, it seems that the authors show results from single seeds and therefore it is hard to see what is the stability of their results.
- The authors don’t talk about the extra cost of transmitting the indices of the non-zero values in the main text (although they do mention the CSC format in the table caption), which can be an important additional communication cost in the case of unstructured pruning. Furthermore, at section 5 at the end, the authors mention $0.5(r_{mask }- sp)$ savings and it is not clear to me how is that derived and whether it does take into account sending the indices of the non-zero values as well.

As for other general feedback:
- At the first bullet before the related work, the authors argue that with the way their sparsity is realised  the privacy is better preserved. Could they perhaps elaborate more? To me it seems that this can actually be worse for privacy as each user can have its own sparsity pattern and thus reveal additional information compared to a server imposed sparsity pattern which is the same for all users.
- This work seems to be geared towards upload compression, although to me it seems that download compression is also feasible. Have the authors experimented with such an idea? 
- The algorithm needs some improvements, e.g., add an if statement for the different choices, as it can be confusing as it is now. Furthermore, the details Figure 3 are quite hard to see, so I would encourage the authors to use a different colour palette and intensity for the lines. 
- At the caption of table 1 the authors mention a 7.4x improvement over vanilla SWAT but the 7.4x seems to be on the dense model, i.e., when SWAT is not employed. Could the authors elaborate on what they compare against?
- Sometimes the authors refer to top-K and sometimes to top-K%. It would be better if they pick one for consistency. 

summary_of_the_review: Overall, while ZeroFL is simple, it does seem to be a straightforward extension of SWAT to the federated setting with a very minor modification. The experimental evaluation is rather weak (experiments on a single dataset) and there is an absence of other critical baselines (such as federated dropout). These are the primary issues that lead me to recommending rejection.
review_title: null

Meta-review: This paper considers the problem of on-device training for federated learning. This is an important problem since, in real-world settings, the clients have limited compute and memory, and local training needs to be efficient. The paper shows that the standard sparsity based speed-up techniques that consider top-K weights/activations during forward and/or backward pass do not work well in the federated setting and proposes several solutions to mitigate this issue. The proposed solutions are demonstrated to work well on several datasets.

In their initial assessment, given that this is largely an empirical insights driven paper, the reviewers mainly expressed concerns about the experimental evaluation (e.g., only one dataset CIFAR10 and one architecture ResNet18) and lack of more baselines (e.g., Federated Dropout). The authors responded in detail to the reviews and also conducted additional experiments and the reviewers and authors engaged in discussion. As the discussion converged, the reviewers agreed that the revised manuscript addresses their key concerns and their assessment, on an average, are now learning largely towards a borderline accept.

I also read the reviews, the discussion, and read the paper. I think the paper is a good initial attempt at providing a general approach to enable on-device federated learning when the clients are lightweight devices (e.g. edge devices). Even though the study is somewhat preliminary, the current manuscript, after the revision during the discussion phase, is significantly improved version of the original submission and does address the key concerns from the reviewers. Overall, I would rate the paper for a borderline acceptance.
Decision: Accept (Poster)