Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper makes the following claims about the existing frameworks and benchmarks -
Limitation with existing benchmarks and datasets
 - Lack of versatility in the data
 - Synthetically generated partitions of conventional datasets
Not practical enough
 - System speed and availability is missing
Not reproducing large scale deployments
 - Lack of user-friendly API for automated integration

This paper fixes those aforementioned problems by making the following contributions -
- Integrate existing datasets in a way which is more suitable for practical FL
- Modular Framework
- Experiments on some of the existing problems/techniques.

strengths: The main strength of this paper is the introduction of a realistic and large scale dataset that would bring the community very close to benchmarking FL in the real world.

The authors do a good job in identifying key issues with the existing way of benchmarking FL experiments and provide a solution to some of them. The benchmark experiments that blend resource and statistical efficiency would open up a new direction of methodological FL works that can build upon this benchmark.

The proposed FAR simulator is significantly more scalable than the existing frameworks and hence would be useful for the upcoming FL works that aim at evaluating multiple FL under such setups.

The API design appears to be flexible and modular for ease of integration. The amount of code required to run FL experiments is indeed a few lines of code.

weaknesses: FAR Platform
- Practicality of the benchmark - Saying that a limited number of clients is not a practical setup is debatable. I would argue that a limited number of clients is a more practical setup when big institutions want to train a model together without centralizing data (due to privacy reasons).

- Modularity - While L233-244 touches upon the framework’s flexibility and modularity, it is not clear why this would be a preferred design over something like FedML that also provides a similar API. I would suggest if this part can be covered somewhat more rigorously (like comparing the architecture or design etc.), it would bolster their claim. Otherwise, it is difficult for me as a reviewer to agree with the claims made. Another good example could have been writing a similar code as shown in Fig. 15, 16, and Fig.17 but using other frameworks.

- Scalability - Point 1) at L247 should not be considered a practical solution because in the practical world different individuals are not sharing the same GPU. However, for resource-efficient simulation, I see it as a good solution. Once again, it would help if you mention what design primitives were missing from the existing works that make FAR better than the others. I am also not entirely sure what authors mean when they say existing frameworks “fail” to support more than 10 clients. Is their framework not able to accommodate more clients due to memory issues or the models are not converging or something else?

Experiments
For the three categories of experiments mentioned from L293-322, it is not clear how this benchmark helps in addressing those problems? In general, it is not clear what value is added by those experiments to the main theme of the paper.
For example - are you proposing a new way of benchmarking or measuring co-optimizations of the system and statistical efficiency? Or is it just that the benchmark can be useful for researchers working on those problems, in such a case it would be useful to mention how existing frameworks are not suitable in comparison to FAR for such problems.
By looking at the third contribution, I assume the authors want it to present these experiments as a comparison between existing works but it is again not clear to me if it adds any value to the main theme.

At a high level, I see two weaknesses - 
Lack of comparison with existing works as pointed above
Many claims have been phrased too strongly about the existing works. My suggestion would be for the authors to rewrite such phrases.

correctness: The dataset construction appears to be practically motivated and coherently presented. I have some concerns with the experiments shown in the paper which I have mentioned in the "Weaknesses" section. However, the evaluation metrics and criteria are appropriate.
clarity: The paper is well written and the flow of the paper is smooth. However, I believe some of the claims can be significantly toned down (mentioned in weaknesses) and should focus on objective results more than subjective evaluations.
relation to prior work: The paper fixes the scalability issue for a multiple clients setup in comparison to existing works such as Flower, FedML, Leaf and FedEval. However, the experiments and comparisons made are not enough from an objective standpoint. However this is justifiable because of heavy compute required to do large scale training for all the frameworks and datasets.
additional feedback: Once I have the first set of responses from the authors, I believe I can make better suggestions for improvement. For now, I would recommend addressing the items mentioned in the weaknesses.
review_title: The paper proposes a new set of datasets and a framework that facilitates practical and scalable evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a set of benchmarking datasets for federated learning. The big novelty with the datasets is that they account for client (end user) heterogeneity in terms of compute & communication resources and availability. Accounting for them, can provide a different perspective on the performance of the various proposed optimizations for federated learning. 
strengths: The strongest contribution of the paper lies in the way it accounts for client heterogeneity in its evaluations.
weaknesses: The simulation / emulation environment based on client traces seems quite primitive. The reviewer wonders if more realistic simulation and emulation environments from computer networking and systems areas could not be taken advantage of here.

The benchmarks by and large ignore privacy, which is the motivating reason for federated learning. While the paper may not contribute much here, it would have been nice to see a discussion. 
correctness: Seems OK. 
clarity: Overall, the paper is well-written. Though it would have been better if the paper also described all the desiderata for such a benchmarking framework, even if FedScale achieves a subset of the desiderata. 
relation to prior work: Yes. 
additional feedback: As this paper represents a first step, it would be great to have a discussion of open (unaddressed) issues that need to be addressed in the future.
review_title: A step in the right direction for an important problem area

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents an evaluation system for federated learning. The presented system, FedScale, contains multiple datasets and can evaluate both system and statistical performances. The authors also demonstrate that the system is flexible and easily usable. 

The contributions of this paper can be summarized as follows: 
- Comprehensive datasets: 18 datasets with realistic federated data partitions. The datasets are also diverse, in terms of both scale and tasks. 
- Realistic Systems Benchmarking: The paper also incorporates system heterogeneity by simulating computation/communication performance and availability. 
- Easily usable runtime: The paper presents FAR, which is a set of APIs to evaluate FL in terms of many metrics. 
- Experiments and Insights: The paper does benchmarking experiments and reveal some interesting insights, including system & statistics co-optimization. 

strengths: Strengths: 
- Extensive and realistic datasets: FedScale provides 18 realistically partitioned datasets. The datasets span a wide range of tasks: image classification, translation, object detection, recommendation. Also, the scale of the datasets spans a wide range, from thousands to millions. These datasets would be helpful in evaluating FL algorithms. 
- Simulation of systems heterogeneity: Systems heterogeneity is an important problem in FL aside from data heterogeneity. The authors did a very good job in stimulating systems heterogeneity, including the computation/communication performance, and availability. 
- Systems implementation: The authors implement FAR, a usable FL API for evaluation. This would be helpful to researchers in FL. The systems is also efficient and implements optimizations, such as communication/computation overlapping. 
- Good insights: I appreciate the authors' experiments and insights regarding system & algorithm co-optimization in Section 5. 
weaknesses: - Regarding 'Privacy'. In Section 1 and Fig. 2, the authors claim that the system can evaluate privacy. However, in the paper, the authors only showcase how to perform evaluations on backdoor attacks, which from my opinion is not related to privacy. Privacy means that the individual data should not be revealed, while backdoor attacks do not aim to reconstruct original data, but try to compromise the model trained, which should be more related to 'security' instead of 'privacy'. Therefore, I think that the authors should re-consider their claims in Section 1. 
correctness: The benchmark is generally correct and accurate. I appreciate the authors' efforts in trying to make it realistic, in terms of both data and systems heterogeneity. The only thing that I cannot fully agree with is the notion of 'privacy' as mentioned in the Weaknesses part. 
clarity: This paper is written clearly and very easy to follow. 
relation to prior work: I am somewhat familiar with LEAF and FedEval mentioned in Table 1, and I think that the comparisons with these two benchmarks are correct and clear. 
additional feedback: Please see the "Weaknesses" above. 

Minor issues: 
- Table 2 in the main paper contains only 15 datasets, while Table 4 in the appendix contains 18 datasets. Why is the difference? 
- Line 37: What is "privacy efficiency"? I cannot understand why this is related to efficiency. 
- Line 47: "borrow" -> "borrowed". 

Further comments: 
- It would be even better if you can try to implement security features into FedScale, e.g. differential privacy. This would help the research community even further. 

========================================================

After the update, I think that my concerns have been addressed.
review_title: A very helpful toolkit for benchmarking FL. 

Meta-review: In this paper the authors provide a new framework for evaluating federated learning algorithms across a wide variety of datasets and tasks.  Reviewers were generally impressed with the setup, with the primary two pieces of feedback being (a) a clearer discussion of privacy implications and (b) clearer comparison to other federated learning evaluation tools.  In particular for comparison to other evaluation tools, the authors are encouraged to follow reviewer feedback be more precise and qualified in terms of the unique contributions of this approach over prior approaches.  I'd encourage the authors to revise and resubmit.
Decision: Reject