Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: Fedamby is an open source cross-silo federated learning dataset with natural partitions, code examples, and benchmarking guidelines. It contains 7 datasets in the healthcare domain with different scales. 
strengths: 1. Very useful datasets in healthcare domains
2. Partitioned by natural client information
3. Timely topic and relatively unexplored direction 
weaknesses: 1. The authors claim the benchmark to be comprehensive, but the scope is only limited to the healthcare domain. 
2. The main contribution of the paper is introducing new datasets for cross-silo federated learning. However, FLamby's datasets are already available and not collected by the authors. 
3. I think the paper should provide more details on the benchmark results and explain them. It will be very interesting to see if FLamby will enable the developers to get new insight into a cross-silo setting. 
4. Security and privacy are major concerns in the cross-silo settings. (although mentioned in future work) FLambly should enable the developers to explore, for example, different cryptographic algorithms. 
5. The scale of the dataset is relatively small, even for a cross-silo setting. 

correctness: The claims are mostly correct. However, the paper lacks details on the evaluation metrics of the benchmark. 
clarity: The paper is well written 
relation to prior work: The paper clearly discusses previous works and how it differs from them. 
additional feedback: Thanks for submitting to NeurIPS 2022 Datasets and Benchmarks Track. I believe this is an important topic and the proposed datasets will be very useful. However, I believe the paper can be improved by addressing the concerns I listed in the weakness section.
review_title: An useful benchmark for cross-silo FL

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work introduces an aggregation of medical datasets which are relevant for the context of Federated Learning (FL). This means that each dataset is constituted by data gathered from different sources (referred as "clients"), with the same classification task and similar input.

These datasets differ from previously presented dataset suites for benchmarking FL, where the datasets are _cross-device_, where there are many clients, but each client provides a low number of instances. Here, the datasets are _cross-silo_, where the number of clients is reduced, but each client contributes with a large number of instances.

Additionally, the authors provide a benchmark of different FL algorithms, against the baselines of single client training and pooled training. 

The code for downloading, splitting and replicating the benchmark results are made available in a python package.
strengths: - The datasets are easily accessible;
- The choices made in model selection, pre-processing, hyperparameter optimization, appear to be adapted and adequate for each dataset in the suite;
- The benchmark can be replicated and can also be extended with other datasets, models and/or FL strategies, without much development effort;
- The limitations of previous related work are well framed and are partially solved with the presented suite of datasets.
weaknesses: - The presented datasets only encompass medical tasks (e.g, cancer detection);
- The datasets have limited size;
- Most of the datasets have image data types;
- Although no new dataset is presented (the datasets are repurposed from previous uses), there is no discussion on the ethical/social implications nor documentation about the datasets, following one of the frameworks presented in the call.
correctness: Yes, the claims of the submission are correct. The datasets are representative of one of the possible applications of FL, and they represent _cross-silo_ sources of data. The datasets appear to be collected and pre-processed in a sound way.

The benchmark performed is not exhaustive (only simpler FL strategies are employed), but the models, hyperparameter optimization, metrics, and FL strategies appear to be chosen appropriately accordingly to the dataset. 


clarity: The paper is clearly written.
relation to prior work: Yes, the work distinguishes itself from previous contributions. The main difference between this and previous works is the setting of FL (_cross-silo_), and the presentation of a benchmark.
additional feedback: - The placement of images and tables could be improved. If they could be placed inline with the text instead of the top of the page, it would have a better flow with the text.
- Table 1 has cut text, due to the table size.
- It is not clear why the plots 1b, 1c, and 1d have different formats.
- The code snippet is interesting but it would make more sense to include it in supplementary material.
- The discussion on results and their implications (e.g., answering questions such as, "are FL strategies dataset-dependent?", "Which strategy is the better baseline?", etc.) could be extended.  
- Having datasheets (or at least more extended documentation for each dataset, not focusing on the setup).

These are merely suggestions for improvement from my perspective in the paper. I encourage the authors to comment their thoughts on these points.
review_title: Review of "FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Settings"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper is introducing a dataset benchmark suite for cross-silo FL in medical applications. In particular, it gathers 7 previously available datasets, which the author federate, and provides benchmark models and FL glue code (simulated setup) for running experiments on "realistic" cross-silo setups. Coarsely put, one can think of it as LEAF for cross-silo Federated Learning.
strengths: * The paper does a very good job at organising datasets from the medical domain in the cross-silo setting, federating them, and providing a benchmark implementation of a model that works. This can help a lot of people bootstrap by having a set of baselines along with dataloaders they can extend upon.
* The authors do a very good work at qualitatively explaining the setup and task of each dataset, along with the data heterogeneity that describes each one.
* I appreciate the effort that the authors have put in developing out-of-the-box solutions for each task at hand, covering a wide range of tasks and modalities.
weaknesses: * Flamby is not introducing a dataset per se, but rather organises and adapts existing datasets in the realm of cross-silo federated learning to democratise participation by offering a set of baselines.
* The datasets integrated in the Flamby are focusing rather on the smaller side of cross-silo, particularly in terms of number of participating clients (<10). Moreover, the case of vertically split cross-silo FL has not been visited at all. This hurts a bit the generality and scalability of the proposed suite.
* The paper, in its current state, is not a suite for general cross-device FL, but rather more focused on the medical domain.
correctness: * The paper, in its current state, is not a suite for general cross-device FL, but rather more focused on the medical domain. Albeit the authors' desire to become a more general suite, the papers fails to convince otherwise. This is actually one of the main points that I am putting the current score.
    * The authors talk in their related work about "extensive benchmarks with natural splits" being available in the cross-silo setting, referring to vision and HAR use-cases, but do not integrate any of these in Flamby.
    * Maybe it would be better to retitle the paper as "medical cross-silo" in its current form.
* Lines 121-122, "compatible with existing FL software libraries [...]": I am not sure where this compatibility stems from. Is it the fact that one can use the provided dataloaders? The provided simulation script is inherently incompatible with the workflow of many FL libraries and is to be used in place of these, not agnostically on top of these.
    * The authors focus their effort in a PyTorch compatible implementation. Is there an intention of extending it to other framework backends (TFF, etc.)?
* In terms of the local adaptive optimisers, it is unclear how the authors aggregate these.
clarity: The paper is very well written with details about the setup, evaluation and reproducibility of the experiments.

Some minor suggestions include:
* Figures 1,3 have quite small font size.
* The algorithm in Figure 2 is not really a "figure". Maybe a different caption heading would be better. It might as well be moved to the appendix in favour of freeing up some space for results of Figure 3.
* Hyperparameter tuning and Baselines: This section could be written more clearly.
* Appendix, L9, "porigin": typo
relation to prior work: It has been largely the case that many frameworks provide pre-implemented dataloaders for FL datasets (e.g. FedScale, TFF, FedML, FLSim), strategies, models, etc. to enable users bootstrap their research/development.

I see the proposed approach as an attempt to standardise experiments/datasets through benchmarking suites, that are subsequently being inherited by FL frameworks and provided off-the-shelf by the respective tool.

In the related work, it would be conducive to the reader to see what cross-silo datasets major FL frameworks provide and potentially how Flamby's implementation differs if there is an intersection. This could also be relevant to FL framework developers to fix potential issues in their current implementations or provide additional support for extra datasets.
additional feedback: I honestly like this paper and believe it is moving the needle towards a direction if FL worth pursuing. My main concern is that it appears to be more general than it is.

I am also documenting here some missing things that I would like to have seen in the paper:

* Introduction: I would also like to have seen the issue of system heterogeneity as a trait of cross-device vs. cross-silo setups (i.e. discrepancy in the capabilities of participating clients). Moreover, the general availability and reliability of these clients is limited.
* lines 52-53, "[...] the other highlighted [...] synthetic partitioning": While intuitively true, I would expect a citation for this claim.
* On the simulated run setup:
    * Can workers run on different machines through the simulation script? If so, what is the communication protocol. If not, do different clients run sequentially or in multi-treaded/process setup?
    * How is the dataset distributed to different workers?
    * Is GPU acceleration supported? (missing from the paper)
* On experimental setup:
    * How many different runs have been run?
    * Experimental setup and time to convegence missing for assessing the approximate time cost of training.
    * It would be nice to have seen some convergence graphs in the appendix.

I don't think the requirement on the client-side for having a google account to download from google drive is necessary. The authors should adopt such an approach (in place of the GDrive's Python SDK) for open access.

Last, I think that having integrated a DP/SecAgg-bootstrapped simulation would have made the paper much more appealing. Treating privacy - especially in the realm of medical data - as orthogonal/future work can potentially lead to harmful side-effects. Maybe a comment could go to the "broader impact" section of the paper about this.
review_title: Review for FLamby

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes an open-source cross-silo federated dataset and benchmark called FLamby.
 
Summarized contributions
1. Build an open-source federated cross-silo dataset suite including 7 datasets across different tasks in multiple domains in healthcare.

2. Provide guidelines to benchmark FL strategies

3. Make it accessible for benchmark reproducibility and easy integration in different FL frameworks
strengths: They provide standardized dataset APIs for 7 realistic healthcare datasets. 

These dataset APIs are agnostic to existing FL libraries for easy and flexible evaluation. 

It's easy to conduct the benchmark with a few lines of code. 
weaknesses: 1. These 7 datasets are relatively small compared to the cross-device FL datasets. Additionally, limited to the healthcare domain as well as the tasks.

2. No privacy, security, and fairness strategies were introduced and benchmarked. 

3. The baseline FL strategies are directly from cross-device FL. What about cross-silo FL algorithms? 

4. Limited contribution in the benchmark. If all these are simulations inside a cluster, what are the key differences between your benchmark and with existing cross-device FL benchmark? 

Even the authors mentioned some points above in future work, but the current version has very limited contribution to the cross-silo FL.
correctness:  I believe the methodology used for dataset and benchmarking it's correct.
clarity: It's a clear paper, but I would expect more experimental benchmark results. 
relation to prior work: As I discussed in the weakness sections.
additional feedback: Please refer to the sections above.
review_title: A dataset suite and benchmark for cross-silo FL

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper focuses on the cross-silo federated learning setting, where the number of clients are usually small but the data size per clients are usually medium to large. The typical clients for the cross-silo settings can be hospitals, banks or various institutes. The paper doesn't provide new datasets, but rather use the existing dataset and partitioned the datasets by the data collecting hospitals/institutes. The datasets consist of different task type: classification, segmentation or survival. The paper also open-sources the code and provides benchmarks for different FL strategies, including FedAvg, FedProx, SCAFFOLD, FedAdam, and Cyclic learning.
strengths: - The paper provides 7 different datasets in the cross-silo FL settings, across different training tasks from binary/multi-class classification, 3D segmentation and survival. 
- The papers also provide quite extensive FL benchmarks across different strategies, such as FedAvg, FedProx, SCAFFOLD, FedAdam and Cyclic learning, with details experimental setups and hyper-parameters.
- The paper provides details for each datasets with statistics and heterogeneity studies. 
weaknesses: - The paper mentioned that the cross-silo situation is normally for small client number, when client numbers are from 2 to 50, with each clients having medium to large data size. However, the maximum number of clients among these 7 datasets are 6, which is nowhere near 50. As for the size of dataset, the Fed-KITS2019 only has 96 data samples in total, with some clients only has 12 sample. 
- As the paper also mentioned, the datasets are limited to healthcare datasets, which can be limited in terms of cross-silo FL applications. As paper mentioned that there are other stand-alone cross-silo FL datasets, which are not healthcare dataset. 
correctness: - On line 286, the paper said E stands for number of local steps. Does it means number of SGD steps or number of local epochs? It also mentioned that you use E=100 local updates for all datasets, which means that number of local updates are fixed regardless of the number of data samples in each clients. Normally, FL reports in terms of number of local epoch and number of communication rounds. If it means number of local epoch, 100 seems quite a lot, given some of clients only have 12 samples. If it means number of SGD steps, it seems not reasonable to use the same number of SGD steps for different number of data samples. 
clarity: The paper is written quite well and well structured.
relation to prior work: It provides the benchmarks for cross-silo FL scenarios, which as the paper stated, are less easy to find in the current literature. There is LEAF benchmarks that are primarily focus on cross-devices FL settings. 
additional feedback: A few points that need to clarify:

- The paper mentioned that apart form Fed-KITS2019, the pooled performance is not attained by FL strategies. However, by looking at Fed-Camelyon16 in Fig 3, it seems both Cyclic Learning and FedAdagrad are better than pooled? 
- (Minor point) Section 3.1, authors mentioned that FLamby can be compatible with existing FL libraries, just checking have authors implement FLamby datasets in the mentioned FL libraries?
review_title: Good cross-silo medical FL datasets

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: - The paper presents 7 datasets with natural partitions for cross-silo FL settings. The presented datasets are from the healthcare domain, spanning multiple predictive tasks.
- The paper provides detailed descriptions and implementations of how to run standard FL algorithms on these datasets. 
- The paper presents some analysis to illustrate the statistical heterogeneity of the datasets.
- The paper provides some initial benchmark results based on the implementation.

-----------------------

Post-rebuttal update: score raised from 6 to 7
strengths: 1. The size of the benchmark (7 datasets) is considerable. The datasets are also naturally partitioned. 
2. The provided source code is relatively easy to follow, well documented, and has a high degree of completeness. 
3. The predictive tasks are reasonably diverse (including classification, 3D segmentation, and a survival ranking problem).
4. The authors used various metrics to illustrate the statistical heterogeneity across clients (section 3.4)
5. Sufficient details (hyperparameters, training setup, code documentation) are included in the main text and the supplementary for reproducibility.
6. Literature review on past FL benchmarks is quite thorough.

Overall, the proposed benchmark/datasets are a valuable contribution to the FL community, though I encourage the authors to consider making it explicit that FLamby as it is right now mainly covers the cross-silo FL in the healthcare domain (see weaknesses).

weaknesses: 1. While the title/abstract/introduction presents FLamby as an extensive benchmark for cross-silo FL, the datasets focus heavily on the healthcare domain (all 7 datasets) and image tasks (5 out of 7). I would suggest making the focus on healthcare more explicit in the title/abstract/introduction, as FLamby as it is right now is not representative of many other cross-silo settings, such as collaborations between banks, schools, and other institutions or data silos. 
2. All datasets have a small number of clients (2 - 6), which may be specific to the healthcare domain. This may not be sufficient for benchmarking existing algorithms whose utility may benefit from a (slightly) larger client count. Similar to weakness #1, for a comprehensive cross-silo benchmark there should be at least one dataset with say 20-100 clients. It might be worth considering hybrid partitioning (i.e. having both artificial and natural partitions) and giving recommendations for practitioners on how to do that for the presented datasets.
3. Lack of personalization algorithms. The nature of cross-silo FL (particularly the properties that the clients are stateful, resource-abundant, and have relatively large local datasets) often necessitates model personalization. This is also suggested in Fig 3 where simple local training seems to be a strong baseline for many datasets. 
4. (Minor) The presented tasks tend to require domain-specific knowledge (for data preprocessing, training, tuning, etc) and might make it slightly harder for practitioners to develop/adapt/tune FL algorithms. Compared to LEAF, the medical computing tasks and baseline models like Cox Model / nnU-Net / V-Net can be unfamiliar to many FL practitioners.

I would be happy to adjust my rating if the authors could address these concerns.
correctness: - The presented datasets are constructed in a sound manner (e.g. authors emphasized having data naturally partitioned, and they performed good analysis on the resulting data heterogeneity).
- The metrics for evaluating the datasets look reasonable.
- The chosen baseline methods are reasonable, though the authors should consider personalization baselines as per weakness #3.
- The experiment setup is also reasonable.

clarity: The paper is generally well-written and easy to follow.

Some minor issues:
- It may be worth simplifying the dataset names; e.g. “Fed-LIDC-IDRI” and “Fed-TCGA-BRCA” is a bit complicated, error-prone, and not very informative. 
- Throughout the paper and the code, it seems that the term “center” or “local” is used to refer to “clients” in FL. For consistency please consider using the term “client” throughout.
- Fig 1(a) is not very easy to interpret; is the x-axis the pixel values? The goal seems to be to illustrate the heterogeneity across the two clients, but color histograms may not be the most informative figure, especially when multiple colors overlap. One possible alternative for better visualization is to juxtapose some form of “mean image” from each of the silos. If the authors insist on keeping Fig 1(a) please consider separating the colors.
- For Fig 1, consider changing “Local” in legends to “Client” for consistency and to avoid confusion.

relation to prior work: The literature review is fairly comprehensive and the paper discussed key distinctions from previous work.
additional feedback: - If the goal of the paper is to provide a comprehensive benchmark and evaluation, I think it is also acceptable (and encouraged) to incorporate existing (non-medical) cross-silo datasets into the benchmark, as standardizing the evaluation methods and providing recommendations/code would by itself be considered a good contribution. Otherwise, as per weakness #1, I would encourage the authors to make clear that FLamby is focused on the healthcare domain.
- L175: is there supposed to be a footnote (3)?
- L275: is the average client performance a weighted or a uniform average?
- L284 (Federated setup): if I understand correctly, the setup here (eq 1) basically says the number of FL rounds is determined by the time to train the “pooled” dataset, but (1) this might not be a good strategy since in practice the convergence of training on central, “pooled” dataset is often not the same as the convergence of FL training due to heterogeneity and (2) this heuristic might be a bit overcomplicated and one can simply train till convergence by setting a large enough $T_\text{max}$.
review_title: Good benchmark for cross-silo FL with a focus on medical computing tasks and some minor weaknesses

Meta-review: metareview: This paper is a great fit for this specific track of NeurIPS. As underlined by the reviewers, it will greatly benefit the federated learning for healthcare community as it provides an holistic starting point containing curated datasets with natural partitioning and baselines. Those elements currently are unfortunately missing for this community, which will be addressed with this article! 
confidence: 5: The area chair is absolutely certain
Decision: Accept