Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: pFL-bench is a comprehensive benchmark suite for personalized FL (pFL). It contains more than 10 datasets in diverse application domains and more than 20 pFL baseline implementations. The paper also demonstrates the utility of pFL-bench by conducting an extensive experiment. 
strengths: 1. Numerous relevant datasets and use cases
2. Extensive evaluation
3. Good documentation
weaknesses: - The datasets are relatively small in size (i.e. the number of clients.) and do not have a natural split. 
correctness: The experiments are sound and the evaluation methods are designed appropriately and performed correctly.
clarity: The paper is well-written. 
relation to prior work: The paper discusses the difference from previous work and is fairly comprehensive. 
additional feedback: Please refer to the section above.
review_title: A comprehensive benchmark suite for Personalized FL

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This is a benchmark paper studies personalization in the federated learning (FL) setting. In FL, users/clients jointly train a model without sharing their personal local data. In personalized FL (pFL), each client can learn their own model in addition to the global model. This paper provides empirical study on ~10 datasets (sampled EMNIST, 3 CIFAR-10, COLA, SST-2, Cora, Pubmed, Citeseer, MovieLens1M and MovieLens10M), and (including the combination) multiple baseline methods (FedAvg, FineTune, pFedMe, FedBN, Ditto, FedEM). 


======================== After discussion
I still appreciate the workload of this draft. However, I also have concerns about the choices of datasets, and presentation/discussion of FL algorithms, see https://openreview.net/forum?id=2ptbv_JjYKA&noteId=ICd1jefTRoW for details. 

strengths: I am impressed by the general workload of performing experiments on multiple datasets and baseline methods, while also having some concerns on the settings and insights from the benchmark. 
weaknesses: The decisions on some of the datasets seem to be ad-hoc: why sample 5% of EMNIST clients, why use \alpha={5,0.5,0.1} for CIFAR-10 and split it to 100 clients? 
A related issue, I would strongly recommend the authors follow [Wang 2021 A Field Guide to Federated Optimization section 3.1] to discuss the practical setting, e.g., cross-device vs. cross-silo. The number of clients in the datasets in table 1 are relatively small; the authors mentioned client sampling, but I am not sure if it is always enforced, or only for EMNIST. 

The insights of combining various methods are not quite clear. I appreciate the extensive experiments, but table 2 and table 3 are hard to interpret: Some methods are not personalization specific, such as FedOPT, if it consistently improves the performance, maybe it should be used by default. It is also not clear which version of FedOPT is used, is it FedADAM? Some methods like Ditto and FedBN assumes each client maintain their local states, it is not clear to me how they can be applied to unseen clients. 


Clarification questions:
For \bar{Acc}, is there a training/validation partition on each client? What is the percentage of data for training?
The repository https://github.com/alibaba/FederatedScope/tree/master/benchmark/pFL-Bench seems to only provide scripts for EMNIST. 

Minor:
I am not sure how useful Eq (1) is. The presentation looks problematic to me. 
It seems to be a variant of the population risk where the objective of the global model is decomposed to ERM for training G and validation R. The relationship between population risk and ERM is not entirely new, for example, see Eq (1) & (2) in  [Wang 2021 A Field Guide to Federated Optimization]. It is confusing that personalization only happens for C, not unseen clients \bar{C}. 

correctness: Good
clarity: Good.
relation to prior work: Yes. 
additional feedback: NA.
review_title: Impressive workload, insights unclear, clarification questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper is introducing a benchmark suite for personalised Federated Learning, with a focus on cross-device settings. It provides 9 pre-split datasets and implements 11 personalised FL baselines methods for these tasks. Implementation is based on FederatedScope.
strengths: * Personalised FL is a promising research direction of distributed training in the wild that has received attention in the last years. Having a benchmark suite that acts as the common denominator and the golden standard between different techniques can be beneficial for the community.
* The authors have gone the extra mile and implemented various techniques from the literature that can act as initial baselines in the area of personalised FL. Moreover, certain techniques can be combined for further gains. It's also positive that they have integrated datasets representing different modalities.
* I like the brief theoretical background that the authors provide about how personalisation in FL is being pursued in the literature.
weaknesses: * The paper felt at times more like a survey of personalised FL techniques rather than a benchmarking suite for personalised FL.
* Datasets introduced here are simply gathered from pre-existing literature, and merely federated. This has been done various times before and especially for the selected vision tasks.
    * Some widely used FL datasets are missing, from the NLP (e.g. Reddit, StackOverflow), Speech (e.g. SpeechCommands) or other modalities (e.g. HAR)
* There is lack of support/integration of techniques from the system heterogeneous FL and privacy realms. The tradeoff dynamics between personalisation, privacy and fairness are the most interesting and underexplored in the field.
* The suite is offering implementations based on FederatedScope, which might not be easily "intergratable" with other FL frameworks (e.g. TFF, Flower, FLSIm, etc.)
* The authors argue a lot about the reproducibility of their implementation. However, in cross-device settings, stochasticity may come from client availability (or lack thereof), failures or even asynchronicity of aggregation. It is not clear how PFL-Bench would support that level of determinism/reproducibility.
correctness: The computational/memory/network costs associated with training have been very coarsely presented and lack definitions and runtime footprints in the main text. The authors are measuring certain proxy performance metrics instead of runtime ones (e.g. FLOPs vs. training latency) and are missing other important ones (e.g. peak memory usage) that would be important from a deployability point of view and especially in the cross-device setting.

Some of the datasets are counted as different while the only thing that changes is the LDA alpha parameter. I would count them uniquely.

The authors start motivating their work by supporting that different approaches in personalised FL are evaluated in different settings and are not directly comparable with one another. However, they reach the conclusion in L229 that there is no dominating technique across the implemented ones (see free lunch theorem).
Given that new work tends to emphasise on the positive results, even under the presence of such a benchmarking suite, the trend of presenting favourable results/setups where a new approach shines  may not be avoided. Thus common baselines/setups may be trickier than offering a benchmarking suite.

Wrt evaluation:
* Why does FEMNIST in the "Global-Train" method perform so poorly?
* Bottom accuracy being zero in table 4 means that it is essentially worse than random.
clarity: The paper is generally well-written and easy to follow. However, there are certain presentation issues, laid out below:

* Maybe a more explicit distinction between cross-silo and cross-device settings should be made in the paper and what datasets/techniques correspond to each realm.
* Figure 4: It is very difficult to make out different lines per graph. Maybe the authors should consider an alternative presentation method for this dense information.
* Tables 6-19 should not be inline with the references in the appendix.
* In section 5.3 (some are only made clearer in the relevant tables of the appendix):
    * Are FLOPs counted as the amount of floating operations in training or inference (f/w propagation only)?
    * It is not clear in what granularity are the communication bytes reported. Is it upstream+downstream across users and communication rounds until convegence (with early stopping)?
    * Similarly for the communication rounds, the setting is a bit counter-intuitive. Typically, one either fixes the number of communication rounds or they count them until a certain accuracy threshold is reached or they perform early stopping. It is unclear how the communication rounds were selected per technique here.
* The metrics for fairness could be presented more like a set of commonly accepted benchmark metrics for assessing fairness in FL. Also, fairness in participation could be quantified/included.

In terms of wording, the terms "practicability", "systematical", "un-participated" and "FL course" should be reconsidered.
relation to prior work: I see the proposed approach as an attempt to standardise experiments/datasets through a common, already bootstrapped benchmarking suite. However, there have been several FL frameworks with partial support for datasets/personalisation approaches. Maybe a table representing the coverage of techniques/datasets in major FL frameworks would be of great utility to the community.
additional feedback: Further support:
* The suite seems to be running everything in a simulated manner. Are there any plans to run across remote workers or on-device? This is particularly important for the case of cross-device setups.
* Will the benchmark support pretrained global models for the users to bootstrap their research?
* It is further unclear if the frameworks supports personalisation per cohort instead of per client.

Limitations of work have very briefly mentioned if at all. I would urge the authors to present limitations of their system more spherically.
review_title: Review for pFL-Bench

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors have clearly invested a lot of time and effort in this very thorough benchmark.
It is the first to focus on personalised federated learning.
This is new although the reviewer doubts the pertinence of having those methods evaluated separately from non-personalised federated learning methods.
The article is pleasant and clear to read. The supplementary material gives useful details about the datasets, methods, results and the implementation.
The reviewer was not able to run any experiment or look at any code as the choice of a docker image makes it cumbersome.
Overall, this is a very comprehensive set of datasets and models that can foster FL innovation and fits nicely with the purpose of the NeurIPS Datasets and Benchmarks track.
strengths: - a comprehensive list of datasets and methods for personalised FL
- a clear description of the methods used to generate split datasets
- a concise and well referenced explanation of the benchmarked methods
- a large and interesting collection of experimental results

weaknesses: - It would have been have been nice to see more non-personalised FL methods benchmarked.
- The reviewer would have enjoyed more metrics and figures about the heterogeneity of the split datasets (looking at both inputs and labels) that is a key variable for the topic of this article.
- The github repo should be more user-friendly and the code easier to install and run.
- Some experimental results do look surprising.

correctness: The experimental design is sound and rigorous and the evaluation metrics are relevant and informative.
clarity: The paper is clear and pleasant to read. Between the main article and the supplementary material, the level of detail given is appropriate for this track.
relation to prior work: The related works section is well referenced and gives good context.
additional feedback: NA
review_title: An exhaustive bunch of datasets and models for personalised federated learning with a lot of experimental results

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper benchmarks "standard" federated learning algorithms and some recent personalized federated learning (pFL) algorithms on 11 federated learning datasets, spanning a wide range of machine learning tasks including image classification, natural language processing, graph classification, and recommender systems. The comparison includes different criteria to measure average accuracy, fairness, and energetic efficiency. 

The experimental results demonstrate on one hand that personalization is a promising solution to handle statistical heterogeneity in federated learning, and on the other hand that no method can consistently outperform others across all metrics and datasets.
strengths: The paper is well motivated and helps to overcome the absence of a standard benchmark for personalized federated learning. The paper proposes 11 datasets with different tasks, sizes, and levels of heterogeneity, and conducts extensive experiments to evaluate a reasonable amount of "standard" and personalized FL algorithms in terms of test accuracy, fairness and efficiency. To the best of my knowledge, this work is the most complete benchmark in the field of personalized federated learning. 

The authors made a significant effort to include all the aspects of personalized federated learning in their comparison. They provide weighted average accuracy on seen and unseen clients, as long as the equally-weighted accuracy, the bottom decile accuracy and the standard deviation of the average accuracy. The authors also provide results to quantify the effect of heterogeneity on different methods, and give an estimation of the computation, memory and communication cost of the of the algorithms they study.

The authors also propose a clear plan to maintain the benchmark and to extend to include additional aspect of personalized federated learning. 

weaknesses: Despite the effort of the authors and the overall relevance of the paper, I think it still need some improvement. 

First, the generalized FL formulation discussed in Section 3.1 is not totally accurate; for example, the formulation in Eq. (1) does not capture Ditto [35]. The true objective of personalized FL methods is to solve for each client $i$ the following optimization problem
$$\min_{\theta_{i}} {\mathbb{E}}_{\mathcal{D}_i}[f(\theta_i; x, y)],$$
corresponding to the selection of the parameters that minimize the true risk of client $i$. Of course client $i$ can not minimize its true risk, because he does not have access to the local data distribution $D_i$, instead client $i$ only has access to a finite, usually small, number of samples drawn from $D_i$. Most papers, considers the minimization of an empirical risk computed using the aggregated dataset and the local dataset of a client, with some reguralization on the distance between the global and local models, see for example Eq. (1) in (Hanzely et al., 2021). I highly recommend the authors to update Section 3.1, good references include (Mansour et al, 2020) and (Mohri et al, 2019). 

Second, some experimental results are unexpected, I do not understand why FedAvg outperforms (by a large margin) Global-Train on FEMNIST for example. I think that the implementation/execution of Global-Train need to be double checked.

Third, the benchmark is based on FederatedScope, which creates an entry barrier to the usage of the benchmark as it requires first learning how to use FederatedScope. The benchmark will be used and adopted in more papers, if the code was provided in native PyTorch or TensorFlow because they are a lot more popular. However, I understand the choice of the authors to use FederatedScope.

correctness: The paper is overall correct. I have few comments on some claims that I included in Section Weaknesses.
clarity: The paper is clear and easy to follow. The writing quality is acceptable but need to be improved, and the paper has some typos. 

Line 93: that are perform well -> the perform well

Line 109: that do not participate the FL training -> that do not participate to the FL training

Line 118: that are not contribute -> that do not contribute
relation to prior work: The paper has a decent  related work discussion. 
additional feedback: I want to thank the authors for their efforts. This paper is valuable for the personalized FL community and I recommend acceptance. I strongly recommend the authors to updated Section 3.1 and to double check the implementation of Global-Train.

I encourage the authors to include other methods/datasets in the comparison, and suggest to add the results of this study to [paperswithcode](https://paperswithcode.com/task/personalized-federated-learning). 

## References 

Hanzely, Filip, Boxin Zhao, and Mladen Kolar. "Personalized federated learning: A unified framework and universal optimization techniques." arXiv preprint arXiv:2102.09743 (2021).

Mohri, Mehryar, Gary Sivek, and Ananda Theertha Suresh. "Agnostic federated learning." In International Conference on Machine Learning, pp. 4615-4625. PMLR, 2019.
review_title: Good paper, sigificant contribution to the field of personalized federated learning 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, they propose a new benchmark system pFL-Bench for personalized federated learning. pFL-Bench contains 11 datasets and 20 pFL baselines. They make the following claims about the existing FL benchmark systems

1. They mostly benchmarked on general FL algorithms

2. None of them support the new clients generalization evaluation

3. Few existing benchmarks simultaneously support comprehensive evaluation for trade-offs among accuracy, fairness, and systematical costs

This paper fixes those above by making the following contributions
1. A modular and easy-to-extend pFL codebase with more than 20 competitive pFL baseline implementations

2. Systematic evaluations under containerized environments in terms of generalization, fairness, system overhead, and convergence.

strengths: Propose a new evaluation protocol and conduct experiments in great detail. Report performance of different pFL algorithms in the aspect of generalization, fairness, and system efficiency.
weaknesses: 1. Lack of comparison with the existing benchmark system. How hard is it for other frameworks (Flower, FedML, FedScale) to adapt this new evaluation protocol? Do they provide APIs to do so? Is there any key system design difference that makes pFL not be supported by other frameworks?

2. Lack of analysis and comparison of existing pFL evaluation protocols. What are these protocols and what new aspects does your new unified evaluation protocol bring to us. Unlike the general FL optimization, pFL optimization can be diverse, and so are the evaluation metrics. I didn’t see discussions on different evaluation metrics on pFL. Can your evaluation protocol cover all these parts? If not, why not provide flexible evaluation APIs to support a comprehensive evaluation. 

3. Lack of insights and takeaways from evaluation results. I did not see an in-depth analysis of the experiments. Which pFL algorithm is better at different aspects under different scenarios. Which aspects are largely ignored or misunderstood by pFL evaluations before and what are the insights for future pFL based on your results.

4. Other aspects: 1. Evaluations on system efficiency are not convincing, since the clients' heterogeneous system dynamics are not considered currently. 2 I notice the dataset and model is relatively small scale compared to other FL benchmarks. 3. Communication bytes and convergence round is insufficient to picture the system cost in practical FL. These can’t tell much without the device characteristic and network condition. 




correctness: The results seem largely correct. The choice of evaluation methods is appropriate, which is a strength of the paper. However, I expect to see more discussion on the benchmark results.
clarity: The paper is well written.
relation to prior work: As I discussed in the weakness part.
additional feedback: Please refer to the sections above.
review_title: A benchmark system for personalized federated learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduced a benchmark on personalized federated learning with an unified experiment setup on existing commonly used datasets. Specifically, this paper took 11 datasets covering image, text, graph and recommendation domains, and benchmarked 4 different personalized setups. The benchmark included evaluation on generalization on trained/unseen participants, fairness between participants, as well as training efficiency for different personalization methods. 

strengths: 1. This paper introduced the first unified experimental setup for personalized federated learning. This is critical since existing papers might use different settings which makes it hard to compare the results fairly with each other.
2. The number of datasets and number of methods used in the benchmark are comprehensive, and evaluation criterions could provide a more holistic view of each method.

weaknesses: The dataset and evaluation setup seems to be missing the following description:
1. For setup such as fine-tuning, there needs to be a train/test split for local data within each client, how is this splitting done?
2. Are the train/val/test partitioned by clients, or partitioned within each client, or partitioned randomly as if the dataset is centralized?
3. More in-depth explanation for the accuracy definition in table 2 and 3. For example, what is bottom individual accuracy? I could not find a clear definition in the paper.

For text domain data, it would be nice if there are results on the datasets with a larger number of clients such as StackOverflow or Reddit Comments to understand the personalized approach on larger datasets.

correctness: I believe the methodology used for benchmarking personalized federated learning is correct but more details about the dataset split need to be clarified.

clarity: This paper is well-written and organized for the most part. I have raised what else could be clarified in the weakness section.

relation to prior work: This paper clearly stated its difference to prior work.

additional feedback: Questions:
1. Why subsample 5% of FEMNIST in Table 1?
2. What does s = 2 refer to in Table 2 and 3 for FEMNIST?
3. For SST numbers in Table 2, why does the accuracy of un-participated clients can sometimes be higher than that of participating clients?
review_title: A comprehensive benchmark on personalized federated learning

Meta-review: metareview: This work highly benefited from the rebuttal period, and, following the reviewers comment, I now recommend it to be accepted. It offers a nice overview of personalized FL. After checking, I would however concur with the reviewers in asking the authors to be particularly careful with the given code that could benefit from an extra documentation and a maintenance in time  — adding more flexibility with other FL frameworks. 
confidence: 5: The area chair is absolutely certain
Decision: Accept