Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper asks the question of whether the Q-Fair Federated Learning (Q-FFL) algorithm can help or obviate personalization. This federated learning algorithm attempts to instill fairness by weighting losses more heavily for clients with large losses, controlled via an exponential weighting parameter q. By equalizing the loss for under-performing clients, the authors hypothesize that Q-FFL may reduce the need to use personalization to boost individual client performance. However, they find through an offline study that Q-FFL does not reduce and can even increase the number of underperforming clients. They instead propose to use distillation to train local models that still benefit from global learnings. Preliminary results show this helps reduce the number of underperforming clients, indicating this as a promising new direction.
strength_and_weaknesses: Strengths:
1. Presents detailed investigation of whether a fairness algorithm can also reduce the need for personalization, yielding the surprising finding that it does not reduce disparities between clients overall.
2. The background information is well written and provides a clear overview of fairness and federate learning.
3. Authors present results from a new algorithm based on distillation that can reduce the number of under-performing clients.

Weaknesses:
1. The main result is a negative result -- that Q-FFL did not reduce the need for personalization. This is unsurprising since Q-FFL is not a personalization algorithm. Since the authors present an alternative algorithm that does use fairness to reduce the need for personalization, it would be better if they led with this and used Q-FFL as a baseline.
2. Along the same lines, the results with their new approach seem too preliminary and under-developed. It is likely the authors will need to follow-up with another publication to present this approach in more detail. I strongly suggest to the authors to develop this work further and make it the main subject of their publication.
3. Color scheme in figure 3 makes it hard to distinguish red and orange dots, especially on such a small figure. Maybe use black for one of these symbols.
clarity,_quality,_novelty_and_reproducibility: Clarity - A well-written paper with useful background section.
Quality - Analyses appear competent and thorough.
Novelty - Novelty is hurt by the fact that the bulk of the work uses a previously published algorithm, just evaluated for a different purpose (suitability for personalization). The authors also present a novel and promising algorithm, but the work is preliminary.
Reproducibility - The authors do not open source their work, but the algorithm and datasets are open-source. The new approach is not open-source but may not be too difficult for a read to re-implement.
summary_of_the_review: I am marginally uninclined to accept this paper. I believe there is a very promising piece of work here. The authors present a strong and clear background for the problem of fairness in federate search and explore an interesting question about whether fairness can reduce the need for personalization. This is an important topic, and the authors have made progress by showing one existing algorithm q-FFL does not meet this criteria, and they propose an alternative approach that performs better. However, the work is quite preliminary, meaning the authors focus most of the paper on the under-performing baseline. This paper would benefit by giving the authors more time to flesh out their new approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors talk about the idea of utilizing fairness in reducing the need for personalization. The authors have shown experimentally that the fairness doesn't always help in providing a better starting point for personalization. The authors have proposed a knowledge distillation during FL training to improve the local accuracy. Experimental evidences are provided. 
strength_and_weaknesses: *Strength*
- The idea is interesting and empirical resutls are provided. 

*Weakness*

- The paper is not well organized and difficult to follow. 

- The contributions are not clear and properly placed in comparison to related literature. 

- What is G in equation (2) ? 

- What is he proposed/new part in the paper is not clearly mentioned. 

- There is no theoretical justification. 

- It looks like the main contribution is in Sec 4.3 but that is also not clearly discussed. The paper is not publishable in its current form. 
clarity,_quality,_novelty_and_reproducibility: The paper is not clearly written, and novelty is limited. 
summary_of_the_review: Overall, the idea is interesting but it is not executed well. The paper requires major revision before publications. The main idea is to show that fairness it not suitable for personalization and authors have proposed to change the loss function with knowledge distillation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper evaluates Q-Fair Federated Learning (Q-FFL) in personalized federated learning with its underperforming clients and proposes using knowledge distillation during FL training. And experiments with different datasets show a 50% reduction in underperforming clients in the language task with no increase for the image task. 
strength_and_weaknesses: Strength:
1. There is a good effort to evaluate Q-Fair Federated Learning (Q-FFL) with comprehensive and sufficient analysis. And all of the formulas in the paper are derived from others' papers, rather than serving the methods proposed in the paper.

Weakness:
1. Use too many words to describe relevant knowledge, but not enough to the author's ideas.
2. The whole ‘Method’ part seems to be the details of the experiment and the analysis of the results. I think they should be put in the experimental position rather than the method.
3. The baseline performance occupies too much space and compresses the space for other experiments.
4. Suggest the author apply the knowledge distillation to other federated learning methods to illustrate the universality.
5. Inadequate citation of papers.

clarity,_quality,_novelty_and_reproducibility: Limited clarity, Limited quality, Limited novelty, Highly reproducibility.
summary_of_the_review: Although it is a good effort to evaluate Q-Fair Federated Learning (Q-FFL) with comprehensive and sufficient analysis. But this paper needs to use more words to describe the author’s ideas, not the other’s.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the performance fairness problem of federated learning. The authors empirically show that Q-Fair Federated Learning fails to improve performance fairness. They also show that knowledge distillation is a better way for fair federated learning without additional personalization mechanisms.
strength_and_weaknesses: Strengths:
- The problem studied in this paper is interesting.
- The analysis in this paper can provide some reference for future fair federated learning studies.

Weaknesses:
- The scope of this paper is quite narrow and the findings are not significant. The study is limited to Q-FFL and only one baseline method is used. Thus, the conclusion drawn from the results may not be universally valid. In addition, the advantage of knowledge distillation over other types of techniques is not verified.
- Using knowledge distillation to teach local models is a quite common strategy (e.g., [1]), thereby the technical contribution of this paper is very limited. 
- The coverage of related work is quite limited. The authors should present a more comprehensive review of fair federated learning and personalized federated learning.
- The experimental setting is unclear. It would be a hard task to reproduce the results without sufficient details.

[1] Yu, T., Bagdasaryan, E., & Shmatikov, V. (2020). Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758.
clarity,_quality,_novelty_and_reproducibility: The quality of this work is below the bar. The originality is very limited. The reproducibility is not so satisfactory.
summary_of_the_review: This paper has limited contribution, flawed evaluation, and insufficient literature review. Thus, my recommendation is rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The bulk of the contribution lies in verifying that the Q-Fair Federated Learning (Q-FFL) algorithm does not always help in providing a better starting point for personalization. The authors then follow up with knowledge distillation during FL training to improve the local accuracy. 

While the reviewers have found this work interesting, they also voiced (A) several similar major concerns about this work and provided useful recommendations to this work. In particular, the reviewers think that a greater focus should be placed on their proposed work, the claim/findings should be investigated in a more general context, the qualitative and quantitative comparison with related work should be improved, a theoretical analysis would be useful, among other concerns.

The authors have made a major revision to their manuscript in response to the reviewers' feedback. However, we all agree that the revised version will need another thorough, careful review and hence, it is more appropriate to be resubmitted to a future ML-related venue.
justification_for_why_not_higher_score: See major concerns described in (A) above.
justification_for_why_not_lower_score: N/A.
Decision: Reject