Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a federated learning algorithm which is fair and gives consistent performance across different clients in the following sense.  To achieve fairness, some measure of disparity is considered and enforced as a constraint for each client (i.e. each client has an upper bound on how high the disparity measure can go).  To achieve consistency, the objective is to find model parameters that minimize the maximum loss incurred across the clients.  This contrasts with prior work which does not directly address the fairness locally at each client.  Compared to [21], this paper adds constraints on a disparity measure for each client, in addition to the min-max objective.  Compared to [6], this paper considers a fairness constraint at each client as opposed to a single, global fairness constraint.

The proposed gradient descent-based algorithm first aims to satisfy the fairness constraints, and then it aims to find a Pareto optimal solution (w.r.t the clients’ utilities) while continuing to satisfy the fairness constraints.  The proposed algorithm replaces the (non-smooth) max’s in the problem formulation with a smooth surrogate, which is basically the log-sum-exp trick.

The proposed method is evaluated on three datasets with respect to prediction accuracy as the loss/utility metric and two different fairness metrics: demographic parity and equal opportunity.  The results show that the proposed method typically performs significantly better with respect to the fairness measure, with a small loss in accuracy compared to other methods which do not achieve the same level of fairness.

main_review: This paper considers two currently relevant topics - fairness in ML and federated learning.  The main contribution here is the problem formulation and the experimental results, which are interesting.  I would not consider the algorithm design (including the tricks such as replacing max with log-sum-exp) very novel.  Below I outline some issues I have with the current presentation.  Overall, I would be okay with accepting this paper.

 - The authors claim that their method converges in theory, but no claim is made as to the convergence rate.
 - The experiments seem to focus on reporting the accuracy and fairness performance.  The proposed method involves solving an LP at each iteration (for both stages of the algorithm).  What is the impact on the runtime of solving these?  How does the overall runtime of the proposed method compare to the baselines considered?  Are the improvements to fairness/accuracy coming at a significant computational cost compared to other methods?
 - The number of clients considered in the experiments seems rather small.  How does the proposed method perform in situations where there are significantly more clients?

Minor comments:
 - Line 33 - “in FL scenario” -> “in a FL scenario”
 - Line 42 - “There have been researches” -> “There has been prior work”
 - Line 170 - “in deep model” -> “in a deep model”


limitations_and_societal_impact: Some discussion about the choice of fairness metric may be useful (this paper looks at demographic parity and equal opportunity).  I do not see a major potential for negative societal impact directly stemming from this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper first introduces a new task about federated learning in which fairness and consistency are jointly considered, then proposes a solution named FCFL to solve the problem. After introducing fairness, the problem is transferred to a min-max optimization problem, and the two-stage optimization algorithm works well on it. The main contributions are as follows:

1) Introduce a new task which jointly considers model fairness and consistency in federated learning,
2) Propose an optimization method called FCFL to address the problem,
3) Conduct experiments on a synthetic dataset and two public datasets and compare the proposed algorithm with three baseline methods, and show that the proposed method outperforms the rest.
main_review: Both method and task in this paper are new, and the proposed method is a good utilize of several existing approaches. Related works are cited in a proper way. The idea of the task is interesting, as few work has ever considered fairness in a federated learning setting and introducing it makes the whole problem a min-max optimization. Most of the paper is well-written, a few details can be further improved (e.g. introduce full name of FCFL when it first appears).The paper is technically sound, both theoretical analysis and experimental results are sufficient. However, I believe more models in the area of federated learning should be taken into consideration. Some recent works are focusing on the inconsistency caused by non-iid in federated learning and proposed good solutions (e.g.  FedNova). Weakness of the work should also be discussed, especially the possible usage of this framework. The results are reproducible, with some help of the authors. The experimental results are nice, researchers should be interested to further build on them. The datasets are existed. Since this is a brand new task, no previous methods can be used to compare with, but the results still show that performance of the framework is generally good.
limitations_and_societal_impact: The authors has discussed the limitations in the appendix. However, potential negative social impact is not discussed. Moreover, more limitations should be considered, for example, from the aspect of how important fairness is in federated learning, as it has never be mentioned before.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies how to solve the model fairness and performance consistency among clients in federated learning. There exist work on model fairness and performance inconsistency separately as mentioned in the paper and the contribution of this work is to provide a FL framework which considers both effects. The problem is formulated as a constrained min-max pareto optimization problem. The authors use a two-stage optimization method to approximate the optimal solution. 
main_review: The paper brings local model fairness into federated learning that allows clients to set its own “fairness budget”. The model fairness is an important problem in recent machine learning communities to avoid the model being biased by the sensitive attributes. The problem is quite interesting and practical. It is not evident to see how to combine fairness with performance consistency in federated learning. The authors gave a solution easy to follow and performs well in logistic regression problems.  The paper is well organized and the ideas are easy to follow although it makes use of the existing techniques.    

I have the following comments:
1. The main algorithm 1 in Appendix for FCFL is presented in a centralized way. It would be better to present it from both the client and the server perspectives.  It is not clear from the main paper how to parallelize the calculations of FCFL.
2. The experiments part is a little bit weak as only logistic regression problems are considered. It would be better to show the neural network performance. 
3. It seems the definition of $\triangle DP_k$ in Eq. 2 is for binary classification. Just for curiosity, what does the  $\triangle DP_k$ look like for multiple classification problems? 

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a framework for finding solutions that achieve both demographic parity and performance consistency across local clients in FL. They propose first achieving min-max performance with multi-client fairness and then finding a solution that is Pareto optimal and demonstrate their method on experiments with both synthetic and real-world data. 
main_review: Proposing individual fairness criteria for each client is novel (as far as I am aware) and can be very important in many domains. 

The proposed multi-client fairness definition can be better explained and justified. How are the $\epsilon$ related to one another? When should they be the same? What kind of disparity is introduced when $\epsilon_k$ 's are different across different k’s? Can small $\epsilon_k$ across a large k lead to large global disparity? 


Does there always exist a Pareto-optimal Min-Max Performance model that satisfies MCF? It seems like the set of $\mathcal{H}^{FP}$ is defined independently from the $\mathcal{H}^{FU}$? How do we know $H^*$ the intersection is non-empty? 

Not sure what line 193-194: “minimizing the current maximum value of all objectives directly in Eq(7) can cause another worse objective and can be computational hard when faced with a tremendous amount of clients” means. Is this saying runtime is exponential in k? 

This submission could benefit from grammar fixes but the grammar issues did not prevent me from understanding the paper. 

limitations_and_societal_impact: The authors have not addressed limitations and potential negative impacts. The authors should consider what types of additional disparities or unintentional concequences their local definition of fairness may introduce.
review_title: null

Meta-review: This paper studies the notion of fairness concerns in Federated Learning settings where a vanilla global model may be particularly bad on some clients versus others. Their solve this issue by maximizin the performance of the model on the worst client, leading to minimax style of fairness gurantees. The authors complement this analysis with an empirical evaluation of their method. 
Decision: Accept (Poster)