Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discusses the use of bounded group loss (BGL) in the context of fair federated learning. The core objective is to learn a classifier that satisfies some predefined constraints on the expected loss function of each group on average across all clients. It is expected that the client-conditional data distribution may differ between clients.

The paper further assumes that the bounded loss function is convex w.r.t. the model parameters, and proposes a distributed saddle point optimization algorithm to recover a model that minimizes empirical risk subject to these group constraints. A convergence proof for the algorithm is also provided

strength_and_weaknesses: Strengths:

The paper allows the direct specification of target group-specific loss minimums, which are a natural way of specifying group constraints. The resulting algorithm is simple and well grounded, and the resulting fairness-utility tradeoffs seem comparable to existing approaches.

Weaknesses:

I think the novelty of BGL is limited, the proposed objective is equivalent to FedMinMax, which is discussed on the paper,  with an additional static bias added to the group-conditional loss terms. The formulation as a saddle point presented in this paper is not fully compatible with the bounds on the Lagrangian weights presented in Eq 2, and Algorithm 1 makes it seem like the weights asymptotically reach the norm bound B, which begs the question on why is that a separate optimization parameter from \beta in Eq 2.

The theoretical comparisons with existing methods, in particular group DRO, which uses a similar exponential weight update rule for lambda and thresholds on the group losses (albeit with an additional relu), and FedMinMax, which in its paper formulation has a lower bound constraint on lambda  which makes both objectives theoretically equivalent, is sorely lacking.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easily reproducible
summary_of_the_review: The proposed method is sensible, but there is limited novelty, and uncertainty if the method is not equivalent to existing approaches as it stands.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a group federated learning algorithm with formal convergence/BGL guarantees.
strength_and_weaknesses: -Privacy leakage/secrecy of the proposed algorithm is not discussed. It is unclear how much privacy leakage occurs. The proposed algorithm requires the exchange of additional information "r" per round, so one *must* discuss the implication of this. Federated learning algorithms with frequent synchronization/additional data exchange may be subject to an unacceptable level of information leakage, making it no different from sharing all the local datasets with every client. To avoid this kind of pitfall, one must clearly discuss the privacy aspect of new algorithms.

-BGL seems to be ok if it's used for regression or used for classification without any preferred labels. ACS Emp./COMPAS are the tasks with preferred outputs, so I am not sure if using BGL makes sense in these settings.

-Some of the experimental results look weird. In most of the experiments, FedAvg, which should minimize the error rate better than any other constrained counterparts, does not seem to achieve the lowest error rate. Sometimes it performs the worst among all tested methods. Why?

-How are the hyperparameters chosen for the baseline methods?

-Some missing baselines. Also, please add the performance of the standard reduction method with central data (upper bound). 

[1] FairFed: Enabling Group Fairness in Federated Learning https://arxiv.org/abs/2110.00857
[2] GIFAIR-FL: A Framework for Group and Individual Fairness in Federated Learning https://arxiv.org/abs/2108.02741
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written.
summary_of_the_review: See my comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a fair learning objective for federated settings via Bounded Group Loss. The authors propose a scalable federated solver to find an approximate saddle point for the objective. Theoretically, they provide convergence and fairness guarantees for the method. Empirically, they show that their method can provide high accuracy and fairness simultaneously across tasks from fair ML and
federated learning.


strength_and_weaknesses: Strength:
- The topic of fair federated learning is relevant.
- The technical idea of reducing to federated saddle point optimization is natural.

Weaknesses:
- The selection of Bounded Group Loss is not very well motivated. To my understanding, this selection is due to its convexity instead of its practical interest. The advantage of the proposed algorithm in Bounded Group Loss is not surprising since the baselines are not defined for this fairness measure.
- Lack of discussion on techniques. The reduction to federated saddle point optimization is natural, and hence I expect to see a comparison with federated saddle point optimization algorithms in the literature. For example, [Hou et al., Efficient Algorithms for Federated Saddle Point Optimization, 2021] and [Shen et al., FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation, 2021].
clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is ok, but lacks some literature search for federated saddle point optimization.
Quality: The optimization approach is standard.
Novelty: The provable guarantee for fair federated learning is novel.
summary_of_the_review: The paper proposes a fair learning objective for federated settings via Bounded Group Loss. The selection of Bounded Group Loss is not very well motivated. The writing is ok but lacks some literature search for federated saddle point optimization. Overall, I do not recommend acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies fair federated learning (FL) where each group of clients are guaranteed to have the same loss function value upper bound. Theoretical guarantees for both model convergence (for convex loss functions) and fairness are provided, together with experimental results that support the claims.
strength_and_weaknesses: Strength:
+ Given bounded group loss (BGL) as the fairness criterion, this paper develops a FL solution that optimizes empirical risks for different groups under absolute loss constraints. 
+ Theoretical analysis of convergence and fairness guarantee is provided.

Weakness:
- The biggest issue of this work is the fairness criterion that, in my opinion, violates the core principle of fairness. The BGL definition in Def. 1 uses an absolute average loss threshold for every (restricted) group $a \in A$. The authors' argument to move away from the equity between different groups and move towards such absolute, one-size-for-all quality threshold is highly flawed. First, the authors argue that this makes the problem convex, but such consideration is based on whether the authors can technically solve the problem as opposed to what is the right metric for fairness. Second, the argument that this metric "boosts the worst group's prediction quality" is solely focused on one group's performance, as opposed to the fairness **among all groups**. Fundamentally, fairness must consider all stakeholders. Last but probably the most importantly, using a single threshold could essentially lead to one group (the so-called worst group) barely meeting this threshold, while other groups achieve significantly smaller loss values. The authors consider this case to be fair, but in reality, such prediction outcome discrepancy can still lead to unfair decision making. Additionally, choosing such threshold $\zeta$ becomes non-trivial, as it significantly affects BGL. Having the algorithm fairness depend critically on hyperparameter tuning is dangerous and should be avoided when possible. As the whole work is built on such absolute threshold, I believe this is a critical issue that should be addressed.
- As the authors stated in the Remark, $G$ is linear in $\lambda$ and convex in $w_0$. I'm not sure why the optimal solution has to be defined w.r.t. a saddle point instead of the true optimum.
- The novelty of this work is limited given the state of the art. The algorithmic approach described in Sec. 4.1 is a standard alternative optimization procedure. The technique used in the convergence analysis in Sec. 4.2 is very similar to the prior approach, e.g., in Li et al. 2019b. 
clarity,_quality,_novelty_and_reproducibility: - I'm not sure how Fig. 1 is supposed to be interpreted. I do not think it gives a clear motivation of the problem. I suggest the authors to either modify this figure (and caption) to make it more illustrative, or remove it all together. Instead, I suggest the authors to add Alg. 1 to the main paper, as opposed to leaving it entirely in the appendix.
summary_of_the_review: This work is technically sound given the premise of the fairness formulation. However, this premise does not seem to satisfy the core principle of fairness -- treating all groups fairly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contribution of this work lies in using bounded group loss for achieving group fairness in federated learning with provable fairness guarantees.

Such a fairness notion BGL is proposed by Agarwal et al. (2019) and relatively well received, albeit in a context different from FL. To date, there is no one fairness notion that "rules" them all, hence resulting in several fairness notions being proposed. So, at this time, we should remain open to the consideration of various fairness notions, including BGL.

After reviewing the authors' rebuttal and an active discussion, a general concern shared by most reviewers is that it is not immediately clear (beyond the theoretical guarantees and empirical results) why real-world applications need to consider BGL (over other fairness notions) specifically in the context of FL. Furthermore, the authors have mentioned in their rebuttal: "More importantly, compared to baselines that are explicitly designed to optimize the DP / EO gap, our method achieves comparable / even stronger fairness-utility trade offs, even though we did not directly optimize for these criteria." No explanation was given for the observation of this result, begging the question regarding the relationship of BGL to the other evaluated fairness notions in the context of FL. The above seems to have inspired the reviewers to ask various questions regarding the motivation of this work and its relation to the other fairness notions in the context of FL.

The authors are strongly encouraged to revise their work based on the above feedback and the reviewers' suggestions.
justification_for_why_not_higher_score: See the aforementioned general concern, and the unclear motivation of the work and its relation to the other fairness notions in the context of FL.
justification_for_why_not_lower_score: N/A.
Decision: Reject