Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a regularization-based training loss modification that promotes larger margins for minority subgroups in order to improve fairness constraint generalization.
strength_and_weaknesses: Strengths:
1. Margin-based approach makes sense in theory and by intuition.
2. Good comparisons against the well-known reductions-based approaches by Agarwal et al. (2018).
3. Good empirical results.

Weaknesses:
- Would be interesting to have comparisons against more than a single work; namely, a glaring baseline that's missing is the TensorFlow Constrained Optimization package, even more so given that this work focuses on over-parameterized models (NNs).
  - The same authors of the TFCO package also published work specifically focused on generalization of fairness constraints:
> Cotter, Gupta, Jiang, Srebro, Sridharan, Wang, Woodworth and You. "Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraints".
  - These references should at least be discussed in related work, but in truth they should also serve as baselines for empirical results.
- Bad fairness constraint generalization can only mean that performance on one subgroup generalizes better than performance on the other group; it would be interesting to see group-wise loss (or group-wise TPR or FPR), and how these change with difference values for fairness constraint tolerance.
  - Since this difference in generalization between different subgroups likely stems from the imbalanced nature of the dataset, it would be interesting to see comparison against methods from the imbalanced learning literature (not only the single method from fairness literature).

Minor point:
- Most references have double parentheses.
clarity,_quality,_novelty_and_reproducibility: - Paper is well written and clearly presented.
- This work has clear close ties with other papers in the generalization literature, but I believe its application to fairness _with imbalanced data_ is novel enough to warrant a publication.
summary_of_the_review: The paper presents a somewhat novel approach to improving fairness constraint generalization based on enlarging margins of minority subgroups. It is missing some discussion of important related work, and important baselines for fairness constraint generalization as well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In settings where the goal is to learn a classifier that does not exhibit disparate impact, it is common to constrain the learning process via fairness constraints or an adversarial learning procedure. Typically, the disparate impact effects are reduced on the training set, but bias is still observed on the test set. In this paper, the authors propose a new margin-based loss function that is then minimized via exponentiated gradient descent. The intuition behind the loss is that the decision boundary of the final classifier should be further away from small sub-groups in the dataset.
strength_and_weaknesses: **Strengths**\
The paper addresses an important problem and the choice of the loss does indeed make intuition sense. Further, the empirical results indeed suggest that one makes significant improvements using the proposed loss function modifications in this work. 

**Weaknesses**
- Clarification about the importance of group sizes: I am confused about why the bounds in the paper and the key intuition is around group size as opposed to some other measure of group difficulty. One could envision a case where a smaller group is perhaps more homogenous, so it still easy to 'learn' with smaller samples versus a larger group whose inputs have more feature noise. I would've expected a term in these bounds that captures group difficulty. Any insight on this issue?
- Adult dataset: I'd suggest the authors consider the recently introduced adult dataset by Ding et. al. (Retiring Adult dataset) as a replacement for the standard Adult dataset.
- On first reading, it was unclear what parts of section 5 are new to this work, and which ones aren't. For example, I know the ExpGrad formulation is due to Agarwal, but several parts of that section are not well-delineated. For example, is Theorem 5.1 also based on the modification of the equivalent theorem from the Agarwal paper? 


clarity,_quality,_novelty_and_reproducibility: The paper is well-written and mostly free of typos. While certain parts of the work were dense, at a high level the work was easy to follow.

The key insight of this work is mostly original even though significant portions of the work build on the previous Cao et. al paper as well as the formulation of Agarwal et. al.

Overall, the work addresses an important problem with a new margin-based loss function that improves the generalization of the fairness property of a model at test-time.
summary_of_the_review: Overall, the work addresses an important problem with a new margin-based loss function that improves the generalization of the fairness property of a model at test-time.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work studies the generalization of fairness constraints  in Machine Learning systems.  The authors argue that while current algorithms are optimized for generalizing on the whole dataset, their efficacy on minority classes for imbalanced datasets have not been extensively covered They introduce their Flexible Imbalance Fairness Aware (FIFA) classification, combined with logit based losses.  The authors conclude their work by empirically showcasing the performance on relevant datasets for over-parameterized models.
strength_and_weaknesses: Strengths
* The paper is well written, clear to follow.
* It motivates the problem and lists the related work reasonably.It tackles an important problem in the domain, especially with the advent of large  overparameterized models.


Weakness
* The novelty of the technical contributions itself doesn't seem quite significant, specifically the integration with existing classification based approaches.
* The overall empirical evaluation is  reasonably extensive, answering the questions raised in the introduction. However, it would be nice if the authors could elucidate further the  complexity/real world mapping of these datasets to real world challenges we observe.
clarity,_quality,_novelty_and_reproducibility: The work is clear, reasonably easy to follow and seems novel to the best of my knowledge. The reviewer would be able to reproduce these based on the details provided.
summary_of_the_review: Overall, the paper tackles an important problem in the domain, especially with the rapid growth of large overparameterized. The paper is clear, well written and easy to follow. While the empirical significance is reasonably novel, the introduced technical novelty isn't as significant.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new approach to ensure fairness on problems with label imbalance. The idea is to construct a flexible imbalance-fairness-aware (FIFA) loss, which builds on existing label-aware margin losses, to incorporate a margin dependent on the labels but also the sensitive attributes.

All three reviewers had a positive view of the paper, finding it a generally well-written, well-executed work that has clear intuitions and theoretical support, building on margin losses for label imbalance. One potential critique was that the final proposal is a somewhat natural combination of existing ideas (LDAM, and fairness reductions); nonetheless, the paper does take care to spell out the motivations and theoretical underpinnings for their final solution. Overall, the paper could be of interest to the community, and could help inspire more work in the area.
justification_for_why_not_higher_score: Decent, but not groundbreaking technical novelty.
justification_for_why_not_lower_score: Three positive reviews owing to a well-executed paper. From my reading, I concur with this view.
Decision: Accept: poster