Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work examines when does a fairness constrained model improve in overall accuracy on a target dataset which has a different distribution (of loss conditioned on some covariates) from the train dataset. It provides necessary and sufficient conditions under which this happens. Experiments on synthetic data validate the theoretical results. Real world experiments on COMPASS and Adult income datasets are promising.
main_review: Significance: This is my main concern. The significance of the theoretical results to evaluation and design of fair classifiers under shifts is not discussed. The condition provided in Theorem 4.3 (when to expect improvement) are hard to relate to or compute for any practical setting, thus limiting its applicability. Experiments on real world dataset also do not shed light on how the conditions can be computed. The problem of assessing future performance of the classifier under shifts is very important and the work provides a novel perspective based on geometry of risk profiles. Discussing the computable implications of the results will make them more useful for practitioners.

The work does not discuss the motivation for training fair models to correct subpopulation shift in training data. If we anticipate distribution shift, then we can use transfer learning (e.g. domain adapation, generalization) methods. It does not seem realistic to assume that modeler will not use any such methods and will rely on fairness constraints to potentially correct for the shift. 

Originality: The work adds an original, geometrical perspective to studying fairness under distribution shifts. This is enabled by their focus on the space of risk profiles and the subpopulation shift condition. Related work compares with the most directly relevant papers but does not place this work in the larger fair learning under shifts literature. Some works https://cs.stanford.edu/~jure/pubs/contraction-kdd17.pdf, https://arxiv.org/abs/1806.02887, https://arxiv.org/abs/2007.06029.

Quality: The key results are technically sound. The experiments do not adequately test the results.
Not much discussion is provided on whether the conditions in Theorems 3.2, 4.3 can be tested. Both the target distribution P* and fair risk set F are unknown.
Experiments on real data too are not checking whether these conditions hold. The experiments only check for an increase in accuracy in P*. The validation of the theory is incomplete without showing that the conditions that predict this increase also hold or can reasonably be expected to hold for real data.

Clarity: The writing is clear. The figures can be explained better. Please see suggestions at the end.


Overall, I like the new geometrical perspective to reason about model performance under shifts. I would encourage authors to develop the ideas to make them more practical and to consider their implications on better evaluation or training procedures.


More detailed comments:

Replacing risk with model output in the fairness definition of demographic parity and equalised odds is assuming that the loss function used in learning is the same as that for fairness consideration. The advantage of enforcing independence with model output is that it will ensure independence with any fairness loss function. For many reasons such as ease of optimization (use of surrogate loss) or application context, the two loss functions can be different. This critical assumption should be stated.

The target for comparison in the unbiased data is assumed to be the unconstrained Bayes optimal model. I disagree with the claim that assumption 3 is implicitly assumed in many previous works, lines 229-235. The cited works are not suggesting to just use the unconstrained Bayes classifier on representative data. The classifier for unbiased data can be trained with fairness constraints too if suitable for the application context.

Label bias described in Example 4.2 differs from the subpopulation shift assumption. It is usually defined as an invariant distribution of outcome given some or all covariates, whereas the label bias and the resulting risk profile is additionally conditioning on the outcome. Please define subpopulation shifts in the Problem setup to avoid any confusion.


Questions that I would like authors to address:

Is the risk profile in 4.2 defined with respect to biased distribution P tilde?

Are the conditions for checking improvement in the theorems testable? What data does it require? 

How to interpret the threshold on P* in Theorem 3.2 which is used to check whether FRM improves?

In Theorem 4.3 assumptions, which risk set or profile is used, 3.1 or 4.2?

Please clarify how the results can be used to better evaluate and design fair classifiers.

---

After the response, my concerns on the definition of subpopulation shift, and interpretation of Theorem 3.2 and Assumption 3 are addressed.
limitations_and_societal_impact: Please discuss limitations of the work e.g. limited to two fairness metrics (risk parity and conditional risk parity), limited to subpopulation shift (and not label shift), limited ability to verify if the conditions in theorems are satisfied in real data.


Comments to improve presentation:

Please consider contrasting the insights from this work with the fair learning under shifts literature. How does this work help in training better models under shifts?

What is the space plotted in Figure 2? Although clear from Figure 1, but useful to mention in caption.

In line 208, what is the fairness constraint the unbiased Bayes classifier will satisfy? Doesn’t it minimize the unconstrained risk? It is explicitly assumed later in Theorem 4.3 assumption 3.

Instead of linear optimization problem e.g. in line 78, please be specific that constraints are linear. Linear optimization suggests that loss function is linear which is not the case here.

Is it necessary to define risk profiles in terms of strict equality of risks across groups? Will the results extend to a more realistic case when there is small difference in risks across groups?

Line 132: “trivial random variables” -> V is an empty set.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors analyze the effect of enforcing algorithmic fairness (performance parity) in the presence of subpopulation shift. They argue that model performance can be summarized in terms of risk profiles, as risk minimization problems are equivalent to linear optimization wrt risk profiles. Basically, the authors consider ML problems that can be mapped to risk minimization. The paper also provides a way to analyze subpopulation shift by decomposing the training bias into (1) a portion which is orthogonal to the fair constraint, and (2) a robustness of $R^\star$ to changes in the distribution of the data in the hypothetical target domain ($P^\star$). Lastly, the authors show that fair risk minimization while training does not necessarily improve performance---a theoretical analysis is provided that shows when improvement and harm is done to model performance. Computational results are shown over synthetic data and recidivism data (COMPAS) to support results. 
main_review: ORIGINALITY: The main contribution of this paper is to offer insight and an interesting perspective into bias mitigation caused by population shift. The problem is well-motivated; this is not an uncommon phenomenon in practice. The work seems sufficiently original, although I am not intimate with the frontier of knowledge in this area.  There are already works that evaluate whether FRM reduces/impacts algorithmic bias. One key difference is that the authors are looking to evaluate these criteria in the target domain. The related work seems thorough in differentiating between the contributions of this work and similar works. 
QUALITY: The technical content of the main paper appears correct---I thank the authors for the experimental details in the main text. The optimization review in the Appendix B adds to the general readability/understanding of the techniques used in the proof. I did not look over the supplementary proofs in A but focused on B.8, the permissibility of continuous-valued attributes. 
CLARITY: The paper is structured clearly and is well-written! The notation is clear and intuition is provided for different concepts, e.g., enforcing RP by solving the constrained risk minimization problem using two groups...visual explanation is provided and clearly explained. The goal of the computational results is explained clearly, and adequate experimental details are provided in the main text. 
SIGNIFICANCE: I enjoyed this paper. It sheds light on a well-known phenomenon seen in practice---distribution (subpopulation) shift, and could be useful as a tool for analyzing/creating fairness aware algorithms. I believe this work adequately expands on previous work, e.g., ML problems applicable to risk minimization and considering posterior drift, while focusing on the effect on the target domain. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addresses the following problem: What will the performance of a ML model be on the target domain if one enforces fairness in the training space, while considering the fact that some groups may be underrepresented in the training domain (and hence, biased) and considering subpopulation shift? The paper brings theoretical results as well as experiments to provide insights into this phenomenon. 
main_review: This paper is well-written and well-motivated. The problem statement and the contributions of the paper are clear. Specially, I appreciate the {\em insight} that this paper can bring to the community. One of the main points of the paper is characterizing and analyzing algorithmic fairness under subpopulation shifts, which is defined in the paper.

The theoretical results of the paper are backed by the experiments on the simulated data, as well as COMPAS and Adult data.
 
Note: One should be careful with interpreting results of this paper as the paper assumes the assumption in equation (3.3). This is assumption 2 under Theorem 3.2 and 4.3. It means that this paper does not consider an entire distribution shift from the train domain to the target domain and restricts its attention to subpopulation shift. Hence, one should be cautious about making general statements based on this paper. Perhaps, a discussion about this can help the general reader more.

Minor: Contribution number 3 in Introduction: Line 36, please complete the sentence. 

limitations_and_societal_impact: See my main review.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper studies the impact of constrained fairness risk minimization on model performance when data are affected by subpopulation shifts. The authors focus on two notions of fairness that are defined with respect to the loss of the model, and that they call "risk parity" and "conditional risk parity" respectively. 
Under a series of assumptions, the authors derive the conditions under which a model model trained via constrained risk minimization has overall higher/lower performance on the target data (vs. train data). They then examine the resulting bounds via simulations and experiments. 
main_review: My comprehension of the results was very much hindered by the lack of clearly defined notation (see suggestions below). I also had a hard time understanding how these results differ from prior work and the meaning of the bounds in the results. Two important points:
* It seems that assumption (3.3) is fairly limiting. For example, this assumption rules out the setting of label-bias as defined by [1] and thus . In addition, I looked at the paper in reference [16] (line 92), and I have the impression that (3.3) is an additional assumption imposed on the subpopulation shifts setting, rather than being “(slightly) less restrictive”. 
* That said, it is unclear to me why the bounds per se, in their current form, should be of interest, or at least I do not think that the result in theorem 3.2 was properly discussed. We should probably interested in understanding by how much performance will change, although I am not sure this is analytically possible. 

Some (hopefully) constructive suggestions are: (i) move related work up and clearly explain how this work fits into the existing literature, (ii) clearly define all notation, (iii) improve the explanations of the results, (iv) add an organization paragraph in the introduction. 

Some of the typos and other minor issues:
* definition 3.1: is this definition based on prior work?
* line 36 “during improves”
* line 63: needs citation and \mathcal{l} has never been introduced
* line 65: the original study should be cited
* ERM has never been defined
* (3.2) I am not sure that the inner product on the RHS makes sense
* line 77: erroneous reference (4.3)
* it’s unclear what the pedix in the expectation means (compare pedix in (3.1) vs. Line 76)
* (3.2) \mathcal{F} has never been defined
* line 75: the bound between 0 and 1 for \tilde{P}_A makes sense only in case of discrete distributions
* (3.3) the expectations E* and \tilde{E} have not been defined
* (4.4) V has never been properly defined
* example 4.2: I believe that the definition of label bias adopted by the authors is one of many, so they may want to cite a paper the authors took this definition from. For example, [1] calls (4.2) “under-representation and labeling bias” (I see that this paper is actually cited in the related work).
* line 185: “even if the Bayes decision rule in the target domain is algorithmically fair” is unclear
* line 147: “.” Should be “,” 

[1] Blum, Avrim, and Kevin Stangl. "Recovering from biased data: Can fairness constraints improve accuracy?." arXiv preprint arXiv:1912.01094 (2019).
limitations_and_societal_impact: The authors should clearly motivate how their results advance and could contribute to our understanding of risk minimization with fairness constraints, e.g., perhaps by referring to the their case study of the COMPAS data.
review_title: null

Meta-review: This paper examines the question of whether enforcing algorithmic fairness in the training domain will improve performance of the model in a target domain in which subpopulations (such as those defined by the sensitive attribute) occur in different proportions from the training domain.  The key assumption behind the results is that the risk profiles (expected loss conditional on group, or group and other discriminative attributes) are the same between the training and test domain.   The authors’ main result crisply characterizes conditions under which target domain performance improves.  Overall this is an interesting paper that studies an important problem, but it is not without its weaknesses.  Given the authors’ responses to concerns raised by reviewers, I believe the major weaknesses can be addressed through moderate revision and am thus recommending this paper for acceptance.  

In revising the submission, the authors should focus on the following dimensions.
1. Clarity in exposition.  Reviewers jgBW VZHD enumerated a lengthy list of typos and confusions that I myself also share.  (Additionally, the difference between distributions, $P^* - \tilde P$ isn’t actually defined where it is first used in line 53.) While the authors note in their response that several reviewers described the work as well-written and clear, I want to emphasize that reviewers who went into depth in their comments noted typos, undefined and ill-defined notation, and confusion about theorem statements.  These issues are minor, but abundant.    The manuscript will be much improved once they are weeded out in revision.
2. Displaying figure 2 sideways is confusing.  If it is desired to keep the figure in this orientation to conserve space, at least rotate the text.  
3. The work would benefit from further discussion of the settings in which the authors believe the results would likely apply.  That is, in what practical settings might the subpopulation shift assumption hold?  Would it hold in something like the motivating gender shades study example?   
4. The COMPAS example is currently highly underdeveloped.  I would either expand on the discussion to explain what this example is illustrating and its implications for training risk assessment tools, or omit it altogether.  
5. In the author response the authors have committed to making a number of revisions to address concerns raised by reviewers.  They are all worthwhile changes that will improve the clarity of the manuscript.  

Decision: Accept (Poster)