Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the fair violations of predictive models under bounded distribution shifts. The main contributions are bounds on the change in fairness metrics (like demographic parity and equal opportunity) between train and test distributions that undergo either covariate shift or label shift. The bounds are further specialised to two applications in covariate or label shifts due to individuals strategically responding classifiers.
strengths_and_weaknesses: Strengths
- Proposes a generic way to write down and analyze the maximum changes in fairness properties under shifts.
- Results are original. Analysis for three common fairness metrics and two broad types of shifts.
- The paper is well-written in general.

Weaknesses
- Significance of the bounds is not clear. The interpretation of the theoretical results does not describe how to apply them in practice.
- The experimental results on Adult Income data are not summarized.

The list of results seem to have non-obvious implications, but these are hard to see. Please address the questions below that will help me to decide whether to improve the evaluation to 6 or 7. Particularly, convey the insights from one of the bounds in more detail.

---

## After the response

The response aptly addressed my concerns on the practical utility and interpretation of the results. So I have increase my score to 8, Strong Accept.

The work makes a great contribution on anticipating robustness of fairness guarantees under distribution shift. It may encourage further work on making the shift assumptions more interpretable and practically useful.
questions: Questions to address in the response

How can a practitioner use these to judge whether to use a model? Do the quantities included in the expressions (like Eq 17) plausibly known beforehand to estimate the max change in fairness?

How does the bounds or the analysis, in general relate to previous work? In what ways does the formulation unify those in the previous work, as claimed?

What are the takeaways from Adult Income results in Figure 4? Since the real data shifts do not neatly fit in either covariate or label shift, how to interpret the difference between the empirical and theoretical changes? I would suggest estimating the type of shifts and doing the comparison between states or time periods which have close to covariate or label shift. Or, to perform a more controlled setting where distribution shifts are simulated but the training data is real. Conditional independence test can be used to detect the types of shifts such as the methods used in this paper https://proceedings.neurips.cc/paper/2020/hash/e2d52448d36918c575fa79d88647ba66-Abstract.html

Relation of the results in Section 3.1, 3.2, and 3.3 to the results in Section 4 is not clear. Are the results in Section 4 special cases of these results?


Minor points and suggestions (I do not expect a response)

Some more related work
https://proceedings.neurips.cc/paper/2020/hash/d6539d3b57159babf6a72e106beb45bd-Abstract.html
https://arxiv.org/abs/2005.03474
https://proceedings.neurips.cc/paper/2021/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html
https://arxiv.org/abs/2206.12796


Notation can be eased. For example, consider skipping \Psi or reusing \Delta for it since \Psi is not used in the results.

Please consider discussing how to bound the fairness change when both covariate and label distributions shift.

Describe the divergence metric D_g in Eq (21).

Figure 2 is not clear.

Describe the term replicator dynamics briefly.

Consider explaining why the divergence between distributions p and q is named ‘pre’metric in Definition 2.1.
limitations: Limitations of the work are not discussed. Please discuss limitations of the way distribution shifts are characterized that is the class of shifts considered and the parameters to quantify the shift, discuss the applicability of the results to help in deployment decisions, and the experiments required to validate the results on real data.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers fairness under domain shift, specifically, bounding the change in the violation of group fairness criteria when the evaluation (target) data distribution differs from the one (source) on which the model is trained.

- First, a framework/distribution shift assumption is proposed for studying this problem.  Namely, the domain shift should be bounded w.r.t. some probability (pre)metric.  Then the worse-case (or adversarial) fairness bound is considered under this assumption.
- The authors derive some results under several simplifying assumptions in section 3, then work out the results for covariate shift and label shift in sections 4 and 5, respectively.
- In section 6, two examples of distribution shifts are considered, which are instances of covariate and label shift.
- Finally in section 7, a visualization of the bounds in sections 4 and 5 are provided on a real-world dataset, and are compared to the true change in fairness violation incurred by a model under simulated domain shift.
strengths_and_weaknesses: Strengths.

1. The paper is complete, clearly written, and easy to follow.
2. The problem studied, fairness under domain shift, is current and of importance to the field.

Weaknesses.

1. I feel that some of the results are weak.

    - It is unclear whether the results in section 3 under the simplifying assumptions would be generally useful; as the authors pointed out on line 183 “direct computation of [terms in theorem 3.2] is difficult”, and the same could be said of theorem 3.4.

        Although the proof of theorem 5.2 uses theorem 3.2, I feel that a direct proof with potentially simpler arguments is also possible, given the label shift assumption.

    - I feel that the conclusion in theorem 4.3 is trivial given the assumption that the change in true positive rate is bounded.  Evaluating the change in TPR requires labels from the target distribution; is there a bound if this information is not available?

2. It is not clear what is being visualized in fig. 4 in section 7.  Is the blue mesh the ground-truth change in DP, and the colormapped mesh the modeled change?  Are the left two plots produced assuming covariate shift, and the right two label shift?  Are there any remarks/comments on the results?
questions: My main questions are embedded in the section above.

1. line 230 missing $x_h\in\{l_h,u_h\}$ under $\max$?

limitations: The problem studied is of interest to the field, but the potential impact of the present work may be limited given the assumptions required in the results.  I am willing to hear the opinions of the other reviewers.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper's primary contribution is a theoretical upper bound on certain fairness definitions, when the target distribution has drifted from the source distribution. The paper then works through how label shift and covariate shift would affect the bounds on certain fairness definitions, leading to a brief empirical analysis of the results. The following are the main contributions:

1) The paper provides a theoretical result that shows an upper bound on the violations distribution shifts can achieve under several different fairness definitions. (Significance: High)

2) The paper provides a walk through of how certain distribution shift characteristics can be analyzed with their framework. (Significance: Medium)

3) The paper provides a unifying framework for several other related works trying to assess the impacts of distribution shift (Significance: High)
strengths_and_weaknesses: Quality:

The paper is of high quality overall, but section 7 needs a bit more work. The paper suggests in the abstract that empirical justification would be given for both demographic parity and equalized odds. However, section 7 seems to only deal with the demographic parity definition without much discussion. Recommendations:

1) Clarify the abstract as to what empirical justification will be given

2) Bolster section 7 with either examples concerning equalized odds and/or give more discussion as to how to interpret the outputs of the empirical justification.

Possibly, including details from Appendix E.2. in section 7 may solve this issue.

Clarity:

Overall, the paper is clear, but the over reliance on notation makes it hard to read at times. Many definitions have overused notation (such as implication symbols) rather than plain English, making interpretation of the definition hard to gauge. As an example, please look at definition 2.4, where the if and only if symbol and plain English are both there.

Originality:

Based on my knowledge of the field and the authors’ citations, I believe that this is original work that will move the field forward. However, I do find that some critical dynamics in fairness papers are missing from the citations, such as Liu et. al. 2018 Delayed Impact of Fair Machine Learning and Creager et. al. 2020 Causal Modeling for Fairness in Dynamical Systems, which takes papers like Liu et. al. 2018 and simplifies them into a single framework as well. These citations will provide more credibility and offer other avenues for unifying frameworks for dynamics in fairness.

Significance:

This paper provides a significant result for the field that could provide a general bound for how much dynamics can violate fairness definitions on a static dataset. Although I am unsure right now how to use it in practice, I believe having these theoretical tools will be helpful in the future. However, the paper could be made more legible by decreasing the usage of notation. Moreover, having a more robust narrative in section 7 will show the results in the real world better.

questions: 1) Could you give a more granular description of Appendix E.2. and why it was relegated to the appendix instead of being written in section 7?
limitations: As this paper is theoretical, there doesn't seem to be any limitations to discuss. If anything, the paper provides upper bounds on the limitations of other papers that explore how distribution shifts affect the violation of fairness definitions.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the transferability of group fairness under distribution shift which is an important problem in reality. Under bounded distribution shift, this paper develops a general bound for the fairness violation in the test time. It also derives bounds when considering particular fairness metrics (including demographic parity and equalized odds) and types of distribution shift (including covariate shift and label shift). 

strengths_and_weaknesses: Strengths:
This is the first paper that provides guarantees of fairness under bounded distribution shift. This paper considers multiple types of distribution shift and multiple fairness metrics. They also compare the derived theoretical bounds with the real change in fairness violation. The theoretical analysis can be applied to any models.

Weakness:
1. This paper has too many notations. I got lost many times. 
2. It would be better if there are more intuitions and explanations. For example, what’s the message we can get from theorem 3.2? Figure 2 is hard to understand without explanation.
3. Many contents are delayed to the appendix, but the appendix is not included in the supplementary material. (Please point it out if I missed it.)
4. I’m concerned about the practical use of the derived bounds. This is because the assumptions this paper considers including the particular distribution shift (covariate shift or label shift) and the availability of the target data are hard to be satisfied in reality. 

questions: 1. The theoretical bounds depend on the source and target data distribution, but in practice, we only have finite samples. What’s the estimation error when we measure the bound in practice? Are there any factors, such as the data size, that affect the measurement in practice?
2. Based on the theoretical bounds, do you have any ideas or suggestions for the algorithms to tackle the distribution shift? How to design policy \pi so that to have a smaller upper bound?
3. Does figure 4 indicates that the actual distribution shift is not a covariate shift, and the bounds become less useful in practice?

Minor issue:
In equation (22), is there a typo under $\max$? Do you miss $x_h$?

limitations: One limitation of this paper is that the theoretical bounds are based on some impractical assumptions which hinder their use in reality. I encourage authors to discuss some potential algorithms inspired by the theoretical bounds.
review_title: null

Meta-review: metareview: Reviewers agreed that this work, looking at the degradation of group fairness metrics under a few types of distribution shift, add to the literature in this space.  The author's/authors' rebuttal was especially effective in belaying concerns from two reviewers, in addition to many of my own.  I agree with Reviewer 84wj's skepticism of the practical implementability of some theoretical results (e.g. verification of the Lipschitz condition for Thm 3.2), but these are not showstopper concerns; still, please do address them in a final/next version.  Overall, though, the paper addresses a timely topic in a strong and reasonably complete way.
confidence: Certain
Decision: Accept