Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for controlling unfairness of a model when the test distribution may differ in the marginal distribution of a feature such as gender, race. It derives a statistical test for checking unfairness on the unknown test distribution via importance weighting combined with user-input on extent of the shift. Experiments on an academic performance dataset shows that the approach achieves low failure rate of learning an unfair classifier.
main_review: The work provides a conceptually clear and flexible framework for learning fair classifiers under demographic shifts. The organization of the concepts (Seldonian framework, problem description, fairness tests) is great and the writing is clear. Isolating the problem to performing the fairness test under an unknown distribution helps to understand the proposed method. The flexibility of the framework to accommodate different fairness measures is an advantage. The probabilistic guarantee for fairness violation is a good feature absent from many fair learning approaches for this particular problem. 
My main issues are (1) with the setup of controlling only fairness and not the accuracy, and (2) the seemingly large number of samples required for good accuracy for the method. Some related work is also missing which can be easily addressed. Few details on the scope of the demographic variable (discrete vs continuous) and relation to covariate shifts need clarity. I mention these issues in detail below.

---
  
## Questions to address in the response:

1. Why is the objective of controlling test unfairness while controlling train error justified, as opposed to attending to both quantities for test set? As pointed out, determination of the fairness test is impacted by demographic shift. So, is the classification loss at deployment time that is to be computed for the shifted distribution. Does this accounted somehow in the algorithm? One strategy is similar to the proposed fairness test procedure to note that loss conditional on T remains the same and somehow importance weighting or bounding the loss. I am wondering if I am missing this important detail in the algorithm on how training and deployment error are related to each other. 

2. Does the demographic shift assumption rely on demographic attribute space \mathcal{T} to be discrete? The specification of marginal shift in terms of lower-upper bounds on each probability of each value of T suggests that. Similarly, conditioning set \zeta around Eq (4) is assumed to be discrete, thus, excluding some fairness metrics, which is fine, but the allowed domains for variables should be specified. Please mention if discreteness is a limiting assumption for the proposed algorithm.

3. Accuracy of Shifty seems to be severely impacted even with known shifts when there are <100K data points in middle plots of Figure 3. Under known shift, the test fairness constraint is not harsher than the fairness constraint on the train set. What is the reason for the drop? Also, it is more practical to have much fewer data points such as of the order of 10K. At least, the sizes of fairness datasets used in literature are in the same or lower range. Does the severe drop in accuracy observed in only this particular dataset of exam scores or others too? It would be worthwhile to either include results for lower sample sizes on this and preferably other datasets, or empirically study and discuss issues limiting the sample efficiency for future work to address.

---

## Minor questions that do not affect my review:
4. In Algorithm 1, does the candidate model \thetha_c in step 2 appear in inputs to fairness testing in step 3?

5. I missed the description of how the candidate models are enumerated and tested one by one. Can you please clarify what is the output of the candidate model selection step 2 in Algorithm 1.

---

## Related work

There have been multiple studies on fairness guarantees under shifts which have been omitted. They are making somewhat different, in some cases weaker, assumptions and have different guarantees. It would be good to discuss differences from these works.

- Biswas and Mukherjee 21 https://dl.acm.org/doi/abs/10.1145/3461702.3462596
- Rezaei et al. 21 https://arxiv.org/abs/2010.05166
- Schumann et al. 19 https://arxiv.org/abs/1906.09688
- Coston et al. 19 https://dl.acm.org/doi/abs/10.1145/3306618.3314236
- Singh et al. 21 https://arxiv.org/abs/1911.00677
- Dai and Brown 20 https://dynamicdecisions.github.io/assets/pdfs/29.pdf

If any of these works have the same problem setup such as Rezaei et al. 21, please consider comparing against them.
  
---

## Suggestions

Please mention if the demographic shift assumption is a type of covariate shift, which is a better known term in literature or how does it differ.

Please describe how the result in Theorem 2 is important and used to support the method.

Please add the reason for choosing the particular numerical optimizer from Endres et al. 2018. Is U_ttest a non-smooth function?

A possible dataset to experiment is the new Adult dataset derived from US Census which has distribution shifts, perhaps demographic and other shifts. See Folktables package https://github.com/zykls/folktables.
summary_of_the_review: The work presents a novel and clear approach to the problem of fair learning under shifts. There are some open issues on the problem setup and experimental results, which I would like authors to respond to. In total, the work is a technically strong contribution to the nascent literature on the problem.

---
From the newly made changes and clarifications in the response, my concerns are addressed. I highly encourage authors to move the results on new dataset, discussion on related work and continuous demographic attributes to the main text.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a class of algorithms for ensuring fairness guarantees when the deployment data is susceptible to a demographic shift, termed as a marginal shift in demographic attributes such as gender or race, in comparison to the training data. Results are provided for two settings, 1) when the exact demographic shift in the deployment setting is known and 2) when the demographic shift in the deployment environment is unknown. This is done by computing a high-confidence upper bound on the prevalence of unfair behavior in the deployment environment, presently done using the Student's t-Test. Comparisons are made with Seldonian and Quasi-Seldonian algorithms, Fairlearn, and Fairness Constraints. The empirical analysis is performed on the university dataset to predict GPA using scores, with gender as the fairness attribute and student's race as the demographic attribute, the marginal distribution of which changes across the training and deployment settings. 
main_review: Positives:

1. The paper is generally well-written and easy to follow.
2. The problem studied is definitely interesting to the community and one of the main challenges with the deployment of models in practical settings. 
3. The high-confidence upper bounds on unfairness in the deployment setting present an interesting approach to ensuring fairness guarantees in the deployment setting.

Negatives:
1. The authors are missing some parallel works with ensuring fairness guarantees with general distribution shift [1,2,3]. 
2. Certain design choices are missing clear explanations such as how was the interpolation factor of 0.3 decided?

Additional concerns:
1. Can the assumptions about $g(\theta)$ be extended to settings where there needs a comparison with respect to another demographic group rather than ensuring it to be under some threshold? 
2. It is not exactly clear how the approach can be extended to definitions such as individual fairness. 
3. In order to better assess the quality of the fair predictions in the deployment setting, it would be helpful to have some empirical analysis when multiple definitions of fairness that may not be compatible are suggested by the user.

References:
1. Singh, H., Singh, R., Mhasawade, V., & Chunara, R. (2021, March). Fairness violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 3-13).
2. Du, W., & Wu, X. (2021). Robust Fairness-aware Learning Under Sample Selection Bias. arXiv preprint arXiv:2105.11570.
3. Dai, J., & Brown, S. M. Label Bias, Label Shift: Fair Machine Learning with Unreliable Labels.
summary_of_the_review: The contributions of the paper are clear. As the paper is missing some related works, the baseline comparisons can be further extended to assess other factors as suggested.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an approach called SHIFTY that provides high-confidence fairness guarantees when the distribution of training and deployment is different. They proposed two approaches to tackle the problem, one based on known distribution shift and another one for when the distribution shift is unknown. They evaluated their approach on a dataset for student success prediction and compared their approach with SOTA in-processing fairness approaches. 
main_review: The paper tackles an interesting and challenging problem. The idea of separating the tasks in a parallel sense is interesting as they divide the problem into the candidate selection and searching for the high-confidence upper bound. The main issue, however, is the data distribution part that it seems that they didn’t describe its procedure, hence, I am not sure whether it was a fair sample selection in which both of the Df and Dc are representative of the source data.

In the introduction, the authors claimed that they allow users to select fairness notion(s) from a large class depending on the application domain. However, according to their approach, it is not clear how they can account for individual fairness. Can the authors elaborate on this?

Furthermore, the authors acknowledge in section D of supplementary material, there is an assumption that each parameter’s intervals are independent, which is often false. Can the authors explain such justifications.

Their method of identifying the distribution shift in both known and unknown shifts relies on user-provided information. Indeed, it is an assumption to the problem which may hold untrue.  Would it be possible to detect distribution shift at the deployment and change the model accordingly? Why there is a need to train models for various unknown distribution shifts? ( It is obvious that considering various shifts has a huge impact on the performance (according to their results in Section 4.2)? 

The structure of the evaluation section is not convincing to me, and the comparison with in-processing fairness approaches does not seem fair. The only valid comparison is wrt to the fairness guarantees. However, it is not clear to me why the authors compared their approach under distribution shift with other methods that do not claim to be fair when the distributions of the train and test sets are different. Also, given the results presented for the experiment with unknown distribution shift,  it seems the Seldonian algorithm performs better that their approach. I'd suggest the authors to compare their approach with methods proposed for concept shift, also it will be useful to add at-least one more dataset for the evaluation to make sure that these results are consistent in various contexts.
summary_of_the_review: I have found the main idea of the paper interesting, however, I have some doubts about their proposed method and the way that they evaluated their approach and compared it with other methods (see previous section). 
Also, I suggest the authors to focus only on group fairness as it is not clear how their approach can be used for individual fairness methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on algorithmic fairness in machine learning (ML) and in particular on the problem of demographic shift. The motivation comes from the fact that the distribution of the underlying population might change between training and deployment -- the papers defines this as a shift in the marginal distribution of a single random variable (race, sex, etc.).

The setup considers a classification task, where sensitive information (the “fairness attribute”) might not be legal to use, but is available information. Given a certain event of interest and a target tolerance value \tau (e.g., with respect to sex, the false-positive rate of the model must below \tau% for female), an algorithm is defined to be fair if the algorithm achieves the target value \tau with high probability (given by some \delta parameter). To account for demographic shift, the model further includes a “demographic attribute” T; the distribution of T can change over time, but the conditional distribution must remain the same (by assumption).

Given this setup, the paper introduces Shifty, an ML algorithm that provides high-confidence guarantees that a fairness property will hold even when a demographic shift has occurred. Shifty works as follows. It takes as input a training datasets, some required fairness constraints and a description of demographic shifts. Then, it partitions the dataset to two subsets, finds a model based on the first dataset. Then, it uses Q and the second dataset to build an upper confidence bound for the model after deployment and given demographic shift. It returns only models that are unfair after deployment with probability at most \delta.

The experiments use test scores data from the Brazilian university system and show the satisfactory performance of the proposed algorithm.

main_review: Comments: 

-I found the formal definition of demographic shift (page 3) somewhat hard to parse and understand intuitively. 

-On top of the previous comment, it is not clear to me why the authors make a distinction between fairness attribute and demographic attribute. What if the college wants to take into account both race and gender as a fairness, legally protected, attribute? Also why is it reasonable to assume that T changes (eq. 2) but the conditional probability (eq. 3) does not? In particular, in what settings it is justified to assume that eq. 3 holds?

-Section 3.1.2 assumes a more realistic scenario. It is hard to think of scenarios where Q would be exactly known so I think this section should be presented as part of the main setup (and not as an extension). However, the description of the method is not very precise. How do the numerical approximation and simplicial homology optimization exactly work in this setting?

-Current appendix A and related work on pages 3-4 could be improved. How does Shifty provide guarantees that the other algorithms cannot? Would it require significant modification of the existing algorithms (and how, if possible) to account for demographic shifts (property B)?

-In terms of writing, the paper is quite clearly written. There are some repetitions in certain places (e.g., section 2)

-In the main contributions list (page 2), I think that the theoretical contribution (#2) is an overstatement. Theorem 1 is rather obvious and no guarantees are provided for the unknown demographic shift case (maybe obtaining distribution-dependent bounds would be possible?)

-Why was only this dataset chosen? Have the authors tried (potentially synthetic) experiments with other datasets (e.g., the law admissions dataset used often in fair ML)?

-How is such a dramatic demographic shift explained in Brazil (fig. 2)?

-The sentence “To evaluate each model’s… E)” about evaluation in Section 4.2 is not very clear. 

summary_of_the_review: I think the paper touches upon an important practical challenge not only in fair ML, but ML models used for decision-making in general. The analysis is natural, straightforward, and technically sound; the most technical aspect is the computation of the upper confidence bound which is standard, so I think there is no particular novelty in its methodological contribution. 

That said, I think the conceptual contribution of the paper is still quite new and useful. In the light of the novel application, I thus think the algorithm is quite novel -- nevertheless, it would be helpful if the authors provided a clearer comparison to the most related works and explained how their approach is novel. 

Finally, I found the evaluation section convincing and well-executed. I would appreciate additional experiments on -- other widely used in fair ML -- datasets (even synthetic experiments).
review_title: null

Meta-review: The paper studies the setting of group-based fairness under the so-called demographic shift, where the marginal distribution of the data remains the same conditional on the subgroup but the subgroup distribution can change. It provides a class of algorithms which give high confidence guarantees under demographic shift in both the known and unknown shift setting.

Overall the paper is a worthwhile contribution: it provides a new angle to the important problem of group-based fairness with good theoretical and empirical results.
Decision: Accept (Poster)