Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new fairness notion called Equalized Selection Rate (ESR) for sequential selection problems in which individuals are seen by the decision-maker sequentially, and there are a limited number of positive decisions available. For example, in a job acceptance setting, individuals are applicants submitting resumes sequentially, and the process ends when the decision-maker has accepted $m$ applicants. The authors show that a pre-trained model able to satisfy statistical parity or equal opportunity can be unfair in this sequential setting. A post-processing method is provided to circumvent this issue. Moreover, the authors show that when sensitive attributes are perturbed according to a local differential privacy notion, ESR fairness is still obtainable. Lastly, an empirical analysis is performed comparing equalized odds and ESR over two real data sets. 
main_review: originality -- I thank the authors for the extensive related work section---related work in classification fairness, selection, and differential privacy is discussed. Work related to sequential decision making is briefly mentioned (a single RL algorithm). To add, the bandit literature is another large area in sequential decision making with numerous works in fairness, e.g., Joseph et al. Fairness in Learning: Classic and Contextual Bandits, which focuses on individual fairness. More closely related to the classification setting (does not deal w/ exploration problem) might be batch bandit/RL algorithms that enforce fairness, e.g., Metevier et al. Bandits with High Prob. Guarantees. ESR is a fairness notion based on predicted outcome and not actual outcome (the selection made and not the "selection that should have been made"), and can be thought of as the action taken by the bandit. To summarize, along with reinforcement learning, I suggest mentioning bandit literature in the related work, as it is another important area in sequential decision making. Discussing the differences in the RL/bandit settings and the one being considered in this work can also strengthen the related work section. 

clarity/quality -- The paper is readable, but there is some room for improvement. In line 183 "It shows that even with the seemingly fair..." this requires more explanation, e.g., an example or intuition. Line 147 offers good motivation for the need of a fairness definition for the sequential setting. Also, notation for probabilities seem to change in Theorem 2 (from brackets to parentheses). I did not view the supplementary material but looked at the work in the main text. Is there a reason the authors switch between parens and bracket notation for probability? 

significance -- The authors assert in the introduction that they show that models enforcing statistical parity / equal opportunity may still be fair in the sequential setting they consider. I only see this in (1) after Corollary 1, line 183, "Note that he condition in Corollary 1 generally does not hold. It shows that...we may still be discriminatory,"  and (2) the comparison to equal opportunity in the experimental section. A comparison to statistical parity seems to be missing entirely in the empirical section of the main text, although it is listed as a contribution. Otherwise, the empirical analysis seems to support the claims made by the paper. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a notion of algorithmic fairness for the setting of sequential selection. Namely, a situation where candidates arrive one after the other, and a learner aims to select some number of whom. The authors propose a notion they term “Equalized Selection Rate (ESR)" which requires that the probability of an accepted qualified candidate is the same across groups in the distribution. The authors further suggest a post-processing approach for deriving a predictor that obeys the ESR constraint via solving a linear program (for fair selection using a binary classifier) or using a Bayesian optimization approach (for fair selection using qualification score). Additionally, the authors consider a case where for privacy concerns, the protected attribute is only accessible after noise is added (for predicted individuals), and show that perfect ESR is still attainable (when training the model on individuals for which the unnoisy protected attributes are included). The paper concludes with empirical evaluation of the proposed approaches.
main_review: The problem formulation studied in the paper is interesting. Rather than considering classification in its most general form, the authors focus on a sequential selection problem, inspired by various real-life scenarios. I appreciate this novel problem formulation and believe it is well-motivated and is worthy of further attention. The paper is well-written and is easy to follow and understand. In terms of novelty of the considered notion, to the best of my understanding, ESR is essentially equivalent to a condition regarding the true positive rates, weighted by the marginal group sizes for qualified individuals. The technical contributions are mainly through the suggested post-processing algorithms, which operate through a reduction to a linear programming problem or using a Bayesian optimization approach, and the extensions to the private case. This solution scheme seems elegant and easy to operationalize, however, as is the case with prior post-processing approaches (e.g. Hardt et al. 2016), it seems like the resulting accuracy-fairness trade-off could be sub-optimal.

In terms of technical contribution, there is no analysis or bounds regarding the accuracy loss that may result from taking the suggested post-processing approach for achieving the ESR notion, as opposed to an in-processing approach. I believe providing such bounds, even under additional assumptions, can significantly strengthen the results of paper. The authors do not have a discussion as for their choice to focus on a post-processing approach with a pre-trained classifier assumption rather than an in-processing approach. One could imagine, for example, attempting to cast the objective of accuracy under the ESR condition as a series of cost-sensitive optimization problems, as done in the reductions approach to fair learning (Agarwal et al. 2018) or one of the online variants. 

The proposed ESR notion should be better motivated. For example, consider a case where the distribution consists of two groups, one with probability 0.9 and the other with probability 0.1. Assume further that the fraction of qualified individuals across both groups is 0.9. Under the ESR condition, it is necessarily the case where qualified candidates from the larger group are denied at a much higher rate. A discussion regarding the advantages and disadvantages of the proposed notion of fairness should appear in the paper.

Questions to authors:

1. Prior work has suggested a post-processing scheme for a different yet closely related fairness definition, equal opportunity, by solving a linear program (Hardt et al. 2016). Can you elaborate on the differences between the approach in the paper and previously suggested post-processing approaches, and especially Hardt et al. 2016, in terms of the difficulty of achieving EO vs. ESR and how does the different objective alter the required solution method?

2. In line 83, the paper states "Hardt et al. in [9] introduce a post-processing algorithm to find an optimal binary classifier satisfying equal opportunity". It is important to notice that the approach of Hardt et al. does not in general guarantee optimality of the equal opportunity predictor learned, but only in very specific cases (Woodworth et al. 2017). Optimality is rather with regards to the class of post-processed predictors, which can be far from general optimality under the equal opportunity condition. Can you discuss how taking a post-processing approach affects the resulting level of accuracy for the notion of ESR comparing to the optimal, not necessarily post-processed, solution over a given base class of predictors? 

The submission appears to be technically sound, although I have only glanced at proofs in the appendix. The paper cites the relevant prior work. 
limitations_and_societal_impact: As stated in the main review, I would like to see a more elaborate discussion about the potential societal effects of the proposed ESR condition, and a discussion regarding taking a post-processing approach rather than an in-processing one.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies how to "fairly" select a single candidate from a repeated i.i.d. classification perspective "selection process". They contribute a new fairness notion that is unique in the selection setting, however, it is not that different from previously contributed group fairness notions. They provide sound technical methods to post-process predictors to satisfy the fairness notion and consider selection in the private attribute setting. 
However, the selection setting is not interesting enough without decisions at time step i affecting those at time step i+1, moreover, as the paper studies the  single selection setting, it is not that different from just classification. 
The paper is novel and interesting in it's combination of all those elements, but there is no sufficient novelty in each direction alone to stand out. The paper is clearly written and easy to follow and technically sound. I think if the paper had a slightly more realistic selection setting (any sort of dynamics from one step to another), then it would be a clear accept. But without that, I think the paper is borderline. 

main_review: Originality: The paper considers a new fairness criterion for selection problems of finite resources. The fairness notion is similar to existing notions and has the same form of equality of group wise probabilities. Fairness in selection problems has previously been studied. Guaranteeing fairness when only privatized attributes are available has been studied in classification. The novelty in this paper is studying this new fairness notion in the selection setting (repeated i.i.d. classification) with private attributes (only at test time).  The postprocessing approach is a very interesting technical result and approach, and not obvious. However, the selection setting considered is only studied for a single selection, does not study dynamics of selections on each other (because it studies single selections) and only studies the setting for binary sensitive attributes (but I understand this may only complicate theory but not the approach). 

Quality: The theorems and technical details are sound and complete. Some crucial experimental details are missing especially how 1) they simulated selection problems from the static datasets and 2) how the optimization was performed and thresholds chosen for EO and the baselines and 3) how exactly they computed the statistics. Furthermore, I am unsure about one of the results in table 1. 

Clarity: The paper is clearly written and easy to follow. The basis of the paper is the following sentence  “decisions are fair if the (qualified) applicants from different groups are
selected at the same rate.”,  this implies in classification that P(R=1|Y=1,A=0)=P(R=1|Y=1,A=1) (EO) but Theorem 1 says otherwise, what the authors actually mean is that decisions are fair if we select the same number of qualified individuals from different groups and not “rates”. I think the proposed notion (ESR) is fundamentally very very different from EO as it is not a conditional statement . 

Significance: I think this paper misses out an incredible opportunity to study selection problems by only focusing on the m=1 case and assumes i.i.d. selections and no impact of previous selections on the following ones. Due to this, I feel the "selection" aspect of the paper is not that novel and so it goes back to studying the ESR notion in classification problems and trying to argue for it (from theorem 1). However, the paper does not attempt to justify ESR in the classification  from an ethical perspective or long term perspective but only cites the EEOC.

Limitations:  1) studying binary attributes, 2) m=1 setting "single selection", 3) i.i.d. between selections which is not a reasonable assumption in reality, 4) not enough argumentation, especially from a societal perspective, for why one wants ESR instead of EO. 

Comments:

 re footnote 1: College admission is a subset selection problem and not sequential selection, hiring can be argued as sequential selection.

How calibrated is the pretrained model?

ESR extension to m>1 is not obvious what it means, also authors don’t write in words what ESR signifies: The probability that we select a qualified A=0 individual vs A=1 individual should be the same, that is not the natural extension of EO.

Theorem 1 is illustrative but is obvious from the definition 1 and so is corollary 1. 

Theorem 2 is nice, I think the proof needs to have all the steps written out (cancelling of terms more clearly) to be more easily readable. 

I want to see a comparison or a theoretical attempt at the difference between post-processing and in-processing with fairness, something like Theorem 5.6 in Equality of Opportunity in Supervised Learning  (Hardt et al., 2016).

I think it is reasonable to remove Assumption 1 and assume one only observes the noisy attribute at test and train, if not I would like to see an impossibility result to justify assumption 1 as it is not reasonable to assume a discrepancy between train and test in observing the sensitive attribute.

In table 1, the result of the 0 probability of selecting the minority group is a bit odd. Consider the following  test: at each step, we sample a random individual from the population, the probability that this individual is Black and qualified is P(A=0,Y=1), if a classifier satisfies EO, then it cannot be the case that the probability of selecting a Black individual is 0 if we repeat this selection process. I need more details on how the testing was accomplished. 

Constraint 17 is too artificial with the ½ threshold. I think it is okay to consider m=1 in theorems, but in experiments not showing m>1 is a big limitation.

What can make the paper really improve: 
1) considering m>1
2) Considering that decisions at time step i, affect the distribution of individuals at time step i+1. 
3) theoretical results to showcase the issue with EO shown in the experiments



---------------------------------------------

Post response: I have read the authors responses. The authors do acknowledge and address some of the limitations of their study, however, I remain on my previous score as the current problem studied is still limited by the lack of any dynamics governing the population or meaningful experimental study.
limitations_and_societal_impact: The papers introduces a fairness constraint on selection problems. Downstream impact of this fairness condition or it's ethical viability are not adequately discussed in the paper. Especially in selection problems, it is important to understand long term impact of a fairness notion rather than short term.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new task of making fair decisions in sequential selection problems. The authors have shown that existing fairness notions are not suitable for this problem and could lead to discrimination. Then the authors propose a new fairness notion called "Equalized Selection Rate (ESR)" for sequential selection problems and a post-processing approach is designed to satisfy the ESR constraint. The paper considers 2*2 different settings, i.e., whether the pre-trained model provides a binary decision or a continuous score, and whether the actual or noisy sensitive attribute is accessed. The paper also conducts experiments on FICO and Adult datasets.
main_review: The paper studies a novel setting in fair machine learning. I can see that there are a lot of theoretical works done by this paper. However, I feel that this paper is not quite well-written as some key points are not explained or missing. In addition, how each section in the appendix is responding to each part in the main paper is not explicit (the authors should refer to the appendix sections in the main paper). These factors lower my score. My detailed comments are as follows.

1. Theorem 1 and A.1 need more explanations in the paper. The derivation of the first equation to Theorem 1 and the equation between Eq. (18) and (19) could show the key difference between the sequential selection setting and traditional supervised learning setting.

2. In the proof to Theorem 2, how the second constraint in Eq. (5) is used? I believe that it is hidden somewhere in the proof but the authors should explicitly show that.

3. The paper defines accuracy as Pr{\tide{Y}=1}. But to me this is just the true positive rate, not the accuracy, since it only considers the case where the individual is predicted positive. Could the authors clarify this?

4. The logic of Sections 4.1 and 3.1 are not consistent. In Section 3.1, the paper defines a probability to flit the decision of the pre-trained model. But in Section 4.1, the paper “trust” the score made by the pre-trained model and only aims to learn a threshold for making decisions. 

I don’t have the expertise to check the differential privacy part of this paper.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: Reviewer all agreed that the paper formulates and partially addresses an interesting and practically relevant question: namely, how to define and guarantee an appropriate notion of fairness in sequential decision-making settings where a limited number of positions are available and diversity among the selected is an important consideration. However, among other suggestions for improvement, reviewers urge the authors to expand their discussion of the related work, the motivation behind proposed fairness notion, and the post-processing approach. Assuming that the authors will add the necessary discussions and revise the paper to reflect the reviewers’ suggestions, I recommend acceptance.
Decision: Accept (Poster)