Fair Sequential Selection Using Supervised Learning Models

We consider a selection problem where sequentially arrived applicants apply for a limited number of positions/jobs . At each time step , a decision maker accepts or rejects the given applicant using a pre-trained supervised learning model until all the vacant positions are filled . In this paper , we discuss whether the fairness notions ( e.g. , equal opportunity , statistical parity , etc . ) that are commonly used in classification problems are suitable for the sequential selection problems . In particular , we show that even with a pre-trained model that satisfies the common fairness notions , the selection outcomes may still be biased against certain demographic groups . This observation implies that the fairness notions used in classification problems are not suitable for a selection problem where the applicants compete for a limited number of positions . We introduce a new fairness notion , “ Equal Selection ( ES ) , ” suitable for sequential selection problems and propose a post-processing approach to satisfy the ES fairness notion . We also consider a setting where the applicants have privacy concerns , and the decision maker only has access to the noisy version of sensitive attributes . In this setting , we can show that the perfect ES fairness can still be attained under certain conditions . 1 Introduction . Machine learning ( ML ) techniques have been increasingly used for automated decision-making in high-stake applications such as criminal justice , loan application , face recognition surveillance , etc . While the hope is to improve societal outcomes with these ML models , they may inflict harm by being biased against certain demographic groups . For example , companies such as IBM , Amazon , and Microsoft had to stop sales of their face recognition surveillance technology to the police in summer 2020 because of the significant racial bias [ 1 , 2 ] . COMPAS ( Correctional Offender Management Profiling for Alternative Sanctions ) , a decision support tool widely used by courts across the United States to predict the recidivism risk of defendants , is biased against African Americans [ 3 ] . In lending , the Apple card application system has shown gender biases by assigning a lower credit limit to females than their male counterparts [ 4 ] . To measure and remedy the unfairness issues in ML , various fairness notions have been proposed . They can be roughly classified into two categories [ 5 ] : 1 ) Individual fairness : it implies that similar individuals should be treated similarly [ 6 , 7 , 8 ] . 2 ) Group fairness : it requires that certain statistical measures to be equal across different groups [ 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . In this work , we mainly focus on the notions of group fairness . We consider a sequential selection problem where a set of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . applicants compete for limited positions and sequentially enter the decision-making system.1 At each time step , a decision maker accepts or rejects an applicant until m positions are filled . Each applicant can be either qualified or unqualified and has some features related to its qualification state . While applicants ’ true qualification states are hidden to the decision maker , their features are observable . We assume the decision maker has access to a pre-trained supervised learning model , which maps each applicant ’ s features to a predicted qualification state ( qualified or unqualified ) or a qualification score indicating the applicant ’ s likelihood of being qualified . Decisions are then made based on these qualification states/scores . Note that this pre-trained model can possibly be biased or satisfy certain group fairness notions ( e.g. , equal opportunity , statistical parity , etc. ) . To make a fair selection with respect to multiple demographic groups , each applicant ’ s group membership ( sensitive attribute ) is often required . However , in many scenarios , such information can be applicants ’ private information , and applicants may be concerned about revealing them to the decision maker . As such , we further consider a scenario where instead of the true sensitive attribute , each applicant only reveals a noisy version of the sensitive attribute to the decision maker . We adopt the notion of local differential privacy [ 16 ] to measure the applicant ’ s privacy . This notion has been widely used by researchers [ 17 , 18 , 19 ] and has been implemented by Apple , Google , Uber , etc . In this paper , we say the decision is fair if the probability that each position is filled by a qualified applicant from one demographic group is the same as the probability of filling the position by a qualified applicant from the other demographic group . We call this notion equal selection ( ES ) . We first consider the case where the decision maker has access to the applicants ’ true sensitive attributes . With no limit on the number of available positions ( i.e. , no competition ) , our problem can be cast as classification , and statistical parity and equal opportunity constraints are suitable for finding qualified applicants . However , when the number of acceptances is limited ( e.g. , job application , college admission , award nomination ) , we can show that the decisions made based on a pre-trained model satisfying statistical parity or equal opportunity fairness may still result in discrimination against a demographic group . It implies that the fairness notions ( i.e. , statistical parity and equal opportunity ) defined for classification problems , are not suitable for sequential selection problems with the limited number of acceptances . We then propose a post-processing method by solving a linear program , which can find a predictor satisfying the ES fairness notion . Our contributions can be summarized as follows , 1 . We introduce Equal Selection ( ES ) , a fairness notion suitable for the sequential selection problems which ensures diversity among the selected applicants.To the best of our knowledge , this is the first work that studies the fairness issue in sequential selection problems . 2 . We show that decisions made based on a pre-trained model satisfying statistical parity or equal opportunity fairness notion may still lead to an unfair and undesirable selection outcome . To address this issue , we use the ES fairness notion and introduce a post-processing approach which solves a linear program and is applicable to any pre-trained model . 3 . We also consider a scenario where the applicants have privacy concerns and only report the differentially private version of sensitive attributes . We show that the perfect ES fairness is still attainable even when applicants ’ sensitive attributes are differentially private . 4 . The experiments on real-world datasets validate the theoretical results . Related work . Learning fair supervised machine learning models has been studied extensively in the literature . In general , there are three main approaches to finding a fair predictor , 1 . Pre-processing : remove pre-existing biases by modifying the training datasets before the training process [ 20 , 21 ] ; 2 . In-processing : impose certain fairness constraint during the training process , e.g. , solve a constrained optimization problem or add a regularizer to the objective function [ 22 , 23 ] ; 3 . Post-processing : mitigate biases by changing the output of an existing algorithm [ 10 , 24 ] . Among fairness constraints , statistical parity , equalized odds , and equal opportunity have gained an increasing attention in supervised learning . Dwork et al . [ 25 ] studies the relation between individual fairness and statistical parity . They identify conditions under which individual fairness implies statistical parity . Hardt et al . in [ 10 ] introduce a post-processing algorithm to find an optimal 1Rolling admission , job application , and award nomination are examples of sequential selection problems with limited approvals/positions . On the other hand , there is no competition in the credit card application as the number of approvals is unlimited . binary classifier satisfying equal opportunity . Corbett-Davies et al . [ 26 ] consider the classification in criminal justice with the goal of maximizing public safety subject to a group fairness constraint ( e.g. , statistical parity , equalized odds , etc. ) . They show that the optimal policy is in the form of a threshold policy . Cohen et al . [ 27 ] design a fair hiring policy for a scenario where the employer can set various tests for each candidate and observe a noisy outcome after each test . There are also works studying both privacy and fairness issues in classification problems . Cummings et al . in [ 28 ] examine the compatibility of fairness and privacy . They show that it is impossible to train a differentially private classifier that satisfies the perfect equal opportunity and is more accurate than a constant classifier . This finding leads to several works that design differentially private and approximately fair models [ 29 , 30 , 31 ] . For instance , [ 29 ] introduces an algorithm to train a differentially private logistic regression model that is approximately fair . Jagielski et al . [ 30 ] propose a differentially private fair learning method for training an approximately fair classifier which protects privacy of sensitive attributes . Mozannar et al . [ 31 ] adopt local differential privacy as the privacy notion and examine the possibility of training a fair classifier given the noisy sensitive attributes that satisfy local differential privacy . In a similar line of research , [ 32 , 33 , 34 ] focus on developing fair models using noisy but not differentially private sensitive attributes . Note that all of the above works assume that the number of acceptances is unlimited ( i.e. , no competition ) , and every applicant can be selected as long as it is predicted as qualified . Our work is closely connected to the literature on selection problems . Unlike classification problems , the number of acceptances is limited in selection problems , and an applicant may not be selected even if it is predicted as qualified . Kleinberg and Raghavan [ 35 ] focus on the implicit bias in selection problems and investigate the importance of the Rooney Rule in the selection process . They show that this rule effectively improves both the disadvantaged group ’ s representation and the decision maker ’ s utility . Dwork et al . [ 14 ] also study the selection problem but consider individual fairness notion . Khalili et al . [ 13 ] study the compatibility of fairness and privacy in selection problems . They use the exponential mechanism and show that it is possible to attain both differential privacy and perfect fairness . Note that the selection in all of these works is one shot , i.e. , the applicant pool is static , and all the applicants come as a batch but not sequentially . Fairness in reinforcement learning and online learning is also studied in the literature . In [ 36 ] , an algorithm is considered fair if it does not prefer an action over another if the long-run reward of the latter is higher than the former . The goal is to learn an optimal long-run policy satisfying such a requirement . Note that this fairness constraint does not apply to classification or selection problems . Joseph et al . [ 37 ] consider a multi-armed bandit problem with the following fairness notion : Arm i should be selected with higher probability than arm j in round t only if arm i has higher mean reward than arm j in round t. This notion is not applicable to our selection problem with a group fairness notion . Metevier et al . [ 38 ] study an offline multi-armed bandit problem under group fairness notions . However , their method does not address the fairness issue in selection problems because their model does not consider the competition among applicants . The remainder of the paper is organized as follows . We present our model and introduce the ES fairness in Section 2 . We then propose a fair sequential selection algorithm using pre-trained binary classifiers in Section 3 . Sequential selection problems using qualification scores are studied in Section 4 . We present our numerical experiments in Section 5 .

Summary: This paper introduces a new fairness notion called Equalized Selection Rate (ESR) for sequential selection problems in which individuals are seen by the decision-maker sequentially, and there are a limited number of positive decisions available. For example, in a job acceptance setting, individuals are applicants submitting resumes sequentially, and the process ends when the decision-maker has accepted $m$ applicants. The authors show that a pre-trained model able to satisfy statistical parity or equal opportunity can be unfair in this sequential setting. A post-processing method is provided to circumvent this issue. Moreover, the authors show that when sensitive attributes are perturbed according to a local differential privacy notion, ESR fairness is still obtainable. Lastly, an empirical analysis is performed comparing equalized odds and ESR over two real data sets.