Fair Sequential Selection Using Supervised Learning Models

We consider a selection problem where sequentially arrived applicants apply for a limited number of positions/jobs . At each time step , a decision maker accepts or rejects the given applicant using a pre-trained supervised learning model until all the vacant positions are filled . In this paper , we discuss whether the fairness notions ( e.g. , equal opportunity , statistical parity , etc . ) that are commonly used in classification problems are suitable for the sequential selection problems . In particular , we show that even with a pre-trained model that satisfies the common fairness notions , the selection outcomes may still be biased against certain demographic groups . This observation implies that the fairness notions used in classification problems are not suitable for a selection problem where the applicants compete for a limited number of positions . We introduce a new fairness notion , “ Equal Selection ( ES ) , ” suitable for sequential selection problems and propose a post-processing approach to satisfy the ES fairness notion . We also consider a setting where the applicants have privacy concerns , and the decision maker only has access to the noisy version of sensitive attributes . In this setting , we can show that the perfect ES fairness can still be attained under certain conditions . 1 Introduction . Machine learning ( ML ) techniques have been increasingly used for automated decision-making in high-stake applications such as criminal justice , loan application , face recognition surveillance , etc . While the hope is to improve societal outcomes with these ML models , they may inflict harm by being biased against certain demographic groups . For example , companies such as IBM , Amazon , and Microsoft had to stop sales of their face recognition surveillance technology to the police in summer 2020 because of the significant racial bias [ 1 , 2 ] . COMPAS ( Correctional Offender Management Profiling for Alternative Sanctions ) , a decision support tool widely used by courts across the United States to predict the recidivism risk of defendants , is biased against African Americans [ 3 ] . In lending , the Apple card application system has shown gender biases by assigning a lower credit limit to females than their male counterparts [ 4 ] . To measure and remedy the unfairness issues in ML , various fairness notions have been proposed . They can be roughly classified into two categories [ 5 ] : 1 ) Individual fairness : it implies that similar individuals should be treated similarly [ 6 , 7 , 8 ] . 2 ) Group fairness : it requires that certain statistical measures to be equal across different groups [ 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . In this work , we mainly focus on the notions of group fairness . We consider a sequential selection problem where a set of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . applicants compete for limited positions and sequentially enter the decision-making system.1 At each time step , a decision maker accepts or rejects an applicant until m positions are filled . Each applicant can be either qualified or unqualified and has some features related to its qualification state . While applicants ’ true qualification states are hidden to the decision maker , their features are observable . We assume the decision maker has access to a pre-trained supervised learning model , which maps each applicant ’ s features to a predicted qualification state ( qualified or unqualified ) or a qualification score indicating the applicant ’ s likelihood of being qualified . Decisions are then made based on these qualification states/scores . Note that this pre-trained model can possibly be biased or satisfy certain group fairness notions ( e.g. , equal opportunity , statistical parity , etc. ) . To make a fair selection with respect to multiple demographic groups , each applicant ’ s group membership ( sensitive attribute ) is often required . However , in many scenarios , such information can be applicants ’ private information , and applicants may be concerned about revealing them to the decision maker . As such , we further consider a scenario where instead of the true sensitive attribute , each applicant only reveals a noisy version of the sensitive attribute to the decision maker . We adopt the notion of local differential privacy [ 16 ] to measure the applicant ’ s privacy . This notion has been widely used by researchers [ 17 , 18 , 19 ] and has been implemented by Apple , Google , Uber , etc . In this paper , we say the decision is fair if the probability that each position is filled by a qualified applicant from one demographic group is the same as the probability of filling the position by a qualified applicant from the other demographic group . We call this notion equal selection ( ES ) . We first consider the case where the decision maker has access to the applicants ’ true sensitive attributes . With no limit on the number of available positions ( i.e. , no competition ) , our problem can be cast as classification , and statistical parity and equal opportunity constraints are suitable for finding qualified applicants . However , when the number of acceptances is limited ( e.g. , job application , college admission , award nomination ) , we can show that the decisions made based on a pre-trained model satisfying statistical parity or equal opportunity fairness may still result in discrimination against a demographic group . It implies that the fairness notions ( i.e. , statistical parity and equal opportunity ) defined for classification problems , are not suitable for sequential selection problems with the limited number of acceptances . We then propose a post-processing method by solving a linear program , which can find a predictor satisfying the ES fairness notion . Our contributions can be summarized as follows , 1 . We introduce Equal Selection ( ES ) , a fairness notion suitable for the sequential selection problems which ensures diversity among the selected applicants.To the best of our knowledge , this is the first work that studies the fairness issue in sequential selection problems . 2 . We show that decisions made based on a pre-trained model satisfying statistical parity or equal opportunity fairness notion may still lead to an unfair and undesirable selection outcome . To address this issue , we use the ES fairness notion and introduce a post-processing approach which solves a linear program and is applicable to any pre-trained model . 3 . We also consider a scenario where the applicants have privacy concerns and only report the differentially private version of sensitive attributes . We show that the perfect ES fairness is still attainable even when applicants ’ sensitive attributes are differentially private . 4 . The experiments on real-world datasets validate the theoretical results . Related work . Learning fair supervised machine learning models has been studied extensively in the literature . In general , there are three main approaches to finding a fair predictor , 1 . Pre-processing : remove pre-existing biases by modifying the training datasets before the training process [ 20 , 21 ] ; 2 . In-processing : impose certain fairness constraint during the training process , e.g. , solve a constrained optimization problem or add a regularizer to the objective function [ 22 , 23 ] ; 3 . Post-processing : mitigate biases by changing the output of an existing algorithm [ 10 , 24 ] . Among fairness constraints , statistical parity , equalized odds , and equal opportunity have gained an increasing attention in supervised learning . Dwork et al . [ 25 ] studies the relation between individual fairness and statistical parity . They identify conditions under which individual fairness implies statistical parity . Hardt et al . in [ 10 ] introduce a post-processing algorithm to find an optimal 1Rolling admission , job application , and award nomination are examples of sequential selection problems with limited approvals/positions . On the other hand , there is no competition in the credit card application as the number of approvals is unlimited . binary classifier satisfying equal opportunity . Corbett-Davies et al . [ 26 ] consider the classification in criminal justice with the goal of maximizing public safety subject to a group fairness constraint ( e.g. , statistical parity , equalized odds , etc. ) . They show that the optimal policy is in the form of a threshold policy . Cohen et al . [ 27 ] design a fair hiring policy for a scenario where the employer can set various tests for each candidate and observe a noisy outcome after each test . There are also works studying both privacy and fairness issues in classification problems . Cummings et al . in [ 28 ] examine the compatibility of fairness and privacy . They show that it is impossible to train a differentially private classifier that satisfies the perfect equal opportunity and is more accurate than a constant classifier . This finding leads to several works that design differentially private and approximately fair models [ 29 , 30 , 31 ] . For instance , [ 29 ] introduces an algorithm to train a differentially private logistic regression model that is approximately fair . Jagielski et al . [ 30 ] propose a differentially private fair learning method for training an approximately fair classifier which protects privacy of sensitive attributes . Mozannar et al . [ 31 ] adopt local differential privacy as the privacy notion and examine the possibility of training a fair classifier given the noisy sensitive attributes that satisfy local differential privacy . In a similar line of research , [ 32 , 33 , 34 ] focus on developing fair models using noisy but not differentially private sensitive attributes . Note that all of the above works assume that the number of acceptances is unlimited ( i.e. , no competition ) , and every applicant can be selected as long as it is predicted as qualified . Our work is closely connected to the literature on selection problems . Unlike classification problems , the number of acceptances is limited in selection problems , and an applicant may not be selected even if it is predicted as qualified . Kleinberg and Raghavan [ 35 ] focus on the implicit bias in selection problems and investigate the importance of the Rooney Rule in the selection process . They show that this rule effectively improves both the disadvantaged group ’ s representation and the decision maker ’ s utility . Dwork et al . [ 14 ] also study the selection problem but consider individual fairness notion . Khalili et al . [ 13 ] study the compatibility of fairness and privacy in selection problems . They use the exponential mechanism and show that it is possible to attain both differential privacy and perfect fairness . Note that the selection in all of these works is one shot , i.e. , the applicant pool is static , and all the applicants come as a batch but not sequentially . Fairness in reinforcement learning and online learning is also studied in the literature . In [ 36 ] , an algorithm is considered fair if it does not prefer an action over another if the long-run reward of the latter is higher than the former . The goal is to learn an optimal long-run policy satisfying such a requirement . Note that this fairness constraint does not apply to classification or selection problems . Joseph et al . [ 37 ] consider a multi-armed bandit problem with the following fairness notion : Arm i should be selected with higher probability than arm j in round t only if arm i has higher mean reward than arm j in round t. This notion is not applicable to our selection problem with a group fairness notion . Metevier et al . [ 38 ] study an offline multi-armed bandit problem under group fairness notions . However , their method does not address the fairness issue in selection problems because their model does not consider the competition among applicants . The remainder of the paper is organized as follows . We present our model and introduce the ES fairness in Section 2 . We then propose a fair sequential selection algorithm using pre-trained binary classifiers in Section 3 . Sequential selection problems using qualification scores are studied in Section 4 . We present our numerical experiments in Section 5 .

Summary: This paper introduces a new task of making fair decisions in sequential selection problems. The authors have shown that existing fairness notions are not suitable for this problem and could lead to discrimination. Then the authors propose a new fairness notion called "Equalized Selection Rate (ESR)" for sequential selection problems and a post-processing approach is designed to satisfy the ESR constraint. The paper considers 2*2 different settings, i.e., whether the pre-trained model provides a binary decision or a continuous score, and whether the actual or noisy sensitive attribute is accessed. The paper also conducts experiments on FICO and Adult datasets.