Unintended Selection: Persistent Qualification Rate Disparities and Interventions

1 Introduction . Algorithmic prediction is increasingly used for socially consequential decisions and may determine individual access to information , education , employment , credit , housing , medical treatment , freedom from incarceration , or freedom from military targeting [ 1–5 ] . This situation raises technical challenges and ethical concerns , particularly regarding the dynamics of systemic inequalities and attendant harms to society [ 6–8 ] . Nonetheless , realistically—and equitably—modeling the dynamics of disparity in machine learning remains an open problem . Research historically considered the fairness of algorithmic predictions in terms of statistical ( in ) consistencies [ 9 ] ( e.g. , across groups [ 10–16 ] or between similar individuals [ 10 , 11 ] ) , preference guarantees [ 17–19 ] , or causal considerations [ 19 , 20 ] but ignored the response of a population to new prediction policies . For instance , the proportions of potential loan applicants in each group that will seek higher wages , falsify income , or forego application might change if banks use new policies to approve or deny loans , possibly counteracting fair intent . We refer this class of fairness definitions as normative present fairness . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Efforts to model such population response [ 21–31 ] and the autonomous dynamical systems arising from mutual recursion with myopically updating prediction policies [ 21–29 ] have intensified , but it has remained to plausibly explain persistent disparities under group-independent prediction policies— i.e. , those that do not discriminate on the basis of group membership—without assuming a setting that is structurally imbalanced between groups . Our paper contributes to these efforts and considers the long-term consequences of machine learning on inter-group disparities when a sequence of classifiers induces dynamics within the rates of strategy adoption in each group . Upon adopting a dynamical framework , we note that yet another operationalization of fairness arises : the asymptotic equality of latent variables ( i.e. , those causally responsible for outcome disparities ) between groups . This notion of long-term fairness need not be consistent with normative present fairness , which may actively combat it , highlighting a tension between ends and means for fairness considerations . 1.1 Our contributions . Herein , we describe an equitable model of population response : one which does not suppose inherent differences between groups of people but endogenizes disparities by appeal to unequal initial conditions , accounting for group-specific environmental conditions as dynamical variables . We reform our notion of “ groups ” ( i.e. , subpopulations ) to appeal to natural boundaries of information exchange rather than artificially imposed classes of people . We thus offer a potentially more meaningful way to group individuals in discussions of fairness , asserting that , when considering such networks of peer exchange , “ sensitive attributes ” such as race , sex , color , etc . might not correspond to meaningful divisions of people , which depend on social context . Finally , we recognize fairness interventions as dynamical control policies that ( un ) intentionally select the future trajectories of a given system . We therefore allow ourselves to consider interventions that explicitly incorporate feedback from dynamical variables—rather than relying on fixed , prescriptive modifications of predictor loss functions . Our first contribution is to propose a model of classifier-induced group-level strategy adoption that is ( 1 ) equitable , i.e. , free from structurally asymmetric assumptions as described above , ( 2 ) capable of explaining persistent disparities under Bayes-optimal , group-independent policies , and ( 3 ) derivable from plausible , localized information exchange between individuals . Specifically , we appeal to the replicator equation , an established model for evolutionary phenomena without mutation , to model how competing strategies for qualification ( which determine true machine learning labels { 0 , 1 } , affecting agent utilities ) replicate within groups ( i.e. , isolated subpopulations that differ only in size and initial proportions of qualified individuals ) . We ground statements with a running example involving loan applications ( elaborated upon in Section 2.2 ) for which qualification ( label Y = 1 ) , interpreted as being in the public interest , implies future repayment of a loan for an applicant with feature profile X . As we avoid assuming inherent differences between groups , we consider the label-conditioned feature distribution Pr ( X | Y = y ) as group-independent and define qualification disparity in terms of differences in group qualification rates Pr ( Y = 1 ) . We formulate our model in Section 2 , emphasizing that only the profile of strategies in each subpopulation is subject to evolution—narrowly qualified by the competition between strategies for replicative success—rather than the subpopulations themselves . The persistence of disparity is thus attributed to classifier policy . Our second contribution , in Section 3 , is a rigorous examination of the dynamical system formed by the replicator equation and an updating , group-independent , Bayes-optimal classifier policy , including a characterization of its equilibrium states with linear stability analysis . We identify the set of stable interior states of the system as a stable hyperplane and show that any initial state with non-zero total qualification disparity , defined in Section 3 , will continue to exhibit non-zero disparity asymptotically if the state attracts to the stable hyperplane ( Theorem 10 ) . In this sense , we claim that qualification rate disparity persists indefinitely for this setting . Our final contribution , in Section 4 is to consider a dynamics-aware fairness intervention based on feedback control that parametrically violates classifier group-independence ( and therefore , in our setting , equalized odds [ 12–14 ] and envy-freeness [ 17 , 18 ] ) to achieve long-term fairness . We use simulation to contrast this feedback control policy to a group-independent classifier ; a policy subject to demographic parity [ 10 , 11 ] ; and “ laissez-faire ” , group-specific policies . We conclude by discussing the limitations of our model and our findings and by outlining potential future work . 1.2 Related work . Our work chiefly contributes to the literature on fairness in machine learning but also builds on prior work on “ statistical discrimination ” . The most relevant publications are those that have highlighted the importance of studying the dynamics and long-term consequences of machine learning , fairness constraints , and models of population response . In particular , Liu et al . [ 30 ] use Markov transitions to model agent responses to classification without considering classifier retraining ; D ’ Amour et al . [ 22 ] and Zhang et al . [ 23 ] reapply Markov transitions to agent attributes in the presense of classifier retraining ; Zhang et al . [ 32 ] model agents ’ decisions of whether to engage with classification based on perceived accuracy and intra-group disparity ; Coate and Loury [ 21 ] , Hu and Chen [ 27 ] , and Liu et al . [ 26 ] considered economical “ best-response ” models to agent labels with classifier updates ; and Heidari et al . [ 24 ] considered an imitation-based model of social learning in which agents choose between the strategies of other agents to maximize utility and minimize effort . Tang et al . [ 33 ] also studied the delayed and accumulated impacts of past deployed policies , but did not study the fairness implication of such impacts . Similarly , literature on “ fair bandit/reinforcement learning '' [ 34–36 ] has largely focused on technical aspects of imposing normative present fairness in a sequential setting . Our proposed model synthesizes prior conceptual innovations : First , Coate and Loury [ 21 ] , Hu and Chen [ 27 ] , and Ensign et al . [ 37 ] each considered incomplete information available to a classifier as a means to equitably endogenize persistent predictor bias , but did not consider incomplete information available to individual agents . Second , the class of response functions considered by Mouzannar et al . [ 28 ] allows group-level strategic responses to depend on existing qualification rates and may be used to endogenize persistent disparity under group-independent policies ; the cited work does not explore this direction , but , like us , the authors assume “ groups are ex-ante equal in all respect except for their qualification profiles ... and any potential coupling between groups can only happen through the different and interacting selection rates induced by the policies ” [ 28 , p. 362 ] . Atop this foundation , we provide a plausible mechanism of imitation , motivated by incomplete information available to individual agents , to justify replicator dynamics as a special case of such response functions , and we extend a dynamical analysis for a classifier forced to contend with misclassification errors . To support our use of the replicator equation to model group-level responses to classification , we cite the imitation-based derivation ( s ) of the replicator equation by Björnerstedt and Weibull [ 38 ] ; the characterization of evolutionarily stable strategies conducted by Taylor and Jonker [ 39 ] ; the analogy of memes as attributed to Dawkins [ 40 ] ; and the extensive application of the replicator equation in game-theoretic contexts as explored by Friedman and Sinervo [ 41 ] . 2 Formulation . We defer all proofs and provide them in Appendix B of the supplementary material . We consider countably many agents , n ≥ 2 groups , and a single classifier . Until Section 2.2 , our setting matches that of Coate and Loury [ 21 ] but treats n groups and a more granular classifier utility function . We ground statements with a running example : a regional bank ( classifier ) serving several isolated communities ( groups , subpopulations ) by offering standardized loans for which every individual ( agent ) applies . Alternative examples include hiring decisions [ 21 ] or college admissions . Agents belong to groups , interpreted in Section 2.2 and consistent with isolated communities in our running example , with known relative frequencies µg ∈ ( 0 , 1 ) . We vectorize these frequencies as µ. G : = { 1 , 2 , ... , n } ; ∀g ∈ G , µg : = Pr ( G = g ) ; ∑ g∈G µg = 1 ; µ : = ( µ1 , µ2 , ... , µn ) ( 1 ) For all statements of probability , we assign uniform probability mass to each agent . In addition to relative size µg , each group has a qualification rate sg ∈ ( 0 , 1 ) , which we vectorize as our state variable s. We denote the global qualification rate as s : sg : = Pr ( Y = 1 | G = g ) ; s : = ( s1 , s2 , ... , sn ) ; s : = ∑ g∈G µgsg = 〈µ , s〉 ( 2 ) Assumption 1 . No community is completely ( un ) qualified ; ∀g , sg ∈ ( 0 , 1 ) . In our banking example , a qualified ( Y = 1 ) individual will repay a loan in full if accepted ( Ŷ = 1 ) , and we presume this outcome to be desirable . The fraction of qualified individuals in community g is represented by sg . Assumption 1 states that no community is completely ( un ) qualified , and , because µg ∈ ( 0 , 1 ) , neither is the total population , i.e. , s ∈ ( 0 , 1 ) . The featureX of an agent qualified as ( Y = y ) is sampled according to a probability density function qy . In our banking example , we may interpret X as a “ credit score ” known to the bank . qy ( x ) : = pX ( x | Y = y ) ; y ∈ { 0 , 1 } ( 3 ) Assumption 2 . The qualification-conditioned distribution of features qy ( x ) is group-independent . Assumption 2 ensures that qualified individuals are statistically indistinguishable in terms of feature X across different communities—as are unqualified individuals . Given an agent ’ s qualification Y , learning G gives no additional information about X . Assumption 3. qy is differentiable and strictly positive for each y . The values of X are ordered and unified such that q1 ( x ) /q0 ( x ) is strictly increasing in X : ∀x , y , qy ( x ) ∈ ( 0 , ∞ ) ; d dx ( q1 ( x ) q0 ( x ) ) > 0 ( 4 ) Assumption 3 ensures that the feature X is “ well-behaved ” : In our example , as credit scores increase as x , the odds that individuals with that credit score x will pay off loans also increases . Finally , a classifier observes the feature X of each agent , from which it must predict the agent ’ s correct label Y using a deterministic policy π that , unless otherwise stated , remains ignorant of G. Assumption 4 . The classifier learns the true distribution Pr ( Y | X ) before choosing policy π.1 Assumption 5 . The classifier maximizes its expected utility u with risk-neutral preferences . This utility u is linear in each outcome fraction Pr ( Y = y , Ŷ = ŷ ) , and the coefficients2 Vy , ŷ ∈ ( −∞ , ∞ ) are independent of feature value X and group membership G. The classifier receives higher utility from correct predictions ( Ŷ = Y ) . Ŷ : = π ( X ) ; u ( π ) : = 1∑ y , ŷ=0 Vy , ŷ Pr ( Y = y , π ( X ) = ŷ ) ; Vy=ŷ > Vy 6=ŷ ( 5 ) In our example , Assumption 5 is consistent with a bank maximizing expected net profit , where the bank expects net profit proportional to Vy , ŷ from each individual qualified as y and approved as ŷ , independent of credit score X or community G. By Assumption 4 , the bank selects policy π knowing the stochastic relationship between qualification Y and credit score X for the region it serves . With group-independent classifier policies , having excised assumptions of inherent differences between groups in our formulation , we emphasize that unequal group qualification rates cause any statistical group-level disparities of prediction outcomes . We therefore consider eliminating differences in group qualification rates as a realization of long-term fairness in this setting . Theorem 1 . Discounting sets of measure zero , the u-maximizing , group-independent policy π is parameterized by the feature threshold φ ∈ [ −∞ , ∞ ] such that π ( x ) = 1 if and only if x > φ , where φ depends only on the global qualification rate s̄ . ŷ = π ( x ) = { 1 x > φ 0 otherwise ; q1 ( φ ) q0 ( φ ) = ξ · 1− s s ; ξ : = V00̂ − V01̂ V11̂ − V10̂ ( 6 ) When a solution in φ to the threshold equation , Eq . ( 6 ) , does not exist , φ is either ±∞ . Corollary 1.1 . The classifier ’ s feature threshold φ responds inversely to s : dφds < 0 , ds dφ < 0 . 1In practice , this distribution may be learned from sufficient data . 2We will abuse notation to write , e.g. , Vy , ŷ as V10̂ , to disambiguate the order of indices on V and , later , U . Assumption 6 . V is such that ξ ∈ ( 0 , ∞ ) ( ξ is defined in Theorem 1 , Eq . ( 6 ) ) . The threshold equation , Eq . ( 6 ) , is a reprise of Coate and Loury [ 21 ] restricted to group-independent classifier policies . In our example , the bank maximizes its utility by approving individuals with credit scores greater than φ and denying everyone else . Interpreting Assumption 6 , there exist populations for which the bank prefers to accept some applicants and reject others .

Summary: The authors consider how qualification disparities between groups can arise without structural differences between different groups. The authors formulate the problem in terms of a classifier that interacts with groups of agents, where the agents change their strategy over time in response to the classifier, and the classifier learns new classification thresholds. The authors characterize the dynamics of this system and its fixed points, showing that there exist fixed points with unequal qualification rates between groups. The authors then discuss how imposing different fairness constraints on the classifier affect the trajectory of the system, as well as a feedback control intervention that explicitly aims to reduce qualification disparities.