Unintended Selection: Persistent Qualification Rate Disparities and Interventions
Authors: Reilly Raab
Yang Liu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Fairness
Evolution
Qualification Rate Disparity
Intervention
Replicator Dynamics
Feedback Control

Abstract: Realistically---and equitably---modeling the dynamics of group-level disparities in machine learning remains an open problem. In particular, we desire models that do not suppose inherent differences between artificial groups of people---but rather endogenize disparities by appeal to unequal initial conditions of insular subpopulations. In this paper, agents each have a real-valued feature $X$ (e.g., credit score) informed by a ``true'' binary label $Y$ representing qualification (e.g., for a loan). Each agent alternately (1) receives a binary classification label $\hat{Y}$ (e.g., loan approval) from a Bayes-optimal machine learning classifier observing $X$ and (2) may update their qualification $Y$ by imitating successful strategies (e.g., seek a raise) within an isolated group $G$ of agents to which they belong. We consider the disparity of qualification rates $\Pr(Y=1)$ between different groups and how this disparity changes subject to a sequence of Bayes-optimal classifiers repeatedly retrained on the global population. We model the evolving qualification rates of each subpopulation (group) using the replicator equation, which derives from a class of imitation processes. We show that differences in qualification rates between subpopulations can persist indefinitely for a set of non-trivial equilibrium states due to uniformed classifier deployments, even when groups are identical in all aspects except initial qualification densities. We next simulate the effects of commonly proposed fairness interventions on this dynamical system along with a new feedback control mechanism capable of permanently eliminating group-level qualification rate disparities. We conclude by discussing the limitations of our model and findings and by outlining potential future work.

Unintended Selection: Persistent Qualification Rate Disparities and Interventions
Reilly Raab
Yang Liu
Computer Science and Engineering University of California, Santa Cruz Santa Cruz, CA 95064 reilly@ucsc. edu
Computer Science and Engineering University of California, Santa Cruz Santa Cruz, CA 95064
yangliuQucsc.edi
Abstract
Realistically- and equitably-modelin the dynamics of group-level disparities in machine learning remains an open problem. In particular, we desire models that do not suppose inherent differences between artificial groups of people-but rather endogenize disparities by appeal to unequal initial conditions insular subpopulations. In this paper. agents each have real-valued feature X (e.g., credit score) informed by "true" binary label representing qualification (e.g., for a loan). Each agent altemately (1) receives a binary classification label Y (e.g., loan approval) from Bayes-optimal machine learning classifier observing X and (2) may update their qualification Y by imitating successful strategies (e.g., seek a raise) within an isolated group G of agents which they belong. We consider the disparity of qualification rates Pr(Y 1) between different groups and how this disparity changes subject a sequence of Bayessoptimal classifiers repeatedly retrained on the global population. We model the evolving qualification rates of each subpopulation (group) using the replicator equation, which derives from a class of imitation processes. We show that differences in qualification rates between subpopulations can persist indefinitely for set of non-trivial equilibrium states due to uniformed classifier deployments, even when groups are identical in all aspects except initial qualification densities. We next simulate the effects commonly proposed fairness interventions on this dynamical system along with a new feedback control mechanism capable of permanently eliminating group-level qualification rate disparities. We conclude by discussing the limitations of our model and findings and by outlining potential future work.
Introduction
Algorithmic prediction is increasingly used for socially consequential decisions and may determine individual access to information, education, employment, credit, housing, medical treatment, freedom from incarceration, freedom from military targeting ([]5). This situation raises technical challenges and ethical concerns, particularly regarding the dynamics of systemic inequalities and attendant harms society ( Nonetheless, realistically- and equitably-
the dynamics of disparity in
machine learning remains an open problem.
Research historically considered the fairesss of algorithmic predictions in terms of statistical (in)consistencies 99] (e.g., across groups [6 or between similar individuals [10 II), pref- erence guarantees [1711), or causal considerations [19 20 but ignored the response of population to new prediction policies. For instance, the proportions of potential loan applicants in each group that will seek higher wages, falsify income, forego application might change if banks use new policies to approve or deny loans, possibly counteracting fair intent. We refer this class of fairness definitions normative present fairness.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Efforts to model such population response [21-31 and the autonomous dynamical systems arising from mutual recursion with myopically updating prediction policies [2T 29 have intensified, but has remained to plausibly explain persistent disparities under group-independent prediction policies- Le.. those that do not discriminate on the basis of group memhership- without assuming a setting that is structurally imbalanced between groups. Our paper contributes to these efforts and considers the long-term consequences of machine learning on inter-group disparities when a sequence of classifiers induces dynamics within the rates of strategy adoption in each group. Upon adopting dynamical framework, we note that yet another operationalization of fairness arises: the asymptotic equality of latent variables (i.e., those causally responsible for outcome disparities) between groups. This notion of long-tery term faimess need not be consistent with normative present faimess, which may actively combat it, highlighting& tension between ends and means for fairness considerations.
and
1.1 Our contributions
Herein, we describe an equitable model of population response: one which does not suppose inherent differences between groups of people but endogenizes disparities by appeal to unequal initial conditions, accounting for group-specific environmental conditions as dynamical variables. We reform our notion of "groups" (i.e., subpopulations) to appeal to natural boundaries of information exchange rather than artificially imposed classes of people. We thus offer a potentially more meaningful way to group individuals in discussions of fairness, asserting that, when considering such networks of peer exchange, "sensitive attributes" such as race, sex, color, etc. might not correspond to meaningful divisions of people, which depend on social context. Finally, recognize fairness interventions as dynamical control policies that (unjintentionally select the future trajectories of a given system. We therefore allow ourselves to consider interventions that explicitly incorporate feedback feedbbackeee from dynamical variables- -rather than relying on fixed, prescriptive modifications of predictor loss functions.
Our first contribution is to propose model of classifer-induced group-level strategy adoption that is (I) equitable, ie., free from structurally asymmetric assumptions as described above, (2) capable of explaining persistent disparities under Bayes-optimal, group-independent policies, and (3) derivable from plausible, localized information exchange between individuals. Specifically, we appeal to the replicator equation, established model for evolutionary phenomena without mutation, to model how competing strategies for qualification (which determine true machine learning labels {0, 1}. affecting agent utilities) replicate within groups (i.e., isolated subpopulations that differ only in size and initial proportions of qualified individuals). We ground statements with a running example involving loan applications (elaborated upon in Section 2.2 for which qualification (label Y = 1), interpreted as being in the public interest, implies future repayment of a loan for an applicant with feature profile X. As avoid assuming inherent differences between groups, we consider the label-conditioned feature distribution Pr(X Y =y=1 y) as group-independent and define qualification disparity in terms of differences in group qualification rates Pr(Y = 1). We formulate our model emphasizing that only the profile of strategies in each subpopulation is subject to evolution narrowly qualified by the competition between strategies for replicative success-rather than the subpopulations themselves. The persistence of disparity is thus attributed to classifier policy.
in Section -
Our second contribution, in Section is rigorous examination of the dynamical system formed by the replicator equation and an updating, group- booup-nddependent Bayes-optimal classifier policy, including characterization of its equilibrium states with linear stability analysis. We identify the set of stable interior states of the system as a stable hyperplane and show that any initial state with non-zero total qualification disparity, defined in Section 3 will continue exhibit non-zero disparity asymptotically if the state attracts to the stable hyperplane (Theorem 10P. In this sense, we claim that qualification rate disparity persists indefinitely for this setting.
Our final contribution, in Section 4 is to consider a dynamics-awar fairness intervention based on feedback control that parametrically violates classifier group-independence (and therefore. in our setting, equalized odds 12 and envy-freeness [7,188] to achieve long-term fairness. We use simulation to contrast this feedback control policy to group-independer classifier; policy subject to demographic parity [10 [I]: and "laissez-faire faire", group-specific policies. We conclude by discussing the limitations of our model and our findings and by outlining potential future work.
1.2 Related work
Our work chiefly contributes to the literature on fairness in machine learning but also builds on prior work on "statistical discrimination". The most relevant publications are those that have highlighted
2

---

the importance of studying the dynamics and long-term consequences of machine learning, fairness constraints, and models of population response. In particular, Liu et al. [0] use Markov transitions to model agent responses to classification without considering classifier retraining: D Amour et al.
[22] and Zhang et al. [23] reapply Markov transitions to agent attributes in the presense of classifier retraining; Zhang et al. [32] model agents' decisions of whether to engage with classification based on perceived accuracy and intra-group disparity; Coate and Loury [21], Hu and Chen [27], and Liu et al. [26] considered economical "best-response" models to agent labels with classifier updates; and Heidari et al. 224] considered an imitation- -based model social learning in which agents choose
between the strategies of other agents to maximize utility and minimize effort. Tang et al. 33 also
studied the delayed and accumulated impacts of past deployed policies, but did not study the faimess implication of such impacts. Similarly, literature "fair bandiv/riinorremmen learning" largely focused on technical aspects of imposing normative present fairness in sequential setting.
B6 has
Our proposed model synthesizes prior conceptual innovations: First, Coate and Loury [21]. Hu and Chen [27] and Ensign et al. (37] each considered incomplete information available a classifier as a means to equitably endogenize persistent predictor bias, but did not consider incomplete information available to individual agents. Second, the class of response functions considered by Mouzannar et al. (8 allows group-level strategic responses to depend existing qualification rates and may be used to endogenize persistent disparity under group-independent policies; the cited work does not explore this direction, but, like us, the authors assume "groups ex-ante equal in all respect except for their qualification profiles...add any potential coupling between groups can only happen through the different and interacting selection rates induced by the policies" [28 p. 362]. Atop this foundation, we provide a plausible mechanism of imitation, motivated by incomplete information available to individual agents, to justify replicator dynamics extend a dynamical analysis for a classifier forced to contend with misclassification errors.
special case of such response functions, and
To support use of the replicator equation to model group-level responses to classification, cite the imitation-based derivation(s) of the replicator equation by Björnerstedt and Weibull [38]; the characterization of evolutionarily stable strategies conducted by Taylor and Jonker (39]: the analogy of memes as attributed to Dawkins [4]]: and the extensive application of the replicator equation in theoretc contexts explored by Friedman and Sinervo [4).
2 Formulation
We defer all proofs and provide them in Appendix B of the supplementary material.
We consider countably many agents, 2 2 groups, and single classifier. Until Section 2.2 setting matches that Coate and Loury [21] but treats n groups and a more granular classifier utility function. We ground statements with running example: regional bank (classifier) serving several isolated communities (groups, subpopulations) by offering standardized loans for which every individual (agent) applies. Alternative examples include hiring decisions [21] college admissions.
Agents belong to groups, interpreted in Section 2.2 and consistent with isolated communities in our running example, with known relative frequencies My E (0, 1). We vectorize these frequencies as μ.
G = (1,2 n3: Vg E G, M9 = Pr(G = =g]; Σ Ug = 1; f4 == (M1, Pn)) gEG
(1)
For all statements of probability, we assign uniform probability mass to each agent.
In addition to relative size lg. each group has qualification rate $9 (0, 1), which we vectorize as our state variable s. We denote the global qualification rate as s:
8g = Pr(PY==IGG =9); = (8,,,8,,..,, sn)i
B
Σ Mg = (1,
(2)
gEg
Assumption 1. No community is completely (un)qualified;
4g. 8y E (0, 1).
In our banking example, a qualified (Y 1) individual will repay a loan in full if accepted (Y- 1), and we presume this outcome to be desirable. The fraction of qualified individuals in community is represented by 8g. Assumption T states that no community is completely (un)qualified, and, because Mg E (0, 1), neither is the total population, i.c., 3 e (0, 1).
3

---

Table 1: Agent-specific variables forming Markov chain.
Variable
Meaning
Domain
Realizations
G
group
G 21,2,...,71 n.
9, h, i,
qualification
[0,1] ie., (unqualified, qualified)
y
X
feature
(-00, ())
Y
classification
[0,1] ie., (reject, accept)
The feature X an agent qualified as (Y 3/) is sampled according to a probability density function qy. In our banking example, may interpret X as a "credit score" known to the bank.
g, = px Y=
= y); E [0,
Assumption 2.
The qualification-conditann distribution features 4y(x)) is group-independent.
Assumption ensures that qualified individuals are statistically indistinguishable in terms feature X across different communities as are unqualified individuals. Given an agent's qualification Y, learning G gives no additional information about X.
Assumption 3. qy differentiable and strictly positive for each y. The values of X are ordered and unified such that q1 (o)/go(() is strictly increasing in X:
d
41 2
Vx, ys (x(x)) E 20);
> 0
dx
T0 x
Assumption 3 ensures that the feature X is "'wellbehaved"" In our example, as credit scores increase as x, the odds that individuals with that credit score will pay off loans also increases.
Finally, a classifier observes the feature of each agent, from which it must predict the agent's correct label Y using a deterministic policy æ that, unless otherwise stated, remains ignorant of G. Assumption 4. The classifier learns the true distribution Pr(Y X) before choosing policy T
Assumption 5. The classifier maximizes its expected utility u with risk-neutral preferences. This utility u is linear in each outcome fraction Pr(Y = =y Ý = 9), and the coefficient Vy.9 E (-00, 20) are independent of feature value X and group membership G. The classifier receives higher utility from correct predictions (Y =Y).
p
= :X)=
(X)); u(T)= = Σ Vy Pr(Y =. 0); Vy=ū
V Vusti
(5)
v.9=0
In our example, Assumption is consistent with a bank maximizing expected net profit, where the bank expects net profit proportional Vy.9 from each individual qualified y and approved as ÿ. independent of credit score X or community G. By Assumption 4 the bank selects policy π knowing the stochastic relationship between qualification Y and credit score X for the region serves.
With group-inden indepenedeen classifier policies, having excised assumptions of inherent differences between groups in formulation, emphasize that unequal group qualification rates cause any statistical group-level disparities of prediction outcomes. We therefore consider eliminating differences in group qualification rates realization of long-term fairness in this setting.
Theorem 1. Discounting sets of measure zero, the u-maximizing, group-inder inup---ndeeedent policy x is
parameterized by the feature threshold é E [- oo) such that (xx) if and only ifx > O where & depends only the global qualification rate 8.
{1 x>6
1 3 (0) = (o(
Vi
Vii
ÿ (=)
E
(6)
otherwise
3
Vii
V
When solution in & to the threshold equation, Eq. 6 does not exist, 6 is either to0.
Corollary 1.1. The classifier's 's feature threshold 6 responds inversely de
de
0, 0.
In practice, this distribution may learned from sufficient data.
We will abuse notation to write, e.g., Vy.9 as Via. disambiguate the order of indices V and, later, U.

---

Assumption 6. V is such that & E (0, 20) (5 is defined in Theorem 1 Eq. (6
The threshold equation, Eq. 6 is reprise of Coate and Loury 21 restricted to croup-independent classifier policies. In example, the bank maximizes its utility by approving individuals with credit scores greater than & and denying everyone else. Interpreting Assumption there exist populations for which the bank prefers to accept some applicants and reject others.
2.1 T'mee-deennennee
We model our system in discrete time for semantic reasons, acknowledging that a learning process consistent with Assumption 4 requires time, although the mathematics generalize to continuous time without issue 3) Where required, we will denote time-dependence in square brackets [4]. Where we omit this explicit dependence, in all prior expressions, it is understood that all variables in expression correspond to the same time t.
Assumption 7. The relative sizes of groups Fg. quulifeattin-conditione feature distributions 4y and classifier utility coefficients V.. all iime-independent. All prior assumptions hold independently for each time step.
2.2 Replicator dynamics
Anticipating algorithmic classification, how do agents decide whether become qualified? In banking example, we imagine that individuals must "invest" or "apply capital" to be able to pay back loans and that the "rationality" of doing so depends on what they know about the classification policy and potential outcomes- which they must estimate from incomplete information provided by peer examples. To deal with the uncertainty updating personal qualification by imitating the strategies of others based on popularity and "success". For example, if your friend chose become qualified for a loan and now runs small business, the success of the business may induce you to seek qualification yourself by first building credit history; many your neighbors receive loans despite being unqualified and appear successful investing in speculative assets, you may infer that qualification is waste of resources.
limited examples, we imagine the emergent heuristic of
Agents respond to classifter
Update of variables
Classifier chooses policy
Px(z, Y v)
Reject Aroepi
s/t 1| -a(d
Y
group 5 qualification rate Bg
classifier policy a
Y
fitness W/ of strategy
d(8)
Figure 1: Our model appeals to the replicator equation Eq. 7 to model population response and considers a Bayes-optimal, group-independent classifier policy x with Teature threshold ø (Eq. (6). right pane). When coupled (middle pane), these equations give rise to an autonomous dynamical system. We wish to understand how the vector group gualification rates
state variable, changes in time (left pane).
Björnerstedt and Weibull 38 have shown that imitation in this form, whereby agents stochastically update strategies weighted by success and popularity yields the (continuous time) replicator equation, which we use in its discrete time form, as detailed by Friedman and Sinervo
41
Wit W
L + 1] [4
Wa W18 Wo - -g); Vy, W, 2 0
"The continuous replicator equation appears in Björnerstedt and Weibull 38 & Friedman and Sinervo 41 'Weighting by popularity effects "preferential attachment" for the success of a strategy in a subpopulation and thus introduces dynamical inertia. Replicator dynamics also arise when agents update strategies with (Poisson distributed) expected periodicity that is affine in the success of one's current strategy (38).
5

---

we model as independent of feature X and group G. Following Björnrsteedt and Weibull 38 we may derive the fitness Wy in terms of expected "success" U. of each qualificatoon-laassffcaaiio outcome (v, ū):
Here, Wy is the fitness of strategy y. which, by Theorem
Assumption 8. The fitness of strategy (Y =y), denoted Wy is affine in the average success Uy.b of qualification (Y = y) with classification (Y = ÿ). Uy. is time- feature- and group-independent. Without loss of generality, we restrict Uy E [0, 00) and drop the constant bias term from each W
Pr(Y = 9 = y, =. n
(8)
8=0
Theorem 2. The fitness y of strategy = y in group g is feature- and group-independen Vy.g, w Wy
Intuitively, "success" U may interpreted as utility or payeff to each agent when agents align strategy adoption with personal incentives, but, fundamentally, W corresponds to the relative success of the strategy in replicating, i.e, spreading between individuals. The strategies (non)qualification are thus subject to evolutionary pressures, competing out-replicate each other in an environment shaped by perceptions of classifier policy, Notably, the fitness a strategy depends only on the classifier- -ot group membership G. Agents remain identically modelled across all groups.
Assumption 9. The success of (non) (non)qualification sensitive to classification, and the expected success for qualified individuals increases with classifier acceptance: Uoi # Vo6: Uii > Uí
Assumption 10. Each group has the properties of a closed population in which qualification, strategy or meme [40], competes with non non-quaiffication free from exchange with other groups.
significant that we model population updates as independent for closed populations, as this restricts our interpretation of groups, which must be functionally impermeable to the exchange of qualification strategies. To precisely delineate between real-world examples of "groups" akin to disassociating "cultures", which also imply boundaries of exchange but generally intersect. Having noted that "sensitive attributes" such as race, sex, color, etc. may not correspond to meaningful divisions between people (which depend on social context), we instead qualify a group by the extent to which it satisfies Assumption TO As an open question, we ask whether imposing arbitrary demographic dependent policies may catalyze the formation of groups of strategic peers, but will consider insular social groups or isolated communities as canonical examples.
3 Dynamics
The threshold equation, Eq. (6):
(it((s(t),, and the replicator equation, Eq. 7
: sit + [](o(t). s(t)).
may be coupled yield an autonomous dynamical system s|( + (s{{() that evolves in time. To analyze it, we first generate a useful set of coordinates to compliment 8 and track qualification rate disparities, defined by the differences in 8g between groups. We then note the importance of (i(0)= Wo( to the overall dynamics of the system, and use it to identify non-trivial equilibrium states. To interpret this section for our example, we ask how community-specii
loan qualification rates change as individuals imitate successful strategies in their isolated communities, while assuming that the bank maximizes profit using group- -oop-inddeenndent credit thresholds for loan approval.
Definition 3. Define the (signed) qualification distance from group h to group g as
8(g. h) == 8g 8h, g, h E 21, ..., n1
(10)
We next define the vector D comprising (n 1) linearly-independent qualification distances between sequential pairs of subpopulations:
D = (11, 2), 8(2,),, 5(n - 1()-1,n))
(11)
The components of D and value of together yield a complete set of coordinates to describe the state of the dynamical system, which we may exchange for the original vector of qualification rates s = ($1, 52, 8x) via a non-orthogonal, linear change of basis (See Appendix B):
a-1 h
8g = 8 +
8(h, + 1)
ΣΣ
in 5 h 1)
Vg E
(12)
h=
h=1 1=1
Let us denote the state vector in new coordinate system a (8(1,2), (((,,)),((,,3) 8(2,3), .... .....8@n- 8(n- 1,m),

---

Definition 3.1. For 2 1, define a state's p-total qualification rate disparity as the p-norm of D;
|Pl| = 6(g.9 ((o.e+1)l""*
(13)
Remark 4. States with common & value form a hyperplane 3= (p. s) (Eq. (2) by definition.
Theorem 5. The nullity of any p-total qualification rate disparity is preserved in time.
p p21:
|it|l =0 + | + 1]1pp-
(14)
Theorem5 Shighlights a weak notion of the persistence of disparity within the system sans intervention: Any state that possesses some non-zero total qualification disparity (defined as some chosen p-norm of D) must always exhibit some non-zero total qualification disparity with any finite time horizon. In example, if some communities start more qualified than others, the qualification rates of different communities will not naturally equalize in any given lifetime. Note that this statement is insufficient to address the limit + oo, however. For stronger result that includes this limit (Theorem 10 we first characterize the system's equilibrium states.
3.1 Equilibrium
Definition 6. The system as a whole is at equilibrium when, for all E 9 simultaneously, 8y is stationary in time:
at equilibrium
Vgg E G, 36o s.t. VE 2 to. sg = s [4o]
(15)
replicator dynamics an instance of the more general family of monotone dynamics, with which all equilibria are shared [8 41).
Note:
Theorem 7. Disregarding boundary states by Assumption 7 the replicator equation, Eq. implies
sgn (3(t + 3(t) sgn (Wi(dlt)) Wolo(())
(16)
Theorem 8. It is necessary and sufficient for system at equilibrium that W = Wo or for the system to occupy some vertex of the state space.
[i Wo (gg 8g (0
(internal equilibrium)
at equilibrium
(17)
(trivial equilibrium)
Theorem g indicates that the conditions for internal equilibrium are described by the zeros of the function (Ø))) Wo( as depicted in Fig. 2 and, by the thresh- old equation, Eq. (6). has dynamical dependence only It follows that only certain values of 3 support internal equilibrium, and each value corresponds to a hyperplane in state space (Remark 4
W1 Wi-1i Wi
U1 Usp
Un Uni
Theorem 9. (Ø) Wo(Ø) is strictly quasi-concave in 6. This guarantees that no more than two zeros of the function W1 Wo exist.
Figure 2: Wi (d) Wo(d) (blue curve) a strictly quasi-concave function of Φ. ø* denotes the unique local extremum. The di- rection of the arrows is consequence of Theorem 7 and Corollary [.]
We denote the possible zeros of W, Wo as o+ and Φ= where the sign in the superseript indicates the local slope of the function. These zeroes correspond to parallel hyper- planes in state space that comprise all interior equilibria of the system. Whether + corresponds an (un)stable equilibrium hyperplane may be determined by Theorem and the sign of 06 linear stability analysis.
à (W1 Wo): Only s(6)) )) is stable, and we will verify this fact with
Theorem 10. If the state of the system asymptotically approaches an internal equilibrium, the nullity of p-total qualification rate disparity is preserved in the limit of infinite time.
121: lim (W1 Wo) = = (IlIeell, = + lim
(9t'll,=0)
(18)
Theorem 10 formalizes the critical observation that any state that attracts to the stable equilibrium hy- perplane, unless initially free from qualification disparity, will forever exhibit some total qualification disparity. This is more robust notion of the persistence of disparity in system than Theorem
7

---

3.2 Stability
For our regional banking example, we may imagine that qualification rates settle into a stable pattern in which some communities have a higher average qualification rate than others. How robust is this pattern of inequality to small fluctuations of qualification rates? Using linear stability analysis (i.e., linearizing the response of the system to small perturbations about equilibrium and asking "do perturbations amplify or dissipate?"", we show that only the o -hpperplnnee acts as a stable attractor.
First, let us denote the evaluation of an expression at equilibrium by placing a vertical line to the right of the expression with "eq" as a subscript. In light of Theorem @ and Eq. (7). we also introduce the shorthand Weq to denote an equilibrium value of W1, Wo, or, equivalently, any W . It should be noted that the value of Weq E [0, oo) still depends on the particular equilibrium state of the system.
W mq
Wo
Wi
W
Eg
(19)
og oq
mq
We linearize the system at equilibrium by constructing the Jacobian J E R corresponding to discrete time-evolution and identifying its eigenvectors and eigenvalues:
år dr
dr dr
(20)
06(1.2 where r, the state vector in (D, coordinates, is interpreted as a column vector.
08(2.3) 08(n 1, n) 0s
Theorem 11. The Jacobian J simplifies scalar multiplied by a matrix with single non-zero column v in the last position.
611.2)(1-111))
(W1 (d ) do
Wo)) / V. Y
1
dΦ
812
(21)
Weg
dz
5in-Lm][(-
leg
Σ Eyse nu (1=5a)
The eigenvalues of J determine the stability of the system at equilibrium.
Corollary 11.1. At equilibrium, any state displacement vector with zero 3 component is an eigenvec- tor of J with eigenvalue O, while v is an eigenvector of J with eigenvalue X:
X == Σ 9EG
EEgso(L (g (1-)))))))))))))))))))()))
s0)
1 dø d3
d do
(W1
Wo)
(22)
Won
Perturbing (displacing) a state vector r at an internal equilibrium by altering any combination of coordinates appearing in while leaving fixed- specifies motion the hyperplane occurring in neutrally stable equilibrium (i.e., a displacement vector with zero & component has null eigenvalues at internal equilibrium. See Strogatz 42)). An internal equilibrium is stable perturbations in v. leaving the hyperplane, iff À is negative (and, in discrete-time, > -2 to forbid over-corrections) [42]
Corollary 11.2. As consequence of Corollary 1.1 which states de < 0, the eigenvalue À in
मे Eq. 22 is negative, (and the associated equilibrium hyperplane stable) iff σē d Wo)l en > 0.
This prescribes precisely the value ot+ for the stable equilibrium hyperplane.
Interventions
In the dynamical setting we have characterized, we now explore "faireess interventions". which substitute the set of policies that the classifier may choose from, possibly permitting group-specific decision rules g.. We first observe that for the default policy with group-independent feature threshold é one commonly cited standard of normative present fairesss is automatically satisfied.
Definition 12. Equalized Odds [12H14] requires that a classifier's decisions Y. given by policy n. misclassify (un)qualified agents at equal rates across groups:
4g,h E 9, Vy, D E {0, 1}, Pr(ý = Y Y -
(23)
Theorem 13. For policies defined by group-specific thresholds &.p the equivalence of these feature thresholds (Vg, Φy Φ) necessary and sufficient to satisfy Equalized Odds given the group- independence of each qy (Assumption 2
8

---

By Theorem 13 a indeeeeeee policy satisfies Equalized Odds (e.g., the bank accepts/rejects (un)qualified loan applicants proup-independent rates). yet disparities may persist (Theorem TO). This indicates a counter-example to reliance on Equalized Odds for long-term fairness in our model, viz, the optimal group-independent threshold classifier we have studied so far.
Corollary 13.1. Equalized Odds does not imply long-term fairness in model.
We next ask whether small displacement a group- -ouupinndeenndent threshold o near the t hyperplane. which we interpret as universal subsidy (or penalty), can diminish qualification rate disparities. Theorem 14. (e)) perturbations of a group group-independent & at internal equilibrium induce motion, which, to furs-orrder approximation (i.e., ignoring (c²2) terms), is parallel to the eigenvector v.
As a consequence of Theorem 14 while need not be orthogonal to the equilibrium hyperplane. and a universal subsidy may decrease qualification rate disparity while applied (settling on new equilibrium hyperplane with different, though persistent disparities), the system is stable to such per- turbations at ot as characterized by linear system response and will relax to the original equilibrium state when the intervention is removed. To permanently change qualification disparities, a temporary universal subsidy (penalty) must rely on the non-linear response of the system and is therefore liable to require large perturbations to the classifier's s threshold d. This finding compels us to consider interventions with group-dependent threshold perturbations end, we hereafter generalize our classifier such that it independently classifies each group g according to group- threshold dg. We denote the vector of these thresholds d (01, 02.... .... On) and assume that, prior to some perturbative intervention, Ø9 @ for each E G.
deeunded thresholds. To this
Definition 15. Demographic Parity [1] requires that a classifier's 's decisions Y, given by policy π, are positive (Y' 1, e.g., accepting loan application) at equal rates for all groups:
Vg, h E G, Pr(ř = 1 | = g) = Pr(Ý = = =
(24)
Definition 16. Laissez-Faire allows separate, t-maximizing threshold Φ, for each group.
Theorem 17. Demographic parity requires sign-heterogeneous. group-dependent changes to the Laissez-Faire values of 0g when non-trivial (does not uniformly accept (reject)).
Satisfying demographic parity in our setting requires the solution differential equation in dy (Appendix B), which we do not rigidly constrain. We therefore rely on numerical simulation, rather than analytical tools, to evaluate this intervention for our system.
Feedback control Arbitrary state transitions in the equilibrium hyperplane may be permanently effected by greup-dependent perturbations to o which we derive from linear system response at equilibrium. Specifically, to diminish a specific qualification distance ö(g.g + 1) for given g. + may be perturbed by vector quantity A, (Ag01, A902, Ayda).
Theorem 18. On the stable internal equilibrium hyperplane, infinitesimal perturbattinn of & by 3g
A Φ- = -eōgg,9 9 + 11)
0gg
Og
13g
(25a)
81 81)
80(1 - 8z) 89+1 8y+1) Bg = ----pp (mm + +12 +... +Mg)
8n 8n)
0g == (Ag+1 + pg+2 + + pn),
(25b)
will induce motion in the system preserving and each 8(h, h 1or h # g. The value of 6(g. 9 + 1)
will be diminished by ratio proportional to the strength parameter e >> 0.
Perturbations of the form Ag T may be composed linearly for multiple values of g. In particular, when e is a universal quantity, we may determine the total perturbation to & necessary to simultancously and proportionately decrease all qualification distances for any given state on the stable equilibrium hy hyperplane. Let us denote this total perturbation as AQ Egeg Ag = (Adi, Ad2, ..., AQ.). Component-wise, A is given by
Σ Co
Σ Bno(b,
(26)
8g 8q)
We note that the proposed feedback control mechanism depends only on the known constants My and feedback in terms of current qualification distances 8. In addition, the force of the intervention can be tuned by setting the strength parameter Finally, we remark that this mechanism can be composed with global perturbations of Φ, i.e., universal subsidy in the manner of Theorem 14 to intervene without rejecting any agents that would have been accepted under group-independent policy.

---

(c= 0.1) Feedback Control
Equal qua qualification rates
Demographic Parity
croup-Independent Policy
Laisscz-Fus Faire
Gropp quailfcction 9t
0.9
Group Meneei
0.7
Pr
0.5
0.3
0.1
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
Group qualification rate
Group I qualification rate
Group qualification rate
Group qualification rare
Figure 3: Simulated dynamics for two groups of equal size, subject to different global interventions. Streamlines approximate system time evolution. qo and are Gaussians with unit variance and have means -1 and 1, respectively. Other examples and rendered dynamical variables are provided in Appendix C. For this example, (Uo6 0.1; Ui 5.5; 16 0.5; 1 1.0; Voo 0.5; Voi -0.5; V1 0.25; Vii 1.0).
We compare interventions by appeal to simulation, choosing a setting that guarantees a single, stable average qualification rate 3" under cooup-nddependem policies (GI) (Fig. 3). We consider trade-offs between normative present fairness (F) (e.g., demographic parity (DP) or equalized odds (EO)) and long-term fairness (. for which the dynamics must converge to the line demarcating equal qualification rates. Darker shading (blue) implies a higher absolute acceptance rate for Group 1, which, by the setting's s symmetry, is the same for Group when reflected across the aforementioned line; reflectional asymmetry violates DP. Under compulsory DP (first pane), the system violates FLT, settling into a "patronizing equilibrium under affirmative action",, as coined by Coate and Loury [21, in which agents from a less-qualified group are patronized (e.g., granted loans despite nonqualification) by the classifier (ef. the upper-left corner, with low qualification and high acceptance rates for Group 1). States under GI (second pane). which satisfies the EO notion of ENP by Theorem 13 converge to a line of constant 3 (cf. Remark (4) while preserving qualification disparities (cf. Theorem 10). FLI expected from a laissez-faire (LZ) policy (last pane), which adopts group-specifts policies and thus decouples all group dynamics: Each group must converge to s* separately. Still, LZ satisfies neither the DP (by reflectional asymmetry) nor EO (by Theorem [ notions of FNP. In contrast, feedback control (third pane) achieves FuR by conceding c-saal,, parametric violations of FNP (EO) (See Appendix C for plots of classifier error rates in this setting).
5 Discussion and limitations
The novelty of our contribution is the demonstration of persistent qualification rate disparities in a symmetric setting consistent with plausible mechanisms of population response-sustained by the careless deployment of machine learning and myopic fairness interventions. We submit that, given the many charitable assumptions of our model to achieve perfect structural equality between groups, any reasonable fairness intervention should succeed in responsibly rectifying disparities here, anywhere. Moreover, we have laid bare inherent tensions that can exist between the means and ends of fairness considerations in a dynamical context, demonstrating the potential incompatibility of immediate and long-term notions of faimess.
We acknowledge that our model is simplistic, but such simple cases must be well-understood as a first step towards further, equitable models of population response. We regard the requirement of strictly isolated groups as the most tenuous assumption of our model and conjecture that even relatively weak inter-group exchange of strategies should lead to long-term fairness our default setting. Nonetheless, we believe that a program based on incomplete agent information can successfully endogenize persistent disparities in symmetric settings more robustly. Specifically, future work may consider multiple classifiers with different task domains affecting common population; expect this extension to readily endogenize broken symmetries between group environments and conditions. We also trust that voluntary participation, as considered by Zhang et al. [32] may be modelled as an additional strategy within our framework. Regarding empirical falsifiability, we note that the dynamics of social disparity not exclusive to algorithmic classifiers (7]. and ask whether
model's 's predictions may be contrasted with existing and historical resource allocation problems.
We invite readers consider both our model and application of control theory to society through algorithmic classification, with care. We intend our work to reform the misapplication of machine learning, inappropriate modelling assumptions, and myopic notions of faimess.
10

---

Acknowledgments and Disclosure of Funding
We believe our paper benefited significantly from NeurIPS 2021 review process. We thank our peer-reviewers for their helpful feedback and constructive criticism of the originally submitted version of this paper. We also thank the NeurlPS Area Chairs, Senior Area Chair, Ethics Chairs, and Program Chairs for their discussion of relevant sociotechnical issues and helpful recommendations in preparation of this published version, which we believe our contributions are better framed and clarified.
This work is partially supported by the National Science Foundation (NSF) Program on Fairness in Artificial Intelligence in Collaboration with Amazon (FAI) under grant IIS-2040800 and by NSF grant CCF-2023495.
References
[1] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in algorithm used to manage the health of populations. Science, 3cience,,66666664 366(6464)):44--533, 2019.
[2] Karen Hao. The coming war on the hidden algorithms that trap people in poverty. MIT Technology Review, 2020.
[3] Cade Metz and Adam Satariano. An algorithm that grants freedom, or takes it away. The New York Times, 2020.
[4] Derek Newton. From admissions to teaching to grading, Al is infiltrating higher education. The Hechinger Report, 2021.
[5] Joe Hernandez. A military drone with a mind of its own was used in combat, U.N. says. National Public Radio, 2021.
[6] Kate Crawford and Ryan Calo. There is a blind spot in Al research. Nature News, 538(7625): 311, 2016.
[7] Allison JB Chaney, Brandon Stewart, and Barbara E Engelhardt. How algorithmic confound- ing recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM Conference on Recommender Systems, pages 224-232. ACM, 2018.
[8] Andreas Fuster, Paul Goldsmithh-Pnnkhan Tarun Ramadorai, and Ansgar Walther. Predictably unequal? The effects of machine learning on credit markets. The Effects of Machine Learning on Credit Markets, 2018.
[9] Sam Corbett-Davii and Sharad Goel. The measure and mismeasure of fairness: A critical review of fair machine learning, arXiv preprint arXiv:/806 /808.00033, 2018.
[10] Cynthia Dwork. Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
[11] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork, Learning fair representa- tions. In International conference machine learning, pages 325-333. PMLR, 2013.
[12] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning, 2016.
[13] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without dis- parate mistreatment. Proceedings of the 26th international conference on world wide web, pages 1171-1180, 2017.
[14] Alexandra Chouldechova. Fair prediction with disparate impact: study of bias in recidivism prediction instruments. Big data, 5(2):153-163, 2017.

---

[15] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259-268, 2015.
[16] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv: 1609. 05807, 2016.
[17] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, Krishna P. Gummadi, and Adrian Weller. From parity to preference-based notions of faimesss in classification. In Proceed- ings of the Itt International Conference on Neural Information Processing Systems, NIPS'17 17, page 228 238. Curran Associates Inc., 2017. ISBN 9781510860964.
[18) Berk Ustun, Yang Liu, and David Parkes. Fairnsss without harm: Decoupled classifiers with preference guarantees. In International Conference on Machine Learning, pages 6373-6382. PMLR, 2019.
[19] Matt Kusner, Joshua R Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. arXiv preprint arXiv: 700306685, 2017.
[20] Maximilian Kasy and Rediet Abebe. Fairness, equality, and power algorithmic decision- making. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, pages 576-586, 2021.
[21] Stephen Coate and Glenn C Loury. Will affirmative-action policies eliminate negative stereo- types? The American Economic Review, pages 1220-1240, 1993.
[22] Alexander D' Ammourr, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D Sculley, and Yoni Halpern. Fairness is not static: Deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accouniability, and Transparency, pages 525-534, 2020.
[23] Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellström, Kun Zhang, and Cheng Zhang. How do fair decisions fare long-term qualification? arXiv preprint arXiv:2010. 11300, 2020.
[24] Hoda Heidari, Vedant Nanda, and Krishna Gummadi. On the long-term impact of algorith- mic decision policies: Effort unfairness and feature segregation through social learning. the International Conference on Machine Learning (ICML), 2019.
[25] Min Wen, Osbert Bastani, and Ufuk Topcu. arXiv:1901.08555 2019.
Fairness with dynamics. arXiv preprint
[26] Lydia T Liu, Ashia Wilson, Nika Haghtalab, Adam Tauman Kalai, Christian Borgs, and Jennifer Chayes. The disparate equilibria of algorithmic decision making when individuals invest rationally. In Proceedings of the 2020 Conference Fairness, Accountubility, and Trunsparency, pages 381-391, 2020.
[27] Lily Hu and Yiling Chen. A short-term intervention for long-term fairness in the labor market. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1389-1398. International World Wide Web Conferences Steering Committee, 2018.
[28] Hussein Mouzannar, Mesrob Ohannessian, and Nathan Srebro. From fair decision making to social equality. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 359-368. ACM, 2019.
[29] Joshua Williams and J Zico Kolter. Dynamic modeling and equilibria in fair decision making. arXiv preprint arXiv: 1911 1.063377 2019.
[30] Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pages 3150-3158. PMLR, 2018
12

---

[31] Lily Hu, Nicole Immorlica, and Jennifer Wortman Vaughan. The disparate effects of strategic manipulation. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 259-268. 2019.
[32] Xueru Zhang, Mohammad Mahdi Khalili, Cem Tekin, and Mingyan Liu. Group retention when using machine learning in sequential decision making: The interplay between user dynamics and fairness. In Advances in Neural Information Processing Systems, pages 15243-15255 2019.
[33] Wei Tang, Chien-Ju Ho, and Yang Liu. Bandit learning with delayed impact of actions, arXiv preprint arXiv:2002.10316 2020.
[34] Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. In Advances in Neural Information Processing Systems, pages 325-333 2016.
[35] Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David Parkes. Calibrated fairness in bandits. arXiv preprint arXiv:1707.001 717777777877,,. 2017.
[36] Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in reinforcement learning. Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1617-1626. JMLR. org, 2017.
[37] Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkata- subramanian. Runaway feedback loops in predictive policing. In Conference of Fairness, Accountability, and Transparency, 2018
[38] Jonas Björnerstedt and Jörgen W Weibull. Nash equilibrium and evolution by imitation. Technical report, IUI Working Paper, 1994.
[39] Peter D Taylor and Leo B Jonker. Evolutionary stable strategies and game dynamies. Mathe- matical biosciences, 4011-2): 1978.
[40] Richard Dawkins. The Selfish Gene. Oxford University Press, 1976.
[41] Daniel Friedman and Barry Sinervo. Evolutionary Games in Natural, Social, and Virtual Worlds. Oxford University Press, 2016.
[42] Steven Strogatz. Nonlinear Dynamics and Chaos with Student Solutions Manual: With Applications to Physics, Biology, Chemistry, and Engineering. CRC press, 2018.
[43] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
13


Reviews:
[{"review_id": "og6vDv0Cs4S", "reviewer": "Reviewer_R95V", "summary": "The authors are introducing a dynamical system (agents, groups, classifiers, etc.) The agents seem to have fuzzy-like membership of the groups (randomly assigned with some prob.) and they have qualification rates for groups, etc. They have a classifier built upon a specific utility function and it is shown that it is equivalent to having a policy that works by using a simple threshold on the probability of the label conditioned on the feature (Theorem 1). This idea of fitness and replicator equations is used (Eq. 8) to build a system to analyze the stability of the dynamic system. Section 3 which covers that part is a multitude of mathematical arguments showing stability and equilibrium. Section 4 discusses what might happen if there were interventions to make the classifier follow some fairness criterion, like the equal opportunity classification. And it seems to be a direct consequence of this framework. \n\n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe new contribution is the feedback control mechanism which seems to do much better at bringing everyone to a good equilibrium compared to group based fairness measures (The Fig. 3 shows this behavior in an empirical setting as well). Section 4 seems to be interesting. It discusses what might happen if there were interventions to make the classifier follow some fairness setup. Equal opportunity classification seems to be a direct consequence of this kind of setup. \n\nThe goal is clear, but a paragraph or section on the applications/practicality in real-world settings could make the significance more pronounced.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The writing is technical but not unclear. The writing has to be technical in this case. However, some sections could still use some updates. For example, the introduction could be written such that it sets the stage to explain what is being addressed in this paper, and why it would be important and novel in more details. \n\nThe impact it would have due to the assumptions that might constraint the practical applicability of it should be addressed in a paragraph or two as well. For example, the global knowledge of the classifier is one of the assumptions that might make it impractical. In a way, the setup is just relying on the conditional distribution to make predictions. Knowing the conditional probability, is it a common assumption in this space? This needs more detailed explanations and justifications. Also, the claim that this, in practice, can be learned from enough data should be followed by some tangible example or clarification.\n\nAll in all, section 5 is good in the sense that the authors discuss their own limitations.\n\n\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "e164sSGcsxB", "reviewer": "Reviewer_2tow", "summary": "The paper considers the problem of fairness. Given a binary quality, a classifier, and populations groups, how does the interaction of the evolution of the quality and the classifier accuracy evolve? The authors propose a model for this that crucially avoids tying group membership with particular attributes (like race or religion), and instead assume just that the groups are isolated. They then analyze the dynamical system and show that if there are initial disparities, they may persist forever, and finally apply fairness interventions from literature to their model.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an excellent paper and was a pleasure to read, despite it being outside of my area. It addresses a key problem and does so crisply and elegantly. I cannot speak to the technical depth of the results (which is unfortunate for a theory paper) but the quality of the technical writing is great. I don't have many comments, except for I would add a graphic or something of the model to help the reader, and that the interventions section currently is almost incomprehensible for someone not familiar with the related work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "NFDpf9-HsvJ", "reviewer": "Reviewer_YFGZ", "summary": "The authors consider how qualification disparities between groups can arise without structural differences between different groups. The authors formulate the problem in terms of a classifier that interacts with groups of agents, where the agents change their strategy over time in response to the classifier, and the classifier learns new classification thresholds. The authors characterize the dynamics of this system and its fixed points, showing that there exist fixed points with unequal qualification rates between groups. The authors then discuss how imposing different fairness constraints on the classifier affect the trajectory of the system, as well as a feedback control intervention that explicitly aims to reduce qualification disparities.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well-written, and the claims are well-supported. The statements are refreshingly precise. My main concern is that the paper could do more to relate its results to existing work in the classical theory of statistical discrimination from economics and with more recent work in fair ML that has built on it.\n\nMost pressingly, I think it is important to discuss how this model is similar or different from the Statistical Discrimination model (https://en.wikipedia.org/wiki/Statistical_discrimination_(economics)) proposed on the 70's by Phelps (https://www.jstor.org/stable/1806107) and Arrow (https://econpapers.repec.org/paper/priindrel/30a.htm, https://www.jstor.org/stable/2646963). In addition, there has been work in this line that considers the impacts of various interventions such as affirmative action (see Coate and Loury 1993 https://www.jstor.org/stable/2117558). The evolution dynamics seem quite similar between these models, so I would not be surprised if they are equivalent. I think the paper also needs to engage more with the Hu and Chen paper that was cited.\n\nHowever, even if the evolutionary model here is equivalent to previous statistical discrimination models, this paper makes several contributions in terms of presenting this evolution in terms that are more accessible terms, providing a transparent analytical strategy, generalizing these models to an arbitrary number of groups, and showing how current proposals for fairness fit into their dynamics.\n\nOne angle that I think the authors could play up is the idea that the only effective intervention here is the feedback control intervention that explicitly considers the dynamics of the system, whereas current fairness metrics take a static view of the system and can interact poorly (or not at all) with the dynamics. Several papers including Liu et al 2018 Delayed Impact (in the context of a Markov model), D'Amour et al 2019 Fairness is not Static (in terms of simple simulations) and Fazelpour and Lipton 2020 Fairness from a Non-Ideal Perspective (from a more abstract philosophical point of view) have argued this point. The authors discuss some of this in related work, but the idea that there are clear unintended consequences of applying fairness criteria obliviously to dynamics is, IMO, underplayed in the current manuscript.\n\nSome smaller concerns:\n\nIn my opinion, the motivation of replicator dynamics is quite heavy-handed, and does not characterize prior work as carefully as it could. First, this paper is not unique in assuming that there are no structural differences between groups except for this initial conditions. In addition to the statistical discrimination work cited above, the Liu et al 2018 delayed impact paper is similar (contrary to the description provided with the citation at the top of section 2.2). Secondly, it seems no less contentious to axiomatically assume replicator dynamics in this paper as it does to assume that there are incentives for investment in qualification in other work. In a social system, \"fitness\" will end up determined by the costs and benefits of qualification, so whether these assumptions are stated up front or tucked into the parameters chosen to characterize a system seems to make little different. I think it would make more sense to frame this work as an extension of previous work (especially if connections can be drawn to the classical statistical discrimination models) where social dynamics can lead to poor equilibria even if there are few structural disparities.\n\nAt the very least, I would recommend focusing more of the discussion 2.2 on a plausible story for the emergence of the replicator dynamics. There is a mention of \"exchange of qualification strategies\" before Assumption 8; elaboration on this point would be useful.\n\n In (16) and (20), the p-norms of D are missing exponents inside of the summation. The norm notation is also mixed up in the proof of Remark 7 in the appendix (e.g., LHS of eq 63a).\n\nThe statement of Remark 7 seems incomplete. The RHS of the iff arrow is just an expression, not a claim.\n\nTheorem 14 should perhaps be rephrased to say that the *decision rule \\hat Y* satisfies equality of opportunity under the stated assumptions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {"ethical_concerns": "A minor concern is the use of the phrase \"cultural evolution\", as it may suggest the notion of qualification as a \"cultural\" trait. At least in the US, discussion of inequality and its intersection with \"culture\" has been fraught, and it may be worth avoiding confusion with this. Given that the notion of \"qualification strategies\" is discussed in the paper, I might suggest \"strategic evolution\" as a potentially better phrase."}}]

Author rebuttal:

Meta-review: (The meta-review was written jointly by AC and SAC after discussion with the program chairs.)

 This paper studies how qualification disparities between groups can arise without structural differences between different groups. That is: when (optimal) machine learning systems make decisions that impact people, how can these systems cause inequality to arise and maintain inequality when it is present? Furthermore, when the system is modified to enforce standard fairness constraints, how does this impact long term inequality?

Overall, the topic is of extreme interest to the ML community (long-term fairness of systems). The reviewers found the paper to be technically clear and impactful, and all recommend acceptance, two strongly (scores = 8,9). The one reviewer only weakly recommending acceptance (score = 6) did not engage in the discussion. However, while the paper's technical contributions are crisp, several sociotechnical issues were identified by the SAC (in collaboration with ethics chairs and an additional AC).

The decision is to conditionally accept the paper. The sociotechnical issues, raised in a separate comment that serves as a sociotechnical review, must be addressed. In particular, the following changes need to be implemented for the paper to be accepted (these are copy-pasted from the sociotechnical review):

1. Review and update analogies drawn from evolutionary biology, so as to minimize the risk of essentialist misinterpretation.

2. Connect technical definitions to some real-world examples, and discuss how the assumptions made in the paper align or not with those real-world examples. Some of the key terms that require connecting to examples are "qualification" and "subpopulation". The best approach would be to introduce a plausible running example that readers can use to understand the implications of the results. Such an example would need to pin down what precisely is meant by "qualification", how the groups (subpopulations) are defined, and what the classifier is doing. If no plausible example can be produced, then the authors need to walk back their statements about the applicability of the results to real-world sociotechnical systems.

3. Be explicit about the operationalization of fairness and explain what the various assumptions and criteria around that operationalization mean in the context of real-world dynamics.

4. Discuss how the replicator dynamics might be realized in real-world examples.

5. Connect the proposed intervention to a clear motivation of mitigating harms to marginalized groups, and more explicitly clarify differences between this intervention and other approaches (such as equalized odds and demographic parity) in service of that overall goal.

UPDATE: The revisions submitted by the authors have been reviewed and the paper has been officially accepted.

Decision: Accept (Spotlight)