Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions
Authors: Tian Xie
Xueru Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Strategic Classification
Long-term Fairness
Primary area: machine_learning_for_social_sciences
TL;DR: This paper studies the dynamics of welfare and fairness where strategic agents interact with an ML system retrained over time with model-annotated and human-annotated samples.

Abstract: As machine learning (ML) models are increasingly used in social domains to make consequential decisions about humans, they often have the power to reshape data distributions. Humans, as strategic agents, continuously adapt their behaviors in response to the learning system. As populations change dynamically, ML systems may need frequent updates to ensure high performance. However, acquiring high-quality *human-annotated* samples can be highly challenging and even infeasible in social domains. A common practice to address this issue is using the model itself to annotate unlabeled data samples. This paper investigates the long-term impacts when ML models are retrained with *model-annotated* samples when they incorporate human strategic responses. We first formalize the interactions between strategic agents and the model and then analyze how they evolve under such dynamic interactions. We find that agents are increasingly likely to receive positive decisions as the model gets retrained, whereas the proportion of agents with positive labels may decrease over time. We thus propose a *refined retraining process* to stabilize the dynamics. Last, we examine how algorithmic fairness can be affected by these retraining processes and find that enforcing common fairness constraints at every round may not benefit the disadvantaged group in the long run. Experiments on (semi-)synthetic and real data validate the theoretical findings.

Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions
Tian Xie
Xueru Zhang
Computer Science and Engineering the Ohio State University Columbus, OH 43210 xie. 13799os7. edu
Computer Science and Engineering the Ohio State University
Columbus, OH 43210 zhang 12807 (osu.0uuuu.... edu
Abstract
As machine learning (ML) models are increasingly used in social domains to make consequential decisions about humans, they often have the power reshape data distributions. Humans, as strategic agents, continuously adapt their behaviors response the learning system. As populations change dynamically, ML systems may need frequent updates to ensure high performance. However, acquiring high- quality human-annotated samples be highly challenging and even infeasible in social domains. A common practice to address this issue using the model itself to annotate unlabeled data samples. This paper investigates the long-term impacts when ML models are retrained with model-annotated samples when they incorporate human strategic responses. We first formalize the interactions between strategic agents and the model and then analyze how they evolve under such dynamic interactions. We find that agents are increasingly likely to receive positive decisions as the model gets retrained, whereas the proportion of agents with positive labels may decrease over time. We thus propose refined retraining process stabilize the dynamics. Last, we examine how algorithmic fairness can be affected by these retraining processes and find that enforcing common fairness constraints every round may not benefit the disadvantaged group in the long run. Experiments on beemi-)synthette and real data validate the theoretical findings.
Introduction
As machine learning (ML) is increasingly used auto- mate human-related decisions (e.g., in lending, hiring, col- lege admission), there growing concern that these cisions are vulnerable to human strategic behaviors. With the knowledge of decision policy, humans may adapt their behavior strategically in response to ML models, e.g., by changing their features at costs to receive favorable out- comes. A line of research called Strategic Classification studies such problems by formulating mathematical mod- els to characterize strategic interactions and developing algorithms robust to strategic behavior [ 2]. Among existing works, most studies focus one-time model de- ployment where an MI. model is trained and applied to a fixed population of strategic agents once.
VocalAordadee
Thining data sit
1 1
Tramiin 090se1
Figure 1: Illustration of updating the training data from 1++ during the re- training process with strategic feedback
However, practical ML systems often need to be retrained periodically to ensure high performance on the current population. As the ML model gets updated, human behaviors also change accordingly.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To prevent the potential adverse outcomes, it is critical to understand how the strategic population is affected by the model retraining process. Traditionally, the data used for retraining models can be constructed manually with human annotations (e.g., ImageNet). However, acquiring a large amount of aannoonnnnntttedddeeeo samples can be highly difficult and even infeasible, especially in human-related applications, e.g., in automated hiring where an MI. model is used to identify qualified applicants, even an experienced interviewer needs time to label an applicant.
Motivated by a recent practice of automating data annotation for retraining large-scale ML models we study strategic classification in a sequential framework where an ML model periodically retrained by a decision-men maasii with both human and model-ane -nnottaed samples. These updated models are deployed sequentially on agents who may modify their features to receive favorable outcomes. Since ML models affect agent behavior and the agent strategic feedback can further be captured when retraining the future model, their interactions drive both to change dynamically over time. However, it remains unclear how the two evolve under such dynamics and what long-term effects one may have on the other.
To further illustrate our problem, consider an ex- ample of college admission where new students from a population apply each year. In the t-th year, an ML model ft is learned from a training dataset St and used to make admission de- cisions. For students who apply in the (t + 1)-th year, they will best respond to the model fi in the previous year (e.g.. preparing the ap- plication package in way that maximizes the chance of getting admitted). Mcanwhile, the college retrains the classifier fe+1 using a new training dataset S,+1 consisting of previous training data St, new aamoanaaddtttedd samples, and new model-annotated samples (i.e., previous applicants annotated by the most recent model fi). The model f++1 is then used to make admission decisions in the (t + 1)-th year. This process continues over time and demonstrate how the training dataset S, is updated to S i in Fig. [ Under such dynamics, both the ML system and the strategic population change over time and may lead to unexpected long-term consequences. An illustrating example is given in Fig. 2
Figure 2: Evolution of the student distribution and ML model at t = 0 (left). t = 5 (middle), and 14 (right): each student has two features. At each time, a classifier is retrained with both human and model- annotated samples, and students best respond to be admitted (Fig. D. Over time, the learned classifier (black lines) deviates from ground truth (green lines).
In this paper, we examine the evolution of the ML model and the agent data distribution. We ask: 1) How does the agent population evolve when the model is retrained with strategic feedback? 2) How is the ML system affected by the agent's 's strategic response? 3) If agents come from multiple social groups, how can model retraining further impact algorithmic fairness? Can imposing group fairness constraints during model training bring long-term societal benefits?
Compared to prior studies on strategic classification under sequential settings 6 6 that mainly focused on developing classifier robust to strategic agents, we study the long-term impacts of model retraining with agent strategic feedback. Instead of assuming actual labels are available while retraining, we consider more practical scenarios with model--nnotated samples. Although the risks on accuracy (3] and fairness [8] of using model-annotated samples to retrain models have been highlighted, ours is the first to incorporate strategic feedback from human agents. In App. C we discuss more related works in detail. Our contributions are summarized as follows:
1. We formulate the problem of model retraining with human strategic feedback (Sec.
2. We theoretically characterize the evolution of the expected acceptance rate (i.e., the proportion of agents receiving positive classifications), qualification rate (i.e., the proportion of agents with positive labels), and the classifier bias (i.e., the discrepancy between acceptance rate and qualification rate) under the retraining process. We show that the acceptance rate increases over time under retraining, while the actual qualification rate may decrease under certain conditions. The dynamics of classifier bias are more complex depending on the systematie bias of human- annotated samples. Finally, we propose an approach to stabilize the dynamics (Sec. 3
2

---

3. We consider settings where agents come from multiple social groups and investigate how inter- group fairness can be affected by the model retraining process; we also investigate the long-term effects of fairness intervention at each round of model retraining (Sec. 4) We conduct experiments on (semi-)synn sstnii-)ytttticc and real data to verify the theorems (Sec. App.
App.
2 Problem Formulation
Consider a population of agents who are subject to certain ML decisions (c.g., admission/hiring decisions) and join the decision-making system in sequence. Each agent has observable continuous features X E Rd and a hidden binary label Y E {0, 1} indicating its qualification state ("1" being qualified and "0" being unqualified). Let Pxy be the joint distribution of (X,Y) which is fixed over time, and Px. Py be the corresponding marginal and conditional distributions. Assume Px. Prix continuous with non-zero probability mass everywhers in their domain. For agents who join the system at time t, the decision-make maker uses a classifier f. F - {0, }} to make decisions. Note that the decision-maker does not know Pxy and can only learn from the training dataset at / [
Agent best response. Agents who join the system at time can adapt their behaviors based on the latest classifier fi-1 and change their features X strategically. We denote the resulting data distribution as y Specifically, given original features X = æ, agents have incentives to change their features at costs to receive positive classification outcomes, i.e., by maximizing utility
21 = arg max; (fi-1(=) c(z, 2)}
(1)
where distance function 2(x,2) 2 0 measures the cost for agent to change features from to In this paper, we consider c(zr 2) = (z r) B(= - x) for some d x d positive semidefinite matrix B, allowing heterogeneous costs for different features. After agents best respond, their data distribution changes from Pxy to P-y- In this paper, we term Pxy agent prior-best-response distribution and Py post-best-response distribution. We consider natural settings that (i) agents need time to adapt their behaviors and their responses are delayed [10]: they act based on the latest classifier ft-1 they are aware of, not the one they receive; (ii) agent behaviors are benign and feature changes can genuinely affect their underlying labels, so feature-label relationship Pr1x Prix is fixed over time 9
Human-annotater samples and systematic bias. At each round f, we assume the decision-maker can draw limited number of unlabeled samples from the prior-besttresponse distribution Px [ With some prior knowledge (possibly biased), the decision-maker can annotate these features and generate human-anmotated samples S.,t- We assume the quality of human annotations is consistent, so S.,t at any is drawn from a fixed probability distribution D with marginal distribution D& Px- Because human annotations may not be the same as true labels, 1 can be biased compared to Py|x- We define such difference as the decision- maker's systematic bias, formally stated below.
Definition 2.1 (Systematic bias). Define u(Do, P)) Ex-Px 1 1x (1(x) Pyix (1/=:). The decision-maker has a systematic bias if p(D?, P) > (overestimation) or (underestimation).
Def. 2.1 implies that the decision maker has systematic bias when it labels a larger (or smaller) proportion of agents qualified compared the ground truth. In App. B we present numerous examples of systematic bias in real applications where the decision-maker has different systematic biases towards different demographic groups. Generally, the systematic bias may or may not exist and we study both scenarios in the paper.
Model-annotated samples. In addition to human-annotated samples, the decision-maker at each round & can also utilize the most recent classifier fz-1 to generate model-annotated samples for training fr. Specifically, let (*t-1) 1 be N post-best-response features (T) acquired from the
We consider natural settings where the human-annotated samples are drawn from fixed Px and the process independent of the decision-making process, i.e., the agents who best respond to fi-1 are classified by model f. and the decision- maker never confuses them with human annotations. Instead, human annotation a separate process for the
maker to obtain additional information about the whole population (e.g.. by first acquiring data from public datasets or third parties, and then labeling them to estimate the population distribution yxy). ). We provide an additional example and also consider the situation where human- annotated samples are drawn from sest-beet-r-ppooeee distribution P in App. D.I
3

---

agents coming at t 1, the decision maker uses fi-1 to annotate the samples and obtain model- annotated samples Sm.t-1 {a} f. f 1=1 (=1' Both human and model-annotated samples
are used to retrain the classifier at I.
Classifier's retraining process. With the human and model-annotated samples, we next introduce how the model is retrained by the decision-make over time. Denote the training dataset at é as St. Initially, the decision-maker trains fo with uaman--nnotttted training dataset S S0.0. Then the decision-maker updates ft every round to make decisions about agents, and it learns f, using empirical risk minimization (ERM) with training dataset St. Similar to studies in strategic classification (21. we consider linear classifier in the form of ft function (e.g., logistic function) and hy H. At each round 2 L, consists of three components: existing training samples S N new
1 (x) 2 0) where h, E + [0. 1] is the scoring
and K new human-annotann annotated samples:
S S
US 2
(2)
Since annotating agents usually time-consuming and expensive, we have N >> K practice. The complete retraining process is shown in Alg. (App. A).
Given the probability that agents are qualified, i.e.,
distribution rx,, we can define the associated qualification rate as the
Q(P*) ((yy))rk [v]
For the classifier St deployed on marginal feature distribution PX= we define acceptance rate as the probability that agents are classified as positive, i.e.,
A(fe, P') Ex [f(())]
Since S, is randomly sampled at all t, the resulting classifier ft and agent best response are also random. Denote D' as the probability distribution of sampling from S, and recall that D y is the distribution for human-annotated So,t we can further define the expectations of Q(P*), A(fe, P!) over the training dataset:
= Es. (Q(P)) = Es, (A(S.PP))
where is the expected actual qualification rate of agents after they best respond, note that the expectation is taken with respect S, because the distribution Pay is the result agents responding to fe-1 which is trained with S is the expected acceptance rate of agents at time t.
Dynamics of qualification rate & acceptance rate. Under the model retraining process, both the model ft and agent distribution Pxy change over time. One goal is to understand how the agents and the ML model interact and impact each other in the long run. Specifically, we are interested in the dynamics of the following variables:
1. Qualification rate q: it measures the qualification of agents and indicates the social welfare. Acceptance rate ae: it measures the likelihood that an agent can receive positive outcomes and indicates the applicant welfare.
3. Classifier bias A, la: -:l: : it is the discrepancy between the acceptance rate and the true qualification rate, measuring how well the decision-male maker can approximate agents' actual qualification rate and can be interpreted decision-maker welfare.
In the rest of the paper, we study the dynamics of gt, at; As and we aim to answer the following questions: 1) How do the qualification rate 4r, acceptance rate or, and classifier bias A, evolve under the dynamics? 2) How can the evolution of the system be affected by the decision-maker" retraining process? 3) What are the impacts of the decision-maker" systematic bias? 4) If we further consider agents from multiple social groups, how can the retraining process affect inter-group fairness?
3 Dynamics of the Agents and Model
In this section, we examine the evolution qualification rate qt, acceptance rate as and classifier bias A, We aim to understand how applicunt welfare (Sec. 3.1. social welfare (Sec. (.2). and decision-maker welfare (Sec. 3.3) are affected by the retraining process the long run. We first introduce some assumptions used for the theorems.

---

Assumption 3.1. Hypothesis class H can perfectly learn the training data distribution D i.c. Shg H such that (x) (1|z).
With Assumption 3.1 we avoid the effects of learning error on the system dynamics; this allows to focus on the dynamic interactions between strategic agents and ML system. Although the theoretical analysis relies on the assumption, our experiments in Sec. and App. showow consistent results with theorems. We further assume the monotone likelihood ratio property holds for D&y and PXY.
Assumption 3.2. Let r[m] be the m dimension of r E R then Dy (1|zz) and Prix (1/zr) are continuous and monotonically increasing in 2z/m], Vm 1, d while other dimensions are fixed.
Note that the Assumption 3.2 is mild and widely used in previous literature [12 [3]. It can be satisfied by many distributional families such exponential, Gaussian, and mixtures of exponential/Gausiin. implies that agents are more likely to be qualified as feature value increases.
3.1 Applicant welfare: dynamics acceptance rate
We first examine the dynamics of ag Es. [A(ft, P')]. Intuitively, under Assumption 3.1 all classifiers can fit the training data well. Then the model-annotated samples Sm,t1 generated from post-best-response agents would have higher qualification rate than the qualification rate of training data S As a result, the training data S. augmented with Sm,t-1 has higher proportion of qualified agents than the qualification rate of S thereby producing a more "generous" classifier fc with larger dr. This reinforeing process can be formally stated Thm. [.3
Theorem 3.3 (Evolution of at). Under the retraining process, the acceptance rate of the agents that join the system increases over time, i.e., 0t > a--1, Vt 2
We prove Thm. 3.3 by mathematical induction in App. G.3 and Fig. 3 illus- trates the theorem. When agents best respond, the decision-maker tends to ac- cept more agents. We further show that when the number of model-annotated samples N is large com- pared to the number of human-annotated samples K, the classifier will ulti- mately accept all agents in the long run (Prop. 3.4).
point best responses
h
J-1
f
fe
Figure 3: Increasing acceptance rate from at to 0t-1- Unquali- fied/qualified agents are shown as circles/squares, while the admit- ted/rejected agents are shown in red/blue. New agents coming at t + 1 are shown in hollow. The left plot shows the training set S, containing unqualified (red circle) and 2 qualified agents (blue square) and a is 0.5. The middle plot shows the agents coming t best respond to fi-1. After the responses, 3 of 4 agents are qualified (blue square) and is still unqualified (blue circle). However, all 4 agents are notated as "qualified" (blue). The right plot shows the training set S containing all points of the left and middle plot, plus new human-annotatte points (hollow points). All blue points are labeled as and the red points are O. So S:41 has more samples with positive label (0.7), resulting in fz+1 accepting a higher proportion of agents.
Proposition 3.4. For any ,xy, Do, B, there exists a threshold x > 0 such that lim; Jem ag 1 whenever K
<x
N
The specific value of in Prop. 3.4 depends on Pxy, D B, which is difficult to find analytically. Nonetheless, we illustrate in Sec. that when K Since the human--nnottater annotated samples are often difficult to attain (due to time and labeling costs), the condition in Prop. 3.4 is easy to satisfy in practice.
N = 0.05, a tends to approach 1 in numerous datasets.
3.2 Social welfare: dynamics of qualification rate
Next, we study the dynamics of qualification rate q: Es.,, (Q(')) Unlike the acceptance rate a which always increases during the retraining process, the evolution of q. is more complicated and depends agent prior-best-respons distribution PxY-
Specifically, let go = Q(P) P( [e] be the initial qualification rate, then the difference between di and qo can be interpreted as the amount of improvement (i.c., increase in label) agents
5

---

gain from their strategic behavior at t. This is determined by (i) the proportion of agents that decide to change their features at costs (depends on Px). and (ii) the improvement agents can expect upon changing features (depends on Xy.x). Thus, the dynamics of y depend on Pxy. Despite the intricate nature of dynamics, we can still derive a condition under which qt decreases monotonically.
Theorem 3.5 (Evolution of q:). Consider the setting where the d-dimensional feature space X E Rd where Fx(x), Px(x), (x(x(1[a) are the cumulative distribution function, probability density function and the labeling function when Y 1. Denote J = { fo)) 0} as the half-space in IRd determined by the classifier So. Under the retraining process, Vi 2 1, 9t41 S qu either of the following conditions holds: (i) Fx and Py (1/) are convex on J; (it) for each dimension
x[d]. i E [d]. Fx(x) and Pr|x (1[z) are convex with respect to z(i].
Note that 9++1 S qt in Thm. 3.5 holds only for t 2 1. Because agent behavior can only improve their labels, prior-best best-ese Oa always serves as the lower bound of q. The half-space in Thm. 3.5 specifies the region in feature space where agents have incentives to change their features. The convexity of Fx and Py (1 [x) ensure that as Se evolves from 1: (i) fewer agents choose to improve their features, and (ii) agents expect less improvement from feature changes. Thus, dr decreases over time. Conditions in Thm. 3.5 can be satisfied by common distributions Pxx (e.g.. Uniform, (eta 1) with a > 1) and labeling functions Pyjx (1/z) (e.g., linear function, quadratic functions with degree greater than I). The proof and a more general analysis are shown in App. G.5
We also show that Thm. 3.5 is valid under diverse experimental settings (Sec. 3 App. D
App.
3.3 Decision-maker welfare: dynamics of classifier bias
Sec. 3.1 and 3.2 show that as the classifier fi gets updated over time, agents are more likely to get accepted (a, increases). However, their true qualification rate (after the best response) may actually decrease. indicates that the decision-maker misperception about agents varies over time. Thus, this section studies the dynamics of classifier bias A |a, qul. Our results show that the evolution of A, is largely affected by the decision-maker's systematic bias p(D", P) as defined in Def. 2.1
Theorem 3.6 (Evolution of Az). Starting from t = at the retraining process and under conditions
in Thm. 3.5
1. If (DD
(o, P) = 0, i.e., the systematic bias does not exist, then A, increases over time.
If p(DD, > i.e., the decision-maker overestimates agent qualification, then D. increases
over time.
If p((",, ()D,,))< O, iee, the decision-maker underestimates agent qualification, De either monotoni- cally decreases or first decreases but then increases.
Thm. 3.6 highlights the potential risks of the model retraining process and is proved in App. G.6 Originally. the purpose of retraining the classifier was to ensure accurate decisions on the targeted population. However, when agents behave strategically, the retraining may lead to adverse outcomes by amplifying the classifier bias. Meanwhile, though systematic bias is usually factor to eliminate when learning ML models, may help mitigate classifier bias to improve the decision-maker welfare in the retraining process, i.e., A, decreases when DD P) < O.
undesirable
3.4 Intervention to stabilize the dynamics
Sec. 3.1 3.3 show that as the model retrained from strategic agents, de. Gt. A are unstable and may change monotonically over time. Next, we introduce an effective approach to stabilizing the system.
From the above analysis, we know that one reason that makes qt- a, A, evolve is agent's best response. i.c., agents improve their features strategically to be accepted by the most recent model, which leads to higher qualification rate of data), eventually causing de to deviate from qt. Thus, to mitigate such deviation, we can improve the quality of model annotation. Our method is proposed based on this idea, which uses probabilistic sampler g when producing modell-anotatee samples.
samples (and the resulting training
Specifically, at each time t, instead of adding Sm-1o = ft- 1) (samples annotated by the model ft-1) to training data S. (2). we use the probabilistic model lx-1(x)) to annotate each sample according to the following: For each sample x, we label it as 1 with probability he- 1x-1(x),, and as O otherwise. Here ( R D (1)z)) is the estimated posterior probability learned from S

---

(e.g., using logistic model). We call the procedure refined retraining process if model-annotated samples are generated in this way based on probabilistic sampler.
Fig. 3 also illustrates the above idea: agents best respond to ft-1 (middle plot) to improve and ft will label both as By contrast, a probabilistic sampler h2 only labels fraction of them as 1. This alleviates the influence of agents' best responses to stabilize the dynamics of or 9r. 4e. Prop. D.2
and App. F33 provide proof' and more experiments for the refined retraining process.
4 Impacts on Algorithmic Fairness
In this section, we further consider agents from multiple social groups and investigate how group fairness can be affected by the model retraining process. Similar to prior studies in fair ML 02 13). we assume the decision-make maker knows the group identity of each agent and uses group-dependee classifiers to make decisions. WLOG, we present the results for any pair of two groups i, Among the two groups, we define the group with a smaller acceptance rate under unconstrained optimal classifiers as disadvantaged group.
4.1 Impacts of systematic bias & model retraining
We first consider the situation where two groups have no innate difference: they have the same rrossobes-ressponee feature distribution and the same cost matrix B to change features. However, the decision-maker has systematic bias in favor of group i more than group j. making i the advantaged group and j the disadvantaged group. We consider the fairness metric demographic parity (DP) [16], which measures unfairness as the difference in acceptance rate across two groups. Extension to other fairness metrics such as equal opportunity [77 is discussed in App. D.2, Thm. [.1 below shows the long-term impacts of refined retraining process (i.e., model- annotated samples generated with probabilistic sampler h2) and original model retraining process (i.e., model-annotated samples generated with model fi) on group unfairness.
Theorem 4.1 (Impacts of model retraining on unfairness). When groups i, have no innate difference, but j is disadvantaged due to systematic bias: (i) If applying original retraining process to both N satisfying Prop. 3.4, then group j will stay disadvantaged until all agents in both
groups with K groups are accepted to achieve perfect fairness; (ii) If applying refined retraining process to both groups, then group j will stay disadvantaged and the unfairness remains the same in the long run; (iã) If applying refined retraining process to group i but the original process group j with N. satisfying Prop. 3.4 then unfairness first decreases after certain rounds of retraining until group j becomes advantaged, then unfairness increases.
K
Thm. shows that original and re- fined retraining processes impact dif- ferently on group faimess as proved in App. G.9 Specifically, the origi- nal retraining process ultimately attains "trivial" perfect fairness by accepting all agents, whereas the refined retrain- ing process stabilizes the dynamies but unfairness always exists. Interestingly, applying disparate retraining strategies to two groups may result in perfect fair- ness in the middle of retraining process. This suggests that it may be beneficial for the dicision-maker monitor the fairesss measure during the retraining and exccute an early stopping mechanism to attain almost perfect DP faireess, i.c., stop retraining models early once unfairness reaches the minimum. As shown in the right plot of Fig. when refined retraining is only applied to group i, unfairness is minimized at t = 5.
38
63
04.
10
62
10
by fari
for
Unferness
Figure 4: Unfairness (DP) when the refined retraining pro- cess is applied to both groups (left), original retraining pro- cess is applied to both groups (middle). refined retraining process is only applied to group (right) under dataset 2.
4.2 Impact of short-term fairness intervention
Sec. 4. proposed a solution of "disparate retraining with early stopping" to mitigate unfairness. A more common method to maintain fairness throughout the retraining process to enforce certain faimeess constraints every time when updating the models. Next, we consider this method where
7

---

refined retraining process is applied to both groups (to stabilize dynamics) and faimess constraint imposed each round of model retraining. Unlike Sec. 4.I] we consider a general setting where two groups may have different feature distributions and different cost matrices B, but feature-label relation Pr|x (1(z) is the same across groups.
Finding fair models. For each group we use superscript to denote the group-specific distribu- tions/metrics listed in Sec. 2 (e.g.. Pxy)). At each round I, the decision-maker first trains a linear model f (x) 2 0) for group According to Assumption 3.1 and Prop. D.2 hg is expected to be rix (1 (1|x)) Mss where μ4, is the systematic bias towards group 8. Denote Of as the original optimal threshold at t without the fairness intervention, and let aj be the acceptance rate can tune the thresholds to get fair-optimal thresholds of (a pair of thresholds satisfying the fairness constraint with the largest aggregated accuracy).
for group s under 8r, then the decision-maie
Noisy agent best response. To ensure there always exists pair of thresholds that satisfy DP fairness constraint (i.e., equal acceptance rate), each group's post-best-respnns distribution needs to be continuous. However, when agents modify their features based on (T). the aggregate response necessarily exhibits discontinuities [8]. To tackle this issue, we consider the noisy best response model proposed by Jagadeesan al. (18], which assumes the agents only have imperfect and noisy information of decision threshold. Formally, given decision threshold 6, each agent best responds to O where is noise independently sampled from a zero mean distribution with finite variance o². Under noisy best response, we investigate the impacts of fairness intervention.
Theorem 4.2 (Impact of fairness intervention). Suppose group j disadvantaged from round 0 to I, i... group j has smaller acceptance rute than group i under unconstruined optimal thresholds {0r 02., Vr S I. Let of be the variance of the noisy best response
We have the following:
(i) Without fairness intervention, Yo,, there always exists feature distributions and cost matrices under which group j switches to be advantaged at +
(ii) With fairness intervention, if (0) 0. abb -0- di then group always remains disadvan- taged 1+1. 1.
Thm. 4.2 shows that without fairness intervention, the originally disadvantaged group j can flip to be advantaged. In contrast, the fairness intervention helps maintain the disadvantaged and advantaged groups when the agent's perception of the decision rule is sufficiently accurate. Note that the bound on
is well-defined. Since group j disadvantaged, ag af always holds. If at < (0) - 0.) aj of
holds, then is guaranteed that aj af41 (see App. 10 for details). This result implies that the disadvantaged group, by losing their chance to become advantaged, may not benefit from fairness intervention in the long run.
5
Experiments
We conduct experiments on two synthetic (Uniform, Gaussian), one semi-synthetic (German Credit 19 ). and one real dataset (Credit Approval (00) to validate the dynamics of az, 4t, As and the unfairness Note that only the Uniform dataset satisfies all assumptions and the conditions in our theoretical analysis, while the Gaussian and German Credit datasets violate the conditions in Thm. 3.5 The Credit Approval dataset violates all assumptions and conditions of the main paper. The decision-maker trains logistic regression models for all experiments using stochastic gradient descent (SGD) over 7 steps. We present the experimental results of the Gaussian and German Credit datasets to illustrate the dynamics of az, qt, A, in this section, while the results for Uniform and Credit Approval data are similar and shown in App. E
Gaussian data. We consider a synthetic dataset with Gaussian distributed Px. Py x is logistic and satisfies Assumption 3.2 but not the conditions of Thm. B.5 We assume agents have two independent features X, X2 and are from two groups i,j with
Table 1: Gaussian Dataset Setting
Pretes) Ni0000
Prgg e]
T.r.T.g
(14 221)-1
100, 0.05, 15.0.5
5 0 5
0
different sensitive attributes but identical joint distribution PXY.
Their cost matrix is B
httpss//gihhubbcoo//sssss//////t haw.://nnnninttttatatmttotttttnr Strateric-ttttt---ee Haaiie--ummn-nttttt

---

and the initial qualification rate is = 0.3. We assume the decision-mann has systematic bias by overestimating (resp. underestimating) the qualification of agents in the advantaged group (resp. disadvantaged group j), which is modeled as increasing y ar)) to be 0.1 larger (resp. smaller) than Pr|x (1 (z) for group (resp. group j). For the retraining process, we let r = N
K = 0.05 (i.e., the
number of model-annotated samples N 2000, which is sufficiently large compared to the number of human-annotated samples K 100). Table summarizes the dataset information, and the joint
distributions visualized in App. F.1
We verify the results in Sec. by illustrating the dynamics of a,, qt, A for both groups (Fig. 5ap. Since our evolution results are in expectation, we perform = 100 independent runs of experiments for every parameter configuration and show the averaged outcomes. The results are consistent with Thm. 3.3 3.5 and 3.6 (i) acceptance rate a, (red curves) increases monotonically; (ii) qualification rate q: decreases monotonically starting from t = 1 (since strategic agents only best respond from t= 1); (iii) classifier bias A, evolves differently for different groups and it may reach the minimum after a few rounds of retraining.
We further test the robustness of system dynamics against agent noisy response, where we assume agents estimate their outcomes f(( = fx()) + with (O, 0.1). We present the dynamics of a;, 4r, A, for both groups in Fig. 6a which are similar to Fig. 5a demonstrating the robustness of theorems.
German Credit dataset [19]. This dataset includes features for predicting individuals' credit risks. It has 1000 samples and 19 numeric features, which used to construct larger-scaled dataset. Specifically, we fit a kernel density estimator for all 19 features to generate 19-dimensional features, the corresponding labels are sampled from the distribution Pyxx which is estimated from data by fitting a logistic classifier with 19 features. Given this dataset, the first 10 features are used to train the classifiers. The attribute "sex" is regarded as the sensitive attribute. The systematic bias is created by increasing/deecresinng Py by 0.06. Other parameters n, r, T, go are the same as Table Since Py|x is logistic function, Assumption 3.2 can be satisfied easily illustrated in App. F.T
We verify the results in Sec. by illustrating the dynamics of at, qt A, both groups (Fig. 5b). The 3.5 336 (i) acceptance rate (red curves) always increases; (ii) qualification rate q: (blue curves) decreases starting from t = 1 (since strategic agents only best respond from 1); (iii) classifier bias A, (black curves) evolve differently for different groups.
results are consistent with Thm. 3:3
Finally, similar to Fig. 5b Fig. 6b demonstrates the results are still robust under the noisy setting.
1.0
1.0
0.8
0.8
0.8
0.
0.6
0.6
0.4
0.4
0.2
0.2
0.2
02
0.0
0.0
10
16
6
8
10
12
14
16
10
(a) Gaussian: group (left) and j (right)
(b) German: group (left) and j (right)
Figure 5: Dynamics of Oz,, qt, As under the perfect information setting
1.0
1.0
0.8
0,8
0.8
0.8
0.6
0.6
0.4
0.4
0.2
02
0.0
10
14
16
10
6
8
10
12
14
16
10 12
14
(a) Gaussian: for group i (left) and j (right)
(b) German: group (left) and j (right)
Figure 6: Dynamics of de, gr, A, under the noisy setting
Additional experiments. We provide more results in App. F to: (i) verify Thm. 3.3 and Thm. 3.6
(ii) Visualize the influence of each factor including training rounds, cost matrices, the ratio between Humann-nnotern and model-ammotated samples, whether the agents are strategic, and whether certain

---

assumptions are violated; (iii) visualize the evolution of unfairness when different retraining strategies are applied different groups.
Conclusion & Limitations
This paper studies the dynamics where strategic agents interact with an ML system retrained over time with model-annotated and human-annotated samples. We rigorously studied the evolution of applicant welfare, decision-maker welfare, and social welfare. Such results highlight the potential risks of retraining classifiers when agents are strategic. The paper also provides a comprehensive analysis on the fairness dynamics associated with the retraining process, revealing that the fairness intervention may not bring long-term benefits. To the negative social impacts, provide mechanisms to stabilize the dynamics and an early stopping mechanism to maintain fairness. However, our theoretical results rely on certain assumptions and we should first verify these conditions before adopting the results of this paper, which may be challenging in real-world applications.
Acknowledgement
This material is based upon work supported by the U.S. National Science Foundation under award IIS- 2202699 and IIS-2416895, by OSU President's Research Excellence Accelerator Grant, and grants from the Ohio State University's 's Translational Data Analyties Institute and College of Engineering Strategic Research Initiative.
References
[1] Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classifi- cation. Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science, page 111-122, 2016.
[2] Sagi Levanon and Nir Rosenfeld. Generalized strategic classification and the case of aligned incentives. In International Conference on Machine Learning. 2C22,2022, 17-23 July 2022, Baltimore, Maryland, USA, pages 23593-12618 2022.
[3] Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 33883-33920, 2023.
[4] George Alexandru Adam, Chun-Hao Kingsley Chang, Benjamin Kains, and Anna Gold- enberg. Error amplification when updating deployed machine learning models. In Proceedings of the Machine Learning for Healthcare Conference, Durham, NC, USA, pages 5-6, 2022.
[5] Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation, page 55-70, 2018.
[6] Saba Ahmadi, Hedyeh Beyhaghi, Avrim Blum, and Keziah Naggita. The strategic perceptron. In Proceedings of the 22nd ACM Conference on Economics and Computation, pages 6-25, 2021.
[7] Yiling Chen, Yang Liu, and Chara Podimata. Learning sarategy-awars linear classifiers. Ad- vances in Neural Information Processing Systems, 33:15265 15276, 2020.
[8] Sierra Wyllie, Ili Shumailov, and Nicolas Papernot. Fairness feedback loops: training synthetic data amplifies bias. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2113-2147, 2024.
[9] Ozgur Guldogan, Yuchen Zeng, Jy-yong Sohn, Ramtin Pedarsani, and Kangwook Lee. Equal improvability: A new faimness notion considering the long-term impact. In The Eleventh International Conference Learning Representations, 2022.
10

---

[10] Tijana Zmic, Eric Mazumdar, Shankar Sastry, and Michael Jordan. Who leads and who follows in strategic classification? Advances Neural Information Processing Systems, pages 15257-15269, 2021.
[11] Jon Kleinberg and Manish Raghavan. How do classifiers induce agents to invest effort strategi- cally? page 1-23, 2020.
[12] Xueru Zhang, Mohammad Mahdi Khalili, Kun Jin, Parinaz Naghizadeh, and Mingyan Liu. Fairness interventions (Dis)Incentives for strategic manipulation. In Proceedings of the 39th International Conference on Machine Learning, pages 26239-26264, 2022.
[13] Reilly Raab and Yang Liu. Unintended selection: Persistent qualification rate disparities and interventions. Advances in Neural Information Processing Systems, pages 26053-26065, 2021.
[14] Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang. How do fair decisions fare in long-term qualification? In Advances in Neural Information Processing Systems, pages 18457-18469, 2020.
[15] Lydia T Liu, Ashia Wilson, Nika Haghtalab, Adam Tauman Kalai, Christian Borgs, and Jennifer Chayes. The disparate equilibria of algorithmic decision making when individuals invest rationally. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 381-391, 2020.
[16] Michael Feldman. Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259-268. 2015.
[17] Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, 2016.
[18] Meena Jagadeesan, Celestine Mendler-Dünner, and Moritz Hardt. Alternative microfoundations for strategic classification. In Proceedings of the 38th International Conference on Machine Learning, pages 4687-4697, 2021.
[19] Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994, DOI: https///bb..gg/1..244////CC
[20] Quinlan Quinlan. Credit Approval.
UCI Machine Learning Repository, 2017. DOI:
[21] Quinn Capers IV, Daniel Clinchot, Leon McDougle, and Anthony G Greenwald. Implicit racial bias in medical school admissions. Academic Medicine, 92(3):365-369) 2017.
[22] AJ Alvero, Noah Arthurs, Anthony Lising Antonio, Benjamin W Domingue, Ben Gebre-Medhin Sonia Giebel, and Mitchell I. Stevens. Ai and holistic review: informing human reading in college admissions. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 200-206, 2020.
[23] Elias Bareinboim and Judea Pearl. Controlling selection bias in causal inference. In Artificial Intelligence and Statisties, pages 100-108, 2012.
[24] J Michelle Brock and Ralph De Haas. Discriminatory lending: Evidence from bankers in the lab. American Economic Journal: Applied Economics, 15(2):31-68, 2023.
[25] Omer Ben-Porat and Moshe Tennenholtz. Best response regression. In Advances in Neural Information Processing Systems, 2017.
[26] Mark Braverman and Sumegha Garg. The role of randomness and noise strategic classification. CoRR, abs/2005.08377, 2020.
[27] Zachary Izzo, Lexing Ying, and James Zou. How to learn when data reacts to your model: Performative gradient descent. In Proceedings of the 38th International Conference on Machine Learning, pages 4641-4650, 2021.
11

---

[28] Wei Tang, Chien-Ju Ho, and Yang Liu. Linear models are robust optimal under strategic behavior. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2584-2592, 13-15 Apr 2021.
[29] Itay Eilat, Ben Finkelshtein, Chaim Baskin, and Nir Rosenfeld. Strategic classification with graph neural networks, 2022.
[30] Lydia T. Liu, Nikhil Garg, and Christian Borgs. Strategic ranking. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 2489-2518, 2022.
[31] Tosca Lechner and Ruth Urner. Learning losses for strategic classification. arXiv preprint arXiv:220.133421, 2022.
[32] Guy Horowitz and Nir Rosenfeld. Causal strategic classification: A tale of two shifts, 2023.
[33] Keegan Harris, Hoda Heidari, and Steven Z Wu. Stateful strategic regression. Advances in Neural Information Processing Systems, pages 28728 28741,, 2021.
[34] Yahav Bechavod, Chara Podimata, Steven Wu, and Juba Ziani. Information discrepancy in strategic learning. In International Conference on Machine Learning, pages 1691-1715, 2022.
[35] Kun Jin, Xueru Zhang. Mohammad Mahdi Khalili, Parinaz Naghizadeh, and Mingyan Liu. Incentive mechanisms for strategic classification and regression problems. In Proceedings of the 23rd ACM Conference on Economics and Computation, page 760-790, 2022.
[36] Yatong Chen, Jialu Wang, and Yang Liu. Strategic recourse in linear classification. CoRR, 0ts/2011.000553 2020.
[37] Nika Haghtalab, Nicole Immorlica, Brendan Lucier, and Jack Z Wang. Maximizing welfare with incentive- aware evaluation mechanisms. arXiv preprint S.KV::2011.019555 2020.
[38] Tal Alon, Magdalen Dobson, Ariel Procaccia, Inbal Dalgam-Cohent and Jamie Tucker-Folt Multiagent evaluation mechanisms. Proceedings of the AAAI Conference on Artificial Intelli- gence, 14:177411781, 2020.
[39] Yahav Bechavod, Katrina Ligett, Steven Wu, and Juba Ziani. Gaming helps! learning from strategic interactions in natural dynamics. In International Conference on Artificial Intelligence and Statistics, pages 1234-1242, 2021.
[40] Tian Xie, Xuwei Tan, and Xueru Zhang. Algorithmic decision-making under agents with persistent improvement. arXiv preprint arXiv:2405.01807, 2024.
[41] Tian Xie and Xueru Zhang. Non-linear welfare-aware strategic learning, 2024. URL https //arxiv. baaxivv.rg//a abs/ abbs/2455 2405. 01810
[42] John Miller, Smitha Milli, and Moritz Hardt. Strategic classification is causal modeling in disguise. In Proceedings of the 37th International Conference Machine Learning, 2020.
[43] Yonadav Shavit, Benjamin L. Edelman, and Brian Axelrod. Causal strategic linear regression. In Proceedings of the 37th International Conference on Machine Learning, ICML, 20, 2020.
[44] Keegan Harris, Dung Daniel T Ngo, Logan Stapleton, Hoda Heidari, and Steven Wu. Strategic instrumental variable regression: Recovering causal relationships from strategic responses. In International Conference on Machine Learning, pages $502-8522, 2022.
[45] Tom Yan, Shantanu Gupta, and Zachary Lipton. Discovering optimal scoring mechanisms in causal strategic prediction, 2023.
[46] Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünnerr, and Moritz Hardt. Performative prediction. In Proceedings of the 37th International Conference on Machine Learning, pages 7599-7609, 2020.
[47] Moritz Hardt, Meena Jagadeesan, and Celestine Mendler-Düinner. Performative power. In Advances in Neural Information Processing Systems, 2022.
12

---

[48] Nir Rosenfeld, Anna Hilgard, Sai Srivatsa Ravindranath, and David C Parkes. From predictions decisions: Using lookahead regularization. In Advances in Neural Information Processing Systems, pages 41 4126-4126, 2020.
[49] Melissa Hall, Laurens van der Maaten, Laura Gustafson, and Aaron Adcock. A systematic study of bias amplification. arXiv preprint arXiv:201.111006, 2022.
[50] Julius Adebayo, Melissa Hall, Bowen Yu, and Bobbie Chern. Quantifying and mitigating the impact of label emrors on model disparity metrics. In The Eleventh International Conference on Learning Representations, 2022.
[51] Ali Shirali, Rediet Abebe, and Moritz Hardt. A theory of dynamic benchmarks, In The Eleventh International Conference on Learning Representations, 2022.
[52] Klas Leino, Emily Black, Matt Fredrikson, Shayak Sen, and Anupam Datta. Feature-wise bias amplification. arXiv preprint arXivv:812..00000 2018.
[53] Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. Queens are powerful too: Mitigating gender bias in dialogue generation. arXiv preprint arXiv: 19/1.4844, 2019.
[54] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. pages 5310-5319, 2019.
[55] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Kei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv: 2017.
[56] David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, debt in machine learning systems. Advances neural information processing systems, 28, 2015.
Crespo, and Dan Dennison. Hidden technical
[57] Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkata- subramanian. Runaway feedback loops in predictive policing. In Conference on fairness, accountability and transparency, pages 160-171. PMLR, 2018.
[58] Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. Feedback loop and bias amplification in recommender systems. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 2020.
[59] George Alexandru Adam, Chun-Har Kingsley Chang, Benjamin Haibe-Kains, and Anna Goldenberg. Hidden risks of machine learning applied to healthcare: Unintended feedback loops between models and future data causing model degradation. In Proceedings of the 5th Machine Learning for Healthcare Conference, volume 126 of Proceedings of Machine Learning Research, pages 710-731. PMLR, 2020.
[60] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.114993 2023.
[61] Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. arXiv preprint 23X.:2331000422 2023.
International Conference Artificial Intelligence and Staristics, pages 862-870, 2018. [62] Sven Schmit and Carlos Riquelme. Human interaction with recommendation systems. In
[63] Ayan Sinha, David Gleich, and Karthik Ramani. Deconvolving feedback loops in recom- mender systems. Advances in neural information processing systems, 29, 2016.
13

---

[64] Ray Jiang. Silvia Chiappa, Tor Lattimore. András György, and Pushmeet Kohli. Degenerate feedback loops in recommender systems. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, page 383-390, 2019.
[65] Vivek Gupta, Pegah Nokhiz, Chitradeep Dutta Roy, and Suresh Venkatasubraamaiian. Equalizing recourse across groups, 2019.
[66] Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In Proceedings the Twenty-Eightt International Joint Conference Artificial Intelligence, IJCAI-19, pages 6196-6200, 2019.
[67] Sagi Levanon and Nir Rosenfeld. Strategic classification made practical. In International Conference on Machine Learning, pages 6243-6253, 2021.
[68] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. htk.shttss://arcivee ics. uci. edu. http:////aaaaatttsrrertaaaappppp
[69] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 2021.
14

---

A The retraining process
Algorithm I retraining process
Input:
Joint distribution D&y for any So,, Hypothesis class F, the number of the initial training samples and agents coming per round N, the number of decision-maker maker-labeled samples per round K.
Output: Model deployments over time fo, f2,
1: So -S.0 x {-/) j=1
2: tt = deploy fo ~ F(Se)
3: for t (1.... 00) do
4: N agents gain knowledge of ft- and best respond to it, resulting in {af]i= (x)_) 1))== consists of the model-labeled samples from round 1. samples.
5: Sm,t =
6:
So, ~ D'yy consists of the new K maasiinnmaker-ladd
7: U U
8: Deploy ~ F(S.) on the incoming N agents who best respond to f2-1 with the resulting
joint distribution Pxy-
9: end for
B Motivating examples of the systematic bias
Def. 2.1 highlights the systematic nature of the pocision-makerr bias. This bias is quite ubiquitous when labeling is not a trivial task. almost always the case when the decision- maker needs to make human-related decisions. We provide the following motivating examples of systematic bias with supporting literature in social science:
1. College admissions: consider experts in the admission committee of college that obtains of student data and wants to label all students as "qualified" or "unqualified". The labeling task is much more complex and subjective than the ones computer vision/natural language processing which have some "correct" answers. Therefore, the experts in the committee are prone to bring their "biases" " towards specific population sharing the same sensitive attribute into the labeling process including:
(a) Implicit bias: the experts may have an implicit bias they are unaware of favor'discriminate against students from certain groups. For instance, famous study (21) reveals admission committee members at the medical school the Ohio State University unconsciously have "better impression" towards white students; Alvero et al. (22) finds out that even when members in an admission committee do not access the sensitive attributes students, they unconsciously infer them and discriminate against students from the minority group.
(b) Selection bias:
the experts may have insufficient knowledge of the under-represented pop- ulation due to the selection bias (3 because only small portion of them were admitted before. Thus, experts may expect a lower qualification rate from this population, resulting in more conservative labeling practices. The historical stereotypes created by selection bias difficult to erase.
2. Loan applications: consider experts in big bank that obtains data samples from some potential applicants and wants label them "qualified" "unqualified". Similarly, the experts likely to have systematic bias including:
(a) Implicit bias: similarly, Brock and De Haas 24 conduct lab-in-the-fiee experiment with over 300 Turkish loan officers to show that they bias against female applicants even if they have identical profiles as male applicants.
(b) Selection bias: when fewer female applicants are approved historically, the experts have less knowledge on females (i.e., whether they will actually default or repay), thereby tending to stay conservative.
15

---

C Related Work
C.1 Strategic Classification
Strategic classification without label changes. Our work is mainly based on an extensive line of
literature on strategic classification 12 6 12 14 18 25 26 27 28 29
30
31 32 These
works assume the agents are able to best respond to the policies of the decision- pecision-maker to maximize their utilities. Most works modeled the strategic interactions between agents and the decision-maker leads by publishing a classifier and the agents immediately best respond to it. The carliest line of works focused the performance of regular linear classifiers when strategic behaviors never incur label changes 5 251. while the later literature added noise to the agents' best responses [88] randomized the classifiers (26) and
as a repeated Stackelberg game where the decision-
limited the knowledge of the decision-maker (88 Levanon and Rosenfeld (2 proposed a generalized
framework for strategic classification and strategic hinge loss to better train strategic classifiers, but
the strategic behaviors are still not assumed to cause label changes.
Strategic classification with label changes. Several other lines of literature enable strategic behaviors to cause label changes. The first line of literature mainly focuses on incentivizing improvement actions where agents have budgets to invest in different actions and only some of them cause the label change (improvement) [3] 33 34 35| 36
40 41]. The other line of literature 44 45). These works argue that every strategic
37 38
39
focuses on causal strategic learning [32 42
43
learning problem has a non-trivial causal structure which can be explained by structural causal model, where intervening causal nodes causes improvement and intervening on non-causal nodes means manipulation.
Performative prediction. Several works consider performative prediction as more general setting where the feature distribution of agents is a function of the classifier parameters. Perdomo et al. [46] first formulated the prediction problem and provided iterative algorithms to find the stable points of the model parameters. Izzo et al. [27] modified the gradient- based methods and proposed the Perfgrad algorithm. Hardt et al. [47] elaborated the model by proposing performative power.
Retraining under strategic settings
Most works on learning algorithms under strategic settings consider developing robust algorithms that the decision-maie maker only trains the classifier once [ 18, (8), while Performative prediction [46] focuses on developing online learning algorithms for strategic agents. There are only a few works which permit retraining the Strategic classification models. However, all these algorithms assume that the decision-maker has access to new training dataset containing both agents' features and labels at each round. There no work considering the dynamies under the retraining process with model-ane -nmetl-n samples. Also, few works [14] [5] discussed the long-tern fairness issues under sequential strategic settings. Liu al. [13] modeled strategic behaviors
32 48
completely different way where each individual decides whether or not (a binary choice) to acquire the desired qualification based on a cost drawn from fixed distribution. Moreover, they assumed the decision maker has perfect knowledge of the agent distribution, i.e., can draw infinitely many examples from the agent population. Instead, our work focuses on practical setting where the decision maker must learn from samples and acquiring Numan-annotated samples is quite expensive, motivating the use of model-annotated samples as well. Zhang et al. [14] also simplified the modeling of the qualification changes of agents by using fixed transition matrices.
C.2 Bias amplification during retraining
There has been an extensive line of study on the computer vision field about how machine learning models amplify the dataset bias, while most works only focus on the one-shot setting where the machine learning model itself amplifies the bias between different groups one training/testing round 49 50); Another work theoretically studies dynamic benchmarking [31] to propose a retraining scheme to increase model accuracy with adversarial human aatraaaninttttion.... In recent years, another line of research focuses on the amplification of dataset bias under model-annotaree data where ML
models label new samples on their own and add them back to retrain themselves (2, 53
54
55
56
57 158 59 These works study the bias amplification in different practical fields including resource
allocation (57), computer vision (54), natural language processing (55). generative models (60 61
and clinical trials [59]. The most related work is 3 which studied the influence of retraining in the
16

---

non-strategic setting. Also, there is a work A touching on the data feedback loop under performative setting, but it focused on empirical experiments under medical settings where the feature distribution shifts are mainly caused by treatment and the true labels in historical data are highly accessible. Besides, the data feedback loop also related to recommendation systems. where extensive works have studied how the system can shape users' preferences and disengage the minority population 64]. However, previous literature did not touch on the retraining process.
[58 62
63
C.3
Machine learning fairness in strategic classification
Several works have considered how different fairness metries ( 16 65| are influenced in strategic classification [22 12 15 66). The most related works 12 15 studied how strategic behaviors and the decision-matee 's awareness can shape long-term fairness. They deviated from our paper since they never considered retraining and the strategic behaviors never incurred label changes.
D Additional Discussions
D.1 The source of Human-annotated samples
As stated in footnote 1, for the source of human-annotated samples, consider natural settings where the annaan-ndotated samples are drawn from fixed Px and the process is independent of the decision-making process, i.e., the agents who best respond to ft-1 are classified by model fa and the decision- maker never confuses them with human annotations. Instead, human annotation is separate process for the decision-maker to obtain additional information about the whole population (e.g., by first acquiring data from public datasets or third parties, and then labeling them to estimate the population distribution ).y) Here we provide an additional ** **exmmle***; admission scenario, the admission office uses the model fe to assign decisions at and obtain model- annotated samples for + 1. But for human-annotated samples, the admission committee hopes to acquire a more objective knowledge of "what kinds of students are successful" from the whole student population. Thus, the hope of avoiding additional sampling/selection bias, the committee seeks human annotations from third-party education researchers/experts. The experts can provide qualifications of the public/historical student samples drawn from the whole student population including students who are not applicants for this specific university or even do not attend university,
a university
In this section, we additionally consider the situation where all human-annotern annotated samples t are drawn from the post- resso-ee distribution P This will change 4 to the following:
AN +N 1,K (++))N+K -9'-- (++I)++K d/-1 [++I]N+IK
N
K
-i-1
where o' and a' denote the new qualification rate and acceptance rate, and 9:-1 stands for the qualification rate of the human annotations on features drawn from P Note that the only difference lies in the third term of the RHS which changes from qo to -;-1- Our first observation that q is never smaller than do because the best response will not harm agents' qualifications. With this observation, we can derive Prop. D.I.
Proposition D.1. df 2 holds for any 2 1. If Prop. 3.4 further holds, we also have of -> 1.
Prop. D.I can be proved casily by applying the observation stated above. However, note that unlike az ag not necessarily monotonically increasing.
D.2 Additional discussions on fairness
Other fairness metries. It is difficult derive concise results considering other f'arneess metries including equal opportunity and equal improvability because the data distributions play a role in determining these metrics. However, Theorem 3.3 states the true qualification rate of the agent population is likely to decrease, suggesting the retraining process may do harm to improvability. Meanwhile, when the acceptance rate a, increases for the disadvantageous group, the acceptance rate of the qualified individuals will be likely to be better, but it is not guaranteed because the feature distribution of the qualified individuals also changes because we assume the strategic behaviors are causal which may incur label changes.
17

---

suaffed
sreuud Hud
04
Figure 7: Visualization of distribution: Uniform data (left) and Gaussian data (right)
D.3 Additional Discussion on the training dataset S,
S, reuses the accumulated data instead of only fine-tuning the model using the available data the current round. This is more reasonable because: (i) The distribution shifts caused by agent best responses are not known by the decision maker, only using current data may result in forgetting the previous samples that may still be useful. Since agents' best responses will change and will provide useful information, especially
not break Pr|x. previous samples that current samples do not cover; (ii) the sample size at a single round may be too small even for fine-tuning (e.g., for college admission example, if we only have tens of people applying during some application round).
the feature domain
Moreover, only using current data does not qualitatively change the theoretical results. Since the theoretical results demonstrate monotonic trends of or, qt. only using the most recent data will not alter the trends. We perform additional set of simulations where the decision maker only uses samples from the most recent round (mm,1-1, S,,(1) 1) on Gaussian Data and Uniform-lineat Data with the experimental setups same the ones in the paper (Section 5 for Gaussian Data and App. E for Uniform-ling linear Data), and report the average of de, 9t. Dynamics of de, q are still similar.
1.0
1.0
1.0
0.9
0.8
0.8
0.5
0.4
0.4
0.2
0.2
0.2
0.0
2
10
12
14
10
10 12 14 16
12 14 16
(a) Uniform-linen linear: group (left) and (right)
(b) Gaussian: group i (left) and j (right)
Figure 8: Dynamics of a,, q: A, when S; only contains the most recent samples
D.4 Additional Discussions on refined retraining process
We provide the following proposition to illustrate how the refined retraining process leverages probabilistic sampler to stabilize the dynamics.
Proposition D.2. If the decision-maker uses a probabilistic sampler hx-1(x) = D (1/z) to produce
samples at then D DD
The proof details are in App. G.7 Prop. D.2 illustrates the underlying conditional distribution D is expected to be the same as D x
meaning that the classifier ft always learns the distribution of data, thereby only preserving the systematic bias. However, there is no way to deal with the systematic bias in refined retraining process.
D.5 Additional Discussions on the linearity of the model
Our model is built upon previous works on strategic classification (e.g., 2 decision policy was assumed be transparent and interpretable since human agents expect to face linear classifier which they can understand, especially under high-stake situations such as job hiring, college admission and loan application. Meanwhile, as pointed out by Zhang et al. [12], Raab and Liu [13, Levanon and Rosenfeld (67). in practical circumstances, the decision-maker can first fit a non-linear model to learn the embeddings of agents' preliminary features and then produce the embedded features. The docisiionmmkee then uses linear classifier on the new set of features and
12 32
where the
18

---

agents also best respond with respect to new features. Under this setting, nonlinearity is involved but our results still hold. Practical examples include (i) FICO credit score [[2]: FICO credit score is based complex set of features, and the decision-maker (e.g., bank) simply uses threshold of the score to assign decisions; (ii) Spam classification [67]: The decision- -akonr classifies spam
using a linear classifier, but the features are produced as an embedding by neural network based on number words in the post, number of phone numbers in the post and number of followers of the user.
E Main Experiments for other datasets
We provide results on a Uniform dataset and a real dataset 20 with settings similar Sec.
Uniform data, All settings are similar to the Gaussian dataset except that Px and Py (1/2) change shown in 2
We first verify the results Sec. 3 by illustrating the dynam- ics of az, ft, A, for both groups (Fig. 9.. Since our analysis neglects the algorithmic bias and the evolution results are in expectation, we perform experiments for every parameter configuration and show the averaged outcomes. The results are consistent with Thm. 3.3
Table 2: Gaussian Dataset Setting
Px.Saa) Prx((l)))
n47, P.go
20101) (55 1.0,0.05, 05,0.5
100 independent runs of
3.5 and 3.6 (i) acceptance rate a (red curves) increases monotonically; (ii)
qualification rate qr decreases monotonically starting from t = (since strategic agents only best respond from t 1); (iii) classifier bias A, evolves differently for different groups and it may reach the minimum after a few rounds of retraining.
1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0
14 16
2
4
6
8
10
12
14
16
Figure 9: az, qt. A, for group i (left) and (right)
Next, we present the results of a set of complemen- tary experiments on real data (20] where directly fit Pxjx with Beta distributions. The fitting results slightly violate Assumption 3.2 Also D is not equal to Px- More importantly, logistic models cannot fit these distributions well and produce non non-negee non-negligibbl al- gorithmic bias, thereby violating Assumption The following experiments demonstrate how the dynamics change when situations are not ideal.
Table 3: Description of credit approval dataset
Settings
Group
Group j
(x)1(1
Beta(1.37 (.,,333333233 3.23)
Beta(L.73,3.84)
(x.
Beta(1.50, 4.94)
Beta(1.59,4455
(x)]1
Beta(0.83, 2. 2.83)
Beta(0.66, 2.50)
Beta(0.84, 5.56)
Beta 69, 3.86)
r.
50, 0.05. 15, 0.473
50, 0.05, 15. 10
Credit approval dataset [20]. We consider credit card applications and adopt the data in UCI Machine Learning Repository processed by Dua and Graff [68]. The dataset includes features of agents from two social groups i and their labels indicate whether the credit application is successful. first preprocess the dataset by normalizing and only keeping a subset of features (two continuous X1, X2) and labels, then we fit conditional distributions Px./Y for each group using Beta distributions (Fig, 10) and calculate prior-best besir-best-respone qualification rates % d from the dataset. The details are summarized in Table All other parameter settings are the same as the ones of synthetic datasets in Sec.
We first illustrate the dynamics of az qt. As for both groups under different r. The results are shown Fig- II and are approximately aligned with Thm. 3.3 3.5 and 3.6 (i) acceptance rate at (red curves) has increasing trends; (ii) qualification rate de (blue curves) decreases starting from / = 1 (since strategic agents only best respond from /. 1); (iii) classifier bias A, (black curves) evolve differently for different groups.
19

---

Next, we illustrate situation (iii of Thm. 4.1 on this dataset, where the evolutions of unfairness and A, are shown in Fig. 12 Though the dynamics results, the changes are not smooth. However, this is not surprising because several assumptions are violated, and the overall trends still stay the same.
still approximately aligned with the theoretical
Group i
Group
Groupj
Group
inquilited qualifeed
unqualifle quaffied
unqua ffeed
unqualifed
755
7.5
7.5
7.5
5.0
5.
23
0.4
02
0.8
9.4
0.6
0.8
Ag
Figure 10: Visualization of distribution for Credit Approval dataset.
1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.2
0.0
10 12
(a) Group i in Credit Approval data: r = 0.1 (left), 0.05 (middle) and (right)
1.0
1.0
1.0
0.8
0.8
0.4
0.2
0.0
10
0.0
24
(b) Group j in Credit Approval data: r = 0.1 (left). 0.05 (middle) and (right)
Figure 11: Dynamics of a,, qt, A, for Credit Approval dataset
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
a
for Unfairmess
for Unfairness
for
0.4
0.4
Unfairesss
0.2
0.2-
0.0
0.0
10
12
16
14
10 12
(a) Dynamics of unfairness in Credit Approval data: r 0.1 (left), 0.05 (middle) and 0 (right)
0.25
4. for
for
0.20
0.35
0.15
0.15
0.30
0.10
0.10
0.20
0.05
& for
0.35
0.00
10 12 14
16
2
10
12 14 16
10
12
16
(b) Dynamics of A, in Credit Approval data; 0.1 (left), 0.05 (middle) and (right)
Figure 12: Dynamics of unfairness and A, for Credit Approval dataset
F Additional Results on SyntheticSSemi-synhhiii Datasets
In this section, we provide comprehensive experimental results conducted on two synthetic datasets and one semi-synthetic dataset mentioned in Sec. of the main paper. Specifically, App. F.I gives
20

---

the details of experimental setups; App. F.2 demonstrates additional results to verify Theorem 3.3 Theorem 3.6 under different (i.e., ratios of numan-annotatee examples available at each round). The section also gives results on how de, Qr. A change under long time horizon or when there is
no systematic bias; App. F33 further demonstrates the results under refined retraining process; App.
F.4 provides fairness dynamics under various values of on different datasets; App. [.3 illustrrass
how the results the main paper still hold when strategic agents have noisy best responses; App.
F compares the situations when agents are non-strategic with the ones when they are strategic, demonstrating how agents' strategic behaviors produce more extreme dynamics of ar, Qt. Ar.
F1 Additional Experimental Setups
Generally, we run all experiments on a MacBook Pro with Apple MI Pro chips, memory of 16GB and Python 3. 3.9.13. All experiments are randomized with seed 42 to run n rounds. Error bars are provided in App. H All experiments train ft with a logistic classifier using SGD as its optimizer. Specifically, we use SGDClassifier with logloas to fit models.
Synthetic datasets. The basic description of synthetic datasets and 2 is shown in Sec. and App. We further provide the visualizations of their distributions in Fig. 7
German Credit dataset. There are 2 versions of the German Credit dataset according to UCT Machine Learning Database [19], and we are using the one where all features are numeric. Firstly, we produce the sensitive features by ignoring the marital status while only focusing on sex. Secondly, use MinMaxScaler normalize all features. The logistic model itself can satisfy Assumption 3.2 with minimal operations: if feature has coefficients smaller than 0, then just negate it and the coefficients will be larger than 0 and satisfy the assumption.
F.2 Additional Results Verify Thm. 3.3 Thm. 3.6
Dynamics under different r. Although experiments in Sec. 5 and App. D galready demonstrate the validity of Thm. 3.3 Prop. and Thm. [.6, the ratio r of wumman-annotated examples is subject to change in reality. Therefore, we first provide results for E {0, 0.05, 0.1, 0.33 in all datasets. only has small values since human-annotated examples are likely to be expensive to acquire. Fig. 17 shows all results under different values. Specifically, Fig. 17a and 17b show results for synthetic dataset 1, Fig. 17c and [7d]show results for synthetic dataset 2. while Fig. 17c and [7 show results for German Credit data. On every row, r 0.3, 0.1, 0.05, from the left to the right. All figures demonstrate the robustness of the theoretical results, where o₂ always increases and decreases starting from
A, also has different dynamics as specified Thm. 3.6
Prop. 3.4 demonstrates that when r is small enough, a, will increase towards 1. Therefore, we provide results in all datasets when r to see whether increases to be close to 1. As Fig. 15 shows, a₂ is close to 1 after tens of rounds, validating Prop. 3.4
Dynamics under a long time horizon to verify Prop. 3.4
Dynamics under a different B. Moreover, individuals may incur different costs to alter different
features, so we also provide the dynamics of ac, gt. A, when the cost matrix B
6 in two
synthetic datasets. Fig. 16 shows the differences in costs of changing different features do not affect the theoretical results.
Dynamics when all samples are human-annotated. Though this is unlikely to happen under the Strategic Classification setting as justified in the main paper, we provide illustration when all training examples are human-annotated (i.e., r = 1) when humans systematically overestimate the qualification in both synthetic datasets. Theoretically, the difference between d, and 4 should be relatively consistent, which means A, is only due to the systematic bias. Fig. [5d verifies this.
F.3 Additional Results on refined retraining process
In this section, we provide more experimental results demonstrating how refined retraining process stabilizes the dynamics de, gt: A, but still preserves the systematic bias. Specifically, we produce
plots similar to Fig. 17 Fig. 18 but the only difference is that we use probabilistic samplers for
21

---

-onell-nnnotated examples. From Fig. [8 it is obvious the deviations of a, from Gr have the same directions and approximately the same magnitudes as the systematic bias.
F,4 Additional Results Fairness
In this section, we provide additional results on the dynamics of unfairness and classifier bias under We aim to illustrate situation (iii) in Thm. [4.7 where the refined retraining process is applied on the advantaged group while the original process is applied on the disadvantaged group j. From Fig. 19a 19c and 19e| can see unfairness reaches minimum in the middle of the retraining process, suggesting the earlier stopping of retraining brings benefits. From Fig. 19b 19d and 19n, we can see A for the disadvantaged group reaches minimum in the middle of the retraining process, but generally not at the same time when unfairness reaches minimum.
different r (the same settings in App. F22
F,5 Additional Results Noisy Best Responses
Following the discussion in Sec. we provide dynamics of a, (t, A, both groups under different similar App. F2 but when the agents have noisy knowledge. The only difference that the agents' best responses are noisy in that they only know a noisy version of classification outcomes: fe(z) = (x) + where is Gaussian noise with mean 0 and standard deviation 0.1. Fig shows that Thm. 3.3 to Thm. 3.6 are still valid.
F.6 Comparisons between Strategic and Non-strategic Situations
In this section, we show the absence of strategic behaviors may result in much more consistent dynamics of de It: A, illustrated in Fig. 2T
F.7 Additional Results a Dataset with Non-Linearity.
ACSIncome-CA dataset (69 is a larger dataset consisting of over 150K records for agents and their annual income. The decision- maker wants predict whether person has annual income > 50000. first trains a 2-d embedding using a neural network and 53 original
We assume the decision-make features then regards the embedding the new feature. We divide the agents into 2 groups based their ages. Similar to the credit approval dataset, we then fit Beta distributions on the groups and then verify the monotonic likelihood assumption (Fig. [3). We then plot the dynamics of a, gr, A, for both groups when the systematic bias is either positive or negative. The results show that similar trends still hold for this large dataset (Fig. 14
Group
Group
Group
inqualited
unqualited
unqua lied quaiifeed
imqueeiiie quo fled
aualifed
qualfied
10
10
10
6A
02
1.0
0.0
3.6
3.8
0.6
Figure 13: Feature distribution of Income Dataset.
22

---

0.6
0.6
0.4
0.4
0.2
0.0
8 10
10 12 14
Figure 14: Dynamics of de, St. A of the income dataset: the left plot is for group i and the right plot is for group j. We run 10 trials for each experiment and agent cost matrices are the same as the experiments in the main paper. The results are similar to the ones in the main paper.
1.0
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
(a) Uniform data when
0 and T' is large: Group
(b) Gaussian data when
0 and ''' is large: Group i
(left). Group (right)
(left), Group (right)
1.0
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
02
0.2
0.0
0.0
40
10 12
14
6
10
14
(c) German Credit data when = 0 and T is large: Group i (left), Group j (right)
(d) Dynamics when all data is human-annotatee (i.c,
r= 1): Uniform data (left), Gaussian data (right).
Figure 15: Dynamics of ae, qt. A, on all datasets when and T is large or when all examples are annotated by humans.
0.6
0.6
0.6
0.4
0.4
0.2
0.0
10
0.0
10 12
14
(a) Uniform data with different B Group i (left), Group j (right)
(b) Gaussian data with different B Group (left), Group j (right)
Figure 16: Dynamics of a gt. A synthetic datasets. Except B, all other settings are same in Sec. 5
23

---

100
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.0
(a) Group i in Uniform data: r 0.3, 0.1, 0.05, 0 from the leftmost to the rightmost
1.0
1.0
1.0
1.0
0.6
0.6
0.5
0.6
0.4
0.4
0.4
0.2
0.0
0.0
(b) Group j Uniform data 0.3, 0.1, 0.05, 0 the leftmost the rightmost
1.0
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
14
16
2
6
10
(c) Group Gaussian data r 0.3, 0.1,0.05,. 0.05,
the leftmost the rightmost
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0.0
0.0
2
10
12
14
16
0
2
10
14
16
10 12
16
0
10
12
14
16
(d) Group in Gaussian data
0.3, 0.1, 0.05, from the leftmost the rightmost
1.0
1.0-
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4-
0.4
0.4
0.2
0.2
10
(c) Group i in German Credit data
0.3,0 0,1, 0.05, from the leftmost to the rightmost
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.4
0.4-
0.4
0.4
0.2
0.2-
0.2
0.2
0.0
(f) Group j in German Credit data = 0.3, 0.1, 0.05, from the leftmost to the rightmost
Figure 17: Dynamics of a,, gt, A, on all datasets.
24

---

0.8
0.8
0.8
0.2
0.2
(a) refined retraining process for Group i Uniform data: 0.1 (left),
0.05 (middle) and 0 (right)
1.0
1.0
0.0
0.8-
0.8
0.6-
0.4
0.2
0.0
0.0
12
(b) refined retraining process for Group j in Uniform data: 0.1 (left),
0.05 (middle) and ( (right)
1.0
1.0
1.0
0.6
0.4
0.4
0.0
12
0.0
10
12
(c) refined retraining process for Group in Gaussian data: 0.1 (left),
0.05 (middle) and (right)
0.6
0.6
0.6
0.4
0.4
0.0
0.0
(d) refined retraining process Group y in Gaussian data:
(left), 0.05 (middle) and (right)
0.8
0.8
0.8
0.4
0.4
0.2
16
10
12
(c) refined retraining process for Group i in German Credit data: 0.1 (left). 0.05 (middle) and 0 (right)
1.0
1.0
10
0.8
0.4
0.4
0.1
0.2
10
12
14
16
10 12
14
10 12
(f) refined retraining process for Group in German Credit data: 0.1 (left), 0.05 (middle) and (right).
Figure 18: Illustrations of refined retraining process on all 3 datasets.
25

---

0.8
0.8
0.8
arfor
a: fort
tetor
a:fori Unfairesss
0.4
0.4
Unfaimeess
0.4
Unfairess
0.0
0.0
0.0
10
12
14
16
10
12
14
16
6
10
14 16
(a) Dynamies of unfairness in Uniform data (right)
0.1 (left), 0.05 (middle) and
0.25
0.30
2 for
0.25
0.35
0.15
A fort
for Ag
0.10
0.05
A.
(b) Dynamics of A in Uniform data r 0.1 (left). 0.05 (middle) and 0 (right)
0.8
08
0.8
autori
a: Unfairnsss
a:
for
0.4
04
Unfaimess
0.4
+
Unfaimess
0.2
0.0
0.0
0.0
24
16
10
12
(c) Dynamics of unfaimess in Gaussian data r 0.1 (left), 0.05 (middle) and (right)
for
0.20
for
for
0.20
for
0.35
0.15
0.00
0.05
0.00
6 8 10
36
10 12
14
16
10 12
(d) Dynamics of Ar in Gaussian data r 0.1 (left), 0.05 (middle) and (right)
1.0
0.6
0.6
0.6
for Unfaimess
a: for
for
0.4
Unfaimess
0.4
Unfaimess
0.2
0.2
0.2
0.0
24
10 12
14
14
(e) Dynamics unfairness German Credit data
0.1 (left), 0.05 (middle)
and 0 (right)
0.25
0.30
0.30
0.25
0.20
0.15
0.15
Ae Aofori
0.10
fr for)
fofori
0.05
8.05
for]
0.00
0.00
0.00
(f) Dynamics of A; in German Credit data : r 0.1 (left), 0.05 (middle) and (right)
Figure 19: Dynamics of unfairness and A, all datasets.
26

---

0.8
0.6
0.0
0.6
0.4
6
10
12
14
10
14
16
10
12
(a) Noisy retraining process for Group in Uniform data:
0.1 (left),
0.05 (middle) and 0 (right)
1.0
1.0
0.6
0.6
0.6
0.4
0.0
32
14
0.0
10
14
16
0.0
12
(b) Noisy retraining process for Group j in Uniform data: (middle) (right)
0.1 (left),
1.0
0.8
0.6
0.2
0.2
0.2
(c) Noisy retraining process for Group in Gaussian data: 0.1 (left),
0.05 (middle) and 0 (right)
0.8
0.8
0.8
0.2
0.2
0.2
0.0
12
10
16
10
12
(d) Noisy retraining process for Group in Gaussian data: 7 0.1
(left). 0.05 (middle) and (right)
1.0
1.0
0.4
0.4-
0.4
0.2
0.2
0.2
(c) Noisy retraining process for Group i German Credit data: 0.1 (left), 0.05 (middle) and (right)
10
0.0
0.8
0.8
0.6
8.0
0.0
0.0
10
16
10
12
(f) Noisy retraining process for Group j in German Credit data: 0.1 (left), 0.05 (middle) and 0 (right)
Figure 20: Illustrations of noisy retraining process on all 3 datasets.
27

---

0.8
0.8
0.8
0.2
0.2
0.2
(a) Nonstrategic retraining process for Group in Uniform data
0.8
6.8
0.0
0.6
0.2
0.2
022
12
(b) Nonstrategic retraining process for Group in Uniform data
0.8
0.8
0.8
0.6
0.2
0.2
0.2
(c) Nonstrategic retraining process for Group in Gaussian data
1.0
0.8
0.8-
0.8
0.4
0.2
12
10
16
(d) Nonstrategic retraining process for Group in Gaussian data
1.0
0.8
0.8
0.8
0.4
0.4
0.4
0.2
19
10
16
10 12
(e) Nonstrategic retraining process for Group i in German Credit data
1.0
0.8
0.8
0.8
0.4
0.2
0.2
0.2
0.0
10
16
10
(i) Nonstrategic retraining process for Group j in German Credit data
Figure 21: Illustaaioons of nonstrategic retraining process on all 3 datasets: r 0.1 (left), 0.05 (middle) and (right)
28

---

G Derivations and Proofs
G.1 Dynamics of the Expected Qualification Rate of St
Let us begin by defining 7, := Es, [Q(S:)] which is the expected qualification rate of the training dataset While it is difficult to derive the dynamics of a, and q: explicitly, we can first work out the dynamics of d using the law of total probability (details in App. G.I. i.c.
q.
(N+(t-1)K ((+1)N+IK 9.-1 + (++1)N+HK Q6-1 + (++1)N+IK 40
N
(4)
To derive q. := Es, [Q)s,)), we first refer to to get [S (t + 1)N +K Then, by the law of total probability, the expected qualification rate of S, equals to the weighted sum of the expected qualification rate of S,-1, S and the expectation of fi-1(r) over Sm,t1 as follows:
K (+ Es, (9(8 + (C+I)N+(K (A ([+1)N+IX Es,, - (Q(S [+1]]++K
The second expectation is exactly the definition of d Moreover, note that Q(S..t-)) Q(So.0) (Sa) = Yo for any L. >> 0, the third expectation is exactly To- so the above equation is exactly (4).
G.2 Derivation of factors influencing a,, Qt
As stated in Sec 2
get the factors influencing the evolution of a, q by finding all sources affecting p and the expectation of fa(x) over P-
We first work out the sources influencing fr and the expectation of fz( over P P'(X):
de since St trained with St, 4t is key factor influencing the elassifier.
hy may not model 1 accurately enough. However, we ignore this by Assumption 3.1 where realizability a common assumption in theoretical proof, while the experiments do not ignore this source.
BR we now know factors influencing the expectation of fe(x) over D'x- then the only left factor is the ones accounting for the difference between D and P Note that only the best responses of agents can change the marginal distribution Px. We then denote it as 8
Then, with Pxy known, P is only influenced by fi-1 (i.c., the agents' best responses to the classifier at - 1). fi-1 is also dependent on the above factors. So we get all factors.
G.3 Proof Theorem 3.3
proving the following lemma:
Lemma G.1. Assume & 2 2 and the following conditions hold: (i) q. > 7h-1 2 21-2i 2i (ii) Vi E X,
Drix(11az) 2 xx (1)x); (iii) Vx, (1:)(1) 2 Jr-2(1). Let J B._ [[-1)]J, =
ES.DD'y [fe], we have the following results:
Vx, f((x) 2 (()
2. There exists non-zero measure subset of values that satisfies the strict inequality.
Proof. We first prove (i). Note that assume hú models D well in Assumption 3.1 so f--1
(resp. f.) outputs I if D (1)z) (resp. y (1(z)) is larger than some threshold 0. Then, according to the above condition (ii), V0, if D (1|=)) > 0, D' xx (1/x) 2 (1 > 0. This demonstrates that (x) = implies (x)
and (i) is proved.
Next, according 2 if q. > 7. this means Ey~D; [v] EyyDY [v]. Since Dj D D ei-
ther there exists at least one non-zero measure subset of x values satisfying D > D (1/x) or Dr is more "skewed" to the larger values of x (because of monotonic likelihood assumption 3 For the second possibility, note that the only possible cause for the feature distribution in
29

---

the training dataset D'x to gain such skewness is agents' strategic behaviors. However, since (z) 2 (2) always holds, J--1 sets a lower admission standard where some values that able to best respond to / and improve will not best respond to J thereby impossible to result in a feature distribution shift to larger x values while keeping the conditional distribution unchanged. Thus, only the first possibility holds.
Then prove Theorem 3.3 using mathematical induction to prove stronger version:
Lemma G.2. When 02 01-1-
q
T--1 D' (1 Z (1)x) f(()) 2 ft-1( and finally
Proof. starts from 2, but we need to prove the following claim: q "almost equal" to Zo- so are Dyix and D yjx
Firstly, according to the law of total probability, we can derive Q₁ as follows:
N 2N + K
N 2N K ag 2N
K
7
K 40
(5)
The first and the third element are already multiples To- Also, we know D Px. Then, since D (1(z) falls in H and agents at t 0 have no chance to best respond, we have ap = Ex~Px Px (Dr (1|=)] To- Thus, dg is also equal Zn- and the claim is proved. Still, as Assumption 3.1
assumes realizability of he Ji is the same as Jo.
Next, can prove the lemma by induction:
1. Base case: Similar
5 we are able to derive 72 as follows:
2N +K
N
K
722 3N + 2K
41 +
01 +
Tu
(6)
3N + 2K
3N +
2K
Based on the claim above, we can just regard To as 71- Then we may only focus on the second term. Since T1 and d are "almost equal" and both distributions should satisfy the monotonic likelihood assumption 3.2 we can conclude fu. J are "almost identical". Then the best responses of agents to fo will also make them be classified as 1 by F1. and this will directly ensure the second term to be A(fi, P!) > A(f .P) = A The "larger than" relationship is because strategic best responses at the first round will only enable more agents to be admitted. Thus, the first and the third term stay the same as q while the second is larger, so we can claim T2 > 71- Moreover, the difference between D'y and D are purely produced by the best responses at = 1, which will never decrease the conditional probability of y = 1. Thus, Dr (1|x) 2 D
Together with J, Jo- all three conditions in Lemma G.1 are satisfied. we thereby claim that for every æ admitted by F2()) F (z) and there exists some satisfying the strict inequality. Note that P' is expected to be the same as p2 since fi = fo- Thus, A(f2, p2) > A(fi, p2) A(fi, P'), which 02 > 01 The hase case proved.
Induction step: To simplify the notion, we can write:
IN + (t (t- (t))
N
K (L+ I)N + IK
T.
(1+1)N + IK 7--1 + (1+ IK de-
70
A, 7-- + Br - a + C. To
Note that - can also be decomposed into three terms:
T--1 = (t 1)N + (t ()K
N
K
IN + (1 -1)K 7 + IN + (1 1)K
a4-2 +
7o
I + (t - 1)K
A 7- B, a-2 + C-1 go
30

---

Since the expectation in the second term is just at-1, and we already know ac-1 > 2t-2. we know B ac-1 + Ct 7o > Bt at-2 C. To. Note that B_
C.- we let the ratio be C.
Bi-1
m < 1. Then since B 0t-2 2 + C2-1 do > (Br-1 C--1) 4-1 due to 9- 46-2- we can derive B₁ 0111 + C 2o > m (Br_1 + C--1) 7-1 (Br + C.) 7--1- Then
T > (A₃ + B, +C+ 7-11 = 7--1. The first claim is proved. As a;-1 1 > ac-2 and D stays the same, any agent will not have a less probability of being qualified in St compared to in S
demonstrating D'y (1|x)) 2 Drix rix (1(x) still holds. And similarly, we can apply Lemma G.1 to
get (x) 2 (x) and or > a--1-
Now we already prove Lemma G.2 which already includes Theorem 3.3 We also want to note here, the proof of Theorem 3.3 does not rely the initial To which means it holds regardless of the systematic bias.
G.4 Proof of Proposition 3.4
We prove the proposition by considering two extreme cases:
(i) When K - 0, this means we have no decision-maker annotated sample coming in each round and N all new samples come from the deployed model. We prove lim;-+o dt 1 by contradiction: firstly, by Monotone Convergence Theorem, the limit must exist because a > Qe-1 and G₁ < 1. Let us assume the limit is ā < Then, since K when oc, Tr will also approach T. this means the strategic shift RR approaches O. However, this shift only approaches 0 when all agents are accepted by fr-1 because otherwise there will be a proportion of agents benefiting from best responding to fi-1 and result in a larger 40+1 Thus, the classifier at t + 1 will admit more people and the stability is broken. This means the only possibility is lime-so at-1 and produces a conflict.
(ii) When K + oo, the problem shrinks to retrain the classifier to fit D this will make a = 0o. N
Thus, there exists some threshold X, when K < X, lim4 1. In practice, the X could be very small.
N
G.5 Proof of Theorem 3.5
Firstly, since P differs from PXY only because agents' best respond to ft, we write q: go -re where r is the difference of qualification rate caused by agents' best responses to ft. Qualitatively, r is completely determined by two sources: (i) the proportion of agents who move their features when they best respond; (ii) the increase in the probability of being qualified for each agent. Specifically, each agent that moves its features increases its probability of being qualified from its initial point to point at the decision boundary of ft. For an agent with initial feature x and improved feature x" its improvement can be expressed as (xx Prjx (1|z*) Py|x (1/z).
Next, denote the Euclidean distance between æ and the decision boundary of fe as dg.t- Noticing that the agents will best respond to the decision classifier if and only if the Euclidean distance between her feature vector and the decision boundary is less than or equal to some constant C' no matter where the boundary is (Lemma 2 in 2 and what the cost matrix B is, t as follows:
express the total improvement
T(I) 1
f1. (x( (x)) dr Jd
According to the proof of Theorem 3.3 all agents with feature vector x who are admitted by fo will be admitted by fe (t >> 0), making all agents who possibly improve must belong to J. When Fx convex in or it is convex each dimension separately, we know Px as its derivative will be non-decreasing in each of the d dimensions within J. Similarly, when Pyxx (1[z) is convex in each of its dimensions or the whole J, U is also non-decreasing in each of the d dimensions within J. Then note that Se always lies below f:-1, therefore, for each agent who improves at having feature vector Ti,t, we can find an agent at 1 with corresponding IJ,t- such that both Px Px( 2 Px (wr,)) and U (211-11) 2 ( This will ensure (tt) [[t 1). Thus, de decreases starting from 1. L
31

---

G.6 Proof of Theorem 3.6
P) 2 O: according to Def. 2.1 Gg = -Exx D
[x)
> Epx (xx
9o. Now that do decreasing, so A is always increasing.
2 O. Based on Thm. 3.3 and Thm. 3.5 is increasing, while is
,(D", P) < O: similarly we can derive do 9o < 0, so Ao = |g 40 = 90 So while 9o is still increasing, A, will first decrease. Moreover, according to Prop. 3.4 if K small enough, A, will eventually exceed 0 and become larger again. Thus, A, either decreases or first decreases and then increases.
N
G.7 Proof Proposition D.2
Firstly, So = So.0 and S = So U S,, U UmS.-- Obviously, the first
drawn from Dxy-
Consider Sm,o. since it now produced by labeling features from Px = D& with Do Sm,0 is also drawn from yy- Thus, D Dii
Then prove the cases when t using mathematical induction follows:
1.
Base case: We know S2 Si S,, Sm,1 The first two sets on rhs drawn from D The labeing in the third set is produced by ha (x) 1x Di Thus, the base case is proved.
Induction step: S. S U So,t U. Similarly, we only need to consider the third set. But Note that (x))) = rrixx = D2px the induction step is easily completed.
G.8 Proof of Proposition G.3
At round I, know Att
A-c To reach demographic parity, the acceptance rates need be the 0f
same, so at least one of the following situations must happen: (i) 0 > 6 and 0 > 0/: (ii) 0f 0 and 02 09: (iii) 0f 2 0f and 02 < 0%. Next prove that (i) and (ii) cannot be by contradiction.
Suppose (i) holds, then we can find B. < 0f such that (B. E (e(0) St), (00) S!)) and A 0
E
(AJe
gt
00)) We can indeed find this Ty because e, A are continuous w.r.t. 0. Now noticing that
0/r
A
A 02
> A
but A Air of we will know we can find 0² E (0i, 0i) to satisfy demographic 02
0
parity together with 8'. Since (1 satisfies monotonic likelihood and 0f is the optimal point, (0,,,) < St) must hold. Thus, (F,, satisfy demographic parity and have a lower loss than (0i, R.). Moreover, the pair satisfies (iii), which produces a conflict.
Similarly, we can prove (ii) cannot hold by contradiction, thereby proving (iii) must hold.
G.9 Proof of Theorem
where we can just regard the trajectory of the acceptance rate of group j as moving the one of group vertically downward; Situation (ii) is derived from Prop. D.2 where the systematic bias causes unfairness and it keeps stable; Situation (iii) can be derived from Thm. 3.6] where the acceptance rate of group i stays stable, while the one for monotonically increases.
Situation (i) is directly derived from Thm. 3.3
G.10 Proof of Theorem 4.2
The first fact is that both 0% 0j stay stable as 0.5 #4 and 0.5 My when we use accuracy as the metric. Therefore, to understand which group is disadvantaed/ldvaatagee at 1.4 1, we just need to track the agents who best respond to f; 0/_ First, we need the following proposition which is proved in G.8.
G.8
Proposition G.3. If XX. Xy continuous, then 0; 0g and 0. - 0².
32

---

Note that we want to sacrifice the least accuracy to reach demographic parity. Since the agent best response will never make them worse off, we will know aj corresponding to the acceptance rate of group under the optimal threshold 0 is always larger than or equal to og. Thus, it suffices to show that under the condition of Thm. 4.2 ag < aa.
Suppose of < ob. then agent in group reaches 07 after its best response. Denote this event as E and the threshold agent
Chebyshev's Inequality to bound the probability that an unadmitted
feels Z2 where E(I2)= 0j. Then we can write PPE) P(1, E(L)) 0f 02-03 02) 10 10-0)- )
0/-0z Vag-a! we know P(E) ab ay, and from the simple fact 4f41 a + P(B).
Thus, when oe
we know j will stay disadvantaged at
However, when we do not add the faimess intervention, just consider the situation where all unad- mitted agents in group at improve, while no one in group can improve. This will flip the disaadaantaeedaadvaatgee group.
33

---

H All plots with error bars
1.0
10
1.0
0.8
0.6
0.6
0.6
0.6
0.4
0.2
0.0
0.0
20
12
0.0
10 12
(a) Uniform data with different B: Group (left),
(h) Gaussian data with different B : Group (left),
Group j (right)
Group j (right)
Figure 22: Error bar version of Fig. 16
0.6
0.4
3e
Unfalmess
0.0
0.0
-0.2
10
12
10 12
Figure 23: Error bar version of Fig.
0.8
0.6
0.6
0.6
0.2
02
0.0
0.0
0.0
12
(a) Gt. Ae for group (left) and j (right)
(b) unfaimess (Ieft) and classifier bias A, (right)
Figure 24: Error bar version of Fig.
0.8
1.0
0.8
1.5
1.8
3.0
4.
0.4
0.2
Fer
0.0
-1.0
Uhib
-3.0
2 4
8 10
12
14
16
0
2
8
10 12
14
16
(a) de. Qt.
Ae for group (left) and j (right)
(b) unfaimess (left) and classifier bias A (right)
Figure 25: Error bar version of Fig. 9
1.0
0.8
0.8
0.6
0.4
0.2
02
2 4
0.0
2
10
12
14
(a) Gaussian: Ge. 4t. A for group (left) and j (right)
(b) German: ds, 4r. A, for group i (left) and j (right)
Figure 26: Error bar version of Fig.
34

---

1.0
88-
0.8
0.6
0.6
0.6
0.6-
0.4
0.4
0.2
0.0
0.0
12
10 12
12 14
(a) Uniform-linear: group (left) and j (right)
(b) Gaussian: group i (left) and (right)
Figure 27: Error bar version of Fig. 8
1.0
10
0.8
0.8
0.6
0.4
0.6
0.4
02
0.2
0.0
0
2
10
12
14
16
2
10
14161820
36
24
32
40
16.24
32
40
48
56
61 72
(a) Uniform data when
and large: Group
(b) Gaussian when and T is large: Group (left), Group (right)
(left), Group j (right)
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
12
(c) German Credit data when
O and T is large: (d) Dynamics when all data is human-annotaan (i.e. synthetic datasets.
Group (left), Group (right)
Figure 28: Error bar version of Fig. 15
1.0
1.0
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10
14
(a) Group in Credit Approval data: r = 0.1 (left), 0.05 (middle) and (right)
1.0
10-
0.8
0.8
0.8
0.6
0.6
0.4
0.4
(b) Group j Credit Approval data: = 0.1 (left). 0.05 (middle) and (right)
Figure 29: Error bar version of Fig.
1.0
0.8
0.4
0
2
6
10
12
14
16
3
10
12
14
16
Figure 30: Error bar version of Fig.
14
35

---

1.0
1.0
1.0
1.0
0.8
0.6
0.4
0.4
0.4
0.4
0.0
0.0
0.0
(a) Group i in Uniform data r 0.3, 0.1, 0.05, 0 from the leftmost to the rightmost
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.4
0.4
044
0.4
0.2
0.0-
0.0
0.0
(b) Group j in Uniform data r 0.3, 0.1, 0.05, 0 from the leftmost to the rightmost
1.0
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
14 16
10
16
2
10
(c) Group in Gaussian data 0.3, 0.1, 0.05, 0 from the leftmost to the rightmost
1.0
1.0-
1.0
0.8
0.8
0.8
0.6
0.4
0.4
0.2
0.0
0.0
0.0
0.0
2
(d) Group in Gaussian data : r 0.3, 0.1, 0.05, from the leftmost to the rightmost
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.2
0.0
2
14
16
2
4
6
8
10
12
14
2
4
6
8
10
12
14
16
(e) Group German Credit data r 0.3, 0.1, 0.05, from the leftmost to rightmost
0.8
0.8
0.8
0.8
0.6
0.4-
0.4
0.2
0.2
0.0
0.0-
0.0
2
10
12 14
16
0
10
14
16
0
2
4
10 12
14
0
2
8 10
12
14
16
(f) Group j German Credit data
0.3, 0.1, 0.05, from the leftmost to the rightmost
Figure 31: Error bar version of Fig.
17
36

---

0A
02
0.
refined retraining process Group in Uniform data: 0.1 (left), 0.05 (middle) and (right)
00-
0.3
0.
0.6
02
00
60
0.0
10
22
14
16
12
19
10
14
(b) refined retraining process for Group in Uniform data: 0.1 (lcft), 0.05 (middle) and (right)
02
000
refined retraining process for Group Gaussian data: 0.1 (left), 0.05 (middle) and (right)
10
08
0.8
04
02
(d) refined retraining process for Group j in Gaussian data: 0.1
(left), 0.05 (middle) and (right)
1.0
14
08
68
0.8
06
06
refined retraining process for Group in German Credit data: 0.1 (left), 0.05 (middle) and (right)
(1) refined retraining process for Group in German Credit data: (left), 0.05 (middle) and (right).
Figure 32: Error bar version of Fig. 18
37

---

1.00
0.93
0.03
0.
5.0
-0
92
20
(a) Dynamics of unfairness in Uniform data r 0.1 (left), 0.05 (middle) and
0 (right)
0.4
0-4
0.4
92
-02
-2
0.2
14
12
10
(b) Dynamics of A in Uniform data
0.1 (left), 0.05 (middle) and
(right)
1.2
0.8
2.1
S mnf
04
3:fori
forj
0.2
Unb mezs
-22
-2
-32
(c) Dynamics of unfairness in Gaussian data and (right)
0.1 (left), 0.05 (middle)
8.
0:3
3.4
02
022
03
9.0
-0.1
-01
34
24
10
(d) Dynamics of Ae in Gaussian data (right)
0.1 (left), 0.05 (middle) and
1.0
0.8
Unfeimms
Uniiimess
0.4
Unfwimees
0.2
0.2
0.0
0.
0.0
1b
15
16
14
Dynamics unfairness German Credit data (middle) and 0 (right)
0.1 (left), 0.05
0.5
0.3
0.2
0.2
01
4 tor
0.0
2 for
Aoro
10 12
10
12 14
(l) Dynamics of Ar in German Credit data (right)
0.1 (left), 0.05 (middle) and
Figure 33: Error bar version of Fig.
19
38

---

10:
1.0
1.
38
0.
0.3
Noisy retraining process for Group in Uniform data: 0.1 (left), 0.05 (middle) and (right)
08
0.8
033
08
00
(b) Noisy retraining process for Group in Uniform data: 0.1
(Ieft), 0.05 (middle) and (right)
10
34
12 24
10 32
(c) Noisy retraining process for Group in Gaussian data: 0.1 (left), 0.05 (middle) and 0 (right)
1.0
04
0.0
0.4
Noisy retraining process Group j Gaussian data: 0.1 (left), 0.05 (middle) and (right)
Noisy retraining process Group in German Credit data: 0.1 (left), 0.05 (middle) and (right)
02
00
0.0
0.0
12
16
10
(f) Noisy retraining process for Group in German Credit data:
(..1 (left), 0.05 (middle) and (right)
Figure 34: Error bar version of Fig.
20
39

---

00
(a) Nonstrategic refraining process for Group in Uniform data:
0.1 (left), 0.05 (middle) and (right)
1.0
14
08
0.8
06
0A
02
(b) Nonstrategic retraining process for Group j in Uniform data:
(1.1 (left), 0.05 (middle) and (right)
0.6
02
00
Nonstrategic retraining process for Group in Gaussian data:
0.1 (left), 0.05 (middle) and (right)
10
08
10
36
36
10
14
(d) Nonstrategic retraining process for Group j in Gaussian data: ().1 (left), 0.05 (middle) and (right)
1.0
1.0
08
08
(e) Nonstrategic retraining process Group German Credit
data: r = 0.1 (left), 0.05 (middle) and 0 (right)
(f) Nonstrategic retraining process for Group in German Credit data: 0.1 (left), 0.05 (middle) (right)
Figure 35: Error bar version of Fig. 21
40

---

0.5
200
0.2
-0.25
-22
200
Dynamics unfairness in Credit Approval data: 0.1 (left), 0.05 (middle) and (right)
24
24
02
(b) Dynamics of Ae Credit Approval data: 0.1 (left), 0.05 (middle) and (right)
Figure 36: Error bar version of Fig. 12
Although all our theoretical results are expressed in terms of expectation, we provide error bars for all plots in the main paper, App. E and F randomness is applicable. All the above figures demonstrate expectations as well as error bars (+ standard deviation). Overall, the experiments have reasonable standard errors. However, experiments in the Credit Approval dataset (20) (Fig. (36) incur larger standard errors, which is not surprising because the dataset violates several assumptions. Finally, note that we conduct 50-100 randomized trials for every experiment and should expect much lower standard errors the numbers of trials become large.
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract summarizes the paper's contribution appropriately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have a conclusion section to discuss the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
42

---

Justification:
See main paper and Appendix where all assumptions and proofs are clearly
stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
43

---

Answer: [Yes]
Justification:
The code and dataset are in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.ccppulli//guiii wwwiiccgguie///eeeeeee CoodeSubeiiieeeec ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details are in experiment section and appendix and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have standard errors in the plots for all experiments in App. H
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
44

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See App. F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttps://neurji wwweeerris..ccccpoeeeeeo http://ss...ppuuooocccccsccsllla
Answer:
[Yes]
Justification:
We reviewed the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Conclusion Section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
45

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all papers, datasets appropriately.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsiipteeeee www.reeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
46

---

Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "MUBfcfltDT", "reviewer": "Reviewer_WrhT", "summary": "This paper studies the effects of retraining machine learning (ML) models with data annotated by both humans and the models themselves, especially in social domains where human behavior is influenced by the ML systems. The authors explore how strategic human agents, who adapt their behaviors to receive favorable outcomes from ML models, can create feedback loops that affect the model's performance and fairness over time.\n\nThe paper begins by formalizing the interactions between strategic agents and ML models. It shows that as agents adapt to the decision policies of the model, they increasingly receive positive outcomes. However, this becomes problematic where the proportion of agents with truly positive labels may decrease over time. To address this issue, the authors propose a refined retraining process designed to stabilize these dynamics.\n\nThe authors analyze how fairness constraints imposed during each round of model retraining might impact disadvantaged groups in the long run. They find that enforcing common fairness constraints can sometimes fail to benefit these groups, highlighting the complexity of maintaining fairness in dynamic environments. The empirical section of the paper includes experiments on both semi-synthetic and real datasets. These experiments validate the theoretical findings, demonstrating that the proposed retraining process can mitigate some of the adverse effects of strategic behavior. The results show that while acceptance rates of agents tend to increase, the actual qualification rates may decrease under certain conditions, leading to a growing discrepancy between perceived and actual qualifications.", "questions": "- Can you provide more empirical evidence or case studies to support the validity of the assumptions made in your theoretical analysis?\n\n- Have you considered using more diverse and larger datasets to validate your findings? How might the results change with different types of data?\n\n- How do you envision your findings being applied to other domains beyond college admissions and hiring? Are there specific adjustments needed for other applications?", "limitations": "- The study mainly focuses on specific social domains like college admissions and hiring, and its applicability to other fields remains uncertain.\n\n- The proposed solutions, including the refined retraining process and fairness interventions, may not always be effective in practice. Real-world conditions can vary significantly from controlled experimental settings, which might affect the outcomes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a critical and relatively unexplored issue in ML – the long-term impacts of retraining models in the presence of strategic human behavior.\n\n- The formalization of interactions between strategic agents and ML models provides a robust foundation for analyzing these dynamics.\n\n- The investigation into how retraining processes affect algorithmic fairness is an important contribution of this work\n\n- The use of semi-synthetic and real data to validate the theoretical findings strengthens claims of the study", "weaknesses": "- The theoretical analysis depends on several assumptions, such as the monotone likelihood ratio property and the availability of perfect information about agent distributions, which may not hold true in all real-world scenarios.\n\n- The mathematical models and proofs are complex, potentially making it challenging for practitioners to implement the findings directly.\n\n- While the experiments support the theoretical findings, the datasets used might not fully represent the diversity and complexity found in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. We address your questions point by point as follows.\n\n> Empirical evidence or case studies to support the validity of the assumptions\n\n- The monotonic likelihood ratio property (MLRP): \n\nThis property is rather standard and has been widely used in literature (e.g., [11,12] in the main paper; Jung et al., 2020; Khalili et al., 2021; Barman et al., 2020); it means that each feature dimension is a good indicator of the target variable and an individual is more likely to be qualified when his/her feature value increases. \n\nRegarding the empirical evidence, we demonstrate the distribution of each of the two features in the Credit Approval dataset in Figure 10, where the fitted beta distribution satisfies the monotonic likelihood distribution. Moreover, [11] referred to in the main paper used FICO credit data and verified the monotonic likelihood assumption holds for people's FICO scores in Figure 15 of [11]. FICO has been widely used in the United States to assess people's creditworthiness and is a good practical example. In addition to the lending domain, MLRP has been studied in many other contexts, e.g., research has shown that higher wealth levels are often associated with higher likelihoods of certain favorable economic behaviors (like investment and consumption patterns), empirical data on insurance claims shows that higher levels of risk factors (like age, smoking status) are monotonically associated with higher likelihoods of claims.  \n\n- The perfect information about agent distributions\n\nWe want to clarify that the knowledge of agent distributions is only used when deriving theorems. In all the experiments, the decision-maker updates models by running empirical risk minimization on training data without knowing agent distribution $P_X$. To verify whether the conditions/assumptions in theorems hold, the decision-maker in practice may fit $P_X$ using the initial training set (e.g., Figure 10 for credit approval data). \n\nLast, we emphasize that the assumptions made in the theoretical analysis are sufficient but not necessary, i.e., the results may still hold even when assumptions are violated. This is validated in our experiments, where only the \"uniform-linear\" data satisfies all assumptions in the paper but all results are consistent with theorems.\n\n> Diverse/larger datasets and other domains\n\n- Our paper focuses on strategic classification settings [1,6,7,11,12,17,25,31,35]. According to the previous literature, human strategic behaviors are prevalent in high-stakes domains such as hiring, lending, and college admission, where humans have more incentives to improve their features for favorable outcomes. We believe the datasets adopted in our experiments are sufficiently representative and cover well-known data widely used in strategic classification literature. \n\n- We want to emphasize that our theoretical findings do not have any restriction on the dataset and are broadly applicable to other domains such as recommendation systems, fraud detection, etc., as long as they involve human strategic behavior and the decision-maker uses a linear model to make decisions. Note that the linearity of the decision model is a standard setting in strategic classification. In practice, one can first use a non-linear feature extractor to learn the embeddings of agents' preliminary features and then apply the linear model to the newly generated embedded features. Given the knowledge of both feature extractor and model, agents will best respond with respect to new features, and all our results still hold. The generalization to non-linear settings is also discussed in [13,28] in the main paper and [Levanon and Rosenfeld, 2021]. \n\n- To further validate the above arguments, we provide an additional case study using ACSIncome data [Ding et al., 2022], which contains information on more than $150K$ agents. In this study, the goal is to predict whether a person has an annual income $> 50000$ based on $53$ features such as education level and working hours per week. Specifically, we consider a decision-maker who first learns 2-D embeddings from $53$ original features using a neural network and then regards the embedding as the new feature. A linear decision model is trained and used on these new features to make predictions. We divide the agents into 2 groups based on their ages. Similar to the credit approval data, we then fit Beta distributions on the 2 groups and then verify the monotonic likelihood assumption (Figure 1,2 in the attached pdf). We then plot the dynamics of $a_t, q_t, \\Delta_t$ for both groups when the systematic bias is either positive or negative. The results show that similar trends still hold for this large dataset (Figure 3 in the attached pdf).\n\n### Conclusion\nWhile our theoretical analysis relies on some assumptions and the results are mainly evaluated in domains of lending and admission, the assumptions and datasets are mostly standard and commonly used in strategic classification literature. Nonetheless, we want to emphasize that our results are broadly applicable to other datasets and applications that involve human strategic behavior.     \n\n### References\nChristopher Jung, Sampath Kannan, Changhwa Lee, Mallesh Pai, Aaron Roth, and Rakesh Vohra. Fair prediction with endogenous behavior. In Proceedings of the 21st ACM Conference on Economics and Computation, pages 677–678, 2020.\n\nMohammad Mahdi Khalili, Xueru Zhang, Mahed Abroshan, and Somayeh Sojoudi. Improving fairness and privacy in selection problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8092–8100, 2021.\n\nBarman, S. and Rathi, N. Fair cake division under monotone likelihood ratios. In Proceedings of the 21st ACM Conference on Economics and Computation, pp. 401–437, 2020.\n\nDing, F., Hardt, M., Miller, J., & Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34, 6478-6490.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tOqWXcVMDq", "reviewer": "Reviewer_ovmu", "summary": "The paper explores a scenario where a machine learning system retrains itself over time by collecting data generated through ML system annotation itself as well as human-annotated data while allowing the distribution of training samples to evolve over time. This evolution is influenced by the strategic behavior of samples that may be marked as positive by the system. Based on this model, the paper investigates the dynamic processes of several metrics, including acceptance rate, qualification, and classifier bias.", "questions": "1. Assuming $N \\gg K$ and that strategic agents, using the output of the previous classifier, adapt to the classifier's outcome, it seems straightforward that the acceptance rate should increase. Could you clarify what the non-trivial or surprising aspect of Theorem 3.3 is?\n\n2. To what extent does the conclusion/results in Theorem 3.5 hold for classifiers beyond linear ones?\n\n3. In what situations can the convexity assumption in Theorem 3.5 be validated in practice?\n\n4. What does the term *cumulative* density function exactly refer to in the statement of Theorem 3.5? The domain of a standard cumulative function should be $\\mathbb{R}$ based on the usual definition. How, then, can it be restricted to a half-space $J \\subset \\mathbb{R}^d$ in Theorem 3.5? Is this a typo, or am I missing something?\n\n5. Line 99: What does the continuity of $P(Y | X)$ means here when $Y$ takes values in $\\{0, 1\\}$?\n\n6. Lines 187-188: Why does $\\mathcal{S}{m,t-1}$ have a higher qualification rate than $ \\mathcal{S}{t-1} $? Shouldn't that be the acceptance rate? Or am I missing something?", "limitations": "The authors have discussed the limitations of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem is well-motivated. \n\n2. The high-level model construction is interesting. For instance, the model considers the sequential aspects of the problem and analyzes how certain metrics change over time, which is more realistic than previous works.\n\n3. Additionally, the author incorporates 'systematic bias' into the modeling and discusses numerous real-world examples related to algorithmic fairness, an important topic for the community.", "weaknesses": "1. Some aspects of the results seem trivial or straightforward given the assumptions in the model. (Please see my questions below.)\n\n2. The writing could be improved, as some statements in the paper are confusing. (Please see my questions below.)\n\n3. The figure size, legend, and overall organization could be improved. For example, in Figure 2, the blue and red dots in the legend are too small, making the figure difficult to interpret. Additionally, the y-axis legend in Figure 2 uses different scales, complicating comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. We address your questions point by point as follows.\n\n> Theorem 3.3\n\nThe result in Theorem 3.3 (i.e., $a_t > a_{t-1}$) does **not** require $N >> K$ but only needs $N > 0$. Condition $N >> K$ is only used in Proposition 3.4, under which the acceptance rate converges to $1$ as $t \\to \\infty$ (i.e., all agents are admitted in the long run). For general cases with arbitrary positive $N$, we believe the increasing trend of the acceptance rate (Theorem 3.3) under complex dynamics between retrained model and strategic agents is not straightforward, rigorously proving requires non-trivial efforts by induction (see detailed proof in Appendix). \n\n> Theorem 3.5 beyond linear classifiers?\n\n- Note that the linearity of the decision model is standard setting in strategic classification (e.g., [1,6,7,11,12,17,25,31,35]). In practice, one can first use a non-linear feature extractor to learn the embeddings of agents' preliminary features and then apply a linear model to the newly generated embedded features. Given the knowledge of both feature extractor and model, agents will best respond with respect to new features, and all our results still hold. The generalization to non-linear settings is also discussed in [13,28] in the main paper and [Levanon and Rosenfeld, 2021]. \n\n- To further validate the above arguments, we provide an additional case study using ACSIncome data [Ding et al., 2022] with information on more than $150K$ agents. The goal is to predict whether a person has an annual income $> 50000$ based on $53$ features such as education level and working hours. Specifically, we consider a decision-maker who first learns 2-D embeddings from $53$ original features using a neural network and then regards the embedding as the new feature. A linear decision model is trained and used on these new features to make predictions. We divide the agents into 2 groups based on their ages. Similar to the credit approval data, we then fit Beta distributions on the 2 groups and then verify the monotonic likelihood assumption (Figure 1,2 in the attached pdf). We then plot the dynamics of $a_t, q_t, \\Delta_t$ for both groups when the systematic bias is either positive or negative. The results show that similar trends still hold for this large dataset (Figure 3 in the attached pdf).\n\n> In what situations can the convexity assumption in Theorem 3.5 be validated in practice?\n\n- In lines 236-238 we discussed the situations where the conditions of Theorem 3.5 hold theoretically (e.g. when $P_X$ belongs to Uniform distribution, Beta distribution, or Gaussian distribution when the first decision policy admits 50\\% or more agents). In practice, the decision-maker can fit the empirical distribution of the real data to get $F_X$ and then verify the convexity. Particularly, if the cdf does not have analytical form, we can verify the convexity empirically by sampling the points from the domain and check the convexity inequality $\\forall x_1, x_2 \\in J , \\lambda \\in (0,1), f(\\lambda x_1 + (1 - \\lambda) x_2) \\leq \\lambda f(x_1) + (1 - \\lambda) f(x_2)$.\n\n- The assumptions are sufficient but not necessary, i.e., Theorem 3.5 may still hold even when assumptions are violated. This is mentioned in lines 238-239 and validated in experiments. Among all datasets adopted in experiments, only the \"uniform-linear\" dataset satisfies all assumptions in Theorem 3.5, while the Gaussian dataset satisfies Theorem 3.5 only if the initial qualification rate of agents is larger than 0.5. The German Credit and Credit Approval dataset only satisfies the linear classifier assumption and the monotonic likelihood assumption. However, the empirical results on all datasets demonstrate the validity of Theorem 3.5. Thus, we believe the phenomenon in Theorem 3.5 should exist in reality even when some assumptions are not strictly satisfied.\n\n> Cumulative density function\n\n$F_X$ can have a large domain such as $\\mathbb{R}$, but Theorem 3.5 only needs it to be convex in a **subset** $J$ of its domain. Formally, $\\forall x_1, x_2 \\in J , \\lambda \\in (0,1), f(\\lambda x_1 + (1 - \\lambda) x_2) \\leq \\lambda f(x_1) + (1 - \\lambda) f(x_2)$. As an example, consider a 1-d Gaussian distribution $X \\sim N(\\mu, \\sigma^2)$, we know it is convex if $J = (-\\infty, \\mu)$.\n\n\n> Continuity of $P(Y|X)$\n\n$P_{Y|X}(y|x)$ (where $y$ can be $0,1$) is the conditional probability for an agent with feature $x$ to have label $y$. This is a function of $x$ and we assume it is continuous in $x$. For example, the logistic function $P_{Y|X}(1|x) = \\frac{1}{1+exp(-\\beta^Tx)}$ is continuous when $x \\in \\mathbb{R}$.\n\n> $S_{m, t-1}$ in lines 187-188\n\nYes, your understanding is correct. Because $S_{m, t-1}$ includes the model-annotated samples at $t-1$, the actual qualification of each sample is unobserved and is annotated using the pseudo-label generated by the model. In this sense, the \"qualification rate\" of $S_{m, t-1}$ is in fact the \"acceptance rate.\" The reason we used \"qualification rate\" is because from the decision-maker's perspective, when updating its model using empirical risk minimization, it doesn't distinguish model-annotated samples from human-annotated ones and would regard pseudo-label as the actual qualification. In other words, the dataset the decision-maker uses to update the model has more and more fractions of people with positive labels (regardless of whether labels are acquired from humans or model), i.e., qualification rate of $S_{t}$ (augmented by $S_{m, t-1}$) is higher than that of $S_{t-1}$. We will add more clarification to the paper.\n\n### References\n\nLevanon, S., & Rosenfeld, N. (2021, July). Strategic classification made practical. In International Conference on Machine Learning (pp. 6243-6253). PMLR.\n\nDing, F., Hardt, M., Miller, J., & Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34, 6478-6490.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YOnYlFcdOf", "reviewer": "Reviewer_JSvV", "summary": "This paper addresses the dynamics of machine learning systems when retrained with model-annotated and human-annotated samples in the presence of strategic human agents. It explores how these dynamics affect various welfare aspects, including those of applicants, decision-makers, and the broader social context. The work emphasizes the potential risks and unintended consequences of retraining classifiers in strategic settings, emphasizing the complex interplay between agent strategies and ML system updates.", "questions": "Please see the weaknesses part.", "limitations": "The authors have adequately addressed the limitations and societal impact.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an underexplored aspect of machine learning: the interaction of strategic human agents with ML systems over iterative retraining cycles. The incorporation of both model-annotated and human-annotated data into the retraining process, alongside the strategic adaptations of agents, presents a novel problem formulation.", "weaknesses": "1. The theoretical results depend on assumptions that may not hold in more complex or varied real-world scenarios, and the Gaussian, German Credit, and Credit Approval datasets in the experiment violate some conditions in the theoretical analysis. This limitation may affect the generalizability of the results.\n2. It would be beneficial to organize the experiments section of the main paper, providing a more detailed display of results related to the refined retraining process, which is a main contribution of this work.\n3. The study primarily examines binary classification scenarios with only two outcome classes (e.g., admitted/not admitted). It would be beneficial if the authors could explore how their findings and methodologies might be applicable to more complex machine learning tasks, such as multi-class classification or ranking problems, which are more common in real-world settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. We address your questions point by point as follows.\n\n> Assumptions of the theoretical results and the generalizability\n\n- Our paper focuses on strategic classification settings [1,6,7,11,12,17,25,31,35]. According to the previous literature, human strategic behaviors are prevalent in high-stakes domains such as hiring, lending, and college admission, where humans have more incentives to improve their features for favorable outcomes. We believe the datasets adopted in our experiments are sufficiently representative and cover well-known data widely used in strategic classification literature. While the Gaussian, German Credit, and Credit Approval datasets violate some conditions, the experimental results still show similar trends as the theoretical results, demonstrating the robustness and practical value of the theory. \n\n- The linearity assumption of the decision model is standard setting in strategic classification. In practice, one can first use a non-linear feature extractor to learn the embeddings of agents' preliminary features and then apply a linear model to the newly generated embedded features. Given the knowledge of both feature extractor and model, agents will best respond with respect to new features and all our results still hold. The generalization to non-linear settings is also discussed in [13,28] in the main paper and [Levanon and Rosenfeld, 2021]. \n\n- To further validate the above arguments, we provide an additional case study using ACSIncome data [Ding et al., 2022], which contains information on more than $150K$ agents. In this study, the goal is to predict whether a person has an annual income $> 50000$ based on $53$ features such as education level and working hours per week. Specifically, we consider a decision-maker who first learns 2-D embeddings from $53$ original features using a neural network and then regards the embedding as the new feature. A linear decision model is trained and used on these new features to make predictions. We divide the agents into 2 groups based on their ages. Similar to the credit approval data, we then fit Beta distributions on the 2 groups and then verify the monotonic likelihood assumption (Figure 1,2 in the attached pdf). We then plot the dynamics of $a_t, q_t, \\Delta_t$ for both groups when the systematic bias is either positive or negative. The results show that similar trends still hold for this large dataset (Figure 3 in the attached pdf).\n\n> Organize the experiments section of the main paper\n\nThanks for the suggestion. We will move the content of App F.3 to the main paper.\n\n> The study primarily examines binary classification scenarios with only two outcome classes (e.g., admitted/not admitted). \n\n- Similar to most studies in strategic classification [1,6,7,11,12,17,25,31,35], we focus on binary classification which is a standard setting in strategic classification and has many real applications (e.g., hiring, admission, loan approval) that involve binary decision-making on humans. Compared to existing works, ours is the first that examines the long-term impact of automating data annotation under strategic human agents.\n\n- Although the theoretical modeling and analysis under binary classification is already non-trivial, we may extend the analytical framework and insights to multi-class classification. Specifically, consider strategic agents with categorical labels $Y\\in \\mathcal{Y}$. Instead of improving features toward one specific favorable outcome (e.g., acceptance in lending/hiring, as given in Eq. (1)), each agent may have its own target $y^*\\in \\mathcal{Y}$ and they may best respond based on $x_t = argmax_z {Pr(f_{t-1}(z)=y^*)-c(z,x)}$. For the decision-maker, the model retraining process remains the same, i.e., it augments training data with model-annotated samples and human-annotated samples at every round. Under this broader setting, we may consider two scenarios: (i) all agents have the same target $y^*$; (ii) agents have different target $y^*$. For case (i), because agents move toward one class, the result is similar to a binary setting: instead of having the acceptance rate (resp. qualification rate) increasing (resp. decreasing) over time, the probability $P(f_t(X)=y^*)$ (resp. $P(Y=y^*)$) increases (resp. decreases) under certain conditions. For case (ii), the results will be more complicated because different targets could induce highly diverse agent behavior that disrupts the monotonicity change of distributions $P(f_t(X))$ and $P(Y)$. This is an interesting future direction.\n\n- Last, we agree with the reviewer that it is interesting to extend our work to multi-class classification or ranking, which we hope to study in the future. Indeed, such extension is non-trivial and has been an ongoing effort of the community. For example, [Liu et al., 2022] consider competition among strategic agents and is the first work studying strategic ranking to the best of our knowledge. We hope our paper can provide insights and shed light on future works.\n\n### References\n\nDing, F., Hardt, M., Miller, J., & Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34, 6478-6490.\n\nLiu, Lydia T., Nikhil Garg, and Christian Borgs. \"Strategic ranking.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\nLevanon, S., & Rosenfeld, N. (2021, July). Strategic classification made practical. In International Conference on Machine Learning (pp. 6243-6253). PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OACb7m0ufh", "reviewer": "Reviewer_ynjw", "summary": "This paper investigates an intriguing problem regarding how machine learning models evolve during dynamic retraining using model-annotated samples, incorporating strategic human responses. The authors discover that it becomes increasingly likely for individuals to receive positive decisions as the model undergoes retraining, although the proportion of individuals with positive labels may decrease over time. To stabilize the dynamics, the authors propose a refined retraining process. They also examine how these retraining processes can impact algorithmic fairness and find that enforcing common fairness constraints in every retraining round may not benefit the disadvantaged groups in the long term. Experiments conducted on both synthetic and real-world data validate the findings of this study.", "questions": "See weaknesses", "limitations": "N.A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper thoroughly analyzes how humans, acting as strategic agents, adapt their behavior in response to ML systems and how this behavior impacts the retraining process of ML systems. By formalizing these interactions and analyzing their long-term dynamics, the paper provides a theoretical foundation for understanding and predicting these complex interactions. This in-depth analysis helps uncover potential systemic issues and offers concrete theoretical support for improving models.\n2.\tThe paper not only identifies potential risks of retraining ML models with model-annotated data but also proposes an improved retraining method using a probabilistic sampler to enhance the quality of model-annotated samples. This method aims to stabilize the dynamics of acceptance and qualification rates, reducing classifier bias. The proposed solution is innovative and practical, helping to mitigate negative impacts in real-world applications.\n3.\tThe paper combines theoretical analysis with experiments on semi-synthetic and real data to validate the findings. The experimental results, which show consistent dynamics with theoretical predictions, enhance the credibility and applicability of the research. This approach ensures the reliability of the research findings by providing both theoretical and empirical support.", "weaknesses": "1.\tWhile the conclusions of the paper help understand the impact of human strategic behavior on ML systems, many of these conclusions are somewhat intuitive and straightforward. For instance, the increase in acceptance rates and the potential decrease in qualification rates over time are logically reasonable but do not provide particularly new insights. Delving deeper into underlying mechanisms or revealing more complex interactions could make the research more innovative and impactful.\n2.\tThe scale of the datasets used in experiments is relatively small, which may not fully capture the complexity of system dynamics in large-scale data environments.\n3.\tThe paper primarily focuses on linear models and specific distributions. The research conclusions might not fully apply to non-linear models and complex data distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. We address your questions point by point as follows.\n\n> While the conclusions of the paper help understand the impact of human strategic behavior on ML systems, many of these conclusions are somewhat intuitive and straightforward.\n\nWe believe the theoretical results are not intuitive. In section 3, we show that although the decision-maker keeps retraining the classifier, the increasing $a_t$ and decreasing $q_t$ demonstrate the model becomes more biased (Theorem 3.3 - 3.5). In section 4, we not only display the fairness dynamics of the retraining process but also research the influence of fairness interventions. In particular, Theorem 4.2 reveals that fairness intervention can even prohibit the disadvantaged group from becoming advantaged, which is also counter-intuitive.\n\n\n> The paper primarily focuses on linear models and specific distributions. The research conclusions might not fully apply to non-linear models and complex data distributions.\n\n- Note that the linearity of the decision model is standard setting in strategic classification (e.g., [1,6,7,11,12,17,25,31,35] in the main paper). In practice, one can first use a non-linear feature extractor to learn the embeddings of agents' preliminary features and then apply a linear model to the newly generated embedded features. Given the knowledge of both feature extractor and model, agents will best respond with respect to new features, and all our results still hold. The generalization to non-linear settings is also discussed in [13,28] in the main paper and [Levanon and Rosenfeld, 2021]. \n\n- To further validate the above arguments, we provide an additional case study using ACSIncome data [Ding et al., 2022], which contains information on more than $150K$ agents. In this study, the goal is to predict whether a person has an annual income $> 50000$ based on $53$ features such as education level and working hours per week. Specifically, we consider a decision-maker who first learns 2-D embeddings from $53$ original features using a neural network and then regards the embedding as the new feature. A linear decision model is trained and used on these new features to make predictions. We divide the agents into 2 groups based on their ages. Similar to the credit approval data, we then fit Beta distributions on the 2 groups and then verify the monotonic likelihood assumption (Figure 1,2 in the attached pdf). We then plot the dynamics of $a_t, q_t, \\Delta_t$ for both groups when the systematic bias is either positive or negative. The results show that similar trends still hold for this large dataset (Figure 3 in the attached pdf).\n\n- Finally, it is an interesting direction to extend our setting into non-linear decision policy while the agents can only respond to the classifier with the original features (not the embedding). This setting can be highly intricate because both the decision boundary and the agent best response will vary. We hope our paper can be a starting point and a solid foundation of more future work.\n\n> The scale of the datasets used in experiments is relatively small, which may not fully capture the complexity of system dynamics in large-scale data environments.\n\n-  Our paper focuses on strategic classification settings [1,6,7,11,12,17,25,31,35]. According to the previous literature, human strategic behaviors are prevalent in high-stakes domains such as hiring, lending, and college admission, where humans have more incentives to improve their features for favorable outcomes. We believe the datasets adopted in our experiments are sufficiently representative and cover well-known data widely used in strategic classification literature. \n\n- The additional case study illustrated in the above answer uses a larger dataset.\n\n\n### References\n\nDing, F., Hardt, M., Miller, J., & Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34, 6478-6490.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

We thank the reviewers and AC for reviewing our paper. Here we present a global response to the questions shared by multiple reviewers.

## Assumptions and Applicability of Our Model

- The monotonic likelihood ratio property (MLRP)

This property is rather standard and has been widely used in literature (e.g., [11,12] in the main paper; Jung et al., 2020; Khalili et al., 2021; Barman et al., 2020); it means that each feature dimension is a good indicator of the target variable and an individual is more likely to be qualified when his/her feature value increases.

Regarding the empirical evidence, we demonstrate the distribution of each of the two features in the Credit Approval dataset in Figure 10, where the fitted beta distribution satisfies the monotonic likelihood distribution. Moreover, [11] referred to in the main paper used FICO credit data and verified the monotonic likelihood assumption holds for people's FICO scores in Figure 15 of [11]. FICO has been widely used in the United States to assess people's creditworthiness and is a good practical example. In addition to the lending domain, MLRP has been studied in many other contexts, e.g., research has shown that higher wealth levels are often associated with higher likelihoods of certain favorable economic behaviors (like investment and consumption patterns), empirical data on insurance claims shows that higher levels of risk factors (like age, smoking status) are monotonically associated with higher likelihoods of claims.

- The linearity assumption

The linearity of the decision model is a standard setting in strategic classification (e.g., [1,6,7,11,12,17,25,31,35]). In practice, one can first use a non-linear feature extractor to learn the embeddings of agents' preliminary features and then apply a linear model to the newly generated embedded features. Given the knowledge of both feature extractor and model, agents will best respond with respect to new features, and all our results still hold. The generalization to non-linear settings is also discussed in [13,28] in the main paper and [Levanon and Rosenfeld, 2021]. 

Finally, all assumptions made in our theorems are sufficient but not necessary, i.e., Theorem 3.5 may still hold even when assumptions are violated. This is mentioned in lines 238-239 and validated in experiments. Among all datasets adopted in experiments, only the "uniform-linear" dataset satisfies all assumptions in Theorem 3.5, while the Gaussian dataset satisfies Theorem 3.5 only if the initial qualification rate of agents is larger than 0.5. The German Credit and Credit Approval dataset only satisfies the linear classifier assumption and the monotonic likelihood assumption. However, the empirical results on all datasets demonstrate the validity of Theorem 3.5. Thus, we believe the phenomenon in Theorem 3.5 should exist in reality even when some assumptions are not strictly satisfied.

## Empirical Evidence of Our Model

-  The datasets: our paper focuses on strategic classification settings [1,6,7,11,12,17,25,31,35]. According to the previous literature, human strategic behaviors are prevalent in high-stakes domains such as hiring, lending, and college admission, where humans have more incentives to improve their features for favorable outcomes. We believe the datasets adopted in our experiments are sufficiently representative and cover well-known data widely used in strategic classification literature. 

- **The additional case study**: To further validate the above arguments, we provide an additional case study using ACSIncome data [Ding et al., 2022], which contains information on more than $150K$ agents. In this study, the goal is to predict whether a person has an annual income $> 50000$ based on $53$ features such as education level and working hours per week. Specifically, we consider a decision-maker who first learns 2-D embeddings from $53$ original features using a neural network and then regards the embedding as the new feature. A linear decision model is trained and used on these new features to make predictions. We divide the agents into 2 groups based on their ages. Similar to the credit approval data, we then fit Beta distributions on the 2 groups and then verify the monotonic likelihood assumption (Figure 1,2 in the attached pdf). We then plot the dynamics of $a_t, q_t, \Delta_t$ for both groups when the systematic bias is either positive or negative. The results show that similar trends still hold for this large dataset (Figure 3 in the attached pdf).

### References

Christopher Jung, Sampath Kannan, Changhwa Lee, Mallesh Pai, Aaron Roth, and Rakesh Vohra. Fair prediction with endogenous behavior. In Proceedings of the 21st ACM Conference on Economics and Computation, pages 677–678, 2020.

Mohammad Mahdi Khalili, Xueru Zhang, Mahed Abroshan, and Somayeh Sojoudi. Improving fairness and privacy in selection problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8092–8100, 2021.

Barman, S. and Rathi, N. Fair cake division under monotone likelihood ratios. In Proceedings of the 21st ACM Conference on Economics and Computation, pp. 401–437, 2020.

Ding, F., Hardt, M., Miller, J., & Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34, 6478-6490.

Meta-review: This submission explores novel aspects in the area of strategic classification, namely the dynamics that develop when a population shifts due to strategic responses to published classification rules while the rules keep being adapted over time. All reviewers have found that the questions explored in this submission are novel and relevant. Initial questions raised were clarified during the author response period. 

While the model and results provided in this submission have been found to be somewhat simplistic, the novelty of the questions asked will likely result in follow up studies on the initial set of results presented here. As such, this submission will make an interesting contribution to the program at NeurIPS.
Decision: Accept (poster)