Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challenges
Authors: Usman Gohar
Zeyu Tang
Jialu Wang
Kun Zhang
Peter Spirtes
Yang Liu
Lu Cheng
Venue: Accepted by TMLR ()

Abstract: The widespread integration of Machine Learning systems in daily life, particularly in high-stakes domains, has raised concerns about the fairness implications. While prior works have investigated static fairness measures, recent studies reveal that automated decision-making has long-term implications and that off-the-shelf fairness approaches may not serve the purpose of achieving long-term fairness. Additionally, the existence of feedback loops and the interaction between models and the environment introduces additional complexities that may deviate from the initial fairness goals. In this survey, we review existing literature on long-term fairness from different perspectives and present a taxonomy for long-term fairness studies. We highlight key challenges and consider future research directions, analyzing both current issues and potential further explorations.

Jialu Wang Published in Transactions on Machine Learning Research (07/2025)
Long- Term Fairness Inquiries and Pursuits in Machine Learn- ing: A Survey of Notions, Methods, and Challenges
Usman Gohar Towa State University
ugoharBiastate.cm
Zeyu Tang
zeyutang@hemu.edd edu
Carnegic Mellon University
foldict@ucsc.eeh
University of California, Santo Cruz
Kun Zhang
kunal @cmu edu
Carnegie Mellon University & MBZUAI
Peter Spirtes
psfizSandrew.com. edu
Carnegie Mellon University
Yang Liu
yongliu@ucsc. edu
University of California, Santa Crus
Lu Cheng
lucheng@uic.. edu
University of Nliinois Chicago
Reviewed OpenReview: ottps://openrentt net/ forum? id= mdd=mYYEWvvF
Abstract
The widespread integration of Machine Learning systems in daily life, particularly in high- stakes domains, has raised concerns about the fairness implications. While prior works have investigated static fairness measures, recent studies reveal that automated decision-making has long-ten term implications and that off-the-shelf fairness approaches may not serve the purpose of achieving long-term fairness. Additionally, the existence of feedback loops and the interaction between models and the environment introduce additional complexities that may deviate from the initial fairness goals. In this survey, we review existing literature on long-term fairness from different perspectives and present a taxonomy for long-term fairness studics. We highlight key challenges and consider future rescarch directions, analyzing both current issues and potential further explorations.
1 Introduction
As Machine Learning (ML) algorithms assume increasingly influential roles high-stakes domains tradi- tionally steered by human judgments, extensive body of research has brought attention to the challenges of bias and discrimination against marginalized groups (Mehrabi al.
2021 Cheng et al. 2021b). These issucs are pervasive and manifest in different settings, including finance, legal (c.g., pretrial bail decisions), aviation, and healthcare practices, among others (Gohar et al. 2024 Barocas el. al. 2023). ). The communily has led tremendous research efforts toward measuring and mitigating algorithmic unfairness (Mehrabi et al. 2021). However, the majority such fairness approaches quantify unfairness based on predefined statistical or causal criteria, often assuming constant environment and system, defined as static fairness. This myopic conceptualiizatin of fairness usually focuses short-term outcomes and assesses only the instantaneous im-

---

Published in Transactions on Machine Learning Research (07/2025)
Long-Term Fairness Inquiries and Pursuits
Notion or Focus
Problem Setting &Mitigation
Recourse and/or Effort
Cumulative Utility
Dynamic Modeling
Performative Prediction
Interplay among Dynamics
Meritocratic Ranking
Distribution Shift
Delayed Impact
Pólya Urn
One-Step Feedback
Reinforcement Learning
Causal Modeling and Reasoning
Modeling Framework
Figure 1: A high-level taxonomy for long-term fairness, including notions, common modeling frameworks, and fair learning.
pact of interventions at single snapshot, overlooking system dynamics (dymamic fuirness)) and/or or long-term consequences over a time horizon (Liu et al. 2018 Zhang et al. 2020
Recent works have brought attention to the misalignment of long-term fairness considerations with the fairness objectives optimizod in static settings, and have shown that imposing static fairness criteria often does not guarantee long-term fairness and may even amplify discrimination (Liu et al. 2018: Hu et al., 2019). One way static fairness approaches fall short by failing to account for the future implications of current decisions individuals or groups, undermining their effectiveness. For example, in predictive policing, [Ensign et al.) (2018) observed how an initial higher allocation of police in a specific area leads to more reported incidents, perpeluating increased surveillance and exacerbating the marginalization of those communilies over time. To this end, large body of work has proposed various methods to measure, mitigate, and achieve different aspects of fairness in the long-term sctting rather than enforcing fairness for a single time step.
Simply put, long-lerm fairness constitutes the settings outside of stolic fairness framework and short-term outcomes by addressing fairness over extended periods, rather than focusing solely on immediate outcomes. Importantly, it represents spectrum of temporal fairness, from single-step effects (non-static) to soquential and multi-step fairness over exlended periods, all of which are incorporated within our laxonomy (Figure D to capture the rich complexity of real-world scenarios. While dynamic fairness aligns with this concept by considering evolving dynamics over time (Li et al. 2023), long-term fairness has a much broader scope. This umbrella term has different facets, including sequential fairness (where sequential decisions impact fairness) and fairness over multiple time steps, among others (as depicted Figure D]. In this work, we aim to unify the different strands of literature on long-tern fairness under a common framework. Through this effort, we identify various dimensions and seltings of long-term fairness and provide rich review of existing research to explore diverse approaches and considerations for achieving fair outcomes in dynamic and non-static contexts.
Difference from existing surveys. Several extensive surveys on MT. fairness have been conducted. A subset of these surveys (Chouldechova & Roth 2018 Mehrabi et al.) 2021 Finocchiaro et al. 2021 Pessach & Shmueli 2022 Gohar & Cheng 2023) predominantly focus static fairness specific vision of fairness (e.g., intersections techniques), with only brief and cursory discussions on long-term fairness in some. Meanwhile, other works have incorporated some aspects of long-term fairness. Gajane et al. (2022)
reviewed fairness approaches in Reinforcement Learning (RL), with overlap limited to very brief coverage of RL techniques for long-tend term fairness. Separately, Patro et al. (0022 critically examined the literature on Fair Ranking, discussing the limitations of static settings and exploring modeling frameworks, like simulations, within Ranking Systems only. The survey by Zhang & Liu (2021) focused only on sequential decision algorithms, representing one facet of long-term fairness inquiries, without critical comparative analysis of definitions, methods, and frameworks. Finally, Tang all (2023b) reviewed the fairness concepts in ML, drawing connections to moral and political philosophy. They provide overviews of modcling frameworks and choices in dynamic settings without making them the primary focus. In comparison to these works, our survey specifically concentrates on long-term fairness and offers a comprehensive and up-to-date overview of the field, introducing the taxonomy encompassing long-term fairness notions, mitigation techniques, and evaluation methods. Additionally, we critically examine existing techniques, highlighting key challenges and open research questions. Our literature search methodology is detailed in Appendix A.
2

---

Published in Transactions on Machine Learning Research (07/2025)
Contributions and Outline. Our main contributions are: (1) proposing a taxonomy (Figure ! for long-term fairness, including notions, problem settings, mitigation approaches, and modeling frameworks, examining long-term fair learning methods in multiple standard problem settings and evaluations, and
(2)
(3) highlighting key challenges and future directions. The rest of the paper is organized as follows: Scc-
tion 2 introduces long-term fairness definition, problem settings, and modeling frameworks. Section g dis-
cusses fairness notions, while Section A covers mitigation tochniques in dynamic scttings. Section 5 outlines
application-speciiic works, and Section G presents a critical analysis of the literature, including methodological assumptions, and practical implications for evaluating long-term fairness.. Finally, Section discusses open challenges and future directions.
explores alternative long-term fairness settings. Next, Section
2
Long- Lerm Fairness in ML: An Overview
In this section, we first attempt to define long-term fairness and then proceed to detail the different but intricately woven aspects that form this concept.
2.1 What Is Long- Term Fairness?
Given the dynamic nature real-world scenarios, long-term fairness emerges as a concept focused on achiev- ing and maintaining fairness over a period of time rather than within isolated decisions or instances (Hu & Zhang 2022). Long-term fairness has been described using various terms, often used interchangeably, in- cluding temporal, sequential, non-static, and dynamic fairness. However, these encompass different aspects of the overarching concept under complementary but orthogonal settings. Therefore, our first contribution is to identify and define these problem settings within a unified high-level taxonomy (Figure) 1 for understand- ing the literature on long-term fairness. The taxonomy does not aim to enumerate or exhaust all possible combinations across the three dimensions identified, but rather serves as a flexible organizational scaffold for situating diverse approaches in the literature, with more specific interactions detailed in Figure and Table
2.2 Problem Setting
Long-term fairness was first studied within pipelines, where multiple decisions are required. For instance, a two-stage hiring model involves a candidate selection from the applicant pool, followed by hiring deci- sions (Dwork & Ilvento 2018; Bower et al. 2017). However, long-term fairness has since expanded into multifaceted concept, encompassing various problem settings. In Figure | (red box), we identify the four key problem settings that underpin inquiries into long-term fairness. Dynamic Modeling is the primary setting where the majority of the long-term fairness considerations have been explored. It addresses dynamic factors in the environment to maintain fairness over time, including cases of sequential fairness where fairness isn't achieved over multiple decision rounds due to evolving system dynamics or the failure of static fairness notions when used at each time step (Chouldechova & Roth 2018 Cheng et al.j [221a]] 2022 Yin et al., 2023). It explicitly reprosents the temporal evolution of system, where the current state depends on both past states and actions, as well as feedback from the environment population. Another facet that cap- tures the real-world dynamics is the Distribution Shift setting, where deployment (testing) environments systematically differ from the training environment. This phenomenon can arise due to various factors, such as data non-stationarity (Barrainkua et al. 2023
Delayed impact is a special setting of long-term fairness where the impact of the decisions is not imme- diately observable but manifests gradually or after significant time lag (Liu et al., 2018).. It focuses on a model of delayed outcomes, without which one cannot foresee the impact fairness criterion would have if enforced as constraint on classification system. This framework is distinct because it emphasizes the lag between decisions and their eventual effects, typically without modeling ongoing or complex feedback from the environment. Finally, Performative Prediction is a recently introduced framework that formalizes scenarios involving the specific causal influence that predictions have on their intended targets (Hardt & Mendler-Dinner 2023). Unlike dynamic modeling, which models how predictions influence future states over time, abstracts away temporal dynamics and instead models the data distribution as a function of the deployed model itself. The goal is to find performalice equilibrium, a model that remains optimal even
3

---

Published in Transactions on Machine Learning Research (07/2025)
after inftuencing the data it observes. Perdomo et al. (2020) describe it as a form of distribution shift, but it uniquely focuses on distribution shift resulting solely from model deployment. While the impact of decisions on the environment has been explored in the dynamic modeling setting, performative prediction operates as a distinct framework (see Section G Apart from distribution shift, all these settings intersect with various techniques like bandits, reinforcement learning, performative prediction, and strategic classification, and hence, are not clearly separable, making it challenging to write this survey and develop the taxonomy. To address this complexity and capture these interactions, we identify common modeling frameworks that accommodate these dynamics while offering flexibility in inquiry settings, as detailed below.
2.3 Dynamic Modeling For Long- Term Fairness
The study of long-term fairness dynamics in literature involves various approaches shaped by interactions with the environment and feedback mechanisms. Typically, these works (1) propose a dynamics model tailored to specific domain, (2) expose unfairness from prolonged use of biased policies, and (3) suggest a "fair" policy to mitigate biases. As such, modeling frameworks (Figure | green box) play a pivotal role in addressing the long-term fairness challenge. In this section, we outline commonly utilized frameworks for modoling dynamics in long-term fairnoss.
The Pólya Urn (Mahmood] 2008) model is uniquely powerful framework for capturing reiffreinn dynamics. It simulates stochastic process of drawing balls from an urn that increases the probability of observing certain events. At each time step, ball is drawn and replaced with two of the same color, capturing the rich-get-richer dynamics (Akpinar et al. 2022). Its strength lies in modeling how small inl advantages can compound over time, while remaining mathematically simple, analytically tractable (Helfand 2013). and highly interpretable. As a result, the Pólya urn especially well-suited for studying cunulative advantage and preferential attachment in contexts like resource allocation, social networks, and recommendation systems (Barabási & Albert, 1999).
Pólya Urn:
One-Step Feedback: It explores the impact of decision-making on the underlying population after only a single slep of leedback, distinct from works that capture leedback over multiple time steps. It considers the dynamic response of the population at deployment, influencing fairness without model updates. This single-step evaluation is driven by practical considerations, acknowledging that other variables may dominate over extended periods in real-world scenarios (Liu et al. 2018).
Reinforcement Learning: Long-term fairness involves the model's interaction with the (possibly unknown) environment over time, a dynamic well-suited for RL (Liu et al.) (021). Previous works have applied RL techniques within both the Bandit and Markov Decision Process (MDP) frameworks to capture these dynamics. In both settings, the goal is to derive optimal policy that maximizes cumulative rewards while meeting fairness criteria like Demographic Parity (Calders et al.) 2009). This approach enables the exploration of complex scenarios, balancing the exploitation of existing knowledge (c.g., hiring from a known population) with the exploration of sub-optimal solutions to gather additional data (e.g., hiring from diverse backgrounds).
Causal Modeling and Reasoning: Focusing principled learning and reasoning about under- lying gettaaeeeeaaing processes, causality (Spirtes et al. 1993 Pearl 2009 Guo et al. 2020) has been recognized as a unique tool for better undorstanding and mitigating bias in different contexts. Previous works have proposed to model and reason about causal relations among variables to char- acterize discrimination in static settings (Kilbertus et al. 2017 Kusner et al. 2017 Nabi Shpitser Chiappa & Isaac 20188. ; Going beyond static towards dynamic and long-term settings, causal
2018:
modeling of dynamics has also been explored in the literature to varying extents of detail (Creager
el. al.
2020 Zhang el al. 2020 Tang el. al.] 2023a).
Other frameworks: Apart from these commonly used frameworks, the literature has explored several less studied perspectives that capture different aspects of long-term fairness and its dynamics. small but notable number of works examine incentive effects and strategic behavior (Hu al. 2019),, modeling how individuals may adapt or manipulate their features in anticipation of how an algorithm will respond, often
4

---

Published in Transactions on Machine Learning Research (07 (075)
Long- Term
Faimess Notions
Elloor/Recouree
Curuuaative Utiliy
Meritocratic
Dynamic Interplay
Simulation
Boundec-Elfort Reward
Threshold-
Recourse
Equal
Return Parily
Long- Tarm
Weakly- Meritocratic
Approximale-
Choice
Approximalee Reward
Tier Balancing
Effort Reward
Improvabiity
Equal
Fair Causel
Recourse
Causal
ELBERT
Group Merilocratic
Smooth Fairness
MAB
Causal
Figure 2: Taxonomy of fairness notions used in long-term scttings. The dotted boxes denote the adopted modeling frameworks.
using game-theoretic or best-response frameworks (e.g., Stackelberg games) (Dong et al. 2018 Milli et al. 2019 Zhang et al.j 2022). Other less common frameworks include social settings Heidari et al. 2019)) online learning (Bechavod et al.) 2019), and algorithmic recourse QUstun al., 2019 Von Kügelgen et al. 2022). While they not yet major themes in the long-term fairness literature, they offer useful insights for understanding the broader dynamics and directions for future work.
3 Long- Term Fairness Notions
Static fairness notions often focus on immediate fairness, overlooking long-term equity risks, potentially exacerbating bias over time (D' (D'Amour et al. 2020 Liu et al.) 2018). In contrast, in the long-term fairness setting, the notions capture non-static, dynamic nuances of fairness over time. In this section, we present various long-lerm fairness notions used literature in different settings, as shown Figure D Some of these notions can still be used in static settings, but have been adopted for long-term fairness using the dynamic modcling frameworks discussed in the previous section to capture dynamics. A summary of the main notations used in the paper is provided in the Appendix (Table 2
3.1 Effort-based Fairness
Static notions of fairness predominantly center ensuring similar outcome probabilities across all pro- tected groups (Heidari et al. 2019 Mehrabi et al. 2021). However, these notions often operate under the assumption of a static dataset, treating individuals as immutable entities within the given data context. These approaches fail lo consider that disadvantaged groups may need to exert more effort over time than advantaged groups to achieve similar outcomes, contradicting the principle Equal Opportunity (Roemer & Trannoy 2015 Von Kügelgen et al. 2022).. This oversight persists because the distance to the decision boundary is typically greater for disadvantaged groups. Inspired by this, Heidari al. (2019) proposed "effort unfairness", which captures the disparity in average effort required by disadvantaged group members compared to the advantaged groups. Crucially, effort-based fairness is inherently long- termterm and dynamic since it explicitly acknowledges that individuals can adjust their qualifications or features over multiple de- cision cycles, and that the effort required to reach favorable outcome may accumulate over time. This perspective allows the assessment of fairness as an ongoing process, capturing disparitics that persist or compound across repeated interactions with the system.
They propose Bounded-Effor unfairness as measure to quantify this disparity while constraining it within fixed level of effort. Given data sample denoted by tuple (x,y) where X Rd and y y (0, 1} denote the input vector and ground-truth label, respectively. Formally
Definition 1 Bounded Effort Heidori et al, 2019 Given constont å > 0, classifier f(x) satisfies effort fairness with 8 effortt if
P max f(x Az) 2 0.5, f(x) 0.5 g (()
max f(x Ax) 0.5, f(x) 0.5 u(Ax)58
(1)
1 For ease of reading and comparison, use notations defined in Guldogan 2022 for all effort-based definitions.
5

---

Published in Transactions on Machine Learning Research (07/2025)
holds Vg E G, Ar represents the effort (change) required to improve the feature to x', μ : R - [0, oo] worm which assigns non-negalivve value to impul. veclor, and p(Ar) S 8 constraints effort to be small.
Recognize that the condition f(x) < 0.5 denotes an unfavorable outcome, signifying the individual's un- qualified status. In contrast, condition f(r + Ax) 2 0.5 denotes a positive outcome, siguifying successful effort (Az) to qualify the individual. For simplicity, the features are assumedd to be numerical and monotone (Divesteijnn & Feelders, 2208). The parameter ò constraints the maximum effort to ensure that individuals groups are not required to make disproportionately high efforts for improvement. In contrast to the Bounded-Effort approach, Threshold-Reeun effort to setting a lower bound on the reward. This perspective assesses how the minimum effort required to surpass a defined reward threshold varies among protected groups. Intuitively, this notion captures the idea that a disadvantaged group has to exert greater effort to get the same reward. Finally, Effort-Rewart (Heidari el. al., 2019) unfairness is defined as the difference in maximum utility (difference between reward and effort) for each protected group after additional effort, avoiding reliance on specific thresholds to prevent fairness gerrymandering [Kearns et al. 2018) and reduce result variability.
unfairness (Heidari et al. 2019) shifts the focus from limiting
A closely related notion termed Equal Recourse quantifies the effort to altain desirable outcomes by measuring the average distance of individuals in different groups from the classifier's decision boundary (Gupta et al. 2019). Unlike bounded-effort unfairness, Equal Recourse aims to equalize average efforts across protected groups without imposing limits on effort.
Definition 2 Equal- Recourse Guppt al. 2019). Given the distonce (effort) Dx, f(x) is said achieve equal recourse Vg E
E min ((Az) f(z) < 0.5, = min ((Ar) f(x)= < 0.5 ((x+Ax)20.5
(2)
[(++Az))00 20.5
One drawback of this definition its vulnerability outliers. Given that Equal Recourse computes the minimum required elforts by taking an averge across all samples, the presence of outliers can distort the overall fairness measure. To address this, Guldogan et al. (2022) introduced Equal Improvability, akin to bounded-effort fairness, aiming to equalize average rewards for each group with a constrained effort. Formally:
Definition 3 Equal Improvability (Guldogan et al., 2022). Given a norm μ and a constant 8, f(x) achieves iqual-improvabiiityy fairness with 8 b-bounded-effort if 7g g
PP max [(x 2 0.5 f(x) < 0.5, 9 ((AA))
P max f(x + Ar) 2 0.5 f(x) < 0.5 (3) ((Ar)_ō
However, equal improvability differs from bounded-effort fairness in the conditional part f(x) < 0.5, i.e., it only focuses on equalizing the probability of rejected samples being qualified after applying a limited amount of effort. Von Kügelgen et al. (2022) build on the effort-based notions by introducing a causal version of Equal Recourse called Individual Fair Causal Recourse, focusing solely on rejected samples. They argue that the efforts to allain favorable decisions capture causal relations among features (Karimi et al. 2020 2021
Definition 4 Fair Causal Recourse Von Kügelgen el al. 2022 f(x)) is fuir if the following holds for all
latent variable u and groups 9,9'
min u(aa - ((gg,)) = min p(x' - x(g(,u)) 1(2)
(4)
fla120.55
Finally, Huan et al. (2020) extends this framework to introduce a causal-based notion that enforces equality of effort. However, it does not capture the dynamics of the environment and the model.
3.2 Cumulative Group Fairness
Another line of work CCi al. 2021 Wen et al. 2021 Yin et al. 2023) addressed the long-term nature of fairness by formalizing surrogate fairness notions grounded in static fairness notions, mirroring approaches
6

---

Published in Transactions on Machine Learning Research (07/2025)
These studies define long- term bias as the difference of the cumulative sum of group fairness (e.g., allocation rates) at each time step between different protected groups. These works have operationalized most of these notions within the MDP framework, leveraging RL techniques - a prevalent strategy for addressing long-term fairness dynamics (Zhang Liu 2021). An MDP is defined by a tuple (S, A, P. r,,,,)) where is the state space (e.g.; income levels), A is the action space (accept/reject) p is the initial state distribution, T: S + A(S) is the state transition function, R: S x A + IR is the immediate reward function (e.g., bank's 's earned profits), and y is the discount factor. The goal is to find policy S + ((A) that maximizes cumulative reward my(= == B, -r== R((ss, az)] where 80 ~ P. dr ~ rT-(s)), Sc+1 T(-191,a)) Chi et al. (2021 propose Return Parity, which stipulates that MDPs from distinct demographic groups, while sharing common state and action spaces, should aim to attain roughly equivalent expected rewards under a policy. Formally:
found in other fairness aspects such intersectionality (Gohar & Cheng 2023)
Definition 5 Return Parity (Chi et al, 2021). Given distinct initial distributions po and p1 of two protected groups, g E 10,1) denoting membership in the two groups with policies To, T1 then two MDPs satisfy c-return pority if
(TT))) - m((=))) Sc
In other words, e-return parity ensures that the difference in expected long-term rewards for two demo- graphics modeled using MDPs does not exceed a specified threshold of e, indicating similar outcomes. The reward function can be adaptod to model various fairness notions. Similarly, Wen al. (2021) extends other fairness notions, such as Demographic Parity and Equal Opportunity, to this MDP setting of equalizing long-tern mengtermman rewards for different protected groups. Finally, Yin et al. (2023) follows the same principle, but instead of equalizing long-term reward, their approach focuses on minimizing the violation of the sum of static bias at each time step.
While these notions are promising for modeling fairness beyond single time step, Xu al. 2023b) highlights that they fail to capture the variation in temporal importance. To address this, the authors propose the Equal Long-term Benefit Rate (ELBERT) fairness notion, which considers long-term fairness as the ratio of group supply and group demand (Xu et al., 2023b). They adapt static fairness notion ratios, e-g., Equal Opportunity, by formulating them as the ratio of group supply (qualified and accepted individuals) to group demand (qualified individuals). Unlike previous adaptations of static notions, they first take the sum of group supply and group demand separately before taking ratio. They highlight multiple cases (lending, medical treatment allocation) where temporal variations in group supply and domand load to a false sense fairness.
Definition 6 Long-Term Benefit Rale [uu el al. 20236). Given dústinct initial distributions po and of two protected groups, on MDP is soid to have achieved a Long- Term Benefit. Rate if
(T))
(6)
Po
is equalized across the prolected groups.
Here, (TT) and (T) represent the cumulative group supply and group demand, respectively, which are analogous to the reward function (It) and serve as supply S,(st, 0z) and demand functions Dy(s,, at) to the MDP at each time step.
3.3 Meritocratic Fairness
A body of research leverages bandit settings to model long- tergtermmae fairness. Joseph et al. (2016) initially iden- tified how no-regret algorithms (those avoiding negative foedback) in bandit settings may exhibit increasing unfairness over time. In response, they introduced an individual notion of fairness called Meritocratic Fair- ness, which posits that it is unjust to preferentially select one individual (e.g., for a loan) over another if the former is less qualified than the latter (Joseph et al. 2016), modeled in the contextual bandit setting.
In typical bandit framework, an agent chooses an arm A, from a set of K. arms, denoted by & E [X] at each time step I, and receives a stochastic reward R. These rewards are assumed to be drawn from unknown
7

---

Published in Transactions on Machine Learning Research (07 (07/2025)
distributions specific to each arm. In contextual setting, however, before making decision at each time step 1, the learner observes contextual information =y associated with each arm j. In terms of fairness, each arm is associated with a protected group, where of provides information about an individual from a protected group. Each population is associated with its own underlying function fz: mapping contexts to expected payofls. Formally:
Definition 7 Meritocratic Fairness JJoseph et al, 2016). For any pair of arms j. at time t, if,
5 2 f (x)fo):
(7)
the algorithsm is considered discriminalory, i.e.; if il preferentially chooses the lower quality arm jo- Con- versely, the algorithm is fair if it ensures, with high probability over all rounds and for all pairs of arms j. jo, that whenever fa())) 2 foo (1%)), the algorithm chooses arm with probability at least as high as it chooses
arm Jo.
Liu et al. (017)) complement the aforementioned work by Joseph et al. 2016
2018) by incorporating
hnest constraint and protecting higher qualifications over lower qualifications in an on-average manner. Meritocratic Fairness assumes that k distinet groups, each with a unique mapping from features to expected rewards, are already known. Moreover, it assumes one group member is available and always selected in each round, which is unrealistic (Joseph et al. 2018). To relax this assumption, ,Joseph et al. (2018) introduce the notion of
fairness, which does not assume knowledge of protected group membership.
Definition 8 8- d-Weakly Merilocralic Pairness (Joseph el al., 2018 model is 8-fair if, for all sequences of contezts ; Lt, with probability at least 1 ô over the realization of the history h, for all rounds E [T'] and all pairs of orms j.jo E [
jjh > o Jolh
only if f1(25) > feo (51(2)))
(8)
These notions, however, define fairness with respect to one-step rewards. To capture the desired interaction and evolution of the environment, Jabbari et al. (2017) extends Meritocratic Fairness to the MDP formu- lation, where it requires that an algorithm never probabilistically favors an action with lower long-term reward over an action with higher long-term rewards. Due to the exponential complexity of satisfying this, they propose two relaxations of this notion: 1) Approzimate-choies and 2) Approzimate-action fairness. A Approximate-choice fairness prevents equally good actions from being chosen at very different rates, while approximate-action fairness prevents substantially worse actions from being chosen over better ones. Grazzi et al. 2022) build upon these works to propose the Group Meritocratic Fairness notion in the contextual ban- dit setting. Specifically, they introduce the idea of highest relative rank, which evaluates the reward's 's quality relative to other candidates within the same group. This approach ensures that the chosen candidates are not only selected based on their absolute rewards but also considering their position within their group's 's reward distribution. The aim to address fairness disparitics in scenarios with diverse group reward distributions.
3.4 Cousality-Motivatee Long- Term Fairness Inquiry
Causal modeling of underlying dynamics in long-term settings has been recognized as beneficial for analyzing social implications of automated makiing (Creager et al., 2020 Zhang et al.) 2020). To achieve long- term fairness, understanding the interplay of various processes is crucial (Tang et al. 2023b). Tang et al.) (023a) present detailed causal model, in terms of a directly acyelic graph (DAG) for two key dynamics in decision-disttriuttin interplay: (1) the decision-making process based on data at each time step and (2) the underlying tetta-gnerratng process influencing future distributions. Rather than enforcing fairness criteria at cach time stop, they propose sctting a goal of achieving long-term fairness and evaluating it alongside.
Definition 9 K K-Step Tier Bulancing [Tang el al.) 2023a A sequence of K decision-makiin stralegies (empty set if K = 0) satisfies K-Step Tier Balancing if at the time
Dr.I+K-1- = (Drr..,,Dr-Kk Dr+k-1} step T K the following independence relation holds true (where "11 " denotes statistical independence, and A denotes the protected feature):
HTTK K II Artk:
8

---

Published in Transactions on Machine Learning Research (07/2025)
where HI+K is induced by K- K-step dynamics.
This notion extends beyond observed variables to include latent causal factors, i.e. "tiers", that carry on the influence of current decisions to future data distribution through the interplay between dynamics. Through detailed causal modeling of the dynamics and the corresponding analysis of their characteristics, [Tang] et al. (2233a) formulate the one-step tier imbalance reduction (STTR) term and present possibility and impossibility results to illustrate the feasibility of setting Tier Balancing as a long-term fairness goal.
3.5 Discussion
Existing long-term fairness notions differ based on the modcling framework used and what aspect (c.g.. effort, merit) of fairness they capture dynamically. Cumulative fairness, for instance, extends previous static fairness notions to the long-term context by aggregating fairness measures at each time step using RL. However, recurrent RL policy deployments that enforce static fairness notions at each time step may not be universally suitable for real-world scenarios (Henderson et al.) 2018), necessitating the development of lightweight techniques. Recourse and effort-based notions, adapted from static settings, are only one-step analyses of long-term fairness applied iteratively to address equality of effort. One limitation is that they measure the anticipated effort in response to algorithmic decisions, neglecting the legal (e.g., affirmative action) and philosophical emphasis on actual historical effort. To bridge this gap, future work could focus ensuring parity in outcomes based on the effort already put in and developing algorithms and datascts to capture it in long-term settings. Notably, while effort-based notions consider how individuals respond to incentives, they generally assume straightforward improvements and do not capture complex strategic adaptation, where behavior may change in unexpected or adversarial ways to influence outcomes (Zhang ct al. 2022). Incorporating this strategic dimension is an important direction for future research. Morcover, some notions, like meritocratic and effort. fairness, are only applicable in specific applications, such as hiring, limiting their use. While Tier Balancing is a step towards a generalized natural notion from the dynamic and long-term perspective through causal dynamic modeling, it's imperative that such notions find support in elfective and elficient learning algorithms. Additionally, current notions oflen fail to support multiple protected groups, which fails to fully capture the nuances of fairness (Gohar & Cheng 2023 Incorporating multiple dimensions of intersecting identities is an important challenge to solve. One interesting approach is to extend the Max-Min fairness notion (Ghosh et al. 2021) to long-term settings, optimizing outcomes for the worst-off intersectional subgroups. Finally, these notions are strictly limited to tabular data, and developing long-term fairness notions in other domains (e.g., image, video, and text) is a promising direction.
4 Improving Long- Fairness in Dynamic Setting
Several approaches have emerged to address bias in dynamic settings, leveraging interactions with the en- vironment. to maintain fairness during learning- Table D outlines the representative mitigation methods. Broadly, these approaches are categorized based on whether the decision-making influences the environment and, consequently, future actions. Most mitigation methods consider the dynamics in a multi-step sequen- tial learning framework. In this setting, the system's evolution over time is explicitly modeled, with each state depending on past states, actions, and feedback from the environment. This framework captures both immediate and delayed effects, rich feedback loops, and complex interactions, and is the primary setting for mitigation methods.
4.1 Decisions Decoupled from Environment
In this setting, decision making does not impact the dynamics of the environment, and most of the works assume an online learning setting where the data arrives sequentially and the model learns the dynamics on the go (Bechavod et. al.) 2019). The seminal work of Joseph et al. (2016) proposed no-regret algorithm, based on chained confidence intervals, for dynamic fairness in the stochastic and contextual bandits problem for the meritocratic fairness notion. They prove that the task of learning a provably fair contextual bandit algorithm can be transformed into a "Knows What Knows" (KWIK) learning algorithm (Li al.) 2008))
and propose efficient learning algorithm. Utilizing the MAB framework, [Liu et al. (2017 propose
9

---

Published in Transactions on Machine Learning Research (07/2025)
Example Works
Metric of Interest
Modeling
Time Horizon
Dynamies
Application
Liu
2018
decugrrapic parily merifoctacle falmess
siuglestep
mos. impooceed
lending practice
2016
bandit
sequential
noc impacted
Hentar:
Bechavod
2020
iddividuul faimess risk disparity individual fnimnsss
oaline learning
sequentia
nos. impocteed
Hasiamooo et B..
0015
online learning
sequential
nos impacted
Gillen
bandit
sequentiaa
noc impacted
Pasil
M21
roturn parity
bundit
sequential
nos Impacted
merilocaaic fairure
reinforemmnt kerning
sequealial
impoeeeed
Vinn x
2023]
Won ot ml.
2022
deroggraphic parity
reinforrement karrnngg
sequential
impacted
XL.
M722
returu parily
reinforvement beuning
sequealial
impoesed
Syldique
20211
Giui social wellare function propartional fnirness
reinforeement learning
sequential
impaated
Lau of al
M122
reinforeement larning
sequeutial
impacted
ranking, allocation
Tuny
2120
delayod Impact
bandit reinforement karning
sequential
not impacteri
Wetter el M
2722
delayed iippact
siugle-step
not impealeed
Table 1: A comparative view of long-term fairness learning methods.
based algorithm to achieve calibrated fairness where each arm is selected with the probability equal to that of its reward being the largest while satisfying the smoothness constraint for each round (See See 3.3 with a reward distribution in Bernoulli. Wang et al. (202I) further extend their work to propose a more efficient Thompson- Thompsonn-ammplnng based algorithm by enforcing morit fairness across rounds instead of for each round and an arbitrary reward function. Heidari & Krause (2018) model the sequential individual fairness in an online learning setting. They propose Consistently Follow the Leader algorithm that always picks the outcome label closest to the prediction among the labels satisfying the consistency constraint. The labels generated in such a schema are proven to follow the optimal hypothesis in bounded time steps. Also targeting individual fairness in online learning, Bechavod et al. (2020) reduce the constrained lcarning into the standard online batch classification problem. Thus, the common Mallow-The-Petturbee Leader algorithm can be applied oracle-effciintlly to achieve sublinear Lagrangian regret. Similarly, Gillen et al. (2018) propose an algorithm where the learner aims to satisfy a fairness constraint defined using an unknown distance metric in online lincar contextual bandit sctting. They assume access to only weak feedback, where regulator can flag unfairness but cannot quantify it across individuals. Finally, [Pail et al.) (2021) propose a meta-algorithm, which can be applied to any MAB algorithm (e.g., UCB1) to satisfy a fairness constraint that each arm is pulled at least some pre-specified fraction of the times.
Beyond the bandit setting, Hashimoto et al. (2018) present a distributionally robust optimization (DRO) procedure to mitigate the disparity amplification over time. The core idea to constrain the group-wise worst-case loss in each iteration; thus, the data examples with high risk are up-weighted. Specifically, let P denote the distribution of data examples and B(P,r) denote the chi-squared ball around P of radius r, measured by a certain f-divergence. The DRO loss parameterized by 0 around all possible perturbation with the maximal radius r around P is (f; = suPQeø(P;r) EQ((O). There are two solutions to the above DRO risk. The first solution assumes the diviieegee and solves the dual form of the DRO risk in the following form 0х ming [p((()) AJ,,, where À is dual parameter. The other solution assumes that one can compute the worst case probability distribution Q²(()) E arg max P.r) Eq((0) for given parameter 0 through projected gradient ascent and alternate the optimal (Q) E arg ming Eoε(O). The final solution 8* and Q' will be given by the saddle point of the optimization problem. Moreover, the authors demonstrate that utilizing a one-step minimizer during each iteration reduces the online learning problem to the standard learning problem. Similarly, Hu & Zhang (2022) propose a repeated risk minimization framework to sequentially mitigate long-term fairness on post-intervention data samples.
4.2 Decision Impacts Environment Dynamics
Chi et al. 2021 model the long-term fairness issue in the context of RL. These works typically focus on how the deployment of the decision policy influences the environment and future rewards. In this context, Jabbari el al. (2017) propose a fair polynomial time algorithm using the MDP framework to enforce the notion of approximate meritocratic fairness. Here, the distribution of individuals considered as the state of the environment, and the decision model f is considered as the action a. The decision model is evaluated by the risk E(s, a) and the fairness violation G(s, a) on the data observation. The utility function for the RL agent is the negative of risk and fairness violation subject to the
A series of works (Yin et al. 2023
10

---

Published in Transactions on Machine Learning Research (07/2025)
observed state and the selected decision policy, ie., a) r(s,a)=111 =1-((s,a) and g(s andg(s, - =))((s,)) respectively. The authors assume an unknown state transition probability such that 8,41 ((8),a)). In the RI. context, the goal of long-term fairness can be viewed as maximizing the cumulative reward max BI r(t,, az) so] while preserving the fairness violation over time EiE g(8t, a4)
88] 2cc
Yin et al. (2023) show that a modified variant the Upper-Confidence Bound algorithm can address the long- term fairness with continuous state and action spaces. In particular, the solution built on the assumption that the Q-function can be parameterized as linear form with respect to some weight vector so that w be precisely solved by least squares problem instead. [Wen et al. (2021) propose slightly different MDP for modeling long- term fairness. Instead of using shared reward function for the policymaker, they assumed an additional reward function for individuals, Then, the fairness is measured as the difference of the expected individual reward between groups with the deployed model policy и. To mitigate fairness, Wen et al. (2021) propose prioritizing the policies closer lo holding the fairness constraints and ranking the candidates in lerms of their rewards. The sampled policies are then used to further update the decision model's parameters. The algorithm is proven to be near optimal while enforcing the static fairness constraints. On the other hand, Puranik et al. (022) propose maximizing applicants' accumulated qualification scores with policy learning. Lastly, (Siddique et al. (2020) enforces the fairness constraint that maximizes the utility of the worst-off individuals by formulating it as the generalized Gini social welfare function (Weymark) 1981). Instead of MDP, they adopt multi-objective MDPs, where rewards take the form of a vector instead of a scalar, to impose the specific notion of fairness.
4.3 Discussion
Compared with the standard static fairness interventions, the summarized mitigation methods mainly con- tribule to formulating the dynamics of hunan-model interactions and establishing the corresponding so- lutions based on the group utility shifts. Long-term fairness under unknown dynamics and/or without leveraging demographic information remains largely unexplored but promising, especially considering pri- vacy laws (Chen et al. 2019a) that limit access to such data. From the perspective of temporal learning, it is positive that long-term fairness is focused on the multi-step sequential learning setting. However, the current research typically only focuses on optimizing the model learner at each time step. Understanding the tensions between ensuring fairness at cach time step and cumulatively at the final step needs more in- vostigation. Finally, while RL methods are flexible and learn the dynamics of the environment well, they are data-intensive [Dulac-Arnod Arnold et al.) 2021 Henderson et al.) 2018) and often unsuitable for real-world scenar- ios. Hence, developing techniques that can efficiently estimate the dynamics is crucial. Notably, the recent advance in performative prediction (Perdomo et al.) 2020) provides a new perspective on off-policy learning for modeling the interplay between the model policy and agents. Finally, most current approaches do not account for individuals or groups strategically adapting their behavior in response to deployed algorithms [Estornell et al. 2023). Developing mitigation strategics that robust to such strategic behavior remains open challenge and is key ensuring that fairness interventions remain effective over time.
5 Evaluation of Long- Term Fairness
Practical applications offer a unique context for long-term fairness considerations, depending on which one would expect context-dependent interpretations of technical findings. Therefore, this section, we present application-specéfi fairness inquiries motivated by empirical scenarios.
5.1 Application-Speciin Studies
Credit Application. Liu et al. (2018) focus on lending practices to quantify delayed impacts of decisions (selections) made by instis (e.g, the bank) on different groups. They represent each group through a score distribution, where higher scores indicate a greater likelihood of positive outcomes. Assuming access to a function that provides the expected score change for individuals, they introduce a one-step foodback model and construct outcome curves, which describe the relation between the expected difference in the mean score after the one-step feedback and the natural parameter regimes (e.g., selection rate of the policy).
11

---

Published in Transactions on Machine Learning Research (07 2205))
They found that both Demographic Parity and Equal Opportunity can lead to all possible outcomes, i.c., long-term improvement, stagnation, and decline, in the natural parameter regimes. This study emphasizes the impossibility of foreseeing a fairness criterion's 's impact without carefully modeling the delayed outcome.
Labor Market. The seminal work of Hu & Chen 2018 considers a two-stage dynamic reputational labor market model to capture the long-term and reinforcing nature of disparate outcomes. They propose imposing demographic parity within a temporary labor market to achieve long-term meg- equilibrium in the permanent reputational labor market based on affirmative action. Separately, Celii et al. (2021) investigate the long-term impact of the Rooney Rule (Collins) 2007), a hiring policy aimed at mitigating implicit bias by mandating fecision-makers to include at least one candidate from an underrepresenteed group. They employ a mathematical model of implicit bias, featuring two candidate groups, one of which is underrepresented. Each candidate possesses both true, latent utility and an observed utility, subject to bias by a multiplicative factor. Candidates are then shortlisted by a selection panel based on observed utility. Their findings demonstrate that the rule progressively enhances minority hiring while maintaining overall utility.
Admission-Hiring Pipeline. Kannan el al. (2019) study a two-stage model where a student sequentially (1) gets admitted to a college, and then, if admitted, (2) gets hired by an institute. Economic literature has explored combating discrimination through affirmative actions similar two-stage models, where students invest in human capital (at some cost.) during the first stage and enter the labor market in the second stage (Foster& & Vohra 1992 Coate & Loury 1993). The authors argue that affirmative action interventions have slow effects and advocate studying "downstream effects" for short-term impacts. They consider interventions by colleges in admission rules, grading policies, and information sharing with hiring institutes, and climinating downstream bias in hiring decisions based on demographic group identity.
Predictive Policing, Ensign et al. 2018 explore resource allocation in predictive policing (Mohler et al. 2015), which deploys patrol officers based on historical crime data. They identify a feedback phenomenon present in similar algorithms across contexts like recidivism prediction and lending, where prediction out- comes heavily influence algorithm feedback. Using the Pólya urn model, they explain this feedback occurrence and propose black-box solutions, such as reweighing input incidents. [Ezayn et al. (019)) go one step further by proposing an algorithm which, under simplification assumptions, converges to optimally fair resource allocation, even in the absence of known stationary ground-truth crime rates, avoiding the runaway feedback loop effect.
5.2 Long- Term-Temm Fairness in Recommender Systems
Long-term fairness considerations naturally align with Recommender Systems due to evolving user interac- tions and item popularity, impacting system performance (Liu et al. 2021). Consequently, the RL framework is favored for its ability to model interactive dynamics. The works mostly focus on 1) user-side fairness, addressing equitable treatment for users, and 2) item-side fairness, ensuring unbiased item recommendation stemming from popularity bias due to disproportionate exposure (Chen et al. 2023).
Item-side fairness. Long-term fairness efforts in recommendation systems primarily target item-side fair- ness, addressing the challenge by modeling the sequential interactions between consumers and recommender systems using an MDP framework. A few works in this direction adopt an Actor- Critic RL algorithm leveraging the posicy-based method (actor) with the value-based approach (critic) to learn the policy and value-function simultaneously (Grondman et al. 2012). In this setup, the actor learns an optimal policy through environment exploration, while the critic evaluates actions to determine long-term rewards {Liu et al. 2021). Utilizing this, Liu et al. (2021) propose FairRec that dynamically maintains accuracy and user fairness in the long term by jointly representing the user preferences (at the item and group level) as the states in an MDP model. They use the Wriighted-Proportione Fairness concept, allocating resources (recommendations) proportionally to diverse entities' demands using a weighting factor. This dual-reward system strategically optimizes both accuracy and fairness in the long term. Separately, Ge al. (2021) study the long-term fairness of item exposure, focusing on group item labels. They address the challenge of items categorized into groups based on popularity, which dynamically change with user interaction over time,
12

---

Published in Transactions on Machine Learning Research (07/2025)
leading to shifting group labels. Using Demographic Parity-based fairness- policy optimization method, the authors propose an MDP recommendation model based on the actor-critic architecture. How- ever, they introduce an additional critic designed for the demographic parity fairness constraint. On a related note, Morik et al. 2020) explored the rich-get-richer phenomenon in dynamic Learning-ter to tonnkkk-tt-n approaches (Yang & Ai 2021): Their solution, FairCo, ensures meritocratic fairness by allocating exposure to items proportional to their average relevance. FairCo integrates user feedback and dynamically adjusts its ranking based on the fairness constraint, employing a P-Controller approach for modeling, a popular leedback control mechanism (Bequette] 2003). In another work, Saato et al. (2020) tackles self-selection bias (Chen et al, 2023), where popular items tend to receive more clicks regardless of actual interest. They propose an ideal loss function optimized for relevance rather than the highest click probability. Similarly, Xu et al. (2023a) proposes an online ranking model based on the Max-Min notion of fairness in the batched bandit settings. It trade-off by adding a fairness constraint and utilizing the Upper Confidence Bound (UCB) algorithm. In this regard, uniform data has also been proven to be an effective approach (Akpinar et al. 2022). This method involves randomly selecting items and ranking them using a uniform distribution. Chen et al. (2023) (Section 4.8) summarizes the various methods that use this approach. Lastly, Ferraro et al. (2021) investigates gender fairness in collaborative filtering, revealing disparities in exposure between female and male artists. They introduce an iterative re-ranking method that simulates feedback loops and penalizes male artists, resulting in adjusted rankings without significant performance impact.
balances the fairness-
User-Side Fairness. Relatively fewer works have focused on long-term user-side fairness. Akpinar et al. (2022) employ a simulation framework and Pólya urns study the long-term impacts of fairness interventions in social network connection recommendations. They argue that when used in dynamic ranking settings, typical statistical notions of fairness fail to maintain equity in network sizes across different protected groups over time. This leads to minority groups having smaller network sizes and lower utility.
5.3 Discussion
The majority of the long-term fairness evaluations have either been context-specific (e.g., hiring) or in recom- mendation systems. In recommendation systems, RL methods are favored for modcling dynamics. However, notable drawback of RL methods is their vulnerability to user tampering (Kasirzadeh & Evans 2023 where recommender systems manipulate user opinions and preferences to boost long-term user engagement through their recommendations. Furthermore, assessing long-term fairness in a multi-stakeholder (Abdol-
lahpouri & Burke 2019) setting characterized by competing interests remains largely unexplored. Some other promising applications that. have largely been underexplored include lext, where the advent of Large Language Models (LLMs) has facilitated dynamic conversations in various settings like chatbots and health- care assistants (Gilbert et al. 2023). Studies have indicated that model outputs can be influenced through iterative feedback (Zamfirescu-Peree Pereira et al., 2023 Logan IV et al., 2021), raising fairness concerns. This, coupled with contextual interactions (Zhang et al. 2021 underscores the need to develop evaluation meth- ods capable of assessing long-term fairness to capture these evolving dynamics. For instance, incorporating long-term fairnsss in Reinforcement Learning from Human Feedback (RLHF) (Chaudhari et al.) 2024). a popular LLM optimization method, or prompt engineering to debias and capture diverse group preferences is an interesting direction.
6 Long- Long-Term Fairness Beyond Dynamic Modeling
Besides addressing long-term fairness in dynamic modeling contexts, various problem settings that require distinct consideration have also tackled the problem. This section provides brief overview of these alter- native settings.
6.1 Performative Prediction
Performative prodiction a recently formalized framework that captures how deploying a predictive model can causally alter the environment or population operates on, resulting in a model-induced distribution shift (Hardt. & Mendler-Dünner 2023 A key distinction of performative prediction is that it is often
13

---

Published in Transactions on Machine Learning Research (07/2025)
formulated as stateless: the environment's response at each step depends directly on the model's current action, without explicit dependence on the sequence of past states. This contrasts with dynamic modeling, which explicitly tracks how the state of a system evolves over time based on both past actions and feedback loops. At its core is the concept of a distribution map D 6 > P(X x Y), which captures the dependence of data-generating distribution on the model parameters E θ ignoring other reasons for distribution shift (Mishler Dalmasso, 2022p. This abstraction enables modeling a wide range of mechanisms behind distribution shifts induced by model deployment. [Miller al. (2021) propose a two-stage approach where the learner first estimates model of the distribution map to capture the unknown parameters of the performative effects and then uses the estimated model to construct the proxy of perturbed performative loss. Next, Mendler-D Dünner et al.] (2020) categorize the stochastic optimization in performative prediction into greedy deploy and loay deploy. Greedy deployment involves a sequential program that continually updates model parameters based on current data observations and deploys the updated model. In contrast, lazy deployment progressively approximates repeated risk minimization, collecting multiple rounds of observations to update the model between deployments. The abstract nature of the performative prediction framework also readily lends itself to supporting settings like strategic classification (Perdomo et al.) 2020). The survey by Hardt & Mendler- Dünner (023) offers review of approaches, which can be extended in future work to encompass long-term fairness settings.
6.2 Delayed Impact
Delayed impact describes scenarios whore the effects of model or intervention appear over time, not immediately, which can turn short-term gains like bank loan approvals into long-term harms for some groups (Liu et al. 2018). It can be seen as case of performative prediction without adopting the formal framework (Hardt & Mendler-Dünner 2023). Importantly, delayed impact differs from performative prediction and dynamii modeling in both scope and modeling assumptions. It typically relies on simpler feedback models like one-step transitions and does not require the ongoing, high-dimensional feedback loops that characterize the other frameworks, which further simplifies both theoretical and empirical analysis. couple of works have been brought up to mitigate the delayed impact in the context of multi-armed bandits (Tang et al., 2020) and RL (Weber et al. 2022). The proposed algorithms include two common steps to mitigate the delayed impact. In the first step, the model learns from the historical data through UCB optimization classification problem. In the second step, the delayed impact will be estimated through importance sampling on the realized future data. The estimated delayed impact will be used as a reward signal that can be further used to select the arms or determine the classifier. Delayed impact is best suited for evaluating the downstream consequences of a single intervention or a limited set of interventions in relatively stable environments like college admissions and the labor market, as discussed in Section 5.1
6.3 Distribution Shift
An alternative research avenue explores dynamic learning environments where data distributions evolve over time. For instance, the profile of loan applicants may shift due to macroeconomic trends or changes in self-seloction criteria. This dynamic necessitates continuous fairness maintenance rather than a one-time requirement. A representative work proposes a dynamic learning method that proactively corrects the classifier, estimating future data properties and retraining accordingly without direct model-ennironment interaction (Almuzaini et al.) 2022). Within this framework, two approaches emerge 1) Adaptive, adapting the classifier to a known target environment, and 2) Robust, training classifiers to ensure fairness under perturbations when no larget environment information is available (Barrainkua el. al. 2023). This problem setting differs from other works due to the absence of direct environmental feedback. For an in-depth review of works in this area, we refer readers to the survey on fairness under domain shift (Barrainkua et al. 2023
7 Critical Analyses and Practical Considerations
In this section, we synthesize insights from existing rescarch to critically assess methodological assumptions and limitations and inform practical considerations.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Questioning Normative Assumptions. Many fairness approaches implicitly assume that "positive" de- cisions (e.g., loan approvals, admissions) are inherently beneficial, while "negative" decisions cause harm [Binns 2018 Chouldechova & Roth 2018). These underlying normative assumptions, often implicit, shape what outcomes are desirable and how fairness is operationalized in automated decisions (Cooper 2020). However, this assumption oversimplifies complex realities, e.g., a loan approval does not guaranlee a posi- tive outcome if leads to financial hardship. Existing long-term fairness frameworks, such as effort-basod fairness notions (see Section 3.1), do not account for this nuance and tend to conflate a "positive decision" " by the algorithm with positive outcome. Moreover, long-term fairness cannot be meaningfully assessed independently of how predictions are used in policy. The social consequences of algorithmic decisions depend heavily on how those decisions are used in practice, e.g., in credit lending decisions, some policies may aim to exlend loans proactively to underserved groups to promote financial inclusion, while others may adopt a conservative approach that restricts credit access to minimize defaults, which might be potentially a bet- ter long-term outcome. Zezulla & Genin (2024) demonstrate this dynamic in labor markets, showing that enforcing statistical parity and equal opportunity in risk predictions can undermine ellorts to lower overall long-term unemployment and close gender gaps. These findings underscore the need to evaluate fairness through a long-term lens that accounts for downstream consequences, policy interactions, and evolving so- cial dynamics rather than simply extending static fairness metrics or assuming that positive decisions are inherently beneficial. Therefore, discussions of long-term fairness should explicitly consider how positive (and negative) decisions distribute benefits and burdens across groups and individuals over time.
Fragmented Landscape of Long-Term Fairness Notions. The majority of the static fairness notions, such as statistical parity and equal opportunity, are broadly applicable across diverse domains, resulting in a unified framework for fairness evaluation. This also facilitates the development of standardized benchmarks and comparative studies across models and datasets (Ganesh et al., 2024). In contrast, long-tery ter-termm fairness notions are distinctly fragmented, some are even mutually exclusive, shaped by their close association with specific application domains and the diverse modeling frameworks they rely upon. For example, effort-based fairness, which emphasizes individual progress and cumulative effort, is typically applied outside of the RL, framework, making it suitable for domains like education and employment where sequential decision processes or enviromment feedback are not explicitly modeled. This fragmentation means that these fairness notions and methods are rarely directly comparable or interchangeable, as each emerges from distinct assumptions about temporal dynamics and normative goals inherent to their respective applications and modeling paradigms. This hinders investigation of joint feasibility, or trade-ofis between different long-term fairness notions, unlike
static fairness literature (Fiiedeer et al. 2019 Bao et al. 2021
Feasibility of the Causal Modeling of Dynamics. In terms of principled modeling of underlying dynamics, causal models provide flexible yet precise ways to incorporate understandings of data-generating
processes (D (D'Amour et al. 2020 Creager et al. 2020 Zhang et al.. 2020: Zhang & Liu 2021 Tang et al, 2023a). Existing works tend to (reductively) focus on observed individual-level features when capturing interplays between dynamics, but the complexity of real-world scenarios often calls for more fine-grained description of data generating processes, potentially among both observable variables (since they are casily measured and readily available) and important latent causal factors (variables that play essential roles in practical scenario but are not particularly easy to measure, e.g., socio-economic status). Tang et al. (2023a) introduce a latent causal factor that carries over current state of affairs to future time steps when defining long-term fairness goal, and there remains room to explore the role of additional latent factors not yet captured by the current model. Recent developments in causal discovery and causal representation learning have provided theoretical guarantees of the existence and relationship among latent factors, under mild assumptions (Xic ct al. 2020 Schölkopf ct al.) 2021 Huang et al., 2022 Dong et al. 2024 Kong et al.) 2024 Zhang et al.) 2024 Zheng et al. 2025). The integration of findings from causal representation learning into current modeling approaches remains limited in the current literature on long-term fairness.
The Gap Between Theory and Practice in Long-Term Fairness. The current body of research on long-term fairness has established rigorous theoretical foundations, particularly through causal inference and reinforcement learning frameworks that model fairness dynamics across temporal and foedback-
contexts (Creager et al. 2020 Tang et al. 2023a Yin et al. 2023 Wen et al. 2021). However, practical
15

---

Published in Transactions on Machine Learning Research (07/2025)
application remains constrained by limited and simplified empirical problem settings. This is largely due to the scarcity of longitudinal datasets annotated for fairness- outcomes, challenges in capturing real-world environment dynamics (Henderson et al. 2018), and difficulties in managing confounding fac- tors inherent in obsorvational data (Zhang et al., 2020). As a result, empirical experiments often rely on simplistic simulations, synthetic data, and retrospective analyses, which, while useful, do not yet provide a comprehensive assessment or real-world deployment (Le Quy et al.) (022). For example, the foundational benchmark by D'Amour et al.] (2020) is currently the only established benchmark to simulate dynamical systems, but it is limited in scope and based simplified scenarios. Thus, bridging theory and practice requires the development of richer longitudinal datasets and more sophisticated benchmarks capable of mod- eling complex, weal-world dynamics. Recent advances in large reasoning models offer a promising direction toward this goal.
8 Open Challenges and Future Works
Drawing on the limitations of prior work, we conclude by outlining several important research gaps and future avenues that are yet be explored.
The tension between long-term and short-term fairness. Evaluating the tension and developing techniques for achieving fairness immediately and maintaining it over an extended period presents signif- icant challenge. Approaches strictly oriented toward long-term goals may introduce short-term drawbacks, including reduced utility and fairness. Conversely, focusing predominantly on short-term fairness risks long- term fairness outcomes. Future work should investigate the trade-off between short-term and long-term fairness. Additionally, while the fairness-utility trade-off has been thoroughly examined in static fairness literature (Mehrabi et al., 2021], ; assessing this long-term trade-off and establishing benchmarks to evaluate different methods remain important directions.
Complexity of real-world dynamics. As we have seen in Sections 2.3 and long-term fairness necessi- tales modeling and characterization of dynamics. However, the complexity of real-world scenarios, including limited access to demographic data, presents significant challenges for practitioners to fully understand the interplay of various dynamics and effectively address the discrimination involved. Translating insights from studies focused on specific applications to broader contexts and bridging the gap between empirical fairness evaluations (Gohar al. 2023 and theoretical guarantees (Tang & Zhang 2022) are important and ongoing challenges in the field.
Intersectional identities. Current long-term fairness research has primarily concentrated on single- attribute group identitics, such as gender or age. However, numerous studies reveal that bias is a maaccd phenomenon that spans multiple protected groups and intersecting identities (Gohar Cheng. 2023). This complexity poses unique challenges, notably the exponential number of subgroups. Future work should focus on developing methodologies and frameworks that capture and mitigate bias the intersection of various attributes in long-term settings.
Explainability over time. Explainability in Al has gained tremendous attention due to the need for transparency, which significantly affects acceptance within the broader community (Gross 2007 Shin 2021 ). Long-term fairness notions, such as effort-based fairness, necessitate using explainable methods. It is vital to empower the impacted population to understand how decisions are made and determine strategies to enhance their qualifications in subsequent instances. Furthermore, incorporating explainability methods in dynamic fairness-awen aware recommendation environments is another direction CCen et al. 2019b).
Long-term dynamics in other AI domains. Studies have shown that prompt engincering and fine- tuning in LLMs directly influence the model loward generating the desired outputs (Zanffrescu-Pereiia et al. 2023 Logan IV et al. 2021). In real-world scenarios, user interactions are dynamic and may even be strategic, guiding the LLMs tailoring responses to their ncods through repeated foodback. An important research direction developing evaluation methods that capture these dynamics and assess bias and fairness in LLM (and other large models) outputs. This, coupled with the interaction context, significantly influences
16

---

Published in Transactions on Machine Learning Research (07/2025)
LLM's output Zhang et al., 2021]. On the other hand, it is also crucial to steer the response of LLMs to consistently align with human values represented by different demographies and cultures (oolaman et al. 2023) in multi-round dialogues.
Alignment with policy and regulatory framework in the long run. Trustworthy Al has drawn increasing attention from policy and regulatory efforts. For example, the European Commission Hige-Level Expert Group on AI presented the Ethical Guidelines (Europcan Commission 2019) that put forward a set of key requirements
AT system should meet to be deemed trustworthy. The Organisation for Economic Cooperation and Development also proposed framework to compare implementation tools for promoting Trustwortyy Al (Organisation for Economic Co-operation and Development (OECD) 2020). Consequently, it is important to consider long-term fairness together with other aspects of Trustworthy Al.
9 Conclusion
Recent research highlights the inadequacy of static one-shot fairness approaches in capturing unfairness over extended periods. To offer researchers an understanding of the field, we present unifying review of long-term fairness in multiple problem settings and frameworks. First, we provide readers with a high-level overview of the conceptual frameworks employed to model long-term fairness problems. Next, we propose the first taxonomy on long-term fairness, wherein we explored long-term fairness definitions, mitigation methods, and representative evaluation techniques across various applications. We also discuss briefty other problem settings that lackle this issue. Finally, we critically dissect the literature and discuss open challenges and opportunities of long-term fairness rescarch. Ultimately, we hope our survey serves foundational resource for those new to the field, fostering an understanding of the complexities associated with long-term fairness and drawing attention to the potential societal consequences that may arise without effective long-term fairness measures.
Acknowledgments
Lu Cheng is supported by the National Science Foundation (NSF) Grant #2312862, NSF CAREER #2440542, NSF-Simons SkAI Institute, National Institutes of Health (NIH) #ROLAG091762, and Cisco gift grant. We would also like to acknowledge the support from NSF Award No. 2229881, AT Institute for Societal Decision Making (AI-SDM), the NIH under Contract RO1HL159805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program. Zeyu Tang is supported by the National Institute of Justice (NIJ) Graduate Research Fellowship, Award No. 15PNLJ-24-GG-O1ES -0-L--2--GG--LL-----RSSS RESS. We would also like to thank Nina Mehrabi for feedback on an earlier draft.
References
Himan Abdollahpouri and Robin Burke. Multi-stakkeholder recommendation and its connection to multi-sided fairness. Xix preprint vrXiv:1907.13155 2019.
Nil-Jana Akpinar, Cyrus DiCiccio, Preetam Nandy, and Kinjal Basu. Long-term dynamics of fairness inter- vention in connection recommender systems. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 22-35, 2022.
Abdulaziz A Almuzaini, Chidansh A Bhatt, David M Pennock, and Vivek K Singh. Abcinml: Anticipatory bias correction in machine learning applications. Tn Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1552-1560, 2022.
Michelle Bao, Angela Zhou, Samantha Zottola, Brian Brubach, Sarah Desmarais, Aaron Horowitz, Kristian Lum, and Suresh Venkatasubrammniia. It's compaslicated: The messy relationship between rai datascts and algorithmic fairness benchmarks. arXiv preprint ar wxXiv:2106.054000 2021.
Albert-Lászie Barabási and Réka Albert. Emergence of scaling in random networks. science, 286(5439): 509-51 -12-512,1 1999.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limitations and opportanilies. MTT Press, 2023.
Ainhize Barrainkua, Paula Gordaliza, Jose A Lozano, and Novi Quadrianto. Preserving the fairness guar- antoes of classifiers in changing environments: a survey. ACM Computing Surveys, 2023.
Yahav Bechavod, Katrina Ligett, Aaron Roth, Bo Waggoner, and Steven Wu. Equal opportunity in online classification with partial feedback. Advances in Neural Information Processing Systems, 32, 2019.
Yahav Bechavod, Christopher Jung, and Zhiwei Steven Wu. Metric-free individual fairness in online learning. ArXit, Abs/2000.055474, 2020. URL
B Wayne Bequette. Process control: modeling, design, and simulation. Prentice Hall Professional, 2003.
Reuben Binns, Fairness in machine learning: Lessons from political philosophy. In Conference on fairness, accountability and transparency, pp. 149-159. PMLR, 2018.
Amanda Bower, Sarah Kitchen, Laura Niss, Martin Strauss, Alexander Vargas, and Suresh Venkata- subramanian. Fair pipelines. arXiv preprint arXiv:1707.000 707.00391, 2017.
Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In 2009 IEEE International Conference on Data Mining Workshops, pp. 13 18. IEEE, 2009.
L Elisa Celis, Chris Hays, Anay Mchrotra, and Nisheeth K Vishnoi. The effect of the rooney rule on implicit bias the long term. In Proceedings of the 2021 ACM conference on fairness, accourlability, and transparency, pp. 678 689, 2021.
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms. ACM Computing Surveys, 2024.
Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. Fairness under unawareness: Assossing disparity when protected class is unobserved. In Proceedings of the Conference on Fairness, Accounlabiliby, and Transparency, FAT* '19, pp. 339 348, New York, NY, USA, 2019a. Association for Computing Machinery. ISBN 9781450361255. doi: 10.1145/3287556.3388777 10. 1145/3287560 32875599
URL https://doi.org
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3): 1-39, 2023.
Xu Chen, Yongfeng Zhang, and Zheng Qin. Dynamic explainable recommendation based on neural attentive models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 53 60, 2019b.
Lu Chong, Ahmadreza Mosallanezhad, Yasin N Silva, Deborah L Hall, and Huan Liu. Mitigating bias in session-based cyberbullying detection: A non-compromising approach. In The Joial Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (ACL-IJCNLP), volume 1, 2021a.
Lu Cheng, Kush R Varshney, and Huan Liu. Socially responsible ai algorithms: Issues, purposes, and challenges. Journal of Arlificial Intelligence Research, 71:1137-1181 2021b.
Lu Cheng, Ahmadreza Mosallanezhad, Yasin N Silva, Deborah Hall, and Huan Liu. Bias mitigation for toxicity detection via sequential decisions. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1750 1760, 2022.
Jianfeng Chi, Jian Shen, Xinyi Dai, Weinan Zhang, Yuan Tian, and Han Zhao. Towards return parity in markov decision processes. ar Xix preprint Xrx:211111.1 1044776, 2021.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Silvia Chiappa and William S Isaac. A causal bayesian networks viewpoint on fairness. In IFIP International Summer School on Privacy and Idendity Management, pp. 3-20. Springer, 2018.
Alexandra Chouldechova and Aaron Roth. The frontiers of fairness in machine learning, orXiv preprint arXiv:1810.08814 2018.
Stephen Coate and Glenn Loury. Will affirmative-action policies eliminate negative stereotypes? The American Economic Review, pp. 1220-1240, 1993.
Brian W Collins. Tackling unconscious bias in hiring practices: The plight of the rooney rule. NYUL Reu., 82:870, 2007.
A Feder Cooper. Where is the normative proof? assumptions and contradictions in ml fairness research. arXiv preprint 10407 2020.
Elliot Creager, David Madras, Toniann Pitassi, and Richard Zemel. Causal modeling for fairness in dynamical systems. In International Conference on Machine Learning, pp. 2185 2185-2195. PMTR, 2020.
Alexander D'Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, and Yoni Halpern. Fairness is not static: deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference Fairness, Accountability, and Transporency, pp. 525-534 2020.
Giuseppe Desolda, Lauren S Ferro, Andrea Marrella, Tiziana Catarci, and Maria Francesca Costabile. Human factors in phishing attacks: a systematic literature review. ACM Computing Surveys (CSUR), 54(8):1-33 35,
2021.
Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. Strategic clas- sification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation, pp. 55-70, 2018.
Xinshuai Dong, Biwei Huang, Ignavier Ng, Xiangchen Song, Yujia Zheng, Songyao Jin, Roberto Legaspi, Peter Spirtes, and Kun Zhang. A versatile causal discovery framework to allow causally-related hidden variables. In International Conference on Learning Represenlations, 2024.
Wouter Duivesteijn and Ad Feelders. Nearest neighbour classification with monotonicity constraints. Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2008, Antwerp, Belgium, September 15-19, 2008, Proceedings, Part I 19, pp. 301-316. Springer, 2008.
Gabriel Dulac-Arnokd, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of weal-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, (009))2441-2406 2021.
Cynthia Dwork and Christina llvento. Fairness under composition. arXie preprint arXiv:1806.0611 2018.
Hadi Elzayn, Shahin Jabbari, Christopher Jung, Michael Kearns, Seth Necl, Aaron Roth, and Zachary Schulzman. Fair algorithms for learning allocation problems. In Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, pp. 170 179, 2019.
Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramannia. Runaway feedback loops in predictive policing. In Proceedings of the 2018 Conference Fairness, Ac- counlabilily, and Transparency, pp. 160- 171. PMLR, 2018.
Andrew Estornell, Sanmay Das, Yang Liu, and Yevgeniy Vorobeychik. Group-fair classification with strategic agents. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 389 399, 2023.
European Commission. Ethical guidelines for trustworthy http:://ccc.urppp...eeeaaarrroaa document. cfm?doc 1d=60419 2019.
19

---

Published in Transactions on Machine Learning Research (07/2025)
Andres Ferraro, Xavier Serra, and Christine Bauer. Break the loop: Gender imbalance in music recom- menders. In Proceedings of the 2021 conference on human information interaction and retrieval, pp. 249 254, 2021.
Jessie Finocchiaro, Roland Maio, Faidra Monachou, Gourab K Patro, Manish Raghavan, Andread Stoica, and Stratis Tsirtsis. Bridging machine learning and mechanism design towards algorithmic fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Tronsparency, pp. 489 489-503, 2021.
Dean P Foster and Rakesh V Vohra. An economic argument for affirmative action. Rationality and Society, 4(2):176-188, 1992.
Sorelle A Friedler, Carlos Scheidegger, Suresh Venkataasubaamnninn, Sonam Choudhary, Evan P Hamil- ton, and Derek Roth. A comparative study of fairness-enhancing interventions machine learning. Tn Proceedings of the conference on fairness, accountability, and transparency, pp. 329 338, 2019.
Pratik Gajane, Akrati Saxena, Maryam Tavakol, George Fletcher, and Mykola Pechenizkiy. Survey fair reinforcement learning: Theory and practice. arXiv prepriul arXiv:2205. 2022.
Prakhar Ganesh, Usman Gohar, Lu Cheng, and Golnoosh Farnadi. Different horses for different courses: Comparing bias mitigation algorithms in ml. or preprint arXiv:24 11. 11111111111111111111111111111111 11101, 2024.
Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, et al. Towards long-term fairness in recommendation. In Proceedings of the 14th ACM international conference web search and data mining, pp. 445-453, 2021.
Avijit Ghosh, Lea Genuit, and Mary Reagan. Characterizing intersectional group fairness with worst-case comparisons. In Artificial Intelligence Diversity, Belonging, Equity, and Inclusion, pp. 22 34. PMLR, 2021.
Stephen Gilbert, Hugh Harvey, Tom Melvin, Erik Vollebregt, and Paul Wicks. Large language model ai chatbots require approval as medical devices. Nature Medicine, 29 29(10)2 2398, 2023.
Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth. Online learning with an unknown fairness metric. Advances in neural information processing systems, 31, 2018.
Usman Gohar and Lu Cheng, A survey on intersectional fairness in machine learning: Notions, mitigation, and challenges. arXie preprint arXiv:2305.00m 069669 2023.
Usman Gohar, Sumon Biswas, and Hridesh Rajan. Towards understanding fairness and its composition in ensemble machine learning. In 2023 IEEE/ACM 45th International Conference Soflware Engineering (ICSE), pp. 1533-154: IEFE, 2023.
Usman Gohar, Michael C Hunter, Agnieszka Marczak-Czajee Robyn R Lutz, Myra B Cohen, and Jane Cleland-Huang, Towards engineering fair and equitable software systems for managing low-altitude airspace authorizations. arXiv preprint Xrxi:2240 01.04001.33,, 2024.
Riccardo Grazzi, Arya Akhavan, John IF Falk, Leonardo Cella, and Massimiliano Pontil. Group meritocratic fairness in linear contextual bandits. Advances in Neural Information Processing Systems, 35:24392 24404, 2022.
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. A survey of actor-critic rein- forcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 1291111131113 2012.
Catherine Gross. Community perspectives of wind energy australia: The application of a justice and community fairness framework to increase social acceptance. Energy Policy, 35:2727-2783 2007. URL
https://api.. http://iiiicoiaaoooooo/ooppaa/aa
20

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Ozgur Guldogan, Yuchen Zeng, Jy yong Sohn, Ramtin Pedarsani, and Kangwook Lec. Equal improvability: A new fairness notion considering the long-term impact. ArXiu, abs/2210.06732,, 2022. URL https:
saaiiseematiccccoolar htt.r//ccccccciooooooooooorooaaa 2252877722 252872923
Ruocheng Guo, Lu Cheng, Jundong Li, P Richard Hahn, and Huan Liu. A survey of learning causality with data: Problems and methods. ACM Computing Surveys (CSUR), 53(4):1 37, 2020.
Vivek Gupta, Pogah Nokhiz, Chitradoep Dutta Roy, and Suresh Venkatasubramanian. Equalizing recourse across groups. Xiv preprinl S5:::999003166 2019.
Moritz Hardt and Celestine Mendler-Dinner. Performative prediction: Past and futurr. orXiv preprint
arXiv:2310. 16608, 2023.
Tatsunrii Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang, Fairness without demo- graphics in repeated loss minimization. In Jennifer Dy and Andreas Krause (ods.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1929-1938. PMLR, 10-15 Jul 2018. URL https://proceedings. mlr wttp:/s..w...peee//////00 ppwes//80/hashhaaate html
Hoda Heidari and Andreas Krause, Preventing disparate treatment in sequential decision making. In International Joint Conference on Artificial Intelligence, 2018. URL http:://aap..smmatiiccchhh..ol
:Bg/CorpusID:494872222
Hoda Heidari, Vedant Nanda, and Krishna Gummadi. On the long-term impact of algorithmic decision policies: Effort unfairness and feature segregation through social learning. In 36th International Conference on Machine Learning, pp. 2692-2701, 2019.
NORA Helfand. Polya's urn and the beta- bernoulli process. University of Chicago REU, 2013.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pincau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Lily Hu and Yiling Chen. A short-term intervention for long-term fairness in the labor market. In Proceedings of the 2018 World Wide Web Conference, pp. 1389 1398, 2018.
Lily Hu, Nicole Immorlica, and Jennifer Wortman Vaughan. The disparate effects of strategic manipulation.
In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 259 268, 2019.
Yaowei Hu and Lu Zhang. Achieving long-torm fairness in sequential decision making. In AAAI Conference on Arlificial Intelligence, 2022. URL
247957724
Wen Huan, Yongkai Wu, Lu Zhang, and Xintao Wu. Fairness through equality of effort. In Companion Proceedings of the Web Conference 2020, pp. 743-751, 2020.
Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints. In Advances in Neural Information Processing Systems, vol- une 35, pp. 5549 5561, 2022.
Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamic Morgenstern, and Aaron Roth. Fairness in rein- forcement learning. In International conference on machine learning, pp. 1617-1626. PMLR, 2017.
Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in learning: Classic and con- textual bandits. Tn Neural Information Processing Systems, 2016. URL https://::///pp.. semanticccooll.r lrr
:
Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. Mcritocratic fairness for infinite and contextual bandits. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Sociely, pp. 158 163, 2018.
21

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Sampath Kannan, Aaron Roth, and Juba Ziani. Downstream effects of affirmative action. In Proceedings of the Conference on Fairness, Accounlability, and Transparency, pp. 240-248, 2019.
Amir-Hossein Karimi, Julius Von Kügelgen, Bernhard Schölkopf, and Isabel Valera, Algorithmic recourse under imperfect causal knowlodge: a probabilistic approach. Advances in neural information processing systems, 33:265 277, 2020.
Amir-Hossein Karimi, Bernhard Schölkopf, and Isabel Valera. Algorithmic recourse: from counterfactual explanations to interventions. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 353 362, 2021.
Atoosa Kasirzadeh and Charles Evans. User tampering in reinforcement learning recommender systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Sociely, pp. 58 69, 2023.
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Audit- ing and learning for subgroup fairness. In International conference on machine learning, pp. 2564 2572. PMLR, 2018.
Niki Kilbertus, Mateo Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bern- hard Schölkopf. Avoiding discrimination through causal reasoning. In Proceedlings of the 31st International Conference on Newral Information Processing Systems, pp. 656-666, 2017.
Lingjing Kong, Guangyi Chen, Biwei Huang, Eric Xing, Yuejie Chi, and Kun Zhang, Learning discrete concepts in latent hicrarchical models. In Advances in Neural Information Processing Systems, volume 37, pp. 36938 36975, 2024.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Newral Information Processing Systems, pp. 4066-4076 2017.
Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi. A survey on datasets for fairness-aware machine learning. Wiley Interdisciplinary Reviews: Dala Mining and Knowledgee Discovery, 12(3)2 2022.
Lihong Li, Michael L, Littman, and Thomas J Walsh. Knows what it knows: a framework for self-aware learning. In Proceedings of the 25th international conference on Machine learning, pp. 568 575, 2008
Yunqi Li, Hanxiong Chen, Shuyuan Xu, Yingqiang Ge, Juntao Tan, Shuchang Liu, and Yongfeng Zhang, Fairness in recommendation: Foundations, methods, and applications. ACM Transactions on Intelligent Systems and Technology, 14(5):1 48, 2023.
Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pp. 3150 3158. PMLR, 2018.
Weiwen Liu, Feng Liu, Ruiming Tang, B. Liao, Guangyong Chen, and Pheng-Amn Heng. Balancing accuracy and fairness for interactive recommendation with reinforcement learning. ArXio, abs/2106. Abs//210.13386, 2021. URL https://api. semanticscholar wtw.:oooaa...gooooouii..... 235652076
Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David C Parkes. Calibrated fairness in bandits. arXiv preprint arXic:1707.01880 2017.
Robert I Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13330 2021.
Hosam Mahmoud. Polya urn models. Chapman and Hall/CRC, 2008.
Ninarch Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1 1-35, 2021.
22

---

Published in Transactions on Machine Learning Research (07/2025)
Celestine Mendler-Dünne Juan C. Perdomo, Tijana Zrnic, and Moritz Hardt. Stochastic optimization for performative prediction. ArXin, abs/2006.06888, 2020. URL https://api www.ntissccoolrr..ooom .rg/
CorpuS1D:219636100
John P Miller, Juan C Perdomo, and Tijana Zrnic. Outside the echo chamber: Optimizing the performative risk. In Marina. Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7710 11100-720. PMTRR, 18 24 Jul 2021.
URL https://proceediiii
mlr, greess/v13//milee221. html
Smitha Milli, John Miller, Anca D Dragan, and Moritz Hardt. The social cost of strategic classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 230-239, 2019.
Alan Mishler and Niccolò Dalmasso. Fair when trained, unfair when deployed: Observable fairness measures unstable in performative prediction settings. arXiv preprint or arXin:2202 20500490000099 2022.
George O Mohler, Martin B Short, Sean Malinowski, Mark Johnson, George E Tita, Andrea I, Bertozzi, and P Jeffrey Brantingham. Randomized controlled field trials of predictive policing, Journal of the American Statistical Association, 110(512):1399 1411, 2015.
Marco Morik, Ashudoep Singh, Jessica Hong, and Thorsten Joachims. Controlling fairness and bias in dynamic soaming-to-rante In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieual, pp. 429-438, 2020.
Razich Nabi and Ilya Shpitser. Fair inference outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Organisation for Economic Co-operation and Development for trustworthy ai: for trustworthy tools-for cu-forrtruustorthhyy -e--ri--------ffareor-------t
(OECD).
Tools tools
A framework systems.
compare
implementation
https: oecd www.:.eed..rrr/oonnggddggttaaaaa
-syytmemss
pdf. 2020.
Vishakha Patil, Ganesh Ghalme, Vineet Nair, and Yadati Narahari. Achieving fairness in the stochastic multi-armed bandit problem. The Journal of Machine Learning Research, 22(1):7885-779 2021.
Gourab K. Patro, Lorenzo Porcaro, Laura Mitchell, Qiuyue Zhang, Meike Zehlike, and Nikhil Garg. Fair ranking: critical review, challenges, and future directions. In Proceedings of the 2022 ACM Conference on Fairness, Accounlabilily, und Trunsparency, FAccT '22, pp. 1929 1942, New York, NY, USA, 2022. URL https:
Association for Computing Machinery. ISBN 9781450393522. doi: 10. oobii.org/10/ 10. 11145/3531146 3533238
Judea Pearl. Causality. Cambridge University Press, 2009.
Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pp.
PMLR, 2020.
Dana Pessach and Erez Shuneli. A review on fairness in machine learning. ACM Compuling Surveys (CSUR), 55(3):1 44, 2022.
Bhagyashree Puranik, Upamanyu Madhow, and Ramtin Pedarsani. A dynamic decision-making framework promoting long-term fairness. In Proceedings of the 2022 AAAT/ACM Conference AI, Ethics, and Sociely, AIES '22, pp. 547 556, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392471. doi: 10.1145/351 25144.4.3534127. URL htttp:://doi.rg//// 1145/3514094.38 3534127
John E. Roemer and Alain Trannoy. Chapter equality of opportunity. In Anthony B. Atkinson and François Bourguignon (eds.), Handbook of Income Distribution, volume 2 of Handbook of Income Dis- tribution, pp. 217-300. Elsevier, 2015. doi: httpe///bbaaogr///ooooo///822244
0.00005-9. URL
23

---

Published in Transactions on Machine Learning Research (07/2025)
Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing--ot--l--anddee implicit feedback. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 501-5011000 509, 2020.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612 634, 2021.
Donghee Shin. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable ai. International Journal of Human-Computer Studies, 146:102551, 2021.
Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforce- ment learning with average and discounted rewards. In International Conference on Machine Learning, PMLR, 2020.
pp.
Hannah Snyder. Literature review as research methodology: An overview and guidelines. Journal of business research, (04:333-333) 2019.
Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III, Jesse Dodge, Ellic Evans, Sara Hooker, et al. Evaluating the social impact of generative ai systems in systems and society. arXiv preprint Xrx:222306.000 05949, 2023.
Peter Spirtes, Clark N Glymour, and Richard Scheines. Causalion, prediction, and search. Springer New York, 1993.
Wei Tang, Chien-Ju Ho, and Yang Liu. Bandit learning with delayed impact of actions. In Newral Information Processing Systems, 2020. URL
Zeyu Tang and Kun Zhang, Attainability and optimality: The equalized odds fairness revisited. In Conference on Causal Learning and Reasoning, volume 177, pp. 754-786. PMLR, 2022.
Zeyu Tang, Yatong Chen, Yang Liu, and Kun Zhang. Tier balancing: Towards dynamic fairness over underlying causal factors. In International Conference Learning Represenlations, 2023a.
Zeyu Tang, Jiji Zhang, and Kun Zhang. What-is and how-to for fairness in machine learning: A survey, reflection, and perspective. ACM Computing Surveys, 55(13s):1 1377, 2023b.
Berk Ustun, Alexander Spangher, and Yang Liu. Actionable recourse linear classification. In Proceedings of the conference on fairness, accountability, and bransparency, pp. 10 19, 2019.
Julius Von Küigelgen, Amir-Hosscin Karimi, Umang Bhatt, Isabel Valera, Adrian Weller, and Bernhard Schölkopf. On the fairness of causal algorithmic recourse. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 9584-9594, 2022.
Lequn Wang, Yiwei Bai, Wen Sun, and Thorsten Joachims. Fairness of exposure stochastic bandits. In International Conference on Machine Learning, pp. 10686-10696. PMLR, 2021.
Aline Weber, Blossom Metevier, Yuriy Brun, Philip S. Thomas, and Brun Castro da Silva. Enforcing 2022. URL https://app:: htpp:ppss//ppiieemmttttccoooooop stw.ppiismmmtnicccc....o.
delayed-impact lairness guarantees. ArX,
:tse/CorpuIID:25180000
Min Wen, Osbert Bastani, and Ufuk Topeu. Algorithms for fairness in sequential decision making. In International Conference on Artificial Intelligence and Statistics, pp. 1144-1152. PMLR, 2021.
John A Weymark. Generalized gini inequality indices. Mathematical Social Sciences, 1(4):409 430, 1981.
Feng Xie, Ruichu Cai, Biwci Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In Advances in Neural Information Processing Systems, volume 33, pp. 14891-149M0 2020.
24

---

Published in Transactions on Machine Learning Research (07 (225))
Chen Xu, Xiaopeng Ye, Jun Xu, Xiao Zhang, Weiran Shen, and Ji-Rong Wen. Ltp-mmf: Towards long- term provider max-min fairness under recommendation feedback loops. Xiv preprint 2023a.
Yuancheng Xu, Chenghao Deng, Yanchao Sun, Ruijie Zheng, Xiyao Wang, Jieyu Zhao, and Furong Huang. Equal long-term benefit rate: Adapting static fairness notions to sequential decision making. ar preprint arXiv:2309.03/20 2023b.
Tao Yang and Qingyao Ai. Maximizing marginal fairness for dynamic learning to rank. Proceedings of the Web Conference 2021, 2021. URL https://api. wwt.:/awww...ii....ooooooorrroaa
orgg/ooppulD:221179800
Tongxin Yin, Reilly Raab, Mingyan Liu, and Yang Liu. Long-term fairness with unknown dynamics. arXiv preprint arXix:2304.00 09362, 2023.
JD Zamfirescu-Pereira. Richmond Y Wong, Bjoern Hartmann, and Qian Yang. Why johnny can't prompt: how non-ai experts try (and fail) to design 11m prompts. In Proceedings of the 2023 CHI Conference on Human Pactors in Computing Systems, pp. 1 21, 2023.
Sebastian Zezulka and Konstantin Genin.
From the fair distribution of predictions the fair distribution of social goods: Evaluating the impact of fair machine learning long-term unemployment. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 1984-2006 2024.
Chen Zhang, Yiming Chen, Luis Fernando D'Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. DynaEval: Unifying turn and dialogue level evaluation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 5676-5689, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 441. URL /sttps://aclanthollog. wwwpoggg....gg//eee wrgg//21.acll1aaa00000 acl-long. 441
Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multiple distributions: general setting. In Proceedings of the 41st International Conference Machine Learning, pp. 60057-60075, 2024.
Xueru Zhang and Mingyan Liu. Fairness in dearning-based sequential decision algorithms: survey. In Handbook of Reinforcement Learning and Control, pp. 525 555. Springer, 2021.
Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang. How do fair decisions fare in long-term qualification? Advances in Newral Information Processing Systems, 33: 18457 18469, 2020.
Xueru Zhang, Mohammad Mahdi Khalili, Kun Jin, Parinaz Naghizadch, and Mingyan Liu. Fairness inter- ventions as (dis) incentives for strategic manipulation. In International Conference on Machine Learning, pp. 222294 PMLR, 2022.
Yujia Zheng, Yang Liu, Jiaxiong Yao, Yingyao Hu, and Kun Zhang, Nonparametric factor analysis and beyond. In International Conference on Artificial Intelligence and Statistics, pp. 424-432. PMLR, 2025.
A Appendix
A.1 Survey Methodology
We employed several ostablished approaches to conduct comprchensive scarch for relevant papers under the umbrella of long-term fairness. Initially, we curated a set. of seed papers that have discussed and avaluated
long-term fairness problems (Zhang Liu 2021 Liu et al. 2018 Hu & Chen 2018 D'Amour et al. 2020
Zhang & Liu. 2021 Zhang et al.) 2020). Following common practice (Desolda et al. 2021 Snyder 2019). we used these papers and domain knowledge to identify and develop a set of keywords to search for relevant work: "long-term fairness," " "dynamic fairness," "(non-)static fairness," " and "sequential fairness." Primarily,
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
searched proceedings of relevant conferences and journals that were likely to publish work that fit the definition of long-term fairness. These included but were not limited to FAceT, NeurIPS, AAAT, ICML, AIES, ICLR, and IJCAL. We also followed citation trails to and from these widely cited papers and querying across Google Scholar and Arxiv to identify additional related work.
Framework
Symbols
Description
decision model parameterized by a vector E O
E X B 'nput features
Classification
VEy = {0, 1] ground truth label
g
g g sensitive groups
Ar
Effort (change) required to improve the feature x to x'
S
State Space (e.g., income levels)
A
Action space (e.g., accept )ejept/reject)
S x A + reward function (e.g., bank's profits)
MDP
p
Initial state distribution
T
: x A + ((S) State transition function
T
Discount factor
π
T: m:S-A(A A(4) is policy
Cumulative Reward
K
Set of arms
Time step
MAB
A
A, K. action (arm) selected at time t
R
Stochastic reward at time t.
Table 2: Table of Notations
26


Reviews:
[{"review_id": "SzCRpjHcDS", "reviewer": "Reviewer_EjFb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1) I believe the survey covers the relevant topics in the area of long-term fairness that I am aware of. \n\n2) The taxonomy created in the paper provides a nice template to organize the papers in the field. \n\nWeaknesses:\nI feel like there could be more discussion of how the papers and methods relate to or differ from each other. Please see suggested changes for concrete examples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper surveys work on notions, methods, and frameworks for long-term fairness. The paper taxonomizes these works along multiple axes: 1) fairness notions, 2) modeling of dynamics, and 3) mitigation methods. Outside of this taxonomy they also provide other considerations for long-term fairness. They also survey work that study long-term fairness tailored to specific applications including recommender systems. They also present a number of open questions and limitations that the current literature does not yet address.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some changes that would strengthen the paper:\n\n1) Discussions on whether it makes sense to study all combinations of all dimensions of the taxonomy. Are there certain combinations that do not make sense? For example, is there a certain modeling that only makes sense for particular notions and mitigation methods?\n\n2) Having a table or figure showing where papers fall under all three taxonomy dimensions. I see Figure 2 that relates notions of fairness and modeling. It would be nice to have another figure that also includes mitigation methods and how this relates to the other two dimensions of fairness notions and modeling.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WevqoZ4lRn", "reviewer": "Reviewer_kMgW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think there is value in surveying this line of research, both as a useful resource for researchers and because, ultimately, the long-term effects of our decision-making systems on people is what we should be most concerned about.\n\nThe survey covers a reasonable set of papers; nothing obvious has been left out. The relative emphasis on different papers is puzzling at times, for example, multiple \"effort-based fairness\" methods are defined in detail (despite these not being obviously \"long-term\", see below for further critique) while incentive effects and strategic behavior -- which seem extremely important to any change in high-stakes decision-making -- are mentioned only in citations.\n\nThe taxonomy proposed in this paper is functional, but only just. It pulls together disparate ideas into loose categories, and seems to simultaneously operate at too-coarse and too-specific levels of detail. For example, the Polya urn model (a specific model of one type of dynamic) is given the same prominence in the taxonomy as \"Dynamic Modeling\", which could plausibly encompass all the other settings in the \"Problem Setting & Mitigation\" category as well as all the modeling frameworks including the urn model.\n\nIn places it's hard to determine whether the summaries of various methods and their justifications are weak, or if the underlying research that is being summarized is simply underwhelming. This points to a further weakness: the discuss sections don't critically engage with the methods presented (eg \"when would this method be (in)appropriate to deploy?\", \"are the normative assumptions/assertions underlying the method well-justified\", \"can this method be deployed in pratice\") but instead mostly point out room for straightforward future work (eg \"current notions often fail to support multiple protected groups\"). But future work in an area isn't productive if the work it's building on is unworkable or normatively indefensible.\n\nUltimately, I think this paper is a flawed execution of a good research direction.\n\nFurther issues/notes:\n\n> “For example, in predictive policing, Ensign et al. (2018) observed how an initial higher allocation of police in a specific area leads to more reported incidents, perpetuating increased surveillance and exacerbating the marginalization of those communities over time.”\n\nDoesn’t this summary provide the same myopic, simplistic fairness analysis that this paper is criticizing? It assumes that higher allocations of police to an area (which might grow as the number of reported incidents increases) are bad for the people there. But we can’t conclude this unless we’ve studied the long-term outcomes of the policy on (among other things) crime, safety, subjective well-being, the local economy, etc.\n\nI don’t think “delayed impact” is meaningfully different from dynamic modeling. The “delayed impact” in the Liu et al paper was the effect on a borrowers’ credit score from repaying (or failing to repay) their loan. But this is a mechanical function of the outcome being predicted (default/repayment), not some second-order or downstream effect. In almost every setting we care about, the impact of decisions aren’t immediately apparent: the benefits from admission to college don’t accrue until the student graduates, the benefits from cancer treatment accrue slowly as treatment proceeds, etc. So I don’t see what value “delayed impact” provides as a separate category. To me, the value of the Liu et al paper is that it was one of the first to consider _any_ impact of a decision on the decision subject, rather than (as prior work did) assuming it was good for a borrower to receive a loan even if they defaulted on it.\n\nIt seems like an oversight to study long-term fairness without considering incentive effects. Is that just because there is very little research in this area?\n\nEvery fairness notion in the first section (3.1, Effort-based fairness) assumes there is a “positive outcome” (which would be better termed a “positive decision”, since it is assigned by the classifier). But the most important insight of long-term fairness is that the decision itself does not determine whether the short- and long-term outcomes of the decision are good. It’s not a good outcome to be admitted to university if you eventually drop out with a lot of debt, and it’s obviously not uniformly good to receive chemotherapy – it’s only good if you need the treatment and it works!\n\nFurthermore, what about these effort-based notions of fairness is long-term? Defs 1, 2, and 3 all just constrain a classifier’s mapping from features to classifications, which is also what static fairness notions do.\n\n> “Causal modeling of underlying dynamics in long-term settings has been recognized as beneficial for analyzing social implications of automated decision-making”\n\nIn reality it’s extremely difficult to measure the causal effect of a single decision or policy on humans, let alone construct a correct DAG of interconnected causal effects over time. It’s important that any discussion of causal fairness points out this huge impediment to these methods being practically applicable.\n\nThe “Long-Term Fairness Beyond Dynamic Modeling” section starts with: “Performative prediction refers to the dynamic influence of model decisions on future data distribution.” But what is “dynamic modeling” if it’s not modeling “the dynamic influence of model decisions on future data distribution”? The Performative Prediction section did not illuminate this and didn’t explain why it deserved to be called out in its own section.\n\nSimilarly, delayed impact and distribution shift seem to be exactly the sort of dynamics one might want to model if considering long-term fairness, so it’s weird that they are in the “Beyond Dynamic Modeling” section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper surveys the literature on various fairness methodologies that can be broadly categorized as focused on \"long-term fairness\", rather than static approaches that focus on a single dataset or decision. It constructs a taxonomy of these approaches, presents some in light detail, and discusses where the literature leaves space for future work.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The most important change needed is a clean-up of the taxonomy to address questions like:\n* Why is effort-based fairness a long-term approach?\n* Why are incentive effects and strategic behavior not part of the taxonomy?\n* What distinguishes \"delayed impact\", \"performative prediction\", from \"dynamic modeling\"? And why is this difference meaningful?\n* Why does the urn model warrant special consideration over other dynamic models?\n\nSecondly, the paper presents a lot of methods but does make the reader aware of the huge drawbacks the some have. For example,\n* Causal fairness approaches cannot be deployed in real systems because the real causal DAG is never known (and the outcome of the approaches depend entirely on which stylized DAG is used)\n* Many of the discussed fairness approaches assume that one decision is \"positive\", but in almost every important context (lending, admissions, medicine, criminal justice) neither action is straightforwardly positive or negative. This is particularly concerning in a paper that is supposed to be about the long term!", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K7zk2HfEP7", "reviewer": "Reviewer_dGLx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper addresses an important and timely issue in the fairness literature by focusing on long-term implications, which are often overlooked in favor of short-term metrics.\n- It provides a structured overview of the field, categorizing existing work and pointing out research gaps and challenges.\n- The survey includes many representative works, demonstrating that the authors have performed a thorough review of the literature.\n\n**Weaknesses:**\n- The categorization of fairness into \"short-term\" and \"long-term\" could benefit from a more nuanced treatment. As it stands, the dichotomy seems overly binary. It would be helpful to explore whether a spectrum or taxonomy of temporal fairness notions could better capture the complexity of real-world scenarios.\n- The section discussing online learning as a tool for long-term fairness is promising but lacks sufficient theoretical grounding. It would strengthen the paper to elaborate on the theoretical underpinnings of the approaches mentioned.\n- Additionally, when referring to existing literature, it would be helpful for the authors to more clearly distinguish between empirical and theoretical contributions. This clarification can help readers understand the maturity and foundational depth of different sub-areas.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\nThis paper, *\"Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challenges,\"* provides a comprehensive survey of the current landscape of research on long-term fairness in machine learning. The authors review a wide range of literature, organizing existing works around key fairness notions and methodological approaches. The paper highlights the significance of moving beyond short-term metrics and emphasizes the importance of evaluating fairness over extended time horizons.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Suggestions for Improvement:**\n\n1. Refine the distinction between short-term and long-term fairness. Consider introducing intermediate or hybrid notions that may better reflect the temporal dynamics in machine learning applications.\n2. Expand the discussion on online learning approaches by including theoretical results (where available) and indicating whether referenced works are primarily theoretical or empirical.\n3. Enhance the clarity of the taxonomy or structure used to organize the surveyed works to aid readers in navigating the landscape more effectively.", "broader_impact_concerns": "Since the work is a survey and does not introduce novel algorithms or systems with direct deployment risks, and because it already highlights key fairness challenges, I did not identify any additional broader impact concerns that would require further elaboration.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)