A Causal Lens for Learning Long-term Fair Policies
Authors: Jacob Lear
Lu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: long-term fairness
fair reinforcement learning
causal decomposition
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Fairness-aware learning studies the development of algorithms that avoid discriminatory decision outcomes despite biased training data. While most studies have concentrated on immediate bias in static contexts, this paper highlights the importance of investigating long-term fairness in dynamic decision-making systems while simultaneously considering instantaneous fairness requirements. In the context of reinforcement learning, we propose a general framework where long-term fairness is measured by the difference in the average expected qualification gain that individuals from different groups could obtain. Then, through a causal lens, we decompose this metric into three components that represent the direct impact, the delayed impact, as well as the spurious effect the policy has on the qualification gain. We analyze the intrinsic connection between these components and an emerging fairness notion called benefit fairness that aims to control the equity of outcomes in decision-making. Finally, we develop a simple yet effective approach for balancing various fairness notions.

Published as a conference paper at ICLR 2025
A CAUSAL LENS FOR LEARNING LONG-TERM FAIR POLICIES
Jacob Lear& & Lu Zhang
Department of Electrical Engineering and Computer Science University of Arkansas {jdlear, 1z00 @uarkk.ddk
ABSTRACT
Fairness-aware learning studies the development of algorithms that avoid discrim- inatory decision outcomes despite biased training data. While most studies have concentrated on immediate bias in static contexts, this paper highlights the im- portance of investigating long-term fairness in dynamic dccision-making systems while simultaneously considering instantaneous faimess requirements. In the con- text of reinforcement learning, we propose a general framework where long-term fairness is measured by the difference in the average expected qualification gain that individuals from different groups could obtain. Then, through a causal lens, we decompose this metric into three components that represent the direct impact, the delayed impact, as well as the spurious effect the policy has on the qualifica- tion gain. We analyze the intrinsic connection between these components and an emerging fairness notion called benefit fairness that aims to control the equity of outcomes in decision-making Finally, we develop a simple yet effective approach for balancing various fairness notions.
INTRODUCTION
Artificial intelligence and machine learning decision decision-making systems are being increasingly imple- mented in real-world scenarios Zhang al. 7)); JJonnson et al. (2016); Bakerr & Hawn (2021); Lee & Floridi (2021); Schumann et al. (2020): Berk et al. (2021) 1). Real-world data, influenced by social and historical contexts, often carries biases related to gender, race, and other factors. These biases can be inadvertently embedded in the algorithms, leading to discriminatory outcomes. As result, fairness-aware learning, which aims to satisfy various fairness constraints alongside the usual performance criteria in machine learning, has received increasing attention. A body of literature on fairness hwiness-awarre aware learning focuses on developing fair policies reinforcement learning (RL) Sutton &
Barto (0018). For comprehensive survey fair RL, please refer to Gajane et al. (2022)
Most studies in fairess--waar learning focus only on the immediate implications of bias in a static context. These works typically quantify the fairness of model predictions outcomes in static population. However, real decision-making systems usually operate dynamically, and the decisions made by these systems have long-tern term consequences. The literature has shown that fairness notions and techniques focusing on the immediate bias produced by automated decisions cannot guarantee protect disadvantaged groups in the long run Liu et al. (2018. It has hence been proposed to consider the delayed impact of automated decisions in sequential decision decision-making systems due to the interplay between the decisions and individuals' reactions Liu al. (0220). For example, [hang er al. (2020) proposes to use the equilibrium of the dynamics of population qualifications across different groups of individuals as a measure of the delayed impact of decisions on fairness. this paper, when the context is clear, refer to the instantaneous fairness concerns, such demographic parity Feldman et al.) (2015). equal opportunity Hardt et al. (2016), and causality-base notions like direct and indirect discrimination Zhang al. (011b)) short-term fairness. On the other hand, refer to fairness concerns that arise over time due to dynamic user-decision interactions as long-term fairness, which is the primary focus of this paper.
Given that the objective of RL is typically to maximize the long-term (discounted) reward, long- term fairness has been studied in the context of RI. where Markov Decision Processes (MDPs)

---

Published as a conference paper at ICLR 2025
(0177: : Ge al. (2021): Wen et al. (02T): Yu et al. p002); ; Hu et al. (2023); Yin et al. (2023). However, such requirements may be conflict with long-temn term fairness constraints. For instance, Hu et al. (2024) models the sequential decision-making system with a temporal causal graph and captures long-term and short-term fair- ness as causal effects transmitted through different paths in the temporal causal graph. The authors demonstrated that achieving long-term fairness could be impeded by the requirement of sensitive attribute unconsciousness, L.c., no direct causal effect of the sensitive attribute on the model deci- sion, implying an inherent conflict between long-term and short-tems toortttrrmaa fairness. Thus, understanding the connection between long-term and short-term faimess critical for developing comprehensive strategies that balance both fairness requirements.
utilized to model and learn the system dynamics Jabbari et al.
In this work, we study the specific task of analyzing the long-term fairness that can be achieved in the context of RL. We focus on the impact of the model's ' decisions on individuals' qualifications and aim to understand how long-term fairness intertwines with immediate short-term fairness concerns during sequential decision-making
To this end, we propose a novel decomposition of long-term fairness that differs from existing causal decomposition techniques applied to the advantage func- tion, such as Pan & Schölkopf (2024). We begin by introducing general framework for studying long-term fairness in RL, where we assume a flexible qualification gain function that measures an de-
individual's 's qualification gain they transition between qualification states. Accordingly, fine the expected total qualification gain accumulated from given state while following a specific policy as a state value function, which can be expressed using Bellman's 's equation. Based on that, long-term fairness can be readily formulated based on the disparity in the expectation of the state value function across different groups. We then conduct causal decomposition of the qualifica- tion gain disparity to identify the various sources of inequality, noting that the qualification gain is influenced by both the policy and the environment. We end up obtaining three components of the qualification gain disparity: (1) the Direct Policy Effect (DPE) which represents the direct causal effect of the policy on the long-term qualification gain; (2) the Indirect Policy Effect (IPE) which represents the indirect causal effect of the policy the long-ter qualification gain through the environment; and (3) the Spurious Policy Effect (SPE) which represents the spurious effect only due to the environment. Interestingly, we identify inherent connection between these components and an emerging fairness notion called benefit fairness that aims to control the equalty of the outcome of the decision-making Pleckoo & Bareinboim (2023). Our analysis shows that benefit fairness may not necessarily conflict with long-term fairness, suggesting that it is possible to reconcile short-term and long-term fairness objectives. This connection may offer insights for designing decision-making systems where the long term and short-term objectives are aligned. Finally, we provide simple yet effective approach to strike a balance between qualification gain parity and benefit fairness.
2 RELATED WORK
Early research in long-term fair machine learning focuses on specific applications and scenarios. For example, the authors in Holzer (2007) examine the long-term effects of affirmative action in hiring and highlight its positive impacts on minority and low-income communities. The authors in Hu & Chen (2018) study long-term fairness in two-stage labor market and construct dynamic reputational model that shows how unequal access to resources leads to different investment choices, which in turn reinforces unequal outcomes between groups. The study in Liu et al. (2018) inves- tigates the delayed impact of decisions in lending scenarios using a one-step feedback model and reveals that short-term fairness notions generally do not reshape the population foster long-term improvements.
Reinforcement learning (RL) offers a solution for modeling the long-term impact of decisions through Markov decision processes. Fairness methods in RL designed to address long-term effects have been proposed. The study in Jabbari et al. (2017) presents a fairness constraint that guaran- tees an algorithm will not favor one action over another if the latter has higher long-term reward. However, this notion of fairness is solely based on the reward of each action and does not consider demographic information. The research in Ge et al. (2021) studies long-term fairness and formulates a Constrained Markov Decision Process (CMDP) for recommendation systems. In Wen et al. (2021), faimess constraints are defined based on the average rewards received by individuals across different groups, and two algorithms have been designed to learn policies that meet these faimess constraints. Yu et al. (2022) suggests incorporating fairness requirements into policy optimization by regulariz-
2

---

Published as a conference paper at ICLR 2025
ing the advantage assessment of different actions. Similarly, Hu et al. (2023) incorporates long-tern fairness constraints into the advantage function but proposes a pre- pre-prceesiing technique called ac- tion massaging to address short-term fairness, aiming to balance both requirements. The research in Yin et al. 2023) also demonstrates that achieving long-term fairness may require sacrificing short- term incentives, It also develops probabilistic bounds cumulative loss and cumulative fairness violations. Finally, different from the above work. Henzinger et al. (2023) develops a monitor that continuously tracks events generated by the system in real-time, providing an ongoing assessment of the system's 's faimess with each event.
Another related research direction is performative prediction and performative optimization which have emerged as powerful tools for addressing distributional shifts that occur when deployed models influence the data-genera process Perdomo et al. (2020); Jia et al. (024)). This line of research also offers valuable insights into achieving long-term fairness in dynamic systems. For instance, Jin et al. (2044) examines issues of polarization and unfairness within performative prediction set- tings. They demonstrate that traditional penalty terms for long-term fairess may fail in performative prediction settings and propose new fairness mechanisms that can be incorporated into iterative al- gorithms, such repeated risk minimization, which has been shown to be effective in these settings.
Our research complements related studies by examining the trade-off between long-term and short- term fairness in RL settings from causal perspective and establishing connection between them.
METHODOLOGY
3.1 PRELIMINARIES
We adopt Pearl's structural causal model (SCM) and causal graph framework Pearl (2009) to facili- tate modeling the Markov decision process and formulating long-term fairness. Here, we provide a brief overview of the fundamentals of SCM. A detailed introduction to SCM can be found in Pearl 2010). An SCM is a mathematical framework used in causal inference to represent and analyze the relationships between variables a system. It provides a formal way to describe how changes in one variable can causally affect other variables, enabling the analysis of cause-and- and-effect relationships. An SCM defines the causal dynamics of a system through collection of structural equations. Each SCM is associated with a causal graph that includes set of nodes to represent variables and a set of directed edges to depict direct causal relationships
Causal inference within the SCM is enabled through interventions, described in Pearl (2009).
A hard intervention assigns specific constant values to certain variables, while a soft intervention establishes a functional relationship for some variables in response to others Correa & Barcinboim (2020). A causal path from node X to node Y in a causal graph is a directed sequence of arrows leading from X to Y. The total causal effeet refers to the impact of X on Y when the intervention is transmitted along all causal paths connecting X to Y. If the intervention is restricted to only a subset of these causal paths, the resulting effect is known as the path-specific effect Avin al. (2055.
3.2 FORMULATING LONG-TERM FAIRNESS
Consider a discrete-time sequential decision-making process applied a certain population, de- scribed as a Markov decision process (MDP). Each individual in the population is described by sensitive feature and a qualification state. The sensitive feature S is assumed to be binary vari- able in this paper for ease of representation. i.e., S E {s* }. The qualification state, denoted as X. represents the individual's 's suitability or potential performance in the given context relevant to a specific decision-making task. It could be variable vector and may evolve over time, reflecting the dynamic nature of an individual's attributes or skills. At each time step, a Markov policy (d|x, s) is used to make decisions D E {dd, d') where d represents negative decision d represents positive decision or treatment. A reward R is received following each decision and the utility of the policy is defined the expected total reward received during the process. Meanwhile, individuals may also take actions upon receiving the decisions, which may change their qualification states at the next time step. This interplay is captured by transition probability P P(x,+ |x,, de s).
tre-trratment, and
The above process can be described by the causal graph shown in Fig. F which can represent either randomly sampled individual or a population undergoing the decision cycles. In general, each
3

---

Published as a conference paper at ICLR 2025
time step f. the agent selects action de based on the policy я. This causes the qualification state to transition from X₂ to an intermediate state (x,, de). which subsequently transitions to the next state X++1 according to the transition probability. Without loss of generality, we assume that the sensitive feature 5 may influence any part of the process, including the initial state distribution, the transition probability, the policy function, etc.
A metric of long-term fairness typically con- siders the equilibrium of qualification states across different groups. It can also be treated as the causal effect of the sensitive feature on the qualification states. For instance, the au- thors in Hu al. 2024) investigate long-term fairness by examining the path-specific ef- fect, i.e., the causal effect transmitted through all paths from to XT in the graph. They conclude that the objective of eliminating this causal effect may conflict with the require- ment of sensitive attribute unconsciousness.
x
x1
X2
Ny
B1
Y
B2
Figure I: Causal graph for representing the discrete- time sequential decision-making process where omitted. At each time step the state x, transits to the intermediate state X2, de based on the policy n, and then transits X++1 based on the transition probability. The reward r is received based on the decision/action X,, and the qualification gain is ob- tained based on both xx, de and the next state X
In this work, we explore long-term fairness and establish general formulation within the framework of MDPs. The typical objective of MDPs is to maximize the expected cumu- lative reward for an agent interacting with an environment over time. Taking fairness into consideration, different from previous work that incor- porates long term-termmmm fairness as an advantage regularization (e.g.. Yu et al. (002);; Hu et al. (2023), ). we argue for considering the expected total change in the qualification state when the individual in- teracts with the decision policy. To achieve this, we introduce flexible qualification gain function, denoted as 9d (x x'), which quantifies the increase/decrease in qualifications when an individual from the sensitive group s transitions from state x to state x'. This qualification gain function can be defined in any form long as it satisfies the additive property over intervals, i.e., for all inter- mediate states X1,..., X., between x and x', we have gx x') = g" (x, x1) + ... + g (X,, x') ).). Then, for given trajectory in the sequential prcision-makingg process, the total qualification gain from transitioning from the initial state to the final state is equivalent to the cumulative qualification
gain throughout the process, i.e., Σi (x, Xx+1).
To define the expected qualification gain an individual can achieve through the deployment of this policy, we employ the notations from Hu & Zhang (2022), where the policy deployment is treated as a soft intervention, denoted as do(x), (T). Meanwhile, in line with causality-based fairness notions, we consider the qualification gain individuals would achieve if their sensitive features were altered to different values. This involves performing hard interventions on the sensitive feature S. By conducting both interventions, we consider the interventional expected qualification gain, denoted as Vdo(t,s) (x). Adopting the causal perspective of reinforcement learning Zeng et al. (202)), the interventional expected qualification gain can be defined as the state value function Vdo(T,e)
X = =x] This state value function above captures the cumulative impact of decisions over time, enabling to address the need for equity in decision-making systems over extended periods. A natural long-term fairness requirement, therefore, is to ensure that the average expected qualification gain is equal across different groups over time. Consequently, we define qualification gain parity as a general metric for long-term fairness, as detailed below.
E, 9",,
Definition (Qualification Gain Parity) We say that a policy a exhibits qualification gain parity if (0) BBV (Xx)) (xo) ([(ao((T,s>) (xx)] is equal to zero.
3.3 POLICY OPTIMIZATION WITH CONSTRAINTS
Qualification gain parity can be achieved through policy optimization with constraints. To this end, first express the state value function using Bellman's equation as follows.
V Voo(T. (x) Σ
क (d|x, 8) 1Qdo( (x, d),
(1)
d
4

---

Published as a conference paper at ICLR 2025
where
()) =
Σ P d, s) (x,,xxxx + (xo(z,,)(x())
is the action value function. By following the Policy Gradient Theorem Sut-
ton Barto (2018), the gradient of the state value function Eq. ① is given by x') Ea Vot(dix', 3))(trt
d) where n7 is the probability of transitioning from state x to state x' with policy after an ar- (0) is given by (0)
Va VoVdo(n,s) (x) Ex of n
x')
bitrary number
steps. Thus, the gradient (Eat [V in (d/xx, st [Qdo(=,))) d)] - [Vo In [d[x, s ())a)(().)))) d)]).
where the expectation is over the state visitation distribution and a ((Vao(m.e->xx))))
detailed derivation is included in Appendix A.
Policy Optimization. Our choice for policy optimization is mostly typical as variant of Proximal Policy Optimization (PPO) that incorporates the KL divergence as a penalty [Schulman et al. (2017). Using slight variation of notations, we use Vdo(n,s) (x) and Qdo(x.e) (x) to denote the state/action
value function in terms of utility Tt r(x,, d,). The advantage function for state is given as
Ado(T.s) d) Qdo(s.s) (x, d) VVo(T.N) (x). We then write the objective function as
(P) LUTIL BKL
where LUTIL_ 18 stde xeee)
Toteldd M.:+a)) A ( (Xe, de) is the expected advantage with importance sam-
pling, and LKL E, [K L( (ota( (Sr Xr, de) de)l (st (x,, de)]] is the KL divergence to penalize large divergences between the new and old policies. We use the hat symbol to denote the emperical estimation of a given function or operator.
As PPO updates the policy using minibatch SGD, we can separate the batch of timesteps that were sampled for the minibatches into those belonging to the advantaged group t+ and those for the disadvantaged group t The constraint then becomes Ć,
E
T(d 8,11 Wold(de (de+ (x++ Qdo(Taaass (Xet, de+)) -
T T(d, Told (dh- (x--,
X2 ,
Qdo : do-))
We then integrate the constraint into the full objective function as
J(0) TUTIL gKL. - (C2)
3.4 CAUSAL DECOMPOSITION OF C
As mentioned earlier, it important to study the connection between long-term and short-term fairness requirements as they may be conflicting objectives. To this end, in this section, we perform a causal analysis to decompose the qualification gain parity, enabling us to identify and distinguish various components contributing to bias. We study the various ways in which the policy influences the qualification gain, where we examine the instantaneous impact of the policy as well as the delayed impact through the transitions of the environment.
As shown in Fig. T from a causal perspective, the instantaneous impact of the policy can be captured by the direct edges from (x,, to 9: while the delayed impact can be captured by all other paths from the states to the qualification gain. To examine these different mechanisms through which the policy impact manifests respectively, we leverage the path-specific technique Avin et al. (2005) in causal inference, which can isolate the effect transmitted through a particular causal path while controlling the effect transmitted through other pathways. To achieve this, we construct two hypo- thetical policies that are not actually implemented. One is a baseline policy xp that always makes the non-treatement decision, i.e., To (dd x, 8) (see Fig. a)). The state value function of mo- denoted as Vdn( is given by
Vd (x)
Σ P(x|xx, " (x,') + Vdo(To.s)(x'))
The other is a virtual policy x PS where the qualification gain is assumed to be obtained as if fol- lowing policy no, while the state transitions occur if under policy æ (see Fig. (b)).. Its state value
5

---

Published as a conference paper at ICLR 2025
function, denoted as Vdo(=Ps,N) ,x), can be given by
Vdo(a ps (x)) = Σ
Σ x(d|x, 8) Σ
P(x|Ix d, 8) doo (4)
Facilitated by these hypothetical policies, we can decompose Vdo(z.s) into three components: the component that captures the direct impact of the policy π on the qualification gain, the component that captures the delayed impact, as well as the component that captures the spurious effect solely attributable to the environment, given below.
Vdo(s = ()) - Vdo(aPs,,))xX Vdo(n (o (x)) Vdo(=oo.)) (x) + V (o) (x)
direct impact
delayed impact
sparious effect
The explanation of this decomposition is follows. For deriving the direct impact, we employ the policy T"S the reference policy and assume that the decision- making system shifts from this reference policy to policy и. During this process, the policy π used for generating episodes. However, when calculating the instantaneous qualification gain at each time step, the policy that is used to make the decision shifts from To to æ, which subse- quently influences the next state that used compute the qualification gain via gx(x,, x--1). The direct im- pact is then computed as the difference the qualifica- tion gain. Since the episodes remain unchanged during the shift, this difference eliminates the policy's 's delayed impact, capturing only the direct effect of switching from the baseline policy To to the behavior policy x.
(a)
Ty Fo
Xt
Xt+1
(b)
Xx. x
Xt
Xeut
Se.
X
Similarly, for the delayed impact, employ the policy To as the reference policy where both the episodes and the instantaneous qualification gain are determined by mo- Then, we switch to policy π PS allowing policy æ to gen- erate the episodes. Thus, the change in the qualification gain is due to the delayed impact of policy т. Finally, since Ty unrelated to т,, we can treat the qualification gain achieved, i.e., Vdo(To.s)) environment's transitions, which can be treated as the spurious effect in the impact of policy
Figure 2: Causal graphs for represent- ing the two hypothetical policies (the reward is omitted from the figures). (a) The baseline policy To that always makes the negative decisions. (b) The virtual policy where the state transitions oceur if under ı while the qualifica- tion is obtained as if under mp-
as being solely due to
Correspondingly, by taking S into account and conducting the decomposition for both do(s+) and do(s"), ). we can decompose C2 (0) into three components, referred to the Direct Policy Effect (DPE), the Indirect Policy Effect (IPE), and the Spurious Policy Effect (SPE), as given below.
(+) = DPE + IPE + SPE,
where DPE E ot) (Xo) (o)7=s,--)(x0))
E [Vao(s, (xx) - Vdo( (x)(="s,+))xxo)) (x)()))) IPE E [VdoLTrs .t) (Xp) Vdo(To.st) (xu)] E [Voo (xx)) (o((o..-)(xx)) and SPE
[Vdo(so.>>) (xo)] - [Vao(nouo-) (Xn)
3.5 CONNECTION WITH BENEFIT FAIRNESS
The decomposition of (0) allows us to build connection between qualification gain parity and instantenous fairness requirments. For the former, we pay special attention to the DPE which repre- sents disparity in the instantaneous impact of the policy on the qualification gain. For the latter, we consider benefit fairness proposed in Plecko & Bareinboim| (2023), which does not merely allocate equal treatment each group's entire population but instead takes into account the proportion of the population that can benefit from the treatment, making it more suitable for controlling fairness in the outcomes of decision model.

---

Published as a conference paper at ICLR 2025
In Plecko & Bareinboim (2023), the benefit is defined as the conditional average treatment effect (CATE) which represents the increase in the outcome (e.g., survival) associated with the treatment. In our context, the outcome of interest is the qualification gain an individual receives as a result of the treatment. Hence, the benefit can be defined as the expected increase the qualification gain an individual can achieve if they receive treatment compared to not receiving treatment, i.e.,
A(x, s) (P do(s)) P(x d', do(s))) (x,,,'),
where P(x'Ix, d, do(s)) = P(x||x, d. 8) in our context. Then, benefit fairness is defined as follows.
Definition 2 (Benefit Fairness Plecko & Bareinboim (2023)) We say that a policy x satisfies ben- efit fairess if
Vx, π (d |x, st) = π (dd s-) if s+ = A(x'ss))
In words. benefit faireess means that individuals from different groups who would benefit equally from a treatment should have similar probabilities of receiving that treatment.
The following proposition reveals the connection between the direct impact component of s) (x) and benefit. Please refer to Appendix B for the proof.
Proposition 1 Given policy π and hypothetical policies no, x defined in Eqs. B
(4
we have
(x) - rs,))rs = Σ (x x ( (d |x, 8) s) s).
Vao 28 ss) Vdo(To,s)(x) = (n x +x)) (x + xx)) Σ P(x" xc, di s) (x,x")
where 11 -7=**(x (x + x') the probability of transitioning from state x to state x' with policy πT after any number of steps.
Using this proposition, we can reformulate DPE follows
DPE= E (teflxxs)44 Ax(x) (x)x) - IE E [=)x x~
where the expectation over the state visitation distributions.
By combining the above expression and Definition 2 we observe that the DPE is closely related benefit fairness. When the state visitation distributions in terms of benefit are identical across the two groups, benefit fairness will result in a zero DPE. This implies that both groups have equal opportunities and exposure to the equitable direct impact. However, if the state visitation distributions differ between the two groups, achieving benefit fairness becomes incompatible with eliminating the DPE. This discrepancy arises because differing state visitation distributions indicate that the two groups not equally repre- sented in the sequential equalize the true benefits received by different groups, may inadvertently introduce maintain dis- parities in the DPF. On the other hand, we note that the IPÉ is less sensitive to benefit fairness. Thus, enforcing benefit fairness may result in significant change in the IPE. This analysis highlights the complexity of balancing fairness objectives and the challenges of ensuring equitable outcomes across diverse populations in sequential decision- making systems.
process so that achieving benefit fairness leads to
process. Hence, enforcing
fairness, which aims to
Remark. A key insight we gain from the above analysis is that, if the benefit A is independent of the sensitive feature then achieving benefit fairness aligns with decreasing DPE and also loosely corresponds to the goal of equalizing feature distributions across different groups. This is because, in this context, the state visitation distribution approximates the feature distribution, suggesting that benefit faimess could be achieved simultaneously with a reduced DPE and equalized feature distribu- tion. This insight motivates us to introduce an additional consideration into the design qualification gain function when designing practical decision-making systems, which is to ensure that the benefit A remains independent of S, even though the transition probability may be dependent on S. This may imply that the qualification gain function should be adjusted to account for differences in ben- efits. also underscores the importance of understanding why benefits differ between groups, as highlighted in Plecko & Bareinboim (2023)
7

---

Published as a conference paper at ICLR 2025
Balancing qualification gain parity and benefit fairness. We propose simple yet effective ap- proach to achieve benefit faimess while promoting parity in qualification gains during policy op- timization. Our approach leverages the concept of individual fairness Dwork et al. (2012) which demands that any two similar individuals should receive similar decision outcomes. Individual fair- ness is relevant and applicable to our setting, we require that if two individuals from different groups receive similar benefits from the policy, their probabilities of receiving a positive decision should also be similar. To establish a quantitative metric for benefit fairness, we draw inspiration from the Gini coefficient, commonly used to measure income equality in economic theory Mota al. (202T), which has also been applied to individual fairness Sirohi al. (2024). Mathematically, the Geni coefficient is defined as > Σj=1 |s; أز sj1/2n 2,=1 8j where 8; is the income of
person. Inspired by this, we define a metric that measures the difference in positive decision rates for all pairs of individuals from different sensitive groups, weighted by the inverse distance in their
benefits, as follows.
[ T x A (x', 8-) P(x|s PPxx(s)
Σe X.X' c+ A(X,
Here, is a small positive value that prevents division by zero and also controls the radius within which we require similar positive decision rates. As decreases, similar positive decision rates only enforced when the benefits are nearly identical. Conversely, a large will penalize differences in positive decision rates when the benefits close but not exactly the same. The final objective function is obtained by incorporating A into Eq. (2)
((0) = UTIL 3KL LKLLKE KL (Ox)
(5)
4 EXPERIMENTS
We conduct experiments to evaluate the policy optimization algorithms we have proposed and com- pare them with baselines regarding the achievement of long-term fairness] We also present the results from the decomposition and the performance of our algorithm when taking benefit fairness into consideration, demonstrating how the constraint of benefit fairness influences different compo- nents of qualification parity. We refer to our algorithm with objective function Eq. as PPO-C and the algorithm objective function Eq. (5) PPO-Cb.
4.1 EXPERIMENTAL SETUP
We leverage the simulation environment developed in D' Amour et al. (2020) that is commonly used in related work (c.g., s.. Yu et al. (2022;; Hu et al. (202)). The environment is designed to simulate the process of bank dispersing loans to members of a population. Each individual in the population belongs to either the advantaged group s or the disadvantaged group s-. At each time step, an whether
individual applies for loan and the bank has to make binary decision, de (d), d' to approve or deny the loan. The decision is made in accordance with the current policy лe (d] xt), distribution produced by feedforward neural network from which the decision d, is sampled. By default, the average qualification level of individuals belonging to s+ is higher than that of those belonging
These distributions evolve with time in response to factors in the environment and loan decisions.
To facilitate the idea that the environment can have effects, unrelated to the decision, on an individ- ual's 's qualification level, we also include a term Zdrift E (-1,0, 1}. This represents the change in an individual's credit score which would occur regardless of whether or not loan was received by the individual. As a result, an individual is represented by the 4-tuple (s, Tärifts y). To sample an individual, first we sample the sensitive attribute s ~ P(s), then the credit score x ~ P(x.(s). followed by the ability to repay the loan y P(y|,, s). The credit drift Idrift P(zarifu) is sampled independently of the other attributes. Note that Zarift and y are unobserved for the policy and are not used input for decision-making.
We designed the qualification gain function to reflect the real-world phenomenon where progressing from beginner intermediate requires less effort than progressing from advanced expert. The
'll the code available httpp///giihbbb..mm/////////////
Lens Fair-RI
8

---

Published as a conference paper at ICLR 2025
Setting
Setting 2
Setting 3
Bank Proft
limestep
APG-C
PRO-Cb
F-PPO-L
Figure 3: Utility comparison across different settings.
Qua ificatin Gain Dispa "ty
Setting
Setting
Setting
10
10
302
500
100
350
100
PPO PPO-C
RPO-Co
A/PPO
F-PPO-L
Figure 4: Qualification gain disparity comparison across different settings.
distribution P(x++1 X,, s) is deterministic function of (st, X,, Fariff Me; de) where small amount of mass representing an individual is moved from P(x [s.) to P(xc+1|5t). In the evaluation, we use three different settings for the simulation environment which we will simply refer to Setting 1,2, and 3. These settings differ from each other in initial distributions over credit scores, repayment probabilities, and credit drift likelihoods. The repayment probabilities are learned by using the Home Credit Default Risk dataset Montoya et al. 2018) and a dataset previously released by Lending Club Wagh 2017). The details of the experimental setup provided in Appendix C.
We compare our methods to three baselines: (1) PPO et al. (2017), a variant of the vanilla PPO algorithm incorporating KL-divergence penalty: (2) A-PPO Yu al. (2022). which intro- duces additional regularizations to penalize the advantage function in order to enforce equalized opportunity; and (3) F-PPO-L Hu et al. (202)), which also applies regularization term penalize the advantage function, but with the objective of reducing the 1-Wasserstein distance between group feature distributions.
Due to the large variance in model's performance in both training and evaluation due to the highly stochastic nature of the system, for all results and methods, we perform multiple training runs with different random seeds for initialization and then take an average to better gauge performance.
4.2 RESULTS
Evaluating long-term fairness. We first evaluate the effectiveness of our proposed approaches in achieving qualification gain parity. We compare PPC-C and PPC-Cb against the baseline methods PPO, which purely optimizes for utility, as well as A-PPO and F-PPO-L which optimize for utility and their own fairness objectives. fairly competitive with regard to utility when compared to the baselines. While the inclusion of the additional benefit fairness produce respectable profit in Setting 2, where A-PPO struggles to do so. Since PPO-C and PPO- Ch both optimize for the qualification gain disparity, is unsurprising that they both outperform the baselines in this regard as shown in Figure 4 However, is interesting that PPO-Cb outperforms PPO-C in Setting 1, while in Settings 2 and 3 they exhibit relatively similar performance. This may be attributed to the connection between qualification gain disparity and benefit faimesss, which is further analyzed the following.
Figure 3 we see that PPO-C, the constraint-only variant, is
in PPO-Cb tends decrease the performance, manages to
Evaluating causal decomposition. We then evaluate the causal decomposition and how different components are impacted in the training. In Figure we show C= together with the decompo- sition of the constraint into its constituent components that IPF, DPF, and SPE for PPO-C and

---

Published as a conference paper at ICLR 2025
PRO-C
PRO-C
PRO-Cb
Eptrode Date
0.25
0.25
0.00
0.00
50.5
0.50
0.10
203
602
500
200
1000 6
200
600
500
500
1000
202
602
500
300
1000
200
600
500
500
1000
Training Episodes
Number Training Episodes
Training Epicodes
Number of Training Epicodes
OPE
IPF
SPE
(,09)
DPE
IPF
SPE
C,09)
(a) Setting 1
(b) Setting 2
Figure 5: Qualification gain disparity decompositions.
Setting
Setting Large g
Setting
Setting Large P
0,020
0.00
Faimess
0.00
0.015
0.01.
0.02
0,00
030
Number Tnaining Episodes
Number Training
Number Episodes
Number Training
PPO-Cb
DPE PPO-C
DPE PPO-Cb
Figure 6: Benefit fairness and DPE for PPO-C and PPO-Cb model variants.
PPO-Cb. In both settings, the DPE varies more significantly over time compared with the IPE. This behavior implies that policy optimization is more effective in decreasing DPE, the instantaneous im- pact of the policy, than IPE, the delayed impact of the policy. Comparing the two methods, PPO-Cb exhibits a more pronounced reduction in the DPE than in the IPE, suggesting an inherent connection between the DPE and benefit fairness.
Evaluating benefit fairness. Finally, we evaluate the effectiveness of objective function Eq. (5) in ensuring benefit fairness as well as the connection between benefit fairness and DPE. Figure a demonstration of the effect of increasing the 3* parameter to increase the enforcement of the benefit fairness constraint in PPO-Cb, with PPO-C included further reference. The first two plots are generated using the environment Setting 1, with the left corresponding to the base setting for & that benefit fairness
and the right for the relatively large setting. Comparing the two plots, improves as A increases, showing the effecacy of the regularization. Moreover, we see that the DPE decreases for larger BA, which further reflects the connection between benefit faimess and DPE as demonstrated in our theoretical analysis. This pattern similarly manifests in the last two figures, where Setting is used.
CONCLUSIONS
In this paper, we explored the achievement of long-term fairness in reinforcement learning (RL) from a causal perspective, emphasizing the importance of simultancously addressing instantancous fairness requirements. We proposed a general RL framework where long-term fairness quantified by the difference in the average expected qualification gain that individuals from different groups could obtain. Through a causal decomposition of this disparity, we identified three key compo- nents: the Direct Policy Effect (DPE), the Indirect Policy Effect (IPE), and the Spurious Policy Effect (SPE). Our analysis revealed an intrinsic connection between benefit fairness an emerging short-term faimess concept-and and DPE, which is particularly sensitive policy optimization. Fur- thermore, we developed a simple yet effective approach to balance qualification gain parity with benefit faimess. Experimental results demonstrated the efficacy of our methods. In future work, aim to conduct a more in-depth analysis of the qualification gain function's 's design to account for differences in benefits and align various faimesss notions in sequential decision-making
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by NSF 1910284, 2142725.
REFERENCES
Chen Avin, Ilya Shpitser, and Judea Pearl. Identifiability of path-specific effects. 2005.
Ryan S Baker and Aaron Hawn. Algorithmic bias in education. International Journal of Artificial Intelligence in Education, pp. 1-41, 2021.
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50(1):3-44 2021.
Juan Correa and Elias Bareinboim. A calculus for stochastic interventions: Causal effect identifica- tion and surrogate experiments. In Proceedings the AAAI Conference Artificial Intelligence, volume 34, pp. 10093-10100, 2020.
Alexander Ammour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, David Sculley, and Yoni Halpemn Fairness not static: deeper understanding long term faimess via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 525- 534, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science confer- ence, pp. 214-226, 2012.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 259-268, 2015.
Pratik Gajane, Akrati Saxena, Maryam Tavakol, George Fletcher, and Mykola Pechenizkiy. Survey on fair reinforcement learning: Theory and practice. arXiv preprint arXiv:2205. 10032, 2022.
Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, et al. Towards long-term faimess in recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 445 2021.
Advances in neural information processing systems, volume 29, pp. 3315-3323, 2016. Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In
Thomas Henzinger, Mahyar Karimi, Konstantin Kueffner, and Kaushik Mallik. Runtime monitor- ing of dynamic fairness properties. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 604-614, 2023.
Harry Holzer. The economic impact of affirmative action in the us. Swedish Economic Policy Review, 14(1):41, 2007.
Lily Hu and Yiling Chen. short-term intervention for long-term fairness in the labor market. In Proceedings of the 2018 World Wide Web Conference, pp. 1389-13988 2018.
Yaowei Hu and Lu Zhang. Achieving long-term fairness in sequential decision making. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9549-9557, 2022.
Yaowei Hu, Jacob Lear, and Lu Zhang. Striking balance in fairness for dynamic systems through reinforcement learning. In 2023 IEFE International Conference on Big Data (BigData), pp. 662- 671. IEEE, 2023.
Yaowei Hu, Yongkai Wu, and Lu Zhang. Long-tem fair decision making through deep generative models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 22114 22122, 2024.
11

---

Published as a conference paper at ICLR 2025
Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstem, and Aaron Roth. Fairness in reinforcement learning. In International Conference on Machine Learning, pp. 1617-1626. PMLR, 2017.
Zhuangzhuang Jia, Yijie Wang, Roy Dong, and Grani Hanasusanto. Distributionally robust per- formative optimization. arXiv preprint WxXv:2407.011344, 2024.
Kun Jin, Tian Xie, Yang Liu, and Xueru Zhang. Addressing polarization and unfairness perfor- mative prediction. arXiv preprint arXiv:2406. 16756, 2024.
Kory Johnson, Dean Foster, and Robert Stine. Impartial predictive modeling: Ensuring faimess in arbitrary models. Statistical Science, pp. 1, 2016.
Michelle Seng Ah Lee and Luciano Floridi. Algorithmic fairesss in mortgage lending: from absolute conditions to relational trade-offs. Minds and Machines, 31(1):165- 191, 2021.
Lydia T Liu, Sarah Dean, Esther Rolf, Max Simehowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pp. 3150-3158. PMLR, 2018.
Lydia T Liu, Ashia Wilson, Nika Haghtalab, Adam Tauman Kalai, Christian Borgs, and Jennifer Chayes. The disparate equilibria of algorithmic decision making when individuals invest ratio- nally. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 381-391, 2020.
Anna Montoya, Kirill Odintsov, and Martin Kotek. Home credit default risk, 2018. URL https: //kaggle wwt.cccccccccccccc ionss////// hhw.//ooeeeeeeeeeeeeee www.cc-eedddeuuuete t
risk
Nuno Mota, Negar Mohammadi, Palash Dey, Krishna P Gummadi, and Abhijnan Chakraborty, Fair partitioning of public resources: Redrawing district boundary to minimize spatial inequality in school funding. In Proceedings of the Web Conference 2021, pp. 646-657, 2021.
Hsiao-Ru Pan and Bernhard Schölkopf. Skill or luck? return decomposition via advantage functions. In The Twelfth International Conference on Learning Representations, 2024.
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl. Causal inference. NIPS Causality: Objectives and Assessment, pp. 39-58, 2010.
Juan Perdomo, Tijana Zmic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pp. 7599-7609, PMLR, 2020.
Drago Plecko and Elias Bareinboim. Causal fairness for outcome control. Advances in Neural Information Processing Systems, 37, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. htthhtt:://// /ttp///axvvvv///////a//eee//aava wrg.obggg//.oo. /tbs//100
Candice Schumann, Jeffrey Foster, Nicholas Mattei, and John Dickerson. We need fairness and explainability in algorithmic hiring. In International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 2020.
Anuj Kumar Sirohi, Anjali Gupta, Sayan Ranu, Sandeep Kumar, and Amitabha Bagchi. Graphgini: Fostering individual and group fairness in graph neural networks. arXiv preprint arXiv:2402.12937, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Rijuta Wagh, 2017.
URL
Lending-Club -Lauu---oo------ -or.uu--ooaa--ppootttttt-------- Daaty-Dataaaaa -ata--aaaaaais Analysis
Min Wen, Osbert Bastani, and Ufuk Topeu. Algorithms for faimess in sequential decision making.
In International Conference Artificial Intelligence and Statistics, pp. 1144-1152. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Tongxin Yin, Reilly Raab, Mingyan Liu, and Yang Liu. Long-term faimnsss with unknown dynamics. Advances in Neural Information Processing Systems, 37, 2023.
Eric Yang Yu, Zhizhen Qin, Min Kyung Lee, and Sicun Gao. Policy optimization with advantage regularization for long-term fairness decision systems. NeurIPS, 2022.
Yan Zeng, Ruichu Cai, Fuchun Sun, Libo Huang, and Zhifeng Hao. A survey on causal reinforce- ment learning. arXiv preprint arXiv:2302.05209, 2023.
L. Zhang, Yongkai Wu, and Xintao Wu. Achieving an-discrimination in data release. Proceedings of ACM SIGKDD' 17, 2017a.
Lu Zhang, Yongkai Wu, and Xintao Wu. A causal framework for discovering and removing di- rect and indirect discrimination. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 3929-3935, 2017b.
Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang. How do fair decisions fare long-term qualification? Advances in Neural Information Processing Systems, 13:18457-18469 2020.
13

---

Published as a conference paper at ICLR 2025
APPENDIX A: DERIVING VoCr(()
To derive (x)d we have
VoVaotn.o) (x) Vo Σ r(d|s, xidls,x)QQofff ()Qd((n.)))
Σ ()) + (d|s, x) Qdooo (x, d))
Σ (xca)ls,))XX(oo ++
+ Vo Pixxxxss)oxxxxx+++
Σ Vo + xd)s,x) Σ [P(xxxd))Voo
If we let 6(x) = Ed Voo x)) d), then
VoV VoVeo(T. (x) = xxx) + Σ [d|,, x) ((
= xxx)) + Σ
x 2(1sxx/XX/xxddsVV/
x
= -(x) + Σ
p² (x x (VV
= -xx)= + Σ pT (x + x1 ((xx)) + Σ xxxe(x' + +x", V
= xxx) + Σ pT (x + x 1)0(x)+
+ E Epti(( + Vo
x"
2
p T:8 xx + x x))(x()
By denoting (x + x*) = k== p=:0(x + 1), then we can rewrite Ve VoV*(s, as Ex 17Th (x + x* (x(*))
Plugging these results into (O)=(0) for st, we have that In * (dio(mat))(xx cd) Eria- Volnn s Q (x,,sss>>(x000 d)))
(O), - a (Ezjat
where
a = 2 (E(Vao(H, (xo) - ((Voo))
APPENDIX B: PROOF OF PROPOSITION
Based on the definition, we have that
Vdo(k.e) = Σ (d|x;)) 8) Σ P(x||x, d, (x,(x,x') + (x))
and
Vdot="s .n) (x) = Σ Pxx'xx, d xx," + Σ x(d[x,s s) Σ P(x' |x, d, d, Vo (x))
It follows that
Vdd(n.s)( (x) - VVoo(Fs,o))
= =Gxx0(x) + - (d][x, 8)
Σ
P(x' xx s) (Vao(a.s)(')> - VPS Vddr(,,s(x'')
x²
14

---

Published as a conference paper at ICLR 2025
where
G (x)
Σ
Σ
Σ
π (d] ((xxx - dd, (x,)(xxxx))
By writing P(x' xe, di (x, ( the above expression, we have
(x) s) 5)g x')) and rearrange
(x* π (dd
Σ (P(x1xx -) xxx F(d' x)xxx 8)
By similarly following the policy gradient theorem, we have
Vdd( (x) Vdo(x .)) Σ (x
The second equation in the proposition can be similarly derived.
APPENDIX C: EXPERIMENTAL SETUP DETAILS
Qualification gain function. We would like the qualification gain function to reflect real-world phenomena to some extent. One viewpoint that as an individual's qualification level increases, further progression becomes increasingly more difficult. Progression from beginner to intermediate takes less effort than progression from advanced to expert, so the qualification gain at high levels should have more weight. To encapsulate this idea, the qualification gain function for single transition defined as
x
X
g(x,, xx+1)=
g( 2rg) ji-j(=1
max
where the denominator serves to reduce the range of possible values.
Transition probability. The distribution P(x++1|X, s) is a deterministic function of (s,, X Tarifts Ye, de) where small amount of mass representing an individual moved from (x) to P( i.e.
+ Tdrift +1
if(de = 1, y/ = 1):
X++1
Xx + Tdrift
if(de 1, 3/: = 0), if(de = -)-
Xt + Wedrift
Enviroment settings. In the evaluation of proposed algorithms, for the simulation environment which we will simply refer to as Setting 1,2, and 3. These settings differ from each other in initial distributions over credit scores, repayment probabilities, and credit drift likelihoods.
three different settings
We generated the repayment probabilities by fitting a logistic regression model credit score datasets and used the predicted repayment probability from each credit score level. Setting 1 and 2 both use the same distribution over initial credit scores where Exp-s- [xo] < Ex [Xo]. They differ in the repayment probabilities where Setting uses probabilities generated using the Home Credit Default Risk dataset Montoya et al.) (2018), and the probabilities for Setting are from dataset previously released by Lending Club (207) type of peer-to-peer lending market.
In Setting 3, however, the initial distribution for both groups is the same. What does differ between groups are the credit drift probabilities p(zariff)). which is now dependent on the sensitive attribute. Here, p(Zarift
1| > p(Earift = -1sst) while p(Zarift 1[s < p( 1|s+).
APPENDIX D: INFLUENCES OF 3KL AND BA
We further examine how the penalty coefficients gKL and 3 quantitatively impact faimess and model performance. As shown in Figure 7 larger gkl. can help reduce the model variance. In Figure we show the influence of the strength of enforcing benefit faimess long-term faimesss Notably, we observe that the loan rate for 3 = is more balanced than that for 3 = 0, further implying that benefit fairness may align with long-term fairness in certain conditions.
15

---

Published as a conference paper at ICLR 2025
Jank Profit
20
pr
50 10.0
10
Timestep
Figure 7: Utility earned by the PPO-C variant comparing when the agent was trained with 3K versus 3KL 10.
Loan Comparison Step
ga 0.0
2.0
Group
Group s*
Group s*
Group
0.0
Credit Score
Credit Score
Figure 8: Lending rates for the PPO-Cb variant in setting 2.
16


Reviews:
[{"review_id": "CSiVaW2xPM", "reviewer": "Reviewer_YG39", "summary": "This paper studies long-term fairness through a causal decomposition framework. In particular, the authors first show that the total causal effect can be decomposed into a direct impact, delayed impact, and spurious impact. Moreover, a connection between benefit fairness and the causal effect of switching from a baseline policy to a virtual policy is established. In order to promote long-term fairness, the authors propose to update the PPO objective by introducing a fairness regularizer. Then the experiment section compares various methods on a simulated environment and the proposed method attains improved benefit fairness compared to the other methods.", "questions": "1. Is there a typo in Proposition 1 (main result)? The inner sum already marginalizes the feature $x'$.\n\n2. You have expressed DPE in terms of benefit fairness. Can the same be done for the delayed impact effect?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The decomposition of long-term fairness into direct and delayed impact is interesting, although it is not surprising. The connection between benefit fairness and the proposed measure of fairness (through policy intervention in an MDP) is quite interesting.\n\n2. The experimental section shows that the proposed modification of the PPO objective works, and in particular it can achieve approximate benefit fairness.", "weaknesses": "1. The authors claim that they introduce a general framework to study long-term fairness. However, the proposed framework is based on MDP which has been extensively studied in dynamic fairness.\n\n2. The causal decomposition result is very similar to existing decomposition results of causal fairness for static settings. In fact, the proof is very similar and the only difference seems to be that the indirect effect is replaced with the delayed impact.\n\n3. Finally, the main drawback of the work is that there is no provable guarantee that the proposed method achieves long-term fairness. Since the result is not evaluated across a large class of benchmarks/simulation studies it is difficult to tell whether the method attains long-term fairness in different types of MDPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OW61oYdoH2", "reviewer": "Reviewer_hawr", "summary": "The paper studies long-term fairness in causal reinforcement learning. The paper proposes the difference between the average qualification gain achieved by each group as their notion of long-term fairness and uses a PPO-type optimization objective to enforce fairness. Through a casual decomposition, the authors break down the qualification gain difference into 3 terms: direct impact, delayed impact, and spurious effects. Empirically, the paper studies how to lower the disparity in average qualification gains across groups and study the effect of each of the three terms.", "questions": "-- Can the authors specify how I should interpret the results of Figures 2 and 3 in light of the weaknesses mentioned above?\n\n-- What is the relationship of this notion of fairness, with the ones aiming to equalize rewards across all groups? See https://proceedings.mlr.press/v130/wen21a.html and references within.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- The paper studies long-term fairness which is an important, yet understudied problem.\n\n-- The paper proposes a novel fairness metric for long-term fairness in causal reinforcement learning.", "weaknesses": "-- There are no technical algorithmic novelties in the paper as the optimization problem is inspired by PPO. This is not a major weakness as the paper introduces novel elements like the qualification gain function and proposes a causal decomposition of the qualification gain.\n\n-- The experimental setup does not clearly show the advantage of the approach over prior baselines. For example, the difference between the worst and best approach is a mere 0.2% in all experiments in Figure 2. Are there other settings where this difference is more significant? The qualification gain disparity in Figure 3, does not go to 0 for each of the proposed metrics and seems to increase with increasing horizon. Can the authors provide why this is the case? Are there other settings where the qualification gain disparity goes to 0?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ArAAR0eZ3B", "reviewer": "Reviewer_KhgC", "summary": "This paper uses the causal perspective to study the long-term fairness problem where decisions can", "questions": "1. Why fairness penalty instead of re-weighting in the optimization problem?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The long-term fairness under causal inference perspective seems novel\n2. The fairness penalty decomposition is useful in practice", "weaknesses": "1. Theoretical guarantee not sufficient:\n(1). Convergence of the algorithm\n(2). How the penalty coefficients \\beta^{KL} and \\beta^{\\Lambda} quantitatively impact fairness and model performance (e.g., precision/recall per round)\n\n2. Despite fairness metrics improvement, it is difficult to tell if the model performance improved and if the fairness-aware optimization can result in a Pareto superior result for every group or every individual, there should be more theoretical and numerical evidence on the utilities instead of just utility gaps\n\n3. References: \n(1). The long-term fairness problems are studied in a previous line of work related to repeated distributionally robust optimizations, e.g., \"Fairness Without Demographics in Repeated Loss Minimization\"\n(2). The performative predictions and the strategic classification/regression problems are other research areas closely related to model decision having causal effect on data distribution, please compare with them\n\n4. Similarities and differences between the new metrics and the conventional fairness metrics:\nIt will be more helpful to build connections between the fairness terms in the long-term causal formulation with metrics like demographic parity, equal opportunity, equalized odds in the conventional settings like single shot classification and regression problems, which I think equalized odds (both TPR and FPR being the same) as well as loss disparities do not have counterparts in the causal formulation, and I encourage authors to provide explanations on why not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j45xc8Pd27", "reviewer": "Reviewer_FkQm", "summary": "The paper proposed a general causal framework to characterize long-term fairness in reinforcement learning (RL) settings, where the unfairness is represented by \"qualification gain disparity\" between the advantaged group and the disadvantaged group ($C_{\\pi}$). The authors decomposed the unfairness into three different parts (DPE, IPE, SPE) and used Proximal Policy Optimization (PPO) to optimize the decision policy. Particularly, DPE is closely related to benefit fairness (i.e., individuals from different groups who gain similar qualification improvement should have similar opportunities to get the treatment), and if the benefits are independent of the group attribute, then benefit fairness means with 0 DPE, demonstrating some kind of alignment between the long-term fairness proposed by the authors and benefit fairness. Motivated by this, the authors also added fairness as a separate term to the fair objective function. Thus, there are 2 fairness mechanisms ($PPO-C$, $PPO-Cb$) proposed, and the authors did simulation studies on real datasets to verify their findings.", "questions": "1. Could you make your contribution clearer and review previous work more comprehensively? (weakness 1, 2)\n\n2. Why are IPE and SPE not influenced by the fairness mechanisms? (weakness 3)\n\n3. Could you justify why it helps by directly adding benefit fairness as another regularization term? (weakness 3)\n\n4. Could you provide error bars for Figure 4, or am I missing something? (weakness 4)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Studying Long-term fairness as qualification gain disparity is well-motivated and of social importance.\n2. I like the idea of the causal decomposition using $\\pi_{PS}$.\n3. The experiments demonstrate the effectiveness of the fairness mechanisms.", "weaknesses": "Overall, I think the paper is interesting, but so far it does not situate well in previous literature. I am willing to increase my score if the authors clarify their contribution.\n\n1. Though I appreciate the efforts the authors made to compare their work to previous literature, I still feel the novelty of the paper is not clear enough. Firstly, just as the authors stated, [Hu et al., 2023, Hu & Zhang, 2022, Yu et al., 2022] considered long-term fairness in a causal framework. Especially for the framework proposed by [Hu & Zhang, 2022], the main difference seemed to be: (i) they considered the disparity of qualification rate (not the gain of qualification) as the fairness notion; (ii) they did not consider RL settings. Could the authors try making the comparison clearer?\n\n2. There are other recent papers on strategic classification and performative prediction considering long-term fairness in terms of qualification rate improvement (also known as social welfare) when population distribution can be shaped by the decision policy (e.g., [Jia et al., 2024, Jin et al., 2024]). Moreover, there are at least two fairness notions related to qualification gain: (i) equal improvement [Guldogan et al., 2022]; (ii) bounded effort [Heidari et al., 2019]. It is necessary to review these papers.\n\n3. The fairness mechanisms just stack all terms together and I am not sure about the novelty. Plus, in Figure 5, it seems that IPE and SPE are rarely influenced by the proposed fairness mechanisms. Could you explain why?\n\n4. Why not provide error bars for Figure 4?\n\n> References\n\nJia, Zhuangzhuang, et al. \"Distributionally Robust Performative Optimization.\" arXiv preprint arXiv:2407.01344 (2024).\n\nJin, Kun, et al. \"Addressing Polarization and Unfairness in Performative Prediction.\" arXiv preprint arXiv:2406.16756 (2024).\n\nGuldogan, Ozgur, et al. \"Equal improvability: A new fairness notion considering the long-term impact.\" arXiv preprint arXiv:2210.06732 (2022).\n\nHeidari, Hoda, Vedant Nanda, and Krishna P. Gummadi. \"On the long-term impact of algorithmic decision policies: Effort unfairness and feature segregation through social learning.\" arXiv preprint arXiv:1903.01209 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)