Title: Long-Term Fairness with Unknown Dynamics
Venue: NeurIPS 2023
Accepted: yes
Keywords: Long-term Fairness

Dynamics

Reinforcement Learning
Abstract: While machine learning can myopically reinforce social inequalities, it may also be used to dynamically seek equitable outcomes. In this paper, we formalize long-term fairness as an online reinforcement learning problem for a policy affecting human populations. This formulation accommodates dynamical control objectives, such as achieving equitable population states, that cannot be incorporated into static formulations of fairness. We demonstrate that algorithmic solutions to the proposed fairness problem can adapt to unknown dynamics and, by sacrificing short-term incentives, drive the policy-population system towards more desirable equilibria. For the proposed setting, we develop an algorithm that adapts recent work in online learning and prove that this algorithm achieves simultaneous probabilistic bounds on cumulative loss and cumulative violations of fairness. In the classification setting subject to group fairness, we compare our proposed algorithm to several baselines, including the repeated retraining of myopic or distributionally robust classifiers, and to a deep reinforcement learning algorithm that lacks fairness guarantees. Our experiments model human populations according to evolutionary game theory and integrate real-world datasets.

Long-Term Fairness with Unknown Dynamics

Abstract

While machine learning can myopically reinforce social inequalities, it may also be used to dynamically seek equitable outcomes. In this paper, we formalize long-term fairness as an online reinforcement learning problem for a policy affecting human populations. This formulation accommodates dynamical control objectives, such as achieving equitable population states , that cannot be incorporated into static formulations of fairness. We demonstrate that algorithmic solutions to the proposed fairness problem can adapt to unknown dynamics and, by sacrificing short-term incentives, drive the policy-population system towards more desirable equilibria. For the proposed setting, we develop an algorithm that adapts recent work in online learning and prove that this algorithm achieves simultaneous probabilistic bounds on cumulative loss and cumulative violations of fairness. In the classification setting subject to group fairness, we compare our proposed algorithm to several baselines, including the repeated retraining of myopic or distributionally robust classifiers, and to a deep reinforcement learning algorithm that lacks fairness guarantees. Our experiments model human populations according to evolutionary game theory and integrate real-world datasets.

1 Introduction

As machine learning (ML) algorithms are deployed for tasks with real-world social consequences (e.g., school admissions, loan approval, medical interventions, etc.), the possibility exists for runaway social inequalities (Crawford and Calo, 2016; Chaney et al., 2018; Fuster et al., 2018; Ensign et al., 2018). While 'fairness' has become a salient ethical concern in contemporary research, the closed-loop dynamics of real-world systems with feedback, i.e., comprising ML-driven decisions and populations that mutually adapt to each other (Fig. 5 in the supplementary material) remain poorly understood. We assert that realistic scenarios are often described by fundamentally unknown dynamics : Even with models of human behavior based on rational behavior or evolutionary game theory, utilities and risk-preferences are generally unknown or uncertain. For this reason, we advocate for dynamics-agnostic formulations of fairness, for which reinforcement learning is a natural fit.

In this paper, our primary contribution is to consider the problem of long-term fairness , or algorithmic fairness in the context of a dynamically responsive population, as a reinforcement learning (RL)

problem subject to constraint. In our formulation, the central learning task is to develop a policy that minimizes cumulative loss (e.g., financial risk, negative educational outcomes, misdiagnoses, etc.) incurred by an ML agent interacting with a human population up to a finite time horizon, subject to constraints on cumulative 'violations of fairness', which we refer to in a single time step as disparity and cumulatively as distortion .

Our central hypothesis is that an RL formulation of long-term fairness may allow an agent to learn to sacrifice short-term utility in order to drive the system towards more desirable equilibria . The core practical difficulties posed by our general problem formulation, however, are the potentially unknown dynamics of the system under control, which must be determined by the RL agent online (i.e., during actual deployment), and the general non-convexity of the losses or constraints considered. Additionally, our problem setting generally requires continuous state and action spaces for the RL agent, which preclude familiar methods that yield performance guarantees in discrete settings.

Our secondary contributions are 1) to show that long-term fairness can be solved within asymptotic, probabilistic bounds under certain dynamical assumptions and 2) to demonstrate that the problem can also be addressed more flexibly with existing RL methods. For theoretical guarantees, we develop L-UCBFair , an online RL method, and prove sublinear bounds on regret (suboptimiality of cumulative loss) and distortion (suboptimality of cumulative disparity) with high probability (Section 3.1). To demonstrate practical solutions without strong safety guarantees, we apply R-TD3 , an adaptation of a well-known deep reinforcement learning method (viz., TD3) to a time-dependent Lagrangian relaxation of the central problem. We compare L-UCBFair and R-TD3 to several baselines (Section 3.3), including myopic policies, in interaction with simulated populations initialized with synthetic or real-world data and updated according to evolutionary game theory (Section 4).

Finally, this paper considers fairness in terms of statistical regularities across (ideally) socioculturally meaningful groups . While internal conflict exists between different statistical measures of fairness (Corbett-Davies and Goel, 2018), we show that an RL approach to long-term fairness can mitigate trade-offs between immediately-fair policy decision outcomes (e.g., acceptance rate disparities (Dwork et al., 2012; Zemel et al., 2013; Feldman et al., 2015)) and causally-fair population states (e.g., qualification rate (Raab and Liu, 2021; Zhang et al., 2020)). In particular, we show that our proposed solutions can learn to avoid the familiar trap of pursuing short-term fairness metrics only to widen underlying disparities that demand escalating interventions at the expense of utility (Section 5.2).

1.1 Related Work

Our formulation of long-term fairness bridges recent work on 'fairness in machine learning', which has developed in response to the proliferation of data-driven methods in society, and 'safe reinforcement learning', which seeks theoretical safety guarantees in the control of uncertain dynamical systems.

Dynamics of Fairness in Machine Learning We distinguish long-term fairness from the dynamics of fair allocation problems (Joseph et al., 2016; Jabbari et al., 2017; Tang et al., 2021; Liu et al., 2017) and emphasize side-effects of algorithmic decisions affecting future decision problems. By formalizing long-term fairness in terms of cumulative losses and disparities, we iterate on a developing research trend that accounts for the dynamical response of a human population to deployed algorithmic prediction: both as a singular reaction (Liu et al., 2018; Hu et al., 2019; Hu and Zhang, 2022) or as a sequence of mutual updates between the population and the algorithm (Coate and Loury, 1993; D'Amour et al., 2020; Zhang et al., 2020; Heidari et al., 2019; Wen et al., 2019; Liu et al., 2020; Hu and Chen, 2018; Mouzannar et al., 2019; Williams and Kolter, 2019; Raab and Liu, 2021).

Several prior works have considered the long-term fairness implications in the context of performative stability or optimality (Perdomo et al., 2020) with known, stateless dynamical transitions: Hu and Zhang (2022) characterize the convergence of systems typified by sequential, myopic policies while Hashimoto et al. (2018) contrast myopic policy with a modified objective satisfying distributional robustness. While Mouzannar et al. (2019) and Raab and Liu (2021) address stateful dynamical transitions, the cited works only treat myopic classifiers that optimize immediate utility (subject to fairness constraints) rather than learning to anticipate dynamical population responses. Finally, while Wen et al. (2019) explore reinforcement learning for long-term fairness, the discussion is limited to a tabular explore-then-commit approach over discrete state and action spaces. This approach does not

generalize to continuous spaces, where we provide separate and tighter bounds for both utility and disparity.

Safe Reinforcement Learning L-UCBFair furthers recent efforts in safe RL. While model-based approaches, in which the algorithm learns an explicit dynamical model of the environment, constitute one thread of prior work (Efroni et al., 2020; Singh et al., 2020; Brantley et al., 2020; Zheng and Ratliff, 2020; Kalagarla et al., 2021; Liu et al., 2021; Ding et al., 2021), leading algorithms in this domain are characterized by significant time and space complexity. Among model-free algorithms, the unknown dynamics of our setting preclude the use of a simulator that allows algorithms to search over arbitrary state-action pairs (Xu et al., 2021; Ding et al., 2020; Bai et al., 2022).

While Wei et al. (2022) introduce a model-free, simulator-free algorithm, the tabular approach they consider is only applicable to discrete state and action spaces, while our setting requires continuous state and action spaces. To tackle continuous state space, Ding et al. (2021) and Ghosh et al. (2022) consider linear dynamics: Ding et al. (2021) develop a primal-dual algorithm with safe exploration, and Ghosh et al. (2022) use a softmax policy design. Both algorithms are based on the work of Jin et al. (2020), which proposed a least squares value iteration (LSVI) method, using an Upper Confidence Bound (UCB) (Auer et al., 2002) to estimate a state-action ' Q ' function. In addition to continuous state spaces, L-UCBFair also addresses continuous action spaces. To our knowledge, L-UCBFair is the first model-free, simulator-free RL algorithm to provides theoretical safety guarantees for both discrete and continuous state and action spaces . Moreover, L-UCBFair achieves bounds on regret and distortion as tight as any algorithm thus far on discrete action space (Ghosh et al., 2022).

2 Problem Formulation

We motivate our formulation of long-term fairness with a binary classification task, though the general formulation we propose is more widely applicable. Given this initial task, we sequentially incorporate fairness constraints , then population dynamics , and then cumulative loss and disparity , before fully formalizing the central problem of optimizing cumulative loss subject to expected dynamics and constraints on cumulative disparity.

For our initial binary classification task, suppose that a random individual, sampled i.i.d. from a fixed population, has features X ∈ R d , a label Y ∈ {-1 , 1 } , and a demographic group G ∈ G (where G = [ n ] for n ≥ 2 ). Denote the joint distribution of these variables in the population as s := Pr( X,Y,G ) such that X,Y,G ∼ s . The task is to predict Y (as ˆ Y ) from X and G by choosing a classifier a , where ˆ Y ∼ a ( X,G ) , to minimize the empirical loss L ( s, a ) e.g. = E [ L ( Y, ˆ Y ) ] , where L denotes some individual loss such as zero-one-loss. In general, we allow arbitrary, (unit-interval) bounded loss functions L ( s, a ) ∈ [0 , 1] such that the basic classification task is min a L ( s, a ) .

In this paper, we also wish to consider measures of fairness inherent to the distribution s (e.g., parity of group qualification rates Pr( Y =1 | G = g ) ). Such measures of fairness can only be affected by changes to s and thus require dynamics, which are not captured by the above formulation (Raab and Liu, 2021; Zhang et al., 2020). We will see that such notions of disparity are well-suited to an RL formulation of long-term fairness.

The standard, 'fair' classification task ( without a dynamically responsive population) incorporates constraints on the choice of classifier a , such that the disparity D ∈ [0 , 1] induced on distribution s by a is bounded by some value c ∈ [0 , 1] . Formally, the task is min a L ( s, a ) subject to D ( s, a ) ≤ c . A standard measure of disparity D is the violation of 'demographic parity' (Dwork et al., 2012). For example, when G = { g 1 , g 2 } , D ( s, a ) e.g. = ∣ ∣ ξ s,a ( g 1 ) -ξ s,a ( g 2 ) ∣ ∣ 2 , where ξ s,a ( g ) := Pr( ˆ Y =1 | G = g ) .

State, action, and policy Adopting the semantics of sequence of dependent classification tasks, we identify the time-dependent distribution s τ ∈ S of individuals in the population as a state and the chosen classifier a τ ∈ A as an action of some algorithmic agent at time τ . While state space S is arbitrary, we assume that action space A admits a Euclidean metric, under which it is closed (i.e., A is isomorphic to [0 , 1] m , m ∈ Z > 0 ). We specify that a τ is sampled stochastically according to the agent's current policy π τ , that is, a τ ∼ π τ ( s τ ) . Additionally, we assume s τ is fully observable at time τ ∈ { 1 , 2 , ... } . In practice, s τ must be approximated from finitely many empirical samples, though this caveat introduces well-understood errors that vanish in the limit of infinitely many samples.

Dynamics Moving beyond the one-shot 'fair' classification task above, let us now assume that a population may react to classification, inducing the distribution s to change. In practice, such 'distribution shift' is a well-known, real-world phenomenon that can increase realized loss and disparity when classifiers are fixed (Chen et al., 2022). For classifiers that free to update in response to a mutating distribution s , subsequent classification tasks depend on the (stochastic) predictions made in previous tasks. In our formulation, we assume the existence of dynamical kernel P that maps a state s and action a at time τ to a distribution over possible states at time τ +1 . That is, s τ +1 ∼ P ( s τ , a τ ) .

We stipulate that P may be initially unknown, but we assume that it does not explicitly depend on time and may be reasonably approximated or learned 'online'. While real-world dynamics may depend on information other than the current distribution of classification-specific variables Pr( X,Y,G ) -e.g., exogenous parameters, history, or additional variables of state - we have identified the dynamical state s with the current, fully-observed distribution for simplicity and tractability.

Reward and utility, value and quality functions Because the standard RL convention is to maximize reward rather than minimize loss , for an RL agent, we define the instantaneous reward r ( s τ , a τ ) := 1 -L ( s τ , a τ ) ∈ [0 , 1] and a separate 'utility' g ( s τ , a τ ) := 1 -D ( s τ , a τ ) ∈ [0 , 1] , where r and g do not explicitly depend on time τ . We next propose to optimize anticipated cumulative reward subject to constraints on anticipated cumulative utility. Let the index j represent either reward r or utility g . We use the letter V (for 'value') to denote the future expected accumulation of j over steps [ h, ..., H ] (without time-discounting) starting from state s , using policy π . Likewise, we denote the 'quality' of an action a in initial state s with the letter Q . For j ∈ { r, g } , we define

Note that V and Q marginalize over the stochasticity of state transitions s τ +1 ∼ P ( s τ , a τ ) and the sampling of actions a τ ∼ π τ ( s τ ) . By the boundedness of r and, g , the values of V and Q belong to the interval [0 , H -h +1] .

'Long-term fairness' via reinforcement learning The central problem explored in this paper is

We emphasize that this construction considers a finite time horizon of H steps, and we denote the optimal value of π as π ⋆ . We first assume that a solution to the problem exists bounded within the interior of the feasible set (i.e., strict primal feasibility or Slater's condition ), as in Efroni et al. (2020), Ding et al. (2021), and Ghosh et al. (2022):

Assumption 2.1 (Slater's Condition) . ∃ γ > 0 , ¯ π , such that V ¯ π g, 1 ( s ) ≥ ˜ c + γ .

The Online Setting Given initially unknown dynamics, we formulate long-term fairness for the online setting (in which learning is only possible through actual 'live' deployment of policy, rather than through simulation). As it is not possible to unconditionally guarantee constraint satisfaction in Eq. (1) over a finite number of online steps, we instead measure two types of regret : the suboptimality of a policy with respect to cumulative incurred loss, denoted Regret , and the suboptimality of a policy with respect to cumulative induced disparity, denoted denoted Drtn for 'distortion'.

3 Algorithms and Analysis

Weshow that it is possible to provide guarantees for long-term fairness in the online setting. To do this, we develop L-UCBFair , the first model-free, simulator-free algorithm to provide such guarantees with continuous state and action spaces. Specifically, we prove probabilistic, sublinear bounds for regret and distortion under a linear MDP assumption (Assumption 3.1) and properly chosen parameters (Appendix B.1, in the supplementary material). We defer proofs to the supplementary material.

3.1 L-UCBFair

Episodic MDP L-UCBFair inherits from a family of algorithms that treat an episodic Markov decision process (MDP) Jin et al. (2020). Therefore, we first map the long-term fairness problem to the episodic MDP setting, which we denote as MDP ( S , A , H, P , L , D ) . The algorithm runs for K episodes , each consisting of H time steps. At the beginning of each episode, which we index with k , the agent commits to a sequence of policies π k = ( π k 1 , π k 2 , ..., π k H ) for the next H steps. At each step h within an episode, an action a k h ∈ A is sampled according to policy π k h , then the state s k h +1 ∈ S is sampled according to the transition kernel P ( s k h , a k h ) . s k 1 is sampled arbitrarily for each episode.

Episodic Regret Because L-UCBFair predetermines its policy for an entire episode, we amend our definitions of regret and distortion over all HK time steps by breaking them into a sum over K episodes of length H :

The Lagrangian Consider the Lagrangian L associated with Eq. (1), with dual variable ν ≥ 0 :

L-UCBFair approximately solves the primal problem, max π min ν L ( π, ν ) , which is non-trivial, since the objective function is seldom concave in practical parameterizations of π . Let ν ∗ denote the optimal value of ν ; L-UCBFair assumes bound ν ∗ ≤ V , given parameter V , as described in the supplementary material (Assumption B.1).

Assumption 3.1 (Linear MDP) . We assume MDP ( S , A , H, P , L , D ) is a linear MDP with feature map ϕ : S × A → R d . That is, for any h , there exist d signed measures µ h = { µ 1 h , . . . , µ d h } over S and vectors θ r,h , θ g,h ∈ R d such that, for any ( s, a, s ′ ) ∈ S × A × S ,

Assumption 3.1 addresses the curse of dimensionality when state space S is the space of distributions over X,Y,G . This assumption is also used by Jin et al. (2020) and Ghosh et al. (2022), and a similar assumption is made by Ding et al. (2021). This assumption is well-justified in continuous time if we allow for infinite-dimensional Hilbert spaces Brunton et al. (2021), but in practice we require limited dimensionality d for the codomain of ϕ . In our experiments, we use a neural network to estimate a feature map ˆ ϕ offline which approximately satisfies the linear MDP assumption (Appendix D.1).

3.1.1 Construction

L-UCBFair , or ' LSVI-UCB for Fairness' (Algorithm 1) is based on a Least-Squares Value Iteration with an optimistic Upper-Confidence Bound, as in LSVI-UCB Jin et al. (2020). For each H -step episode k , L-UCBFair maintains estimates for Q k r , Q k g and a time-indexed policy π k . In each episode, L-UCBFair updates Q k r , Q k g , interacts with the environment, and updates the dual variable ν k , which is constant over episode k .

LSVI-UCB Jin et al. (2020) The estimation of Q is challenging, as it is impossible to iterate over all s, a pairs when S and A are continuous and P is unknown. LSVI parameterizes Q ⋆ h ( s, a ) by the linear form w ⊤ h ϕ ( s, a ) , as used by Jin et al. (2020), and updates

In addition, a 'bonus term' β ( ϕ ⊤ Λ -1 h ϕ ) 1 / 2 is added to the estimate of Q in Algorithm 1 to encourage exploration.

Adaptive Search Policy Compared to the works of Ding et al. (2021) and Ghosh et al. (2022), the major challenge we face for long-term fairness is a continuous action space A , which renders the

independent computation of Q k r , Q k g for each action impossible. To handle this issue, we propose an adaptive search policy: Instead of drawing an action directly from a distribution over continuous values, π represents a distribution over finitely many ( M ), V oronoi partitions of A . After sampling a region with a softmax scheme, the agent draws action a uniformly at random from it. In addition to defining a Voronoi partitioning of the action space for the adaptive search policy of L-UCBFair (in the supplementary material), we make the following smoothness assumption, which we use to bound the error introduced by this sampling method:

Assumption 3.2 (Lipschitz) . There exists ρ > 0 , such that ∥ ϕ ( s, a ) -ϕ ( s, a ′ ) ∥ 2 ≤ ρ ∥ a -a ′ ∥ 2 .

Dual Update For L-UCBFair , the update method for estimating ν in Eq. (4) is also essential. Since V π r, 1 ( s ) and V π g, 1 ( s ) are unknown, we use V k r, 1 ( s ) and V k g, 1 ( s ) to estimate them. An estimate for ν is iteratively updated by performing gradient ascent in the Lagrangian (Eq. (4)) with step-size η , subject to the assumed upper bound V for ν (Assumption B.1). A similar method is also used in Ding et al. (2021); Ghosh et al. (2022).

3.1.2 Analysis

We next bound the regret and distortion, defined in Eq. (3), realizable by L-UCBFair . We then compare L-UCBFair with existing algorithms for discrete action spaces and discuss the importance of the number of regions M and the maximum distance ϵ I from any action to the center of its corresponding Voronoi partition.

Theorem 3.3 (Boundedness) . With probability 1 -p , there exists a constant b such that L-UCBFair (Algorithm 1) achieves

For a detailed proof of Theorem 3.3, refer to Appendix B.1. This theorem indicates that L-UCBFair achieves O ( H 2 √ d 3 K ) bounds for both regret and distortion with high probability. Compared to the algorithms introduced by Ding et al. (2021) and Ghosh et al. (2022), which work with discrete action space, L-UCBFair guarantees the same asymptotic bounds on regret and distortion.

3.2 R-TD3

Because assumptions in Section 3.1 (e.g., the linear MDP assumption) are often violated in the real world, we also consider deep reinforcement learning methods as a flexible alternative. Concretely, we apply 'Twin-Delayed Deep Deterministic Policy Gradient' (TD3) Fujimoto et al. (2018), with an implementation and default parameters provided by the open-source package 'Stable Baselines 3' Raffin et al. (2021), on a Lagrangian relaxation of the long-term fairness problem with timedependent multipliers for the reward and utility terms (Eq. (6)). We term this specific algorithm for long-term fairness R-TD3 . While such methods lack provable safety guarantees, they may still confirm our hypothesis that agents trained via RL can learn to sacrifice short-term utility in order to drive dynamics towards preferable long-term states and explicitly incorporate dynamical control objectives provided as functions of state.

To treat the long-term fairness problem (Eq. (1)) using unconstrained optimization techniques (i.e., methods like TD3), we consider a time-dependent Lagrangian relaxation: We train R-TD3 to optimize

where s τ +1 ∼ P ( s τ , a τ ) , λ τ = τ/H .

Strictly applied, myopic fairness constraints can lead to undesirable dynamics and equilibria Raab and Liu (2021). Eq. (6) relaxes these constraints (hard → soft) for the near future while emphasizing them long-term. Thus, we hope to develop classifiers that learn to transition to more favorable equilibria.

Algorithm 1 L-UCBFair

Input: A set of points { I 0 , I 1 , · · · , I M } satisfy Definition B.2. ϵ I = 1 2 ρ (1+ χ ) KH .

for

k

= 1

,

2

, ..., K

do episode

Receive the initial state s k 1 .

end for

for iteration

ξ

(

i,j

end for

= 1

·

, M

· · ·

Λ

-

h

i

i

ϕ

(

, I

,

)

and index

1

)

i

ϕ

(

T

k

(

)

·

, I

)

∈ {

,

r, g

Q

}

do j,h

k

·

, I

(

end for

for

, H

h

= 1

do step

Compute

,

Q

r,h

· · ·

k

k

Take action

end for

h

s

, I

(

∼

i

, Q

)

(

·

|

h

h

k

a

π

g,h

s

k

k

k

h

)

h

i

s

, I

, π

k

(

(

)

and observe

i

I

s

k

h

|

s

+1

h

k

.

3.3 Baselines

We compare L-UCBFair and R-TD3 to a greedy agent as a proxy for a myopic status quo in which policy is repeatedly determined by optimizing for immediate utility, without regard for the population dynamics. This standard is known as 'Repeated Risk Minimization' (Perdomo et al., 2020; Hu and Zhang, 2022), and we implement it using simple gradient descent on the different classes of ( λ -parameterized) objective functions f we consider (Eq. (7)). Having adopted a notion of fairness that relies on 'groups', we presuppose different groups of agents indexed by g , and denote groupconditioned loss as L g . The objectives that are approximately minimized for each baseline are

The two "Maxmin" objectives are related to distributionally robust optimization, which has been previously explored in the context of fairness (Hashimoto et al., 2018), while the two "Myopic" objectives are more straight-forward. While our baselines do not guarantee constraint satisfaction, the two objectives labelled '-Fair' are nonetheless 'constraint aware' in precisely the same way as a firm that (probabilistically) incurs penalties for violating constraints.

4 Simulated Environments

To evaluate the proposed algorithms and baselines, we consider a series of binary ( Y ∈ {-1 , 1 } ) classification tasks on a population of two groups G = { g 1 , g 2 } modeled according to evolutionary

j

1

/

2

)

.

i

)

min

w

k

j,h

, ϕ

(

·

, I

i

)

〉

+

βξ

i,j

, H

]

game theory (using replicator dynamics, as described in Appendix A.1, in the supplementary material). We consider two families of distributions of real-valued features for the population: One that is purely synthetic, for which X ∼ N ( Y, 1) , independent of group G , and one that is based on logistic regressions to real-world data, described in Appendix A.2 (in the supplementary material). Both families of distributions over X are parameterized by the joint distribution Pr( Y, G ) . RL agents are trained on episodes of length H initialized with randomly sampled states.

In order to better handle continuous state space, we make the following assumption, which has been used to simplify similar synthetic environments (Raab and Liu, 2021):

Assumption 4.1 (Well-behaved feature) . For the purely synthetic setting, we require X to be a 'well-behaved' real-valued feature within each group. That is, for each group g , Pr( Y =1 | G = g, X = x ) strictly increases in x.

As an intuitive example of Assumption 4.1, if Y represents qualification for a fixed loan and X represents credit-score, we require higher credit scores to imply higher likelihood that an individual is qualified for the loan.

Theorem 4.2 (Threshold Bayes-optimality) . For each group g , when Assumption 4.1 is satisfied, the Bayes-optimal, deterministic binary classifier is a threshold policy described by a feature threshold a g for group g . That is, if X ≥ a g then ˆ Y = 1 ; otherwise, ˆ Y = -1 .

As a result of Theorem 4.2, we consider our action space to be the space of group-specific thresholds, and denote an individual action as the vector a := ( a 1 , a 2 ) . Nonetheless, we note that Assumption 4.1 is often violated in practice, as it is in our semi-synthetic setting.

5 Experimental Results

Do RL agents learn to seek favorable equilibria against short-term utility? Is Lagrangian relaxation Eq. (1) sufficient to encourage this behavior? We give positive demonstrations for both questions.

5.1 Losses and Disparities Considered

Our experiments consider losses L which combine true-positive and true-negative rates, i.e.,

For disparity D , we consider functions of the form D ( s, a ) = 1 2 ∥ ξ s,a ( g 1 ) -ξ s,a ( g 2 ) ∥ 2 that measure violations of demographic parity ( DP ) (Dwork et al., 2012), equal opportunity ( EOp ) (Hardt et al., 2016), equalized odds ( EO ), and qualification rate parity ( QR ), described in Table 1. We note that QR is inconsequential for the baseline agents, which ignore the mutability of the population state.

5.2 Results

Our experiments show that algorithms trained with an RL formulation of long-term fairness can drive a reactive population toward states with higher utility and fairness than myopic policies. In Fig. 1, the baseline policies (i.e., subfigures (a) and (b)), which focus on short-term incentives drive disparate qualification rates and attain lower long-term utility than the RL agents, indicating that short-term utility is misaligned with desirable dynamics in this example. In subfigure (b) in particular, the baseline policy increasingly violates the static fairness condition with time, agreeing with previous results by Raab and Liu (2021). Meanwhile, the RL algorithms (subfigures (c) and (d)) learn to drive universally high qualification rates, thus allowing

|

-

policies that capture higher utility with time (subfigure (e)), Fig. 3.

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.000000

0.005373

a)

Maxmin-Fair

, (

tp

,

DP

),

λ

=

1

2

Group 1 Qualification Rate

Group 2 Qualification Rate

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.0000

0.0728

b)

Myopic-Fair

, (

tp

,

DP

)

λ

=

1

2

Group 1 Qualification Rate

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Demographic Parity)

Demographic Parity

0

1

c)

L-UCBFair

, (

tp

,

DP

)

Group 1 Qualification Rate

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Demographic Parity)

Demographic Parity

0

1

d)

R-TD3

, (

tp

,

DP

)

Group 1 Qualification Rate

0.2

0.4

0.6

0.8

1.0

Episodic Mean Loss

Maxmin

Maxmin-Fair

Baseline

Baseline-Fair

UCB-Fair

R-TD3

Demographic Parity

Maxmin

Maxmin-Fair

Myopic

Myopic-Fair

L-UCBFair

R-TD3

e) (

tp

,

DP

)

Our central hypothesis, that long-term fairness via RL may induce an algorithm to sacrifice short-term utility for better long-term outcomes, is clearly demonstrated in the purely synthetic environment depicted by subfigures (a1-a3) of Fig. 2, in which the environment provides higher immediate utility (true-positive rates) but lower long-term utility when a policy chooses initially high acceptance rates. In this case, the RL algorithm (subfigure (a2)) drives the system towards high qualification rates by giving up immediate utility maximized by the myopic agent (subfigure (a1)).

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Demographic Parity)

Demographic Parity

0

1

a1) Myopic-Fair

(

tp

,

DP

),

λ

=

1

2

Group 1 Qualification Rate

Group 2 Qualification Rate

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Demographic Parity)

Demographic Parity

0

1

a2)

L-UCBFair

(

tp

,

DP

)

Group 1 Qualification Rate

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Qualification Rate)

Qualification Rate Parity

0

1

b1)

L-UCBFair

(

tp

,

QR

)

Group 1 Qualification Rate

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Qualification Rate)

Qualification Rate Parity

0

1

b2)

R-TD3

(

tp

,

QR

)

Group 1 Qualification Rate

0.6

0.8

1.0

Episodic Mean Loss

Myopic-Fair

UCB-Fair

a3) Task: (

tp

,

DP

)

0.25

0.50

0.75

1.00

Episodic Mean Loss

UCB-Fair

R-TD3

b3) Task: (

tp

,

QR

)

With subfigures (b1-b3) of Fig. 2, we demonstrate the capability of RL to incorporate notions of fairness (e.g., qualification rate parity QR ), that are impossible to formulate in the myopic setting. In subfigures (b1-b3), both RL agents learn to satisfy qualification rate parity by driving the state of the population towards equal qualification rates by group.

Finally, our experiments also show that RL algorithms without theoretical guarantees may be applicable to long-term fairness. In Fig. 2 subfigure (b2), R-TD3 achieves similar qualitative behavior as L-UCBFair (subfigure (b1)), that is, driving high qualification at the expense of short-term utility) while achieving lower episodic mean loss (subfigure (b3)).

0

500

1000

1500

2000

Iterations

0.50

0.52

0.54

0.56

0.58

0.60

Episodic Mean Loss

0

500

1000

1500

2000

Iterations

0.12

0.14

0.16

0.18

0.20

Episodic Mean Disparity

(

tp

+0

.

8

tn

,

DP

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Accumulated Disparity

0

40

(a) TD3, no constraint

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Accumulated Disparity

0

40

(b) R-TD3

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Accumulated Loss

0

85

(c) TD3, no constraint

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Accumulated Loss

0

85

(d) R-TD3

Limitations Despite the potential, highlighted by our experiments, for RL-formulation of fairness to drive positive long-term social outcomes, it is not possible to truly validate such approaches without deployment on actual populations, which may be practically and ethically fraught. In addition, violations of fairness or decreased utility may be difficult to justify to affected populations and stakeholders, especially when the bounds provided by L-UCBFair , while as tight as any known, rely on assumptions that may be violated in practice.

Concluding remarks Machine learning techniques are frequently deployed in settings in which affected populations will react to the resulting policy, closing a feedback loop that must be accounted for. In such settings, algorithms that prioritize immediate utility or static notions of fairness may yield dynamics that are misaligned with these objectives long-term.

In this paper, we have reformulated long-term fairness as an online reinforcement learning problem (Eq. (1)) to address the importance of dynamics. We have shown that algorithmic solutions to this problem (e.g., L-UCBFair ) are capable of simultaneous theoretical guarantees regarding cumulative loss and disparity (violations of fairness). We have also shown that these guarantees can be relaxed in practice to accommodate a wider class of RL algorithms, such as R-TD3 . Finally, we emphasize again that the RL framework of long-term fairness allows notions of disparity inherent to the state of a population to be explicitly treated, while such definitions are inoperable in the standard, myopic framework of fairness. We hope that our contributions spur interest in long-term mechanisms and incentive structures for machine learning to be a driver of positive social change.

Acknowledgements This work is partially supported by the National Science Foundation (NSF) under grants IIS-2143895, IIS-2112471, IIS-2040800, and CCF-2023495.

References

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3682-3689, 2022.

Kianté Brantley, Miro Dudik, Thodoris Lykouris, Sobhan Miryoosefi, Max Simchowitz, Aleksandrs Slivkins, and Wen Sun. Constrained episodic reinforcement learning in concave-convex and knapsack settings. Advances in Neural Information Processing Systems , 33:16315-16326, 2020.

Steven L Brunton, Marko Budiši´ c, Eurika Kaiser, and J Nathan Kutz. Modern koopman theory for dynamical systems. arXiv preprint arXiv:2102.12086 , 2021.

Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM Conference on Recommender Systems , pages 224-232. ACM, 2018.

Yatong Chen, Reilly Raab, Jialu Wang, and Yang Liu. Fairness transferability subject to bounded distribution shift. arXiv preprint arXiv:2206.00129 , 2022.

Stephen Coate and Glenn C Loury. Will affirmative-action policies eliminate negative stereotypes? The American Economic Review , pages 1220-1240, 1993.

Sam Corbett-Davies and Sharad Goel. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023 , 2018.

Kate Crawford and Ryan Calo. There is a blind spot in AI research. Nature News , 538(7625):311, 2016.

Alexander D'Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D Sculley, and Yoni Halpern. Fairness is not static: Deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 525-534, 2020.

Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pages 3304-3312. PMLR, 2021.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive.ics. uci.edu/ml .

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214-226, 2012.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps. arXiv preprint arXiv:2003.02189 , 2020.

Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. Runaway feedback loops in predictive policing. In Conference of Fairness, Accountability, and Transparency , 2018.

Alessandro Epasto, Mohammad Mahdian, Vahab Mirrokni, and Emmanouil Zampetakis. Optimal approximation-smoothness tradeoffs for soft-max functions. Advances in Neural Information Processing Systems , 33:2651-2660, 2020.

Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 259-268, 2015.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Andreas Fuster, Paul Goldsmith-Pinkham, Tarun Ramadorai, and Ansgar Walther. Predictably unequal? The effects of machine learning on credit markets. The Effects of Machine Learning on Credit Markets , 2018.

Arnob Ghosh, Xingyu Zhou, and Ness Shroff. Provably efficient model-free constrained rl with linear function approximation. arXiv preprint arXiv:2206.11889 , 2022.

Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning, 2016.

Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning , pages 1929-1938. PMLR, 2018.

Hoda Heidari, Vedant Nanda, and Krishna P. Gummadi. On the long-term impact of algorithmic decision policies: Effort unfairness and feature segregation through social learning. the International Conference on Machine Learning (ICML) , 2019.

Lily Hu and Yiling Chen. A short-term intervention for long-term fairness in the labor market. In Proceedings of the 2018 World Wide Web Conference on World Wide Web , pages 1389-1398. International World Wide Web Conferences Steering Committee, 2018.

Lily Hu, Nicole Immorlica, and Jennifer Wortman Vaughan. The disparate effects of strategic manipulation. In Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 259-268, 2019.

Yaowei Hu and Lu Zhang. Achieving long-term fairness in sequential decision making. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9549-9557, 2022.

Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in reinforcement learning. In International conference on machine learning , pages 1617-1626. PMLR, 2017.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR, 2020.

Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. Advances in neural information processing systems , 29, 2016.

Krishna C Kalagarla, Rahul Jain, and Pierluigi Nuzzo. A sample-efficient algorithm for episodic finitehorizon mdp with constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8030-8037, 2021.

Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning , pages 3150-3158. PMLR, 2018.

Lydia T Liu, Ashia Wilson, Nika Haghtalab, Adam Tauman Kalai, Christian Borgs, and Jennifer Chayes. The disparate equilibria of algorithmic decision making when individuals invest rationally. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 381-391, 2020.

Tao Liu, Ruida Zhou, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Learning policies with zero or bounded constraint violation for constrained mdps. Advances in Neural Information Processing Systems , 34:17183-17193, 2021.

Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David C Parkes. Calibrated fairness in bandits. arXiv preprint arXiv:1707.01875 , 2017.

Hussein Mouzannar, Mesrob I Ohannessian, and Nathan Srebro. From fair decision making to social equality. In Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 359-368. ACM, 2019.

Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen, Longbo Huang, and Tie-Yan Liu. Reinforcement learning with dynamic boltzmann softmax updates. arXiv preprint arXiv:1903.05926 , 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning , pages 7599-7609. PMLR, 2020.

Reilly Raab and Yang Liu. Unintended selection: Persistent qualification rate disparities and interventions. Advances in Neural Information Processing Systems , 34:26053-26065, 2021.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

Rahul Singh, Abhishek Gupta, and Ness B Shroff. Learning in markov decision processes under constraints. arXiv preprint arXiv:2002.12435 , 2020.

Wei Tang, Chien-Ju Ho, and Yang Liu. Bandit learning with delayed impact of actions. Advances in Neural Information Processing Systems , 34:26804-26817, 2021.

Honghao Wei, Xin Liu, and Lei Ying. Triple-q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In International Conference on Artificial Intelligence and Statistics , pages 3274-3307. PMLR, 2022.

Min Wen, Osbert Bastani, and Ufuk Topcu. Fairness with dynamics. arXiv preprint arXiv:1901.08568 , 2019.

Joshua Williams and J Zico Kolter. Dynamic modeling and equilibria in fair decision making. arXiv preprint arXiv:1911.06837 , 2019.

Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning , pages 11480-11491. PMLR, 2021.

Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning , pages 325-333. PMLR, 2013.

Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellström, Kun Zhang, and Cheng Zhang. How do fair decisions fare in long-term qualification? arXiv preprint arXiv:2010.11300 , 2020.

Liyuan Zheng and Lillian Ratliff. Constrained upper confidence reinforcement learning. In Learning for Dynamics and Control , pages 620-629. PMLR, 2020.

A Simulated Environments: Supplementary

A.1 Replicator Dynamics

Our use of replicator dynamics closely mirrors that of Raab and Liu (2021) as an 'equitable' model of a population: Individuals my be modeled identically, independently of group membership, yet persistent outcome disparities may emerge from disparate initial conditions between groups. For our experiments, we parameterize the evolving distribution Pr( X,Y | G ) , assuming constant group sizes, in terms of 'qualification rates' q g := Pr( Y =1 | G = g ) and update these qualification rates according to the discrete-time replicator dynamics:

In this model, the fitness W g y > 0 of label Y = y in group G = g may be interpreted as the 'average utility to the individual' in group g of possessing label y , and thus relative replication rate of label y in group g , as agents update their labels by mimicking the successful strategies of in-group peers. We model W g y in terms of acceptance and rejection rates with a group-independent utility matrix U :

We choose the matrix U to eliminate dominant strategies (i.e., when agents universally prefer one label over another, independent of classification), assert that agents always prefer acceptance over rejection, and to imply that the costs of qualification are greater than the costs of non-qualification among accepted individuals. While other parameterizations of U are valid, this choice of parameters guarantees internal equilibrium of the replicator dynamics for a Bayes-optimal classifier and 'wellbehaved' scalar-valued feature X , such that Pr( Y =1 | X = x ) is monotonically increasing in x (Raab and Liu, 2021).

A.2 Data Synthesis and Processing

In addition to a synthetic distribution, for which we assume X ∼ N ( Y, 1) , independent of G , for all time, we also consider real-world distributions when simulating and comparing algorithms for 'longterm fairness'. In both cases, as mentioned above, we wish to parameterize distributions in terms of qualification rates q g . As we perform binary classification on discrete groups and scalar-valued features, in addition to parameterizing a distribution in terms of q g , we desire a scalar-valued feature for each example, rather than the multi-dimensional features common to real-world data.

Our solution is to use an additional learning step for 'preprocessing': Given a static dataset D from which ( X ′ , Y, G ) is drawn i.i.d., (e.g., the 'Adult Data Set' Dua and Graff (2017)), where G varies over an individual's sex and Y corresponds to whether an individual has more than 50,000 USD in annual income, at each time-step, we train a stochastic binary classifier ˜ a , such that ˆ Y ′ ∼ ˜ a ( X ′ , G ) with a loss that re-weights examples by label value according to q g . That is, at each time step, we solve:

where

and L is zero-one loss. In our experiments, we choose ˜ a according to logistic regression. We interpret Pr( ˆ Y ′ =1) , drawn from the learned 'preprocessing' function ˜ a , as a new, scalar feature value X ∈ R mapped from from higher-dimensional features X ′ . The threshold policy ultimately operates on X .

Assumption 4.1 is as hard to satisfy in general as solving the Bayes-optimal binary classification task over higher-dimensional features. Nonetheless, we expect Assumption 4.1 to be approximately satisfied by such a 'preprocessing' pipeline. It is important to note that the behavior introduced by retraining a logistic classifier at each time step can yield very different qualitative behavior when compared to the static, purely synthetic distribution, as the distribution of X conditioned on Y and G also mutates, and Assumption 4.1 may be violated.

B Deferred Proofs

Without loss of generality, we assume ∥ ϕ ( s, a ) ∥ ≤ 1 for all ( s, a ) ∈ S × A , and max {∥ µ h ( S ) ∥ , ∥ θ h ∥} ≤ √ d for all h ∈ [ H ] .

Assumption B.1 ( V bound for ν ∗ ) . We define a parameter V that is assumed to bound the optimal value of ν in Eq. (4). For ¯ π and γ > 0 satisfying Slater's Condition (Assumption 2.1),

̸

Lemma B.3. The Voronoi partitioning described above satisfies I i ∩ I j = ∅ , ∀ i = j and ∪ i M =1 I i = A .

In Assumption B.1, V = H/γ , upper-bounds the optimal dual variable ν ∗ as an input to L-UCBFair . Definition B.2. Given a set of distinct actions I = { I 0 , · · · , I M } ⊂ A , where A is a closed set in Euclidean space, define I i = { a : ∥ a -I i ∥ 2 ≤ ∥ a -I j ∥ 2 , ∀ j = i } as the subset of actions closer to I i than to I j , i.e., the V oronoi region corresponding to locus I i , with tie-breaking imposed by the order of indices i . Also define the locus function I ( a ) = min i argmin I i ∥ a -I i ∥ 2 .

̸

̸

Considering that a ′ ∈ I i , according to the given Definition B.2, we can infer that | a ′ -I i | 2 < | a ′ -I j | 2 (since i > j ). However, this assertion contradicts the fact that a ′ ∈ I j , which implies | a ′ -I j | 2 ≤ | a ′ -I i | 2 . Therefore, I i ∩ I j = ∅ for all i = j .

̸

Proof. We will begin by proving I i ∩ I j = ∅ for all i = j . To establish this, we will assume the contrary, that there exist indices i and j such that I i ∩ I j = ∅ . Without loss of generality, assume i > j . We will denote an arbitrary action within the interaction of I i and I j as a ′ ∈ A .

̸

Wethen proof ∪ i M =1 I i = A . Since A is a closed set, for any a ∈ A , there must be a i ∈ { 1 , 2 , · · · , M } , such that d a,i = ∥ a -I i ∥ 2 ≤ ∥ a -I j ∥ 2 = d a,j , ∀ j . If d a,i < d a,j strictly holds for all j , then a ∈ I i . Otherwise define a set J = { j | d a,j = d a,i } , then a ∈ I j ′ where j ′ = argmin j ∈J j .

Theorem B.4. If the number M of distinct loci or regions partitioning A is sufficiently large, there exists a finite set of loci I such that ∀ a ∈ I i , i ∈ M, ∥ a -I i ∥ 2 ≤ ϵ I .

Proof. Since A is closed in euclidean space, denote N the dimension, d = sup a,a ′ ∈A ∥ a -a ′ ∥ 2 . Define an Orthonormal Basis { e 1 , e 2 , · · · , e N } . Randomly choose a point a ∈ A , Set it as I 0 , then we form a set of loci I = { I 0 + ∑ N i =1 ϵk i e i | I 0 + ∑ N i =1 ϵk i e i ∈ A , k i ∈ Z , -⌈ d ϵ ⌉ ≤ k i ≤ ⌈ d ϵ ⌉} . We know that | I | ≤ ( 2 ⌈ d ϵ ⌉ ) N . It's not hard to verify that ∥ a -I i ∥ 2 ≤ ϵ 2 √ 2 N -1 , ∀ a ∈ I i . Taken ϵ I = ϵ 2 √ 2 N -1 yields the statement.

B.1 Proof of Theorem 3.3

Theorem 3.3 (Boundedness) . With probability 1 -p , there exists a constant b such that L-UCBFair (Algorithm 1) achieves for parameter values ς =1 ϵ I ≤ 1 2 ρ (1+ V ) KH √ d , ζ = log(log( M )4 dHK/p ) , and β = O ( dH √ ζ ) .

Outline The outline of this proof simulates the proof in Ghosh et al. (2022). For brevity, denote P h V π j,h +1 ( s, a ) = E s ′ ∼ P h ( ·| s,a ) V π j,h +1 ( s ′ ) for j = r, g . Then

Similar to Efroni et al. (2020), we establish

Bound T 1 and T 2 . We have following two lemmas.

(12)

T 3 is easily bounded if η . The major task remains bound T 1 and T 2 .

Lemma B.5 (Boundedness of T 1 ) . With probability 1 -p/ 2 , we have T 1 ≤ KH ( log( M ) α +2(1 + V ) Hρϵ I √ dK ς ) . Specifically, if α = log( M ) K 2(1+ V + H ) and ς = 1 , we have T 1 ≤ 2 H (1 + V + H ) + 2 KH 2 (1 + V ) ρϵ I √ dK with probability 1 -p/ 2 .

Lemma B.6 (Boundedness of T 2 ) . Ghosh et al. (2022) With probability 1 -p/ 2 , T 2 ≤ O ( ( ν +1) H 2 ζ √ d 3 K ) , where ζ =log[log( M )4 dHK/p ] .

Lemma B.6 follows the same logic in Ghosh et al. (2022), and we delay the proof of Lemma B.5 to Section B.2. Now we are ready to proof Theorem 3.3.

Following the idea of Efroni et al. (2020), there exists a policy π ′ such that V π ′ r, 1 = 1 K ∑ K k =1 V π k r, 1 , V π ′ g, 1 = 1 K ∑ K k =1 V π k g, 1 . By the occupancy measure, V π r, 1 and V π g, 1 are linear in occupancy measure induced by π . Thus, the average of K occupancy measure also produces an occupancy measure which induces policy π ′ and V π ′ r, 1 , and V π ′ g, 1 . Wetake ν = 0 when ∑ K k =1 ( b -V π k g, 1 ( s k 1 )) < 0 ,

otherwise ν = V . Hence, we have

Since V = 2 H/γ , and using the result of Lemma B.15, we have

In this section we proof Lemma B.5 and Lemma B.6.

B.2 Prepare for Lemma B.5

In order to bound T 1 and T 2 , we introduce the following lemma.

Lemma B.7. There exists a constant B 2 such that for any fixed p ∈ (0 , 1) , with probability at least 1 -p/ 2 , the following event holds

for j ∈ { r, g } , where q = √ log [4 ( B 1 +1)log( M ) dT/p ] for some constant B 1 .

We delay the proof of Lemma B.7 to Appendix B.4.

Lemma B.7 shows the bound of estimated value function V k j,h and value function V π j,h corresponding in a given policy at k. We now introcuce the following lemma appeared in Ghosh et al. (2022). This lemma bounds the difference between the value function without bonus in L-UCBFair and the true value function of any policy π . This is bounded using their expected difference at next step, plus a error term.

Lemma B.8. Ghosh et al. (2022) There exists an absolute constant β = C 1 dH √ ζ, ζ = log(log( M )4 dT/p ) , and for any fixed policy π , for the event defined in Lemma B.7, we have

Lemma B.9. Ghosh et al. (2022) With probability at least 1 -p/ 2 , (for the event defined in Lemma B.7)

We also introduce the following lemma. This lemma bound the value function by taking L-UCBFair policy and greedy policy.

Lemma B.10. Define ¯ V k h ( · ) = max a [ Q k r,h ( · , a ) + ν k Q k g,h ( · , a ) ] the value function corresponding to greedy policy, we have

Proof. Define a g the solution of greedy policy,

The first inequality follows from Lemma B.14 and the second inequality holds because of Proposition 1 in Pan et al. (2019).

B.3 Proof of Lemma B.5

Now we're ready to proof Lemma B.5.

Proof. This proof simulates Lemma 3 in Ghosh et al. (2022).

We use induction to proof this lemma. At step H , we have Q k j,H +1 = 0 = Q π j,H +1 by definition. Under the event in Lemma B.13 and using Lemma B.8, we have for j = r, g ,

From the definition of ¯ V k h ,

for any policy π . Thus, it also holds for π ∗ , the optimal policy. Using Lemma B.10 we can get

Now, suppose that it is true till the step h +1 and consider the step h . Since, it is true till step h +1 , thus, for any policy π ,

From (27) in Lemma 10 and the above result, we have for any ( s, a )

Hence,

Now, again from Lemma 11, we have ¯ V k h ( s ) -V k h ( s ) ≤ log( |A| ) α . Thus,

Now, since it is true for any policy π , it will be true for π ∗ . From the definition of V π,ν k , we have

Hence, the result follows by summing over K and considering h = 1 .

B.4 Proof of Lemma B.7

Wefirst define some useful sets. Let Q j = { Q | Q ( · , a ) = min { w T j ϕ ( · , a ) + β √ ϕ T ( · , a ) T Λ -1 ϕ ( · , a ) , H } , a ∈ A } be the set of Q functions, where j ∈ { r, g } . Since the minimum eigen value of Λ is no smaller than one so the Frobenius norm of Λ -1 is bounded.

Let V j = { V j | V j ( · ) = ∫ a π ( a | · ) Q j ( · , a ) da ; Q r ∈ Q r , Q g ∈ Q g , ν ∈ [0 , V ] } be the set of Q functions, where j ∈ { r, g } . Define

the set of policies.

It's easy to verify V k j ∈ V j .

Then we introduce the proof of Lemma B.7. To proof Lemma B.7, we need the ϵ -covering number for the set of value functions(Lemma B.13Ghosh et al. (2022)). To achieve this, we need to show if two Q functions and the dual variable ν are close, then the bound of policy and value function can be derived(Lemma B.11, Lemma B.12). Though the proof of Lemma B.11 and Lemma B.12 are different from Ghosh et al. (2022), we show the results remain the same, thus Lemma B.13 still holds. We'll only introduce Lemma B.13 and omit the proof.

Lemma B.11. Let π be the policy of L-UCBFair corresponding to Q k r + ν k Q k g , i.e.,

and

if ∣ ∣ ∣ Q j -˜ Q j ∣ ∣ ∣ ≤ ϵ ′ and | ν -˜ ν | ≤ ϵ ′ , then ∣ ∣ ∫ a ( π ( a | x ) -˜ π ( a | x )) da ∣ ∣ ≤ 2 αϵ ′ (1 + V + H ) .

Proof.

The last inequaity holds because of Theorem 4.4 in Epasto et al. (2020). Using Corollary B.17, we have

Now since we have Lemma B.11, we can further bound the value functions.

Proof. For any x ,

Using Lemmas above, we can have the same result presented in Lemma 13 of Ghosh et al. (2022) as following.

Lemma B.13. Ghosh et al. (2022) There exists a ˜ V j ∈ V j parameterized by ( ˜ w r , ˜ w g , ˜ β, Λ , ˜ V ) such that dist ( V j , ˜ V j ) ≤ ϵ where

Let N V j ϵ be the ϵ -covering number for the set V j , then,

where ϵ ′ = ϵ H 2 α (1+ V + H )+1

Proof.

(31)

B.5 Prior Results

Lemma B.15. Ding et al. (2021) Let ν ∗ be the optimal dual variable, and C ≥ 2 ν ∗ , then, if we have

Lemma B.16. Jin et al. (2020) For any ( k, h ) , the weight w k j,h satisfies

Corollary B.17. If dist ( Q r , ˜ Q r ) ≤ ϵ ′ , dist ( Q g , ˜ Q g ) ≤ ϵ ′ , and | ˜ ν k -ν k | ≤ ϵ ′ , then, dist ( Q k r + ν k Q k g , ˜ Q r + ˜ ν k ˜ Q g ) ≤ ϵ ′ (1 + V + H ) .

C Additional Figures

//

0

1

2

3

4

5

0

0

.

5

1

s

τ

.

.

.

.

.

.

I

1

I

2

I

3

I

n

O

1

O

2

a

τ

∼

π

(

s

τ

)

s

τ

+1

∼

P

(

s

τ

, a

τ

)

0

1

2

3

4

5

0

0

.

2

0

.

4

0

.

6

s

τ

+1

0.0

0.2

0.4

0.6

0.8

1.0

X

0.020

0.025

0.030

0.035

0.040

Pr(X=x|G=g) (Density)

g=2

0.0

0.2

0.4

0.6

0.8

1.0

X

0.2

0.4

0.6

0.8

1.0

Pr(Y=1|X=x,G=g)

g=2

D Experiments

D.1 Experiment Details

Device and Packages. We run all the experiment on a single 1080Ti GPU. We implement the R-TD3 agent using StableBaseline3 (Raffin et al., 2021). The neural network is implemented using Pytorch (Paszke et al., 2019). Figures were generated using code included in the supplementary material in less than 24 hours on a single Nvidia RTX A5000.

Using a Neural Network to Learn ϕ . We use a multi-layer perceptron to learn ϕ . Specifically, we sample 100000 data points using a random policy, storing s , a , r and g . The inputs of the network are state and action, passing through fully connected (fc) layers with size 256, 128, 64, 64. ReLU is used as activation function between fc layers, while a SoftMax layer is applied after the last fc layer. We treat the outcome of this network as ϕ . To learn ϕ , we apply two separated fc layers (without bias) with size 1 to ˆ ϕ and treat the outputs as predicted r and predicted g . A combination of MSE losses of r and g are adopted. We use Adam as the optimizer. Weight decay is set to 1e-4 and learning rate is set to 1e-3, while batch size is 128.

Note that, ˆ ϕ is linear regarding r and g , but the linearity of transition kernel cannot be captured using such a schema. Therefore, equivalently we made an assumption that there always exists measure µ h such that for given ˆ ϕ , the linearity of transition kernel holds. It's a stronger assumption than Assumption 3.1.

Demographic Parity

Equal Opportunity

Equalized Odds

a) Myopic-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.000000

0.001966

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.000

4.535

1e-5

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.000

5.535

1e-5

b) Maxmin-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.000000

0.000578

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.000000

0.000104

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.000

6.404

1e-5

c)

L-UCBFair

trained

for

2,000

steps.

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Demographic Partity)

Demographic Partity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Equal Opportunity)

Equal Opportunity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Equalized Odds

Equal qualification rates

0

1

d)

R-TD3

trained

for

200,000

steps

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Demographic Partity)

Demographic Partity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Equal Opportunity)

Equal Opportunity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Equalized Odds

Equal qualification rates

0

1

e)

Episodic

Mean

Loss (Over 100 steps)

Maxmin-Fair

Myopic-Fair

UCB-Fair

R-TD3

0.25

0.50

0.75

1.00

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

UCB-Fair

R-TD3

0.25

0.50

0.75

1.00

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

UCB-Fair

R-TD3

0.25

0.50

0.75

1.00

Episodic Mean Loss

Demographic Parity

Equal Opportunity

Equalized Odds

a) Myopic-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.0000

0.0728

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.00000

0.00549

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.0000

0.0138

b) Maxmin-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.000000

0.005373

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.000000

0.005188

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.00000

0.01247

c)

L-UCBFair

trained

for

2,000

steps.

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Demographic Parity)

Demographic Parity

0.0000

0.1677

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Equal Opportunity)

Equal Opportunity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Equalized Odds

Equal qualification rates

0

1

d)

R-TD3

trained

for

200,000

steps

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity (Demographic Parity)

Demographic Parity

0.0000

0.8123

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity (Equal Opportunity)

Equal Opportunity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity (Equalized Odds)

Equalized Odds

Equal qualification rates

0

1

e)

Episodic

Mean

Loss (Over 150 steps)

Myopic-Fair

Maxmin-Fair

UCB-Fair

R-TD3

0.5

1.0

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

UCB-Fair

R-TD3

0.25

0.50

0.75

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

UCB-Fair

R-TD3

0.2

0.4

0.6

0.8

Episodic Mean Loss

Demographic Parity

Equal Opportunity

Equalized Odds

a) Myopic-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.00000

0.04097

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.0000

0.0566

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.00000

0.01892

b) Maxmin-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.00000

0.05166

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.00000

0.02377

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.0000

0.0187

c)

R-TD3

trained

for

200,000

steps

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Demographic Partity)

Demographic Partity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Equal Opportunity)

Equal Opportunity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Equalized Odds

Equal qualification rates

0

1

e)

Episodic

Mean

Loss (Over 100 steps)

Myopic-Fair

Maxmin-Fair

R-TD3

0.2

0.3

0.4

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

R-TD3

0.2

0.4

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

R-TD3

0.2

0.4

0.6

Episodic Mean Loss

Demographic Parity

Equal Opportunity

Equalized Odds

a) Myopic-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.00000

0.06457

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.00000

0.03744

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.0000

0.0221

b) Maxmin-Fair

(

λ

=

1

2

)

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Demographic Disparity

0.000

0.149

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Unequal Opportunity

0.000

0.187

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Odds Disparity

0.0000

0.2072

c)

R-TD3

trained

for

200,000

steps

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Disparity(Demographic Partity)

Demographic Partity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Disparity(Equal Opportunity)

Equal Opportunity

0

1

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

s

1

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

s

2

Equalized Odds

Equal qualification rates

0

1

e)

Episodic

Mean

Loss (Over 150 steps)

Myopic-Fair

Maxmin-Fair

R-TD3

0.2

0.4

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

R-TD3

0.1

0.2

0.3

0.4

Episodic Mean Loss

Myopic-Fair

Maxmin-Fair

R-TD3

0.1

0.2

0.3

0.4

Episodic Mean Loss

D.6 Training Curves: L-UCBFair

D.6.1

0

500

1000

1500

2000

Iterations

0.50

0.52

0.54

0.56

0.58

0.60

Episodic Mean Loss

0

500

1000

1500

2000

Iterations

0.50

0.52

0.54

0.56

0.58

0.60

Episodic Mean Loss

0

500

1000

1500

2000

Iterations

0.46

0.48

0.50

0.52

0.54

0.56

Episodic Mean Loss

0

500

1000

1500

2000

Iterations

0.12

0.14

0.16

0.18

0.20

Episodic Mean Disparity

(a) Demographic Parity

0

500

1000

1500

2000

Iterations

0.12

0.14

0.16

0.18

0.20

Episodic Mean Disparity

(b) Equal Opportunity

0

500

1000

1500

2000

Iterations

0.08

0.09

0.10

0.11

0.12

0.13

0.14

Episodic Mean Disparity

(c) Equalized Odds

0

500

1000

1500

2000

Iterations

0.30

0.31

0.32

0.33

0.34

0.35

0.36

0.37

Episodic Mean Loss

0

500

1000

1500

2000

Iterations

0.29

0.30

0.31

0.32

0.33

0.34

0.35

Episodic Mean Loss

0

500

1000

1500

2000

Iterations

0.29

0.30

0.31

0.32

0.33

0.34

0.35

Episodic Mean Loss

0

500

1000

1500

2000

Iterations

0.015

0.020

0.025

0.030

0.035

0.040

Episodic Mean Disparity

(a) Demographic Parity

0

500

1000

1500

2000

Iterations

0.015

0.020

0.025

0.030

0.035

Episodic Mean Disparity

(b) Equal Opportunity

0

500

1000

1500

2000

Iterations

0.0200

0.0225

0.0250

0.0275

0.0300

0.0325

0.0350

0.0375

Episodic Mean Disparity

(c) Equalized Odds

D.7 Training Curves: R-TD3

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.30

0.35

0.40

0.45

0.50

Episodic Mean Loss

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.325

0.350

0.375

0.400

0.425

0.450

0.475

0.500

Episodic Mean Loss

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.250

0.275

0.300

0.325

0.350

0.375

0.400

0.425

Episodic Mean Loss

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

Episodic Mean Disparity

(a) Demographic Parity

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

1

2

3

4

5

Accumulated Disparity

(b) Equal Opportunity

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.02

0.04

0.06

0.08

0.10

Episodic Mean Disparity

(c) Equalized Odds

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.24

0.26

0.28

0.30

0.32

0.34

Episodic Mean Loss

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.24

0.26

0.28

0.30

0.32

0.34

0.36

Episodic Mean Loss

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.18

0.20

0.22

0.24

0.26

0.28

0.30

0.32

Episodic Mean Loss

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

Episodic Mean Disparity

(a) Demographic Parity

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Episodic Mean Disparity

(b) Equal Opportunity

0

25k

50k

75k

100k 125k 150k 175k 200k

Iterations

0.02

0.04

0.06

0.08

0.10

0.12

Episodic Mean Disparity

(c) Equalized Odds

D.8 Reduction of Utility

0.1

0.3

0.5

0.7

0.9

Group 1 qualification rate

0.1

0.3

0.5

0.7

0.9

Group 2 qualification rate

Reduction of Utility

0.0

0.1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the issue of enduring fairness within a dynamically responsive population, framing it as a reinforcement learning (RL) problem with certain constraints. In this scenario, the state distribution is subject to change dynamically based on the actions of the deployed agent. To address this problem, the authors introduce L-UCBFair and R-TD3. L-UCBFair assumes a simplified linear Markov Decision Process, while R-TD3 incorporates existing deep reinforcement learning methods for a more general setting. According to the numerical experiments conducted, the proposed method consistently surpasses the performance of baseline methods. The RL approach enables an agent to learn how to forego short-term utility to steer the system towards more desirable equilibrium states.\n\nSTRENGTHS:\n**Originality**:\nThe authors uniquely approach the fairness problem in machine learning within a context where the underlying state distribution dynamically evolves based on the agent's decision. This novel setup extends the static fairness issue found in the existing literature. The dynamic viewpoint introduces the concept of cumulative reward and distortion (cumulative fairness disparity), aptly fitting the method of reinforcement learning. The experiments carried out distinctly highlight the benefits of incorporating a reinforcement learning method, particularly in cases of changing underlying environments.\n\n**Quality and Clarity**:\nThe motivation behind this work is cogently presented in the introduction, while the related works section explores a wide range of existing literature, discussing both fairness issues in non-stationary settings and safe reinforcement learning. The demonstration of numerical experiments is lucidly laid out.\n\n**Significance**:\nThe suggested L-UCBFair and R-TD3 methods surpass the performance of baseline methods. The concept of foregoing short-term utility to guide the system towards more optimal equilibrium states is effectively demonstrated. Furthermore, the author presents a thorough theoretical analysis for the proposed L-UCBFair under the linear Markov Decision Process assumption.\n\nWEAKNESSES:\n**Weakness 1**:\nThe depiction of Algorithm 1 could be clearer. There are several terms defined in the paper that aren't elaborated upon, making comprehension more challenging. Enhancing the clarity in Section 3.1.1 could greatly improve overall readability.\n\n**Weakness 2**:\nThe authors introduce R-TD3 as a broader reinforcement learning method, not reliant on the linear MDP assumption. Could the authors shed more light on the advantages or enhancements that R-TD3 offers in comparison to the foundational method, L-UCBFair?\n\n**Weakness 3**:\nIn the numerical experiments, the authors have designed synthetic simulation environments based on straightforward datasets. However, in the existing literature, public simulators dedicated to long-term simulation, such as \"Fairness is not static: Deeper understanding of long term fairness via simulation studies\", are readily available. Given that this has been mentioned in the related works, could the authors assess their proposed L-UCBFair and R-TD3 in these more complex simulation environments? For instance, situations involving lending, attention allocation, or college admissions could provide further insights.\n\n**Weakness 4**:\nReinforcement learning utilizes long-term planning to maximize rewards and diminish distortion, whereas myopic-fair methods – the basis of all baseline methods used in the paper – only address a one-step optimization issue. Given this, the result presented in Figure 1 seems rather self-evident as one would expect a long-term planning-based RL method to surpass more short-sighted methods. Could the authors compare their proposed method with more robust baseline methods that also consider underlying dynamics?\n\nQUESTIONS:\nThe issue of long-term fairness can become quite resource-intensive when the agent interacts with the real-world scenarios. Within the existing body of reinforcement learning literature, model-based reinforcement learning methods have been devised to enhance sample efficiency. Could the authors discuss the potential of developing a model-based reinforcement learning approach to tackle the long-term fairness problem addressed in this paper?\n\nMoreover, several queries were raised in the previously mentioned weaknesses section.\n\nLIMITATIONS:\nThe authors followed the PaperChecklist by providing a discussion on the current limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a binary classification task with long-term fairness. The authors formulate the problem as a constrained MDP where the classifier is an action, the distribution is the state, the distribution will shift reacting to an action. The problem aims to optimize the expected long-term reward under the constraints for long-term fairness in expectation. They develop L-UCBFair which is a model-free algorithm to guarantee long-term fairness with continuous state and action spaces.  Under the linear MDP assumption, they prove sublunar regret and disparity with high probability.  Finally, experiment results are given to demonstrate the performance of the proposed algorithm.\n\nSTRENGTHS:\nI believe the paper has the following strong points.\n+ The paper formulates fairness constrained classification problem as a constrained MDP, which is new to me.\n+ The paper applies linear MDP algorithms and TD3 to the considered problem and prove the regret bound with linear MDP assumption.\n\nWEAKNESSES:\nThe paper can still be improved in the following aspects.\n- The paper seems to directly apply LSVI-UCB and TD3 algorithms to solve the considered constrained MDP problem. It would be much better if the authors can discuss how the algorithm design and proof techniques are different from original works in (Jin et al. (2020)) and  Fujimoto et al. (2018).\n- The paper formulates the fairness-constrained classification problem as a MDP problem, but does not give enough motivation. \n- It usually requires a lot of exploration for the constrained RL to achieve low disparity, so I am afraid that the fairness violation can be very high at early stages.\n- The paper is limit in addressing the complexity problem by using constrained RL for this problem.  The action space (classifier) can be very large, which can cause a very large complexity.\n\nQUESTIONS:\n1. The paper considers a dynamic where the new distribution is affected by the previous distribution and classifier. Can the authors give some concrete motivation examples for such distribution shift?\n2. Can the authors discuss the challenges of solving the MDP with fairness constraints?  What are the differences comparing with standard constrained MDP problems?\n3. How does the complexity of proposed algorithm rely on the space of state and action?\n\nLIMITATIONS:\nThe limitations are listed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method to ensure long-term fairness in reinforcement learning. The method is compatible with different utility measures (e.g., the accuracy of the classifier when the goal is to predict a label) and different fairness constraints (e.g., demographic parity or equal opportunity). The method is an online reinforcement learning method, so the agent has to interact with its environment in order to improve the quality of the policy, either in the real world or in simulation. The authors introduce two measures of regret, one related to the primary utility goal and one related to the fairness measure, and present an upper bound for both regrets achieved by the proposed algorithm. The paper presents empirical results using a mixture of synthetic and real data.\n\nSTRENGTHS:\n- This paper tackles the very important problem of ensuring fairness in a sequential decision-making setting. It uses insights from the - reinforcement learning literature to solve a supervised learning fairness problem.\n- The proposed method extends existing ones to the continuous setting (i.e., continuous states—e.g., attributes—and continuous actions).\n- The paper is very thorough with the mathematical formulation, assumptions, and definitions necessary to fully describe the method.\n\nWEAKNESSES:\n- The proposed method is an online RL method, where the learning process occurs while the policy is applied. This is often not realistic in scenarios where we would like to ensure fairness, as intermediate policies could lead to unfair behavior.\n- My understanding is that the proposed method is an extension of the method presented by Ghosh et al. (2022) for the continuous setting, where the method itself is fairly similar, and the main new contribution is showing that it is possible to achieve the same bound in this new setting. I believe that this could be better clarified in the paper to highlight the novelty of this work more clearly.\n- In the introduction, the authors state that the primary contribution of this work is to consider the problem of long-term fairness as a reinforcement learning problem subject to a constraint. However, previous works have already considered this setting (e.g., the method on which the present work is based).\n- I believe the experiments with R-TD3 are not strongly supporting the argument that an algorithm without theoretical guarantees can still be used for long-term fairness. Applying the method in just two scenarios does not present enough empirical evidence to support the claim that the final policy is still fair, and without any theoretical guarantees, this is essential.\n\nQUESTIONS:\n- Could the authors please provide some motivating examples for when an online RL method would be useful/applicable, considering the fact that the policy is updated while being deployed and the regret bounds are only for the final distortion and not the disparity at each timestep (possibly causing intermediate policies to be unfair)?\n- The color scheme in Figure 1 is slightly unclear. The range for the demographic disparity in the first two plots is between zero and 0.005 or zero and 0.07 (a very small interval), but for the proposed method results it is between zero and 1. This makes it harder to compare the fairness measures. Could the authors please clarify this?\n- Why did the authors select only the Myopic-fair baseline for the second experiment? From the first experiment, it seemed like this wasn’t one of the most competitive baselines, so it would be interesting to see how the other baselines perform in this second scenario as well.\n\nLIMITATIONS:\nThe authors discuss the limitations of the work. No major concerns.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates that reinforcement learning algorithms can be used to satisfy long term fairness constraints in dynamic environments in which a classifier and population interact while finding high-value equilibria. \n\nThe main approach is to treat the problem as a constrained optimization and optimize the Lagrangian function. The authors use a linear UCB-style algorithm to obtain provable regret bounds on both the cumulative value and constraint violations and demonstrate effectiveness of that algorithm and a less-principled-but-still-practical RL alternative using small simulation studies.\n\nAdapting previous work to the interacting classifier population setting requires handling a continuous action space (setting per-group classifier thresholds) - which the authors handle by showing the continuous action space in this setting can be searched in a discretized manner without losing guarantees.\n\nThe simulation experiments demonstrate that the RL approaches can find good operating points where more myopic policies can end up exacerbating qualification rate disparities.\n\nSTRENGTHS:\nSignificance & Originality: The authors address an important question: How to design decision-making algorithms for dynamic settings where the underlying dynamics are not well known and the designers want to be both fair and encourage positive societal outcomes overall.\n\nAdding fairness considerations to an RL algorithm with a suitably specified reward function is not a groundbreaking idea, but this work offers a concrete algorithm and practical model-free approach for this question that (as far as I can tell) thus far have not been proposed. \n\nClarity: Paper is very clear and easy to read up to the experiments, setting and notation are clear, as well as the paper’s goals and contributions.\n\nWEAKNESSES:\nI found the experiments to be a little bit unclear  in terms of what was being demonstrated with each figure / experiment. What is the role of the fully synthetic vs semi-synthetic experiments? I think the paper could be improved with a clearer exposition of the takeaways from each experiment.\n\nThe authors note that real-world experiments would be necessary to really assess impact of this algorithm. I strongly agree with this sentiment and while I understand that truly real-world experiments can be hard to find, the authors could potentially test across a range of dynamic synthetic environments to give a better sense of the generalizability of the proposed approach.\n\nQUESTIONS:\n1. The shading in figures 1a-d suggests that the demographic parity violation can be mapped to the same state space as qualification parity? Is that really true? For any pair x,y on those plots is there a unique value for the demographic parity violation?\n\n2. How would a purely RL-based long-term reward maximizing agent perform as a baseline (with no fairness constraints)? I would think it might also drive the qualification rates up to the top-right of figures 1a-d, in which case can we be sure it’s the fairness constraints that are actually doing the work here and not just the long-term reward optimization in general?\n\nLIMITATIONS:\nAuthors addressed the limitations of their experimental setup with synthetic environments. They might also consider discussing what types of societal inequalities can and cannot be addressed by fairness-in-decision-making style constraints.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}