SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tongxin Yin*
tyin@umich.edu
Reilly Raab*
Santa Cruz, CA 95064
reilly@ucsc.edu
Mingyan Liu
mingyan@umich.edu
Yang Liu
yangliu@ucsc.edu
While machine learning can myopically reinforce social inequalities, it may also be used to dynamically seek equitable outcomes. In this paper, we formalize long-term fairness as an online reinforcement learning problem for a policy affecting human populations. This formulation accommodates dynamical control objectives, such as achieving equitable population states, that cannot be incorporated into static formulations of fairness. We demonstrate that algorithmic solutions to the proposed fairness problem can adapt to unknown dynamics and, by sacrificing short-term incentives, drive the policy-population system towards more desirable equilibria. For the proposed setting, we develop an algorithm that adapts recent work in online learning and prove that this algorithm achieves simultaneous probabilistic bounds on cumulative loss and cumulative violations of fairness. In the classification setting subject to group fairness, we compare our proposed algorithm to several baselines, including the repeated retraining of myopic or distributionally robust classifiers, and to a deep reinforcement learning algorithm that lacks fairness guarantees. Our experiments model human populations according to evolutionary game theory and integrate real-world datasets.
As machine learning (ML) algorithms are deployed for tasks with real-world social consequences (e.g., school admissions, loan approval, medical interventions, etc.), the possibility exists for runaway social inequalities (Crawford and Calo, 2016; Chaney et al., 2018; Fuster et al., 2018; Ensign et al., 2018). While "fairness" has become a salient ethical concern in contemporary research, the closed-loop dynamics of real-world systems with feedback, i.e., comprising ML-driven decisions and populations that mutually adapt to each other (Fig. 5 in the supplementary material) remain poorly understood. We assert that realistic scenarios are often described by fundamentally unknown dynamics: Even with models of human behavior based on rational behavior or evolutionary game theory, utilities and risk-preferences are generally unknown or uncertain. For this reason, we advocate for dynamics-agnostic formulations of fairness, for which reinforcement learning is a natural fit.
In this paper, our primary contribution is to consider the problem of long-term fairness, or algorithmic fairness in the context of a dynamically responsive population, as a reinforcement learning (RL)
problem subject to constraint. In our formulation, the central learning task is to develop a policy that minimizes cumulative loss (e.g., financial risk, negative educational outcomes, misdiagnoses, etc.) incurred by an ML agent interacting with a human population up to a finite time horizon, subject to constraints on cumulative “violations of fairness”, which we refer to in a single time step as disparity and cumulatively as distortion.
Our central hypothesis is that an RL formulation of long-term fairness may allow an agent to learn to sacrifice short-term utility in order to drive the system towards more desirable equilibria. The core practical difficulties posed by our general problem formulation, however, are the potentially unknown dynamics of the system under control, which must be determined by the RL agent online (i.e., during actual deployment), and the general non-convexity of the losses or constraints considered. Additionally, our problem setting generally requires continuous state and action spaces for the RL agent, which preclude familiar methods that yield performance guarantees in discrete settings.
Our secondary contributions are 1) to show that long-term fairness can be solved within asymptotic, probabilistic bounds under certain dynamical assumptions and 2) to demonstrate that the problem can also be addressed more flexibly with existing RL methods. For theoretical guarantees, we develop L-UCBFair, an online RL method, and prove sublinear bounds on regret (suboptimality of cumulative loss) and distortion (suboptimality of cumulative disparity) with high probability (Section 3.1). To demonstrate practical solutions without strong safety guarantees, we apply R-TD3, an adaptation of a well-known deep reinforcement learning method (viz., TD3) to a time-dependent Lagrangian relaxation of the central problem. We compare L-UCBFair and R-TD3 to several baselines (Section 3.3), including myopic policies, in interaction with simulated populations initialized with synthetic or real-world data and updated according to evolutionary game theory (Section 4).
Finally, this paper considers fairness in terms of statistical regularities across (ideally) socioculturally meaningful groups. While internal conflict exists between different statistical measures of fairness (Corbett-Davies and Goel, 2018), we show that an RL approach to long-term fairness can mitigate trade-offs between immediately-fair policy decision outcomes (e.g., acceptance rate disparities (Dwork et al., 2012; Zemel et al., 2013; Feldman et al., 2015)) and causally-fair population states (e.g., qualification rate (Raab and Liu, 2021; Zhang et al., 2020)). In particular, we show that our proposed solutions can learn to avoid the familiar trap of pursuing short-term fairness metrics only to widen underlying disparities that demand escalating interventions at the expense of utility (Section 5.2).
Our formulation of long-term fairness bridges recent work on "fairness in machine learning", which has developed in response to the proliferation of data-driven methods in society, and "safe reinforcement learning", which seeks theoretical safety guarantees in the control of uncertain dynamical systems.
Dynamics of Fairness in Machine Learning We distinguish long-term fairness from the dynamics of fair allocation problems (Joseph et al., 2016; Jabbari et al., 2017; Tang et al., 2021; Liu et al., 2017) and emphasize side-effects of algorithmic decisions affecting future decision problems. By formalizing long-term fairness in terms of cumulative losses and disparities, we iterate on a developing research trend that accounts for the dynamical response of a human population to deployed algorithmic prediction: both as a singular reaction (Liu et al., 2018; Hu et al., 2019; Hu and Zhang, 2022) or as a sequence of mutual updates between the population and the algorithm (Coate and Loury, 1993; D'Amour et al., 2020; Zhang et al., 2020; Heidari et al., 2019; Wen et al., 2019; Liu et al., 2020; Hu and Chen, 2018; Mouzannar et al., 2019; Williams and Kolter, 2019; Raab and Liu, 2021).
Several prior works have considered the long-term fairness implications in the context of performative stability or optimality (Perdomo et al., 2020) with known, stateless dynamical transitions: Hu and Zhang (2022) characterize the convergence of systems typified by sequential, myopic policies while Hashimoto et al. (2018) contrast myopic policy with a modified objective satisfying distributional robustness. While Mouzannar et al. (2019) and Raab and Liu (2021) address stateful dynamical transitions, the cited works only treat myopic classifiers that optimize immediate utility (subject to fairness constraints) rather than learning to anticipate dynamical population responses. Finally, while Wen et al. (2019) explore reinforcement learning for long-term fairness, the discussion is limited to a tabular explore-then-commit approach over discrete state and action spaces. This approach does not
generalize to continuous spaces, where we provide separate and tighter bounds for both utility and disparity.
Safe Reinforcement Learning L-UCBFair furthers recent efforts in safe RL. While model-based approaches, in which the algorithm learns an explicit dynamical model of the environment, constitute one thread of prior work (Efroni et al., 2020; Singh et al., 2020; Brantley et al., 2020; Zheng and Ratliff, 2020; Kalagarla et al., 2021; Liu et al., 2021; Ding et al., 2021), leading algorithms in this domain are characterized by significant time and space complexity. Among model-free algorithms, the unknown dynamics of our setting preclude the use of a simulator that allows algorithms to search over arbitrary state-action pairs (Xu et al., 2021; Ding et al., 2020; Bai et al., 2022).
While Wei et al. (2022) introduce a model-free, simulator-free algorithm, the tabular approach they consider is only applicable to discrete state and action spaces, while our setting requires continuous state and action spaces. To tackle continuous state space, Ding et al. (2021) and Ghosh et al. (2022) consider linear dynamics: Ding et al. (2021) develop a primal-dual algorithm with safe exploration, and Ghosh et al. (2022) use a softmax policy design. Both algorithms are based on the work of Jin et al. (2020), which proposed a least squares value iteration (LSVI) method, using an Upper Confidence Bound (UCB) (Auer et al., 2002) to estimate a state-action “ $Q$ ” function. In addition to continuous state spaces, L-UCBFair also addresses continuous action spaces. To our knowledge, L-UCBFair is the first model-free, simulator-free RL algorithm to provide theoretical safety guarantees for both discrete and continuous state and action spaces. Moreover, L-UCBFair achieves bounds on regret and distortion as tight as any algorithm thus far on discrete action space (Ghosh et al., 2022).
We motivate our formulation of long-term fairness with a binary classification task, though the general formulation we propose is more widely applicable. Given this initial task, we sequentially incorporate fairness constraints, then population dynamics, and then cumulative loss and disparity, before fully formalizing the central problem of optimizing cumulative loss subject to expected dynamics and constraints on cumulative disparity.
For our initial binary classification task, suppose that a random individual, sampled i.i.d. from a fixed population, has features  $X \in \mathbf{R}^d$ , a label  $Y \in \{-1, 1\}$ , and a demographic group  $G \in \mathcal{G}$  (where  $\mathcal{G} = [n]$  for  $n \geq 2$ ). Denote the joint distribution of these variables in the population as  $s := \operatorname*{Pr}(X, Y, G)$  such that  $X, Y, G \sim s$ . The task is to predict  $Y$  (as  $\hat{Y}$ ) from  $X$  and  $G$  by choosing a classifier  $a$ , where  $\hat{Y} \sim a(X, G)$ , to minimize the empirical loss  $\mathcal{L}(s, a) \stackrel{\text{e.g.}}{=} \mathrm{E}\left[L(Y, \hat{Y})\right]$ , where  $L$  denotes some individual loss such as zero-one-loss. In general, we allow arbitrary, (unit-interval) bounded loss functions  $\mathcal{L}(s, a) \in [0, 1]$  such that the basic classification task is  $\min_{a} \mathcal{L}(s, a)$ .
The standard, "fair" classification task (without a dynamically responsive population) incorporates constraints on the choice of classifier  $a$ , such that the disparity  $\mathcal{D} \in [0,1]$  induced on distribution  $s$  by  $a$  is bounded by some value  $c \in [0,1]$ . Formally, the task is  $\min_{a} \mathcal{L}(s,a)$  subject to  $\mathcal{D}(s,a) \leq c$ . A standard measure of disparity  $\mathcal{D}$  is the violation of "demographic parity" (Dwork et al., 2012). For example, when  $\mathcal{G} = \{g_1, g_2\}$ ,  $\mathcal{D}(s,a) \stackrel{e.g.}{=} \left| \xi_{s,a}(g_1) - \xi_{s,a}(g_2) \right|^2$ , where  $\xi_{s,a}(g) \coloneqq \operatorname*{Pr}(\hat{Y} = 1 \mid G = g)$ .
In this paper, we also wish to consider measures of fairness inherent to the distribution  $s$  (e.g., parity of group qualification rates  $\operatorname*{Pr}(Y = 1\mid G = g)$ ). Such measures of fairness can only be affected by changes to  $s$  and thus require dynamics, which are not captured by the above formulation (Raab and Liu, 2021; Zhang et al., 2020). We will see that such notions of disparity are well-suited to an RL formulation of long-term fairness.
State, action, and policy Adopting the semantics of sequence of dependent classification tasks, we identify the time-dependent distribution  $s_{\tau} \in S$  of individuals in the population as a state and the chosen classifier  $a_{\tau} \in \mathcal{A}$  as an action of some algorithmic agent at time  $\tau$ . While state space  $S$  is arbitrary, we assume that action space  $\mathcal{A}$  admits a Euclidean metric, under which it is closed (i.e.,  $\mathcal{A}$  is isomorphic to  $[0,1]^m$ ,  $m \in \mathbf{Z}_{>0}$ ). We specify that  $a_{\tau}$  is sampled stochastically according to the agent's current policy  $\pi_{\tau}$ , that is,  $a_{\tau} \sim \pi_{\tau}(s_{\tau})$ . Additionally, we assume  $s_{\tau}$  is fully observable at time  $\tau \in \{1,2,\ldots\}$ . In practice,  $s_{\tau}$  must be approximated from finitely many empirical samples, though this caveat introduces well-understood errors that vanish in the limit of infinitely many samples.
Dynamics Moving beyond the one-shot "fair" classification task above, let us now assume that a population may react to classification, inducing the distribution  $s$  to change. In practice, such "distribution shift" is a well-known, real-world phenomenon that can increase realized loss and disparity when classifiers are fixed (Chen et al., 2022). For classifiers that free to update in response to a mutating distribution  $s$ , subsequent classification tasks depend on the (stochastic) predictions made in previous tasks. In our formulation, we assume the existence of dynamical kernel  $\mathbf{P}$  that maps a state  $s$  and action  $a$  at time  $\tau$  to a distribution over possible states at time  $\tau + 1$ . That is,  $s_{\tau + 1} \sim \mathbf{P}(s_{\tau}, a_{\tau})$ .
We stipulate that  $\mathbf{P}$  may be initially unknown, but we assume that it does not explicitly depend on time and may be reasonably approximated or learned "online". While real-world dynamics may depend on information other than the current distribution of classification-specific variables  $\operatorname*{Pr}(X,Y,G)$  — e.g., exogenous parameters, history, or additional variables of state — we have identified the dynamical state  $s$  with the current, fully-observed distribution for simplicity and tractability.
Reward and utility, value and quality functions Because the standard RL convention is to maximize reward rather than minimize loss, for an RL agent, we define the instantaneous reward  $r(s_{\tau},a_{\tau})\coloneqq 1 - \mathcal{L}(s_{\tau},a_{\tau})\in [0,1]$  and a separate "utility"  $g(s_{\tau},a_{\tau})\coloneqq 1 - \mathcal{D}(s_{\tau},a_{\tau})\in [0,1]$ , where  $r$  and  $g$  do not explicitly depend on time  $\tau$ . We next propose to optimize anticipated cumulative reward subject to constraints on anticipated cumulative utility. Let the index  $j$  represent either reward  $r$  or utility  $g$ . We use the letter  $V$  (for "value") to denote the future expected accumulation of  $j$  over steps  $[h,\dots,H]$  (without time-discounting) starting from state  $s$ , using policy  $\pi$ . Likewise, we denote the "quality" of an action  $a$  in initial state  $s$  with the letter  $Q$ . For  $j\in \{r,g\}$ , we define
$$
V _ {j, h} ^ {\pi} (s) := \operatorname {E} \left[ \sum_ {\tau = h} ^ {H} j (s _ {\tau}, a _ {\tau}) | s _ {h} = s \right]; Q _ {j, h} ^ {\pi} (s, a) := \operatorname {E} \left[ \sum_ {\tau = h} ^ {H} j (s _ {\tau}, a _ {\tau}) \mid s _ {h} = s, a _ {h} = a \right].
$$
Note that  $V$  and  $Q$  marginalize over the stochasticity of state transitions  $s_{\tau + 1} \sim \mathbf{P}(s_{\tau}, a_{\tau})$  and the sampling of actions  $a_{\tau} \sim \pi_{\tau}(s_{\tau})$ . By the boundedness of  $r$  and,  $g$ , the values of  $V$  and  $Q$  belong to the interval  $[0, H - h + 1]$ .
"Long-term fairness" via reinforcement learning The central problem explored in this paper is
$$
\max  _ {\pi} V _ {r, 1} ^ {\pi} (s) \quad \text {s u b j e c t} \quad V _ {g, 1} ^ {\pi} (s) \geq \tilde {c}. \tag {1}
$$
We emphasize that this construction considers a finite time horizon of  $H$  steps, and we denote the optimal value of  $\pi$  as  $\pi^{\star}$ . We first assume that a solution to the problem exists bounded within the interior of the feasible set (i.e., strict primal feasibility or Slater's condition), as in Efroni et al. (2020), Ding et al. (2021), and Ghosh et al. (2022):
Assumption 2.1 (Slater's Condition).  $\exists \gamma >0,\bar{\pi}$  , such that  $V_{g,1}^{\bar{\pi}}(s)\geq \tilde{c} +\gamma$
The Online Setting Given initially unknown dynamics, we formulate long-term fairness for the online setting (in which learning is only possible through actual "live" deployment of policy, rather than through simulation). As it is not possible to unconditionally guarantee constraint satisfaction in Eq. (1) over a finite number of online steps, we instead measure two types of regret: the suboptimality of a policy with respect to cumulative incurred loss, denoted Regret, and the suboptimality of a policy with respect to cumulative induced disparity, denoted denoted Drtn for "distortion".
$$
\operatorname {R e g r e t} (\pi , s _ {1}) := V _ {r, 1} ^ {\pi^ {*}} (s _ {1}) - V _ {r, 1} ^ {\pi} (s _ {1}) \quad ; \quad \operatorname {D r t n} (\pi , s _ {1}) := \max  \left[ 0, \tilde {c} - V _ {g, 1} ^ {\pi} (s _ {1}) \right]. \qquad (2)
$$
3 Algorithms and Analysis
We show that it is possible to provide guarantees for long-term fairness in the online setting. To do this, we develop L-UCBFair, the first model-free, simulator-free algorithm to provide such guarantees with continuous state and action spaces. Specifically, we prove probabilistic, sublinear bounds for regret and distortion under a linear MDP assumption (Assumption 3.1) and properly chosen parameters (Appendix B.1, in the supplementary material). We defer proofs to the supplementary material.
3.1 L-UCBFair
Episodic MDP L-UCBFair inherits from a family of algorithms that treat an episodic Markov decision process (MDP) Jin et al. (2020). Therefore, we first map the long-term fairness problem to the episodic MDP setting, which we denote as  $\mathbf{MDP}(\mathcal{S},\mathcal{A},H,\mathbf{P},\mathcal{L},\mathcal{D})$ . The algorithm runs for  $K$  episodes, each consisting of  $H$  time steps. At the beginning of each episode, which we index with  $k$ , the agent commits to a sequence of policies  $\pi^k = (\pi_1^k,\pi_2^k,\dots,\pi_H^k)$  for the next  $H$  steps. At each step  $h$  within an episode, an action  $a_h^k\in \mathcal{A}$  is sampled according to policy  $\pi_h^k$ , then the state  $s_{h + 1}^k\in S$  is sampled according to the transition kernel  $\mathbf{P}(s_h^k,a_h^k)$ .  $s_1^k$  is sampled arbitrarily for each episode.
Episodic Regret Because L-UCBFair predetermines its policy for an entire episode, we amend our definitions of regret and distortion over all  $HK$  time steps by breaking them into a sum over  $K$  episodes of length  $H$ :
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \left(V _ {r, 1} ^ {\pi^ {*}} \left(s _ {1} ^ {k}\right) - V _ {r, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right)\right); \quad \operatorname {D r t n} (K) = \max  \left[ 0, \sum_ {k = 1} ^ {K} \left(\tilde {c} - V _ {g, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right)\right) \right]. \tag {3}
$$
The Lagrangian Consider the Lagrangian  $\mathcal{L}$  associated with Eq. (1), with dual variable  $\nu \geq 0$ :
$$
\mathcal {L} (\pi , \nu) := V _ {r, 1} ^ {\pi} (s) + \nu \left(V _ {g, 1} ^ {\pi} (s) - \tilde {c}\right). \tag {4}
$$
L-UCBFair approximately solves the primal problem,  $\max_{\pi} \min_{\nu} \mathcal{L}(\pi, \nu)$ , which is non-trivial, since the objective function is seldom concave in practical parameterizations of  $\pi$ . Let  $\nu^{*}$  denote the optimal value of  $\nu$ ; L-UCBFair assumes bound  $\nu^{*} \leq \mathcal{V}$ , given parameter  $\mathcal{V}$ , as described in the supplementary material (Assumption B.1).
Assumption 3.1 (Linear MDP). We assume  $\mathrm{MDP}(\mathcal{S},\mathcal{A},H,\mathbf{P},\mathcal{L},\mathcal{D})$  is a linear MDP with feature map  $\phi :\mathcal{S}\times \mathcal{A}\to \mathbb{R}^d$ . That is, for any  $h$ , there exist  $d$  signed measures  $\mu_h = \{\mu_h^1,\dots ,\mu_h^d\}$  over  $\mathcal{S}$  and vectors  $\theta_{r,h},\theta_{g,h}\in \mathbb{R}^d$  such that, for any  $(s,a,s')\in S\times A\times S$
$$
\mathbb {P} _ {h} \left(s ^ {\prime} \mid s, a\right) = \langle \phi (s, a), \mu_ {h} \left(s ^ {\prime}\right) \rangle ; \quad r (s, a) = \langle \phi (s, a), \theta_ {r, h} \rangle ; \quad g (s, a) = \langle \phi (s, a), \theta_ {g, h} \rangle .
$$
Assumption 3.1 addresses the curse of dimensionality when state space  $S$  is the space of distributions over  $X, Y, G$ . This assumption is also used by Jin et al. (2020) and Ghosh et al. (2022), and a similar assumption is made by Ding et al. (2021). This assumption is well-justified in continuous time if we allow for infinite-dimensional Hilbert spaces Brunton et al. (2021), but in practice we require limited dimensionality  $d$  for the codomain of  $\phi$ . In our experiments, we use a neural network to estimate a feature map  $\hat{\phi}$  offline which approximately satisfies the linear MDP assumption (Appendix D.1).
3.1.1 Construction
L-UCBFair, or "LSVI-UCB for Fairness" (Algorithm 1) is based on a Least-Squares Value Iteration with an optimistic Upper-Confidence Bound, as in LSVI-UCB Jin et al. (2020). For each  $H$ -step episode  $k$ , L-UCBFair maintains estimates for  $Q_r^k$ ,  $Q_g^k$  and a time-indexed policy  $\pi^k$ . In each episode, L-UCBFair updates  $Q_r^k$ ,  $Q_g^k$ , interacts with the environment, and updates the dual variable  $\nu_k$ , which is constant over episode  $k$ .
LSVI-UCB Jin et al. (2020) The estimation of  $Q$  is challenging, as it is impossible to iterate over all  $s, a$  pairs when  $\mathcal{S}$  and  $\mathcal{A}$  are continuous and  $\mathbf{P}$  is unknown. LSVI parameterizes  $Q_h^\star(s, a)$  by the linear form  $\mathrm{w}_h^\top \phi(s, a)$ , as used by Jin et al. (2020), and updates
$$
\mathbf {w} _ {h} \leftarrow \underset {\mathbf {w} \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \sum_ {\tau = 1} ^ {k - 1} \left[ r _ {h} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) + \underset {a \in \mathcal {A}} {\max } Q _ {h + 1} \left(s _ {h + 1} ^ {\tau}, a\right) - \mathbf {w} ^ {\top} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \right] ^ {2} + \varsigma \| \mathbf {w} \| ^ {2}.
$$
In addition, a "bonus term"  $\beta\left(\phi^{\top}\Lambda_{h}^{-1}\phi\right)^{1/2}$  is added to the estimate of  $Q$  in Algorithm 1 to encourage exploration.
Adaptive Search Policy Compared to the works of Ding et al. (2021) and Ghosh et al. (2022), the major challenge we face for long-term fairness is a continuous action space  $\mathcal{A}$ , which renders the
independent computation of  $Q_{r}^{k}, Q_{g}^{k}$  for each action impossible. To handle this issue, we propose an adaptive search policy: Instead of drawing an action directly from a distribution over continuous values,  $\pi$  represents a distribution over finitely many  $(M)$ , Voronoi partitions of  $\mathcal{A}$ . After sampling a region with a softmax scheme, the agent draws action  $a$  uniformly at random from it. In addition to defining a Voronoi partitioning of the action space for the adaptive search policy of L-UCBFair (in the supplementary material), we make the following smoothness assumption, which we use to bound the error introduced by this sampling method:
Assumption 3.2 (Lipschitz). There exists  $\rho >0$ , such that  $\| \phi (s,a) - \phi (s,a^{\prime})\|_{2}\leq \rho \| a - a^{\prime}\|_{2}$
Dual Update For L-UCBFair, the update method for estimating  $\nu$  in Eq. (4) is also essential. Since  $V_{r,1}^{\pi}(s)$  and  $V_{g,1}^{\pi}(s)$  are unknown, we use  $V_{r,1}^{k}(s)$  and  $V_{g,1}^{k}(s)$  to estimate them. An estimate for  $\nu$  is iteratively updated by performing gradient ascent in the Lagrangian (Eq. (4)) with step-size  $\eta$ , subject to the assumed upper bound  $\mathcal{V}$  for  $\nu$  (Assumption B.1). A similar method is also used in Ding et al. (2021); Ghosh et al. (2022).
3.1.2 Analysis