SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We next bound the regret and distortion, defined in Eq. (3), realizable by L-UCBFair. We then compare L-UCBFair with existing algorithms for discrete action spaces and discuss the importance of the number of regions  $M$  and the maximum distance  $\epsilon_I$  from any action to the center of its corresponding Voronoi partition.
Theorem 3.3 (Boundedness). With probability  $1 - p$ , there exists a constant  $b$  such that  $L$ -UCBFair (Algorithm 1) achieves
$$
\operatorname {R e g r e t} (K) \leq \left(b \zeta H ^ {2} \sqrt {d ^ {3}} + (\mathcal {V} + 1) H\right) \sqrt {K}; \quad \operatorname {D r t n} (K) \leq \frac {b \zeta (1 + \mathcal {V}) H ^ {2} \sqrt {d ^ {3}}}{\mathcal {V}} \sqrt {K}, \tag {5}
$$
for parameter values  $\varsigma = 1$ $\epsilon_I\leq \frac{1}{2\rho(1 + \mathcal{V})KH\sqrt{d}}$ ,  $\zeta = \log (\log (M)4dHK / p)$ , and  $\beta = \mathcal{O}(dH\sqrt{\zeta})$ .
For a detailed proof of Theorem 3.3, refer to Appendix B.1. This theorem indicates that L-UCBFair achieves  $\mathcal{O}\left(H^{2}\sqrt{d^{3}K}\right)$  bounds for both regret and distortion with high probability. Compared to the algorithms introduced by Ding et al. (2021) and Ghosh et al. (2022), which work with discrete action space, L-UCBFair guarantees the same asymptotic bounds on regret and distortion.
3.2 R-TD3
Because assumptions in Section 3.1 (e.g., the linear MDP assumption) are often violated in the real world, we also consider deep reinforcement learning methods as a flexible alternative. Concretely, we apply "Twin-Delayed Deep Deterministic Policy Gradient" (TD3) Fujimoto et al. (2018), with an implementation and default parameters provided by the open-source package "Stable Baselines 3" Raffin et al. (2021), on a Lagrangian relaxation of the long-term fairness problem with time-dependent multipliers for the reward and utility terms (Eq. (6)). We term this specific algorithm for long-term fairness R-TD3. While such methods lack provable safety guarantees, they may still confirm our hypothesis that agents trained via RL can learn to sacrifice short-term utility in order to drive dynamics towards preferable long-term states and explicitly incorporate dynamical control objectives provided as functions of state.
To treat the long-term fairness problem (Eq. (1)) using unconstrained optimization techniques (i.e., methods like TD3), we consider a time-dependent Lagrangian relaxation: We train R-TD3 to optimize
$$
\min  _ {\pi} \underset {a _ {\tau} \sim \pi \left(s _ {\tau}\right)} {\mathrm {E}} \left[ \sum_ {\tau = 1} ^ {H} \left((1 - \lambda_ {\tau}) \mathcal {L} \left(s _ {\tau}, a _ {\tau}\right) + \lambda_ {\tau} \mathcal {D} \left(s _ {\tau}, a _ {\tau}\right)\right) \right], \tag {6}
$$
where  $s_{\tau +1}\sim \mathbf{P}(s_{\tau},a_{\tau}),\lambda_{\tau} = \tau /H$
Strictly applied, myopic fairness constraints can lead to undesirable dynamics and equilibria Raab and Liu (2021). Eq. (6) relaxes these constraints (hard  $\rightarrow$  soft) for the near future while emphasizing them long-term. Thus, we hope to develop classifiers that learn to transition to more favorable equilibria.
Algorithm 1 L-UCBFair
Input: A set of points  $\{I_0, I_1, \dots, I_M\}$  satisfy Definition B.2.  $\epsilon_I = \frac{1}{2\rho(1 + \chi)KH}$ .
$$
\nu_ {1} = 0. w _ {r, h} = w _ {g, h} = 0. \alpha = \frac {\log (M) K}{2 (1 + \chi + H)}. \eta = \chi / \sqrt {K H ^ {2}}. \beta = C _ {1} d H \sqrt {\log (4 \log M d T / p)}, \varsigma = 1.
$$
for episode  $k = 1,2,\dots,K$  do
Receive the initial state  $s_1^k$ .
for step  $h = H,H - 1,\dots ,1$  do
$$
\Lambda_ {h} ^ {k} \leftarrow \sum_ {\tau = 1} ^ {k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {T} + \varsigma \mathbf {I}
$$
for  $j\in \{r,g\}$  do
$$
w _ {j, h} ^ {k} \leftarrow \left(\Lambda_ {h} ^ {k}\right) ^ {- 1} \left[ \sum_ {\tau = 1} ^ {k - 1} \phi \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \left(j \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) + V _ {j, h + 1} ^ {k} \left(s _ {h + 1} ^ {\tau}\right)\right) \right]
$$
for iteration  $i = 1,\dots ,M$  and index  $j\in \{r,g\}$  do
$$
\xi_ {i, j} \leftarrow \left(\phi (\cdot , I _ {i}) ^ {T} \left(\Lambda_ {h} ^ {k}\right) ^ {- 1} \phi (\cdot , I _ {i})\right) ^ {1 / 2}, \quad Q _ {j, h} ^ {k} (\cdot , I _ {i}) \leftarrow \min  \left[ \left\langle w _ {j, h} ^ {k}, \phi (\cdot , I _ {i}) \right\rangle + \beta \xi_ {i, j}, H \right]
$$
$$
\mathbf {S M} _ {h, k} (I _ {i} \mid \cdot) = \frac {\exp \left(\alpha \left(Q _ {r , h} ^ {k} (\cdot , I _ {i}) + \nu_ {k} Q _ {g , h} ^ {k} (\cdot , I _ {i})\right)\right)}{\sum_ {j} \exp \left(\alpha \left(Q _ {r , h} ^ {k} (\cdot , I _ {j}) + \nu_ {k} Q _ {a , h} ^ {k} (\cdot , I _ {j})\right)\right)}
$$
$$
\pi_ {h} ^ {k} (a \mid \cdot) \leftarrow \frac {1}{\int_ {b \in \mathcal {I} (a)} d b} \mathbf {S M} _ {h, k} (I (a) \mid \cdot)
$$
$$
V _ {r, h} ^ {k} (\cdot) \leftarrow \int_ {a \in \mathcal {A}} \pi_ {h} ^ {k} (a | \cdot) Q _ {r, h} ^ {k} (\cdot , a) d a, \quad V _ {g, h} ^ {k} (\cdot) \leftarrow \int_ {a \in \mathcal {A}} \pi_ {h} ^ {k} (a | \cdot) Q _ {g, h} ^ {k} (\cdot , a) d a
$$
for step  $h = 1,\dots ,H$  do
$$
\text {C o m p u t e} Q _ {r, h} ^ {k} \left(s _ {h} ^ {k}, I _ {i}\right), Q _ {g, h} ^ {k} \left(s _ {h} ^ {k}, I _ {i}\right), \pi \left(I _ {i} \mid s _ {h} ^ {k}\right).
$$
$$
\text {T a k e} a _ {h} ^ {k} \sim \pi_ {h} ^ {k} (\cdot | s _ {h} ^ {k}) \text {a n d o b s e} s _ {h + 1} ^ {k}.
$$
$$
\nu_ {k + 1} = \max  \left\{\min  \left\{\nu_ {k} + \eta \left(\tilde {c} - V _ {g, 1} ^ {k} (s _ {1})\right), \mathcal {V} \right\}, 0 \right\}
$$
We compare L-UCBFair and R-TD3 to a greedy agent as a proxy for a myopic status quo in which policy is repeatedly determined by optimizing for immediate utility, without regard for the population dynamics. This standard is known as "Repeated Risk Minimization" (Perdomo et al., 2020; Hu and Zhang, 2022), and we implement it using simple gradient descent on the different classes of ( $\lambda$ -parameterized) objective functions  $f$  we consider (Eq. (7)). Having adopted a notion of fairness that relies on "groups", we presuppose different groups of agents indexed by  $g$ , and denote group-conditioned loss as  $\mathcal{L}_g$ . The objectives that are approximately minimized for each baseline are
$$
\text {M y o p i c :} f (\pi) = \underset {a \sim \pi} {\mathrm {E}} [ \mathcal {L} (s, a) ]. \tag {7a}
$$
$$
\text {M y o p i c - F a i r :} f _ {\lambda} (\pi) = \underset {a \sim \pi} {\mathrm {E}} [ (1 - \lambda) \mathcal {L} (s, a) + \lambda \mathcal {D} (s, a) ], \lambda \in (0, 1). \tag {7b}
$$
$$
\text {M a x m i n}: \quad f (\pi) = \underset {a \sim \pi} {\operatorname {E}} \left[ \max  _ {g} \left(\mathcal {L} _ {g} (s, a)\right) \right]. \tag {7c}
$$
$$
\text {M a x m i n - F a i r :} f _ {\lambda} (\pi) = \underset {a \sim \pi} {\mathrm {E}} \left[ (1 - \lambda) \max  _ {g} \left(\mathcal {L} _ {g} (s, a)\right) + \lambda \mathcal {D} (s, a) \right], \lambda \in (0, 1). \tag {7d}
$$
The two "Maxmin" objectives are related to distributionally robust optimization, which has been previously explored in the context of fairness (Hashimoto et al., 2018), while the two "Myopic" objectives are more straight-forward. While our baselines do not guarantee constraint satisfaction, the two objectives labelled "-Fair" are nonetheless "constraint aware" in precisely the same way as a firm that (probabilistically) incurs penalties for violating constraints.
4 Simulated Environments
To evaluate the proposed algorithms and baselines, we consider a series of binary  $(Y\in \{-1,1\})$  classification tasks on a population of two groups  $\mathcal{G} = \{g_1,g_2\}$  modeled according to evolutionary
game theory (using replicator dynamics, as described in Appendix A.1, in the supplementary material). We consider two families of distributions of real-valued features for the population: One that is purely synthetic, for which  $X \sim \mathcal{N}(Y, 1)$ , independent of group  $G$ , and one that is based on logistic regressions to real-world data, described in Appendix A.2 (in the supplementary material). Both families of distributions over  $X$  are parameterized by the joint distribution  $\operatorname*{Pr}(Y, G)$ . RL agents are trained on episodes of length  $H$  initialized with randomly sampled states.
In order to better handle continuous state space, we make the following assumption, which has been used to simplify similar synthetic environments (Raab and Liu, 2021):
Assumption 4.1 (Well-behaved feature). For the purely synthetic setting, we require  $X$  to be a "well-behaved" real-valued feature within each group. That is, for each group  $g$ ,  $\operatorname{Pr}(Y = 1 \mid G = g, X = x)$  strictly increases in  $x$ .
As an intuitive example of Assumption 4.1, if  $Y$  represents qualification for a fixed loan and  $X$  represents credit-score, we require higher credit scores to imply higher likelihood that an individual is qualified for the loan.
Theorem 4.2 (Threshold Bayes-optimality). For each group  $g$ , when Assumption 4.1 is satisfied, the Bayes-optimal, deterministic binary classifier is a threshold policy described by a feature threshold  $a_{g}$  for group  $g$ . That is, if  $X \geq a_{g}$  then  $\hat{Y} = 1$ ; otherwise,  $\hat{Y} = -1$ .
As a result of Theorem 4.2, we consider our action space to be the space of group-specific thresholds, and denote an individual action as the vector  $a \coloneqq (a_1, a_2)$ . Nonetheless, we note that Assumption 4.1 is often violated in practice, as it is in our semi-synthetic setting.
Do RL agents learn to seek favorable equilibria against short-term utility? Is Lagrangian relaxation Eq. (1) sufficient to encourage this behavior? We give positive demonstrations for both questions.
5.1 Losses and Disparities Considered
Our experiments consider losses  $\mathcal{L}$  which combine true-positive and true-negative rates, i.e.,
$$
\mathcal {L} (s, a) = 1 - \alpha \operatorname {t p} (s, a) - \beta \operatorname {t n} (s, a); \quad r (s, a) = \alpha \operatorname {t p} (s, a) + \beta \operatorname {t n} (s, a), \tag {8}
$$
where  $\alpha, \beta \in [0,1]$ ;  $\mathfrak{tp}(s,a) = \operatorname{Pr}_{s,a}(\hat{Y} = 1, Y = 1)$ ; and  $\mathfrak{tn}(s,a) = \operatorname{Pr}_{s,a}(\hat{Y} = -1, Y = -1)$ .
For disparity  $\mathcal{D}$ , we consider functions of the form  $\mathcal{D}(s, a) = \frac{1}{2} \| \xi_{s,a}(g_1) - \xi_{s,a}(g_2) \|^2$  that measure violations of demographic parity (DP) (Dwork et al., 2012), equal opportunity (EOp) (Hardt et al., 2016), equalized odds (EO), and qualification rate parity (QR), described in Table 1. We note that QR is inconsequential for the baseline agents, which ignore the mutability of the population state.
Our experiments show that algorithms trained with an RL formulation of long-term fairness can drive a reactive population toward states with higher utility and fairness than myopic policies. In Fig. 1, the baseline policies (i.e., subfigures (a) and (b)), which focus on short-term incentives drive disparate qualification rates and attain lower long-term utility than the RL agents, indicating that short-term utility is misaligned with desirable dynamics in this example. In subfigure (b) in particular, the baseline policy increasingly violates the static fairness condition with time, agreeing with previous results by Raab and Liu (2021). Meanwhile, the RL algorithms (subfigures (c) and (d)) learn to drive universally high qualification rates, thus allowing
Table 1: Disparity Functions, where  $\mathcal{D}(s,a) = \frac{1}{2}\| \xi_{s,a}(g_1) - \xi_{s,a}(g_2)\|^2$
policies that capture higher utility with time (subfigure (e)), Fig. 3.
Our central hypothesis, that long-term fairness via RL may induce an algorithm to sacrifice short-term utility for better long-term outcomes, is clearly demonstrated in the purely synthetic environment depicted by subfigures (a1-a3) of Fig. 2, in which the environment provides higher immediate utility (true-positive rates) but lower long-term utility when a policy chooses initially high acceptance rates. In this case, the RL algorithm (subfigure (a2)) drives the system towards high qualification rates by giving up immediate utility maximized by the myopic agent (subfigure (a1)).
With subfigures (b1-b3) of Fig. 2, we demonstrate the capability of RL to incorporate notions of fairness (e.g., qualification rate parity QR), that are impossible to formulate in the myopic setting. In subfigures (b1-b3), both RL agents learn to satisfy qualification rate parity by driving the state of the population towards equal qualification rates by group.
Finally, our experiments also show that RL algorithms without theoretical guarantees may be applicable to long-term fairness. In Fig. 2 subfigure (b2), R-TD3 achieves similar qualitative behavior as L-UCBFair (subfigure (b1)), that is, driving high qualification at the expense of short-term utility) while achieving lower episodic mean loss (subfigure (b3)).
Limitations Despite the potential, highlighted by our experiments, for RL-formulation of fairness to drive positive long-term social outcomes, it is not possible to truly validate such approaches without deployment on actual populations, which may be practically and ethically fraught. In addition, violations of fairness or decreased utility may be difficult to justify to affected populations and stakeholders, especially when the bounds provided by L-UCBFair, while as tight as any known, rely on assumptions that may be violated in practice.
Concluding remarks Machine learning techniques are frequently deployed in settings in which affected populations will react to the resulting policy, closing a feedback loop that must be accounted for. In such settings, algorithms that prioritize immediate utility or static notions of fairness may yield dynamics that are misaligned with these objectives long-term.
In this paper, we have reformulated long-term fairness as an online reinforcement learning problem (Eq. (1)) to address the importance of dynamics. We have shown that algorithmic solutions to this problem (e.g., L-UCBFair) are capable of simultaneous theoretical guarantees regarding cumulative loss and disparity (violations of fairness). We have also shown that these guarantees can be relaxed in practice to accommodate a wider class of RL algorithms, such as R-TD3. Finally, we emphasize again that the RL framework of long-term fairness allows notions of disparity inherent to the state of a population to be explicitly treated, while such definitions are inoperable in the standard, myopic framework of fairness. We hope that our contributions spur interest in long-term mechanisms and incentive structures for machine learning to be a driver of positive social change.
Acknowledgements This work is partially supported by the National Science Foundation (NSF) under grants IIS-2143895, IIS-2112471, IIS-2040800, and CCF-2023495.
Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David C Parkes. Calibrated fairness in bandits. arXiv preprint arXiv:1707.01875, 2017.
Hussein Mouzannar, Mesrob I Ohannessian, and Nathan Srebro. From fair decision making to social equality. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 359-368. ACM, 2019.
Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen, Longbo Huang, and Tie-Yan Liu. Reinforcement learning with dynamic boltzmann softmax updates. arXiv preprint arXiv:1903.05926, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dunner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pages 7599-7609. PMLR, 2020.
Reilly Raab and Yang Liu. Unintended selection: Persistent qualification rate disparities and interventions. Advances in Neural Information Processing Systems, 34:26053-26065, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
Rahul Singh, Abhishek Gupta, and Ness B Shroff. Learning in markov decision processes under constraints. arXiv preprint arXiv:2002.12435, 2020.
Wei Tang, Chien-Ju Ho, and Yang Liu. Bandit learning with delayed impact of actions. Advances in Neural Information Processing Systems, 34:26804-26817, 2021.
Honghao Wei, Xin Liu, and Lei Ying. Triple-q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In International Conference on Artificial Intelligence and Statistics, pages 3274-3307. PMLR, 2022.
Min Wen, Osbert Bastani, and Ufuk Topcu. Fairness with dynamics. arXiv preprint arXiv:1901.08568, 2019.
Joshua Williams and J Zico Kolter. Dynamic modeling and equilibria in fair decision making. arXiv preprint arXiv:1911.06837, 2019.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pages 11480-11491. PMLR, 2021.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pages 325-333. PMLR, 2013.
Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang. How do fair decisions fare in long-term qualification? arXiv preprint arXiv:2010.11300, 2020.
Liyuan Zheng and Lillian Ratliff. Constrained upper confidence reinforcement learning. In Learning for Dynamics and Control, pages 620-629. PMLR, 2020.
A Simulated Environments: Supplementary
A.1 Replicator Dynamics
Our use of replicator dynamics closely mirrors that of Raab and Liu (2021) as an "equitable" model of a population: Individuals may be modeled identically, independently of group membership, yet persistent outcome disparities may emerge from disparate initial conditions between groups. For our experiments, we parameterize the evolving distribution  $\operatorname*{Pr}(X,Y\mid G)$ , assuming constant group sizes, in terms of "qualification rates"  $q_{g}\coloneqq \operatorname *{Pr}(Y = 1\mid G = g)$  and update these qualification rates according to the discrete-time replicator dynamics:
$$
q _ {g} [ t + 1 ] = q _ {g} [ t ] \frac {W _ {1} ^ {g} [ t ]}{\overline {{W}} ^ {g} [ t ]}; \quad \overline {{W}} ^ {g} [ t ] := W _ {1} ^ {g} q _ {g} + (1 - q _ {g}) W _ {- 1} ^ {g}.
$$
In this model, the fitness  $W_{y}^{g} > 0$  of label  $Y = y$  in group  $G = g$  may be interpreted as the "average utility to the individual" in group  $g$  of possessing label  $y$ , and thus relative replication rate of label  $y$  in group  $g$ , as agents update their labels by mimicking the successful strategies of in-group peers. We model  $W_{y}^{g}$  in terms of acceptance and rejection rates with a group-independent utility matrix  $U$ :
$$
W _ {y} ^ {g} = \sum_ {\hat {y} \in \{- 1, 1 \}} U _ {y, \hat {y}} \Pr \Big (\hat {Y} = \hat {y} \mid Y = y, G = g \Big).
$$
We choose the matrix  $U$  to eliminate dominant strategies (i.e., when agents universally prefer one label over another, independent of classification), assert that agents always prefer acceptance over rejection, and to imply that the costs of qualification are greater than the costs of non-qualification among accepted individuals. While other parameterizations of  $U$  are valid, this choice of parameters guarantees internal equilibrium of the replicator dynamics for a Bayes-optimal classifier and "well-behaved" scalar-valued feature  $X$ , such that  $\operatorname{Pr}(Y = 1 \mid X = x)$  is monotonically increasing in  $x$  (Raab and Liu, 2021).
A.2 Data Synthesis and Processing
In addition to a synthetic distribution, for which we assume  $X \sim \mathcal{N}(Y, 1)$ , independent of  $G$ , for all time, we also consider real-world distributions when simulating and comparing algorithms for "long-term fairness". In both cases, as mentioned above, we wish to parameterize distributions in terms of qualification rates  $q_{g}$ . As we perform binary classification on discrete groups and scalar-valued features, in addition to parameterizing a distribution in terms of  $q_{g}$ , we desire a scalar-valued feature for each example, rather than the multi-dimensional features common to real-world data.
Our solution is to use an additional learning step for "preprocessing": Given a static dataset  $\mathcal{D}$  from which  $(X', Y, G)$  is drawn i.i.d., (e.g., the "Adult Data Set" Dua and Graff (2017)), where  $G$  varies over an individual's sex and  $Y$  corresponds to whether an individual has more than 50,000 USD in annual income, at each time-step, we train a stochastic binary classifier  $\tilde{a}$ , such that  $\hat{Y}' \sim \tilde{a}(X', G)$  with a loss that re-weights examples by label value according to  $q_{g}$ . That is, at each time step, we solve:
$$
\min _ {\tilde {a}} \mathrm {E} _ {\tilde {a}, \mathcal {D}} \left[ w (X ^ {\prime}, Y, G) L (Y, \hat {Y} ^ {\prime}) \right],
$$
$$
w (X ^ {\prime}, Y, G) = \left(\frac {1 + Y}{2}\right) \left(\frac {q}{\operatorname * {P r} (Y = 1 \mid G)}\right) + \left(\frac {1 - Y}{2}\right) \left(\frac {1 - q}{\operatorname * {P r} (Y = - 1 \mid G)}\right),
$$
and  $L$  is zero-one loss. In our experiments, we choose  $\tilde{a}$  according to logistic regression. We interpret  $\operatorname*{Pr}(\hat{Y}' = 1)$ , drawn from the learned "preprocessing" function  $\tilde{a}$ , as a new, scalar feature value  $X \in \mathbf{R}$  mapped from higher-dimensional features  $X'$ . The threshold policy ultimately operates on  $X$ .
Assumption 4.1 is as hard to satisfy in general as solving the Bayes-optimal binary classification task over higher-dimensional features. Nonetheless, we expect Assumption 4.1 to be approximately satisfied by such a "preprocessing" pipeline. It is important to note that the behavior introduced by retraining a logistic classifier at each time step can yield very different qualitative behavior when compared to the static, purely synthetic distribution, as the distribution of  $X$  conditioned on  $Y$  and  $G$  also mutates, and Assumption 4.1 may be violated.
B Deferred Proofs
Without loss of generality, we assume  $\| \phi (s,a)\| \leq 1$  for all  $(s,a)\in \mathcal{S}\times \mathcal{A}$ , and  $\max \left\{\| \mu_h(\mathcal{S})\| ,\| \theta_h\| \right\} \leq \sqrt{d}$  for all  $h\in [H]$ .
Assumption B.1 ( $\mathcal{V}$  bound for  $\nu^{*}$ ). We define a parameter  $\mathcal{V}$  that is assumed to bound the optimal value of  $\nu$  in Eq. (4). For  $\bar{\pi}$  and  $\gamma > 0$  satisfying Slater's Condition (Assumption 2.1),
$$
\nu^ {*} \leq \frac {V _ {r , 1} ^ {\pi *} \left(s _ {1}\right) - V _ {r , 1} ^ {\pi} \left(s _ {1}\right)}{\gamma} \leq \frac {H}{\gamma} := \mathcal {V}.
$$
In Assumption B.1,  $\mathcal{V} = H / \gamma$ , upper-bounds the optimal dual variable  $\nu^{*}$  as an input to L-UCBFair.
Definition B.2. Given a set of distinct actions  $I = \{I_0, \dots, I_M\} \subset \mathcal{A}$ , where  $\mathcal{A}$  is a closed set in Euclidean space, define  $\mathcal{I}_i = \{a : \|a - I_i\|_2 \leq \|a - I_j\|_2, \forall j \neq i\}$  as the subset of actions closer to  $I_i$  than to  $I_j$ , i.e., the Voronoi region corresponding to locus  $I_i$ , with tie-breaking imposed by the order of indices  $i$ . Also define the locus function  $I(a) = \min_i \operatorname{argmin}_{I_i} \|a - I_i\|_2$ .
Lemma B.3. The Voronoi partitioning described above satisfies  $\mathcal{I}_i\cap \mathcal{I}_j = \emptyset ,\forall i\neq j$  and  $\cup_{i = 1}^{M}\mathcal{I}_{i} = \mathcal{A}$