SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We will begin by proving  $\mathcal{I}_i \cap \mathcal{I}_j = \emptyset$  for all  $i \neq j$ . To establish this, we will assume the contrary, that there exist indices  $i$  and  $j$  such that  $\mathcal{I}_i \cap \mathcal{I}_j \neq \emptyset$ . Without loss of generality, assume  $i > j$ . We will denote an arbitrary action within the interaction of  $\mathcal{I}_i$  and  $\mathcal{I}_j$  as  $a' \in \mathcal{A}$ .
Considering that  $a' \in \mathcal{I}_i$ , according to the given Definition B.2, we can infer that  $|a' - I_i|_2 < |a' - I_j|_2$  (since  $i > j$ ). However, this assertion contradicts the fact that  $a' \in \mathcal{I}_j$ , which implies  $|a' - I_j|_2 \leq |a' - I_i|_2$ . Therefore,  $\mathcal{I}_i \cap \mathcal{I}_j = \emptyset$  for all  $i \neq j$ .
We then proof  $\cup_{i=1}^{M}\mathcal{I}_{i} = \mathcal{A}$ . Since  $\mathcal{A}$  is a closed set, for any  $a \in \mathcal{A}$ , there must be a  $i \in \{1,2,\dots,M\}$  such that  $d_{a,i} = \|a - I_{i}\|_{2} \leq \|a - I_{j}\|_{2} = d_{a,j}, \forall j$ . If  $d_{a,i} < d_{a,j}$  strictly holds for all  $j$ , then  $a \in \mathcal{I}_{i}$ . Otherwise define a set  $\mathcal{J} = \{j | d_{a,j} = d_{a,i}\}$ , then  $a \in \mathcal{I}_{j'}$  where  $j' = \operatorname*{argmin}_{j \in \mathcal{J}} j$ .
Theorem B.4. If the number  $M$  of distinct loci or regions partitioning  $\mathcal{A}$  is sufficiently large, there exists a finite set of loci  $I$  such that  $\forall a\in \mathcal{I}_i,i\in M,\| a - I_i\| _2\leq \epsilon_I$ .
Proof. Since  $\mathcal{A}$  is closed in euclidean space, denote  $N$  the dimension,  $d = \sup_{a,a' \in \mathcal{A}} \| a - a' \|_2$ . Define an Orthonormal Basis  $\{e_1, e_2, \dots, e_N\}$ . Randomly choose a point  $a \in \mathcal{A}$ , Set it as  $I_0$ , then we form a set of loci  $I = \{I_0 + \sum_{i=1}^N \epsilon k_i e_i | I_0 + \sum_{i=1}^N \epsilon k_i e_i \in \mathcal{A}, k_i \in \mathcal{Z}, -\lceil \frac{d}{\epsilon} \rceil \leq k_i \leq \lceil \frac{d}{\epsilon} \rceil\}$ . We know that  $|I| \leq (2\lceil \frac{d}{\epsilon} \rceil)^N$ . It's not hard to verify that  $\|a - I_i\|_2 \leq \frac{\epsilon}{2}\sqrt{2}^{N-1}, \forall a \in \mathcal{I}_i$ . Taken  $\epsilon_I = \frac{\epsilon}{2}\sqrt{2}^{N-1}$  yields the statement.
B.1 Proof of Theorem 3.3
Theorem 3.3 (Boundedness). With probability  $1 - p$ , there exists a constant  $b$  such that  $L$ -UCBFa  $\mathcal{F}$  (Algorithm 1) achieves
Outline The outline of this proof simulates the proof in Ghosh et al. (2022). For brevity, denote  $\mathbb{P}_hV_{j,h + 1}^\pi (s,a) = \mathbb{E}_{s'\sim \mathbb{P}_h(\cdot |s,a)}V_{j,h + 1}^\pi (s')$  for  $j = r,g$ . Then
$$
Q _ {j, h} ^ {\pi} (s, a) = \left(r _ {h} + \mathbb {P} _ {h} V _ {j, h + 1} ^ {\pi}\right) (s, a) \tag {9}
$$
$$
V _ {j, h} ^ {\pi} (s) = \left\langle \pi_ {h} (\cdot \mid s) Q _ {j, h} ^ {\pi} (s, \cdot) \right\rangle_ {\mathcal {A}} \tag {10}
$$
$$
\left\langle \pi_ {h} (\cdot \mid s), Q _ {j, h} ^ {\pi} (s, \cdot) \right\rangle_ {\mathcal {A}} = \sum_ {a \in \mathcal {A}} \pi_ {h} (a \mid s) Q _ {j, h} ^ {\pi} (s, a) \tag {11}
$$
Similar to Efroni et al. (2020), we establish
$$
\begin{array}{l} \operatorname {R e g r e t} (K) + \nu \text {D i s t o r t i o n} (K) \\ = \sum_ {k = 1} ^ {K} \left(V _ {r, 1} ^ {\pi^ {*}} (s _ {1}) - V _ {r, 1} ^ {\pi_ {k}} (s _ {1})\right) + \nu \sum_ {k = 1} ^ {K} \left(b - V _ {g, 1} ^ {\pi_ {k}} (s _ {1})\right) \\ \leq \underbrace {\sum_ {k = 1} ^ {K} \left(V _ {r , 1} ^ {\pi^ {*}} \left(s _ {1}\right) + \nu_ {k} V _ {g , 1} ^ {\pi^ {*}} \left(s _ {1}\right)\right) - \left(V _ {r , 1} ^ {k} \left(s _ {1}\right) + \nu_ {k} V _ {g , 1} ^ {k} \left(s _ {1}\right)\right)} _ {\mathcal {T} _ {1}} \\ + \underbrace {\sum_ {k = 1} ^ {K} \left(V _ {r , 1} ^ {k} \left(s _ {1}\right) - V _ {r , 1} ^ {\pi_ {k}} \left(s _ {1}\right)\right) + \nu \sum_ {k = 1} ^ {K} \left(V _ {g , 1} ^ {k} \left(s _ {1}\right) - V _ {g , 1} ^ {\pi_ {k}} \left(s _ {1}\right)\right)} _ {\mathcal {T} _ {2}} \\ + \underbrace {\frac {1}{2 \eta} \nu^ {2} + \frac {\eta}{2} H ^ {2} K} _ {\tau_ {3}} \tag {12} \\ \end{array}
$$
$\mathcal{T}_3$  is easily bounded if  $\eta$ . The major task remains bound  $\mathcal{T}_1$  and  $\mathcal{T}_2$ .
Bound  $\mathcal{T}_1$  and  $\mathcal{T}_2$ . We have following two lemmas.
Lemma B.5 (Boundedness of  $\mathcal{T}_1$ ). With probability  $1 - p/2$ , we have  $\mathcal{T}_1 \leq KH\left(\frac{\log(M)}{\alpha} + 2(1 + \mathcal{V})H\rho \epsilon_I\sqrt{\frac{dK}{\varsigma}}\right)$ . Specifically, if  $\alpha = \frac{\log(M)K}{2(1 + \mathcal{V} + H)}$  and  $\varsigma = 1$ , we have  $\mathcal{T}_1 \leq 2H(1 + \mathcal{V} + H) + 2KH^2(1 + \mathcal{V})\rho \epsilon_I\sqrt{dK}$  with probability  $1 - p/2$ .
Lemma B.6 (Boundedness of  $\mathcal{T}_2$ ). Ghosh et al. (2022) With probability  $1 - p / 2, \mathcal{T}_2 \leq \mathcal{O}\left((\nu + 1)H^2\zeta \sqrt{d^3K}\right)$ , where  $\zeta = \log[\log(M)4dHK/p]$ .
Lemma B.6 follows the same logic in Ghosh et al. (2022), and we delay the proof of Lemma B.5 to Section B.2. Now we are ready to proof Theorem 3.3.
Proof. For any  $\nu \in [0, \mathcal{V}]$ , with prob.  $1 - p$ ,
$$
\begin{array}{l} \operatorname {R e g r e t} (K) + \nu \text {D i s t o r t i o n} (K) \\ \leq \mathcal {T} _ {1} + \mathcal {T} _ {2} + \mathcal {T} _ {3} \\ \leq \frac {1}{2 \eta} \nu^ {2} + \frac {\eta}{2} H ^ {2} K + \frac {H K \log M}{\alpha} + 2 K H ^ {2} (1 + \mathcal {V}) \rho \epsilon_ {I} \sqrt {d K} + \mathcal {O} ((\nu + 1) H ^ {2} \zeta \sqrt {d ^ {3} K}) \tag {13} \\ \end{array}
$$
Taking  $\nu = 0, \eta = \frac{\mathcal{V}}{\sqrt{KH^2}}, \alpha = \frac{K\log M}{2(1 + \mathcal{V} + H)}, \epsilon_I = \frac{1}{2\rho(1 + \mathcal{V})KH\sqrt{d}}$ , there exist constant  $b$ ,
$$
\begin{array}{l} \operatorname {R e g r e t} (K) \leq \frac {\mathcal {V} H}{2} \sqrt {K} + 2 H (1 + \mathcal {V} + H) + 2 H ^ {2} K (1 + \mathcal {V}) \rho \epsilon_ {I} \sqrt {d K} + \mathcal {O} \left(H ^ {2} \zeta \sqrt {d ^ {3} K}\right) \\ \leq \left(b \zeta H ^ {2} \sqrt {d ^ {3}} + (\mathcal {V} + 1) H\right) \sqrt {K} = \tilde {\mathcal {O}} \left(H ^ {2} \sqrt {d ^ {3} K}\right). \\ \end{array}
$$
Taking  $\nu = \mathcal{V},\eta = \frac{\mathcal{V}}{\sqrt{KH^2}},\alpha = \frac{K\log M}{2(1 + \mathcal{V} + H)},\epsilon_I = \frac{1}{2\rho(1 + \mathcal{V})KH\sqrt{d}},$
$$
\operatorname {R e g r e t} (K) + \mathcal {V} \text {D i s t o r t i o n} (K) \leq (\mathcal {V} + 1) H \sqrt {K} + (1 + \mathcal {V}) \mathcal {O} \left(H ^ {2} \zeta \sqrt {d ^ {3} K}\right)
$$
Following the idea of Efroni et al. (2020), there exists a policy  $\pi'$  such that  $V_{r,1}^{\pi'} = \frac{1}{K}\sum_{k=1}^{K} V_{r,1}^{\pi_k}, V_{g,1}^{\pi'} = \frac{1}{K}\sum_{k=1}^{K} V_{g,1}^{\pi_k}$ . By the occupancy measure,  $V_{r,1}^{\pi}$  and  $V_{g,1}^{\pi}$  are linear in occupancy measure induced by  $\pi$ . Thus, the average of  $K$  occupancy measure also produces an occupancy measure which induces policy  $\pi'$  and  $V_{r,1}^{\pi'}$ , and  $V_{g,1}^{\pi'}$ . We take  $\nu = 0$  when  $\sum_{k=1}^{K} \left(b - V_{g,1}^{\pi_k}(s_1^k)\right) < 0$ ,
otherwise  $\nu = \mathcal{V}$ . Hence, we have
$$
\begin{array}{l} V _ {r, 1} ^ {\pi^ {*}} (s _ {1}) - \frac {1}{K} \sum_ {k = 1} ^ {K} V _ {r, 1} ^ {\pi_ {k}} (s _ {1}) + \mathcal {V} \max \left(\left(c - \frac {1}{K} \sum_ {k = 1} ^ {K} V _ {g, 1} ^ {\pi_ {k}} (s _ {1}), 0\right) \right. \\ = V _ {r, 1} ^ {\pi^ {*}} \left(s _ {1}\right) - V _ {r, 1} ^ {\pi^ {\prime}} \left(s _ {1}\right) + \mathcal {V} \max \left(c - V _ {g, 1} ^ {\pi^ {\prime}} \left(s _ {1}\right), 0\right) \\ \leq \frac {\mathcal {V} + 1}{K} H \sqrt {K} + \frac {\mathcal {V} + 1}{K} \mathcal {O} \left(H ^ {2} \zeta \sqrt {d ^ {3} K}\right) \tag {14} \\ \end{array}
$$
Since  $\mathcal{V} = 2H / \gamma$ , and using the result of Lemma B.15, we have
$$
\max  \left(c - \frac {1}{K} \sum_ {k = 1} ^ {K} V _ {g, 1} ^ {\pi_ {k}} \left(s _ {1} ^ {k}\right), 0\right) \leq \frac {\mathcal {V} + 1}{K \mathcal {V}} \mathcal {O} \left(H ^ {2} \zeta \sqrt {d ^ {3} K}\right)
$$
In this section we proof Lemma B.5 and Lemma B.6.
B.2 Prepare for Lemma B.5
In order to bound  $\mathcal{T}_1$  and  $\mathcal{T}_2$ , we introduce the following lemma.
Lemma B.7. There exists a constant  $B_{2}$  such that for any fixed  $p \in (0,1)$ , with probability at least  $1 - p/2$ , the following event holds
$$
\left\| \sum_ {\tau = 1} ^ {k - 1} \phi_ {j, h} ^ {\tau} \left[ V _ {j, h + 1} ^ {k} \left(s _ {h + 1} ^ {\tau}\right) - \mathbb {P} _ {h} V _ {j, h + 1} ^ {k} \left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \right. \right\| _ {\left(\varsigma_ {h} ^ {k}\right) ^ {- 1}} \leq B _ {2} d H q
$$
for  $j \in \{r, g\}$ , where  $q = \sqrt{\log[4(B_1 + 1)\log(M)dT / p]}$  for some constant  $B_1$ .
We delay the proof of Lemma B.7 to Appendix B.4.
Lemma B.7 shows the bound of estimated value function  $V_{j,h}^{k}$  and value function  $V_{j,h}^{\pi}$  corresponding in a given policy at  $k$ . We now introduce the following lemma appeared in Ghosh et al. (2022). This lemma bounds the difference between the value function without bonus in L-UCBFair and the true value function of any policy  $\pi$ . This is bounded using their expected difference at next step, plus a error term.
Lemma B.8. Ghosh et al. (2022) There exists an absolute constant  $\beta = C_1dH\sqrt{\zeta},\zeta = \log (\log (M)4dT / p)$ , and for any fixed policy  $\pi$ , for the event defined in Lemma B.7, we have
$$
\left\langle \phi (s, a), w _ {j, h} ^ {k} \right\rangle - Q _ {j, h} ^ {\pi} (s, a) = \mathbb {P} _ {h} \left(V _ {j, h + 1} ^ {k} - V _ {j, h + 1} ^ {\pi}\right) (s, a) + \Delta_ {h} ^ {k} (s, a)
$$
for some  $\Delta_h^k (s,a)$  that satisfies  $\left|\Delta_h^k (s,a)\right|\leq \beta \sqrt{\phi(s,a)^T\left(\Lambda_h^k\right)^{-1}\phi(s,a)}$
Lemma B.9. Ghosh et al. (2022) With probability at least  $1 - p / 2$ , (for the event defined in Lemma B.7)
$$
Q _ {r, h} ^ {\pi} (s, a) + \nu_ {k} Q _ {g, h} ^ {\pi} (s, a) \leq Q _ {r, h} ^ {k} (s, a) + \nu_ {k} Q _ {g, h} ^ {k} (s, a) - \mathbb {P} _ {h} \left(V _ {h + 1} ^ {k} - V _ {h + 1} ^ {\pi , \nu_ {k}}\right) (s, a)
$$
We also introduce the following lemma. This lemma bound the value function by taking L-UCBFair policy and greedy policy.
Lemma B.10. Define  $\bar{V}_h^k (\cdot) = \max_a\left[Q_{r,h}^k (\cdot ,a) + \nu_kQ_{g,h}^k (\cdot ,a)\right]$  the value function corresponding to greedy policy, we have
$$
\bar {V} _ {h} ^ {k} (s) - V _ {h} ^ {k} (s) \leq \frac {\log M}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}}. \tag {15}
$$
Proof. Define  $a_{g}$  the solution of greedy policy,
$$
\begin{array}{l} \bar {V} _ {h} ^ {k} (s) - V _ {h} ^ {k} (s) = \left[ Q _ {r, h} ^ {k} (s, a _ {g}) + \nu_ {k} Q _ {g, h} ^ {k} (s, a _ {g}) \right] (16) \\ - \int_ {a} \pi_ {h, k} (a \mid s) \left[ Q _ {r, h} ^ {k} (s, a) + \nu_ {k} Q _ {g, h} ^ {k} (s, a) \right] d a (17) \\ \leq \left[ Q _ {r, h} ^ {k} (s, a _ {g}) + \nu_ {k} Q _ {g, h} ^ {k} (s, a _ {g}) \right] (18) \\ - \sum_ {i} \mathrm {S M} _ {\alpha} \left(I _ {i} \mid x\right) \left[ Q _ {r, h} ^ {k} (x, I _ {i}) + \nu_ {k} Q _ {g, h} ^ {k} (x, I _ {i}) \right] + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}} (19) \\ \leq \left(\frac {\log \left(\sum_ {a} \exp \left(\alpha \left(Q _ {r , h} ^ {k} (s , I _ {i}) + \nu_ {k} Q _ {g , h} ^ {k} (s , I _ {i})\right)\right)\right)}{\alpha}\right) (20) \\ - \sum_ {i} \mathrm {S M} _ {\alpha} \left(I _ {i} \mid s\right) \left[ Q _ {r, h} ^ {k} (s, I _ {i}) + \nu_ {k} Q _ {g, h} ^ {k} (s, I _ {i}) \right] + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{s}} (21) \\ \leq \frac {\log (M)}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}}. (22) \\ \end{array}
$$
The first inequality follows from Lemma B.14 and the second inequality holds because of Proposition 1 in Pan et al. (2019).
B.3 Proof of Lemma B.5
Now we're ready to proof Lemma B.5.
Proof. This proof simulates Lemma 3 in Ghosh et al. (2022).
We use induction to proof this lemma. At step  $H$ , we have  $Q_{j,H + 1}^k = 0 = Q_{j,H + 1}^\pi$  by definition. Under the event in Lemma B.13 and using Lemma B.8, we have for  $j = r,g$
$$
\left| \left\langle \phi (s, a), w _ {j, H} ^ {k} (s, a) \right\rangle - Q _ {j, H} ^ {\pi} (s, a) \right| \leq \beta \sqrt {\phi (s , a) ^ {T} \left(\Lambda_ {H} ^ {k}\right) ^ {- 1} \phi (s , a)}
$$
Thus  $Q_{j,H}^{\pi}(s,a) \leq \min \left\{\left\langle \phi (s,a),w_{j,H}^{k}\right\rangle +\beta \sqrt{\phi(s,a)^{T}\left(\Lambda_{H}^{k}\right)^{-1}\phi(s,a)},H\right\} = Q_{j,H}^{k}(s,a).$
From the definition of  $\bar{V}_h^k$
$$
\bar {V} _ {H} ^ {k} (s) = \max  _ {a} \left[ Q _ {r, H} ^ {k} (s, a) + \nu_ {k} Q _ {g, h} ^ {k} (s, a) \right] \geq \sum_ {a} \pi (a \mid x) \left[ Q _ {r, H} ^ {\pi} (s, a) + \nu_ {k} Q _ {g, H} ^ {\pi} (s, a) \right] = V _ {H} ^ {\pi , \nu_ {k}} (s)
$$
for any policy  $\pi$ . Thus, it also holds for  $\pi^{*}$ , the optimal policy. Using Lemma B.10 we can get
$$
V _ {H} ^ {\pi^ {*}, \nu_ {k}} (s) - V _ {H} ^ {k} (s) \leq \frac {\log M}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}}
$$
Now, suppose that it is true till the step  $h + 1$  and consider the step  $h$ . Since, it is true till step  $h + 1$ , thus, for any policy  $\pi$ ,
$$
\mathbb {P} _ {h} \left(V _ {h + 1} ^ {\pi , \nu_ {k}} - V _ {h + 1} ^ {k}\right) (s, a) \leq (H - h) \big (\frac {\log M}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}} \big)
$$
From (27) in Lemma 10 and the above result, we have for any  $(s,a)$
$$
Q _ {r, h} ^ {\pi} (s, a) + \nu_ {k} Q _ {g, h} ^ {\pi} (s, a) \leq Q _ {r, h} ^ {k} (s, a) + \nu_ {k} Q _ {g, h} ^ {k} (s, a) + (H - h) \big (\frac {\log M}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}} \big)
$$
$$
V _ {h} ^ {\pi , \nu_ {k}} (s) \leq \bar {V} _ {h} ^ {k} (s) + (H - h) \left(\frac {\log M}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}}\right)
$$
Now, again from Lemma 11, we have  $\bar{V}_h^k (s) - V_h^k (s)\leq \frac{\log(|A|)}{\alpha}$ . Thus,
$$
V _ {h} ^ {\pi , \nu_ {k}} (s) - V _ {h} ^ {k} (s) \leq (H - h + 1) \left(\frac {\log M}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}}\right)
$$
Now, since it is true for any policy  $\pi$ , it will be true for  $\pi^{*}$ . From the definition of  $V^{\pi,\nu_k}$ , we have
$$
\left(V _ {r, h} ^ {\pi^ {*}} (s) + \nu_ {k} V _ {g, h} ^ {\pi^ {*}} (s)\right) - \left(V _ {r, h} ^ {k} (s) + \nu_ {k} V _ {g, h} ^ {k} (s)\right) \leq (H - h + 1) \left(\frac {\log M}{\alpha} + 2 (1 + \mathcal {V}) H \rho \epsilon_ {I} \sqrt {\frac {d k}{\varsigma}}\right)
$$
Hence, the result follows by summing over  $K$  and considering  $h = 1$ .
B.4 Proof of Lemma B.7
We first define some useful sets. Let  $\mathcal{Q}_j = \left\{Q \mid Q(\cdot, a) = \min \left\{w_j^T \phi(\cdot, a) + \beta \sqrt{\phi^T(\cdot, a)^T \Lambda^{-1} \phi(\cdot, a)}, H\right\}, a \in \mathcal{A}\right\}$  be the set of  $Q$  functions, where  $j \in \{r, g\}$ . Since the minimum eigen value of  $\Lambda$  is no smaller than one so the Frobenius norm of  $\Lambda^{-1}$  is bounded.
Let  $\mathcal{V}_j = \left\{V_j \mid V_j(\cdot) = \int_a \pi(a \mid \cdot) Q_j(\cdot, a) da; Q_r \in \mathcal{Q}_r, Q_g \in \mathcal{Q}_g, \nu \in [0, \mathcal{V}]\right\}$  be the set of Q functions, where  $j \in \{r, g\}$ . Define
$$
\Pi = \left\{\pi \mid \forall a \in \mathcal {A}, \pi (a \mid \cdot) = \frac {1}{\int_ {b \in \mathcal {I} (a)} d b} \mathrm {S M} _ {\alpha} \left(Q _ {r} (\cdot , I (a)) + \nu Q _ {g} (\cdot , I (a))\right), Q _ {r} \in \mathcal {Q} _ {r}, Q _ {g} \in \mathcal {Q} _ {g}, \nu \in [ 0, \mathcal {V} ] \right\}
$$
the set of policies.
It's easy to verify  $V_{j}^{k}\in \mathcal{V}_{j}$
Then we introduce the proof of Lemma B.7. To proof Lemma B.7, we need the  $\epsilon$ -covering number for the set of value functions (Lemma B.13Ghosh et al. (2022)). To achieve this, we need to show if two  $Q$  functions and the dual variable  $\nu$  are close, then the bound of policy and value function can be derived (Lemma B.11, Lemma B.12). Though the proof of Lemma B.11 and Lemma B.12 are different from Ghosh et al. (2022), we show the results remain the same, thus Lemma B.13 still holds. We'll only introduce Lemma B.13 and omit the proof.
We now proof Lemma B.11.
Lemma B.11. Let  $\pi$  be the policy of  $L$ -UCBFair corresponding to  $Q_r^k + \nu_k Q_g^k$ , i.e.,
$$
\pi (a \mid \cdot) = \frac {1}{\int_ {b \in \mathcal {I} (a)} d b} S M _ {\alpha} \left(Q _ {r} (\cdot , I (a)) + \nu Q _ {g} (\cdot , I (a))\right) \tag {23}
$$
$$
\tilde {\pi} (a \mid \cdot) = \frac {1}{\int_ {b \in \mathcal {I} (a)} d b} S M _ {\alpha} \left(\tilde {Q} _ {r} (\cdot , I (a)) + \tilde {\nu} \tilde {Q} _ {g} (\cdot , I (a))\right), \tag {24}
$$
$$
i f \left| Q _ {j} - \tilde {Q} _ {j} \right| \leq \epsilon^ {\prime} a n d | \nu - \tilde {\nu} | \leq \epsilon^ {\prime}, t h e n \left| \int_ {a} (\pi (a \mid x) - \tilde {\pi} (a \mid x)) d a \right| \leq 2 \alpha \epsilon^ {\prime} (1 + \mathcal {V} + H).
$$
$$
\begin{array}{l} \left| \int_ {a} (\pi (a \mid x) - \tilde {\pi} (a \mid x)) d a \right| (25) \\ = \left| \sum_ {i = 1} ^ {M} \int_ {a \in \mathcal {I} _ {i}} (\pi (I (a) \mid x) - \tilde {\pi} (I (a) \mid x)) d a \right| \\ = \left| \sum_ {i = 1} ^ {M} \int_ {b \in \mathcal {I} _ {i}} d b (\pi (I _ {i} \mid x) - \tilde {\pi} (I _ {i} \mid x)) \right| \\ \leq \sum_ {i = 1} ^ {M} \left| S M _ {\alpha} \big (Q _ {r} (s, I _ {i}) + \nu Q _ {g} (s, I _ {i}) \big) - S M _ {\alpha} \big (\tilde {Q} _ {r} (s, I _ {i}) + \tilde {\nu} \tilde {Q} _ {g} (s, I _ {i}) \big) \right| \\ \leq 2 \alpha \left| Q _ {r} (\cdot , I (a)) + \nu Q _ {g} (\cdot , I (a)) - \tilde {Q} _ {r} (\cdot , I (a)) - \tilde {\nu} \tilde {Q} _ {g} (\cdot , I (a)) \right| (26) \\ \end{array}
$$
The last inequality holds because of Theorem 4.4 in Epasto et al. (2020). Using Corollary B.17, we have
$$
\left| \int_ {a} \left(\pi (a \mid x) - \tilde {\pi} (a \mid x)\right) d a \right| \leq 2 \alpha \epsilon^ {\prime} (1 + \mathcal {V} + H) \tag {27}
$$
Now since we have Lemma B.11, we can further bound the value functions.
Lemma B.12. If  $\left|\tilde{Q}_j - Q_j^k\right| \leq \epsilon'$ , where  $\tilde{Q}_j \in \mathcal{Q}_j$ , then there exists  $\tilde{V}_j \in \mathcal{V}_j$  such that
$$
\left| V _ {j} ^ {k} - \widetilde {V} _ {j} \right| \leq H 2 \alpha \epsilon^ {\prime} (1 + \mathcal {V} + H) + \epsilon^ {\prime},
$$
Proof. For any  $x$ ,
$$
\begin{array}{l} V _ {j} ^ {k} (s) - \widetilde {V} _ {j} (s) \\ = \left| \int_ {a} \pi (a \mid s) Q _ {j} ^ {k} (s, a) d a - \int_ {a} \tilde {\pi} (a \mid s) \tilde {Q} _ {j} (s, a) d a \right| \\ = \left| \int_ {a} \pi (a \mid s) Q _ {j} ^ {k} (s, a) d a - \int_ {a} \pi (a \mid s) \tilde {Q} _ {j} (s, a) d a + \int_ {a} \pi (a \mid s) \tilde {Q} _ {j} (s, a) d a - \int_ {a} \tilde {\pi} (a \mid s) \tilde {Q} _ {j} (s, a) d a \right| \\ \leq \left| \int_ {a} \pi (a \mid s) \left(Q _ {j} ^ {k} (s, a) - \tilde {Q} _ {j} (s, a)\right) d a \right| + \left| \int_ {a} \pi (a \mid s) \tilde {Q} _ {j} (s, a) d a - \int_ {a} \tilde {\pi} (a \mid s) \tilde {Q} _ {j} (s, a) d a \right| \\ \leq \epsilon^ {\prime} + H \left| \int_ {a} (\pi (a \mid s) - \tilde {\pi} (a \mid s)) d a \right| \\ \leq \epsilon^ {\prime} + H 2 \alpha \epsilon^ {\prime} (1 + \mathcal {V} + H) \\ \end{array}
$$
Using Lemmas above, we can have the same result presented in Lemma 13 of Ghosh et al. (2022) as following.
Lemma B.13. Ghosh et al. (2022) There exists a  $\tilde{V}_j \in \mathcal{V}_j$  parameterized by  $\left(\tilde{w}_r, \tilde{w}_g, \tilde{\beta}, \Lambda, \tilde{\mathcal{V}}\right)$  such that  $\text{dist}\left(V_j, \tilde{V}_j\right) \leq \epsilon$  where
$$
\left| V _ {j} - \tilde {V} _ {j} \right| = \sup  _ {x} \left| V _ {j} (s) - \tilde {V} _ {r} (s) \right|.
$$
Let  $N_{\epsilon}^{V_j}$  be the  $\epsilon$ -covering number for the set  $\mathcal{V}_j$ , then,
$$
\log N _ {\epsilon} ^ {V _ {j}} \leq d \log \left(1 + 8 H \frac {\sqrt {d k}}{\sqrt {\varsigma} \epsilon^ {\prime}}\right) + d ^ {2} \log \left[ 1 + 8 d ^ {1 / 2} \beta^ {2} / \left(\varsigma \left(\epsilon^ {\prime}\right) ^ {2}\right) \right] + \log \left(1 + \frac {\mathcal {V}}{\epsilon^ {\prime}}\right)
$$
where  $\epsilon^{\prime} = \frac{\epsilon}{H2\alpha(1 + \mathcal{V} + H) + 1}$
Lemma B.14.  $|Q_{j,h}^{k}(s,a) - Q_{j,h}^{k}(s,I(a))| \leq 2H\rho \epsilon_{I}\sqrt{\frac{dK}{\varsigma}}$ .
$$
\left| Q _ {j, h} ^ {k} (s, a) - Q _ {j, h} ^ {k} (s, I (a)) \right| \tag {28}
$$
$$
= \left| w _ {j, h} ^ {k} (s, a) ^ {T} \left(\phi (s, a) - \phi (s, I (a))\right) \right| \tag {29}
$$
$$
\leq \left\| w _ {j, h} ^ {k} (s, a) \right\| _ {2} \left\| \phi (s, a) - \phi (s, I (a)) \right\| _ {2} \tag {30}
$$
From Lemma B.16 and Assumption 3.2 we get the result.
B.5 Prior Results
Lemma B.15. Ding et al. (2021) Let  $\nu^{*}$  be the optimal dual variable, and  $C\geq 2\nu^{*}$ , then, if
$$
V _ {r, 1} ^ {\pi^ {*}} \left(s _ {1}\right) - V _ {r, 1} ^ {\pi} \left(s _ {1}\right) + C \left[ c - V _ {g, 1} ^ {\pi} \left(s _ {1}\right) \right] _ {+} \leq \delta ,
$$
$$
\left[ c - V _ {g, 1} ^ {\tilde {\pi}} \left(x _ {1}\right) \right] _ {+} \leq \frac {2 \delta}{C}.
$$
Lemma B.16. Jin et al. (2020) For any  $(k,h)$ , the weight  $w_{j,h}^{k}$  satisfies
$$
\left\| w _ {j, h} ^ {k} \right\| \leq 2 H \sqrt {d k / \varsigma}
$$
Corollary B.17. If  $\mathrm{dist}\left(Q_r,\tilde{Q}_r\right)\leq \epsilon '$ ,  $\mathrm{dist}\left(Q_g,\tilde{Q}_g\right)\leq \epsilon '.$  and  $|\tilde{\nu}_k - \nu_k|\leq \epsilon ',$  then,  $\mathrm{dist}\left(Q_r^k +\right.$ $\nu_{k}Q_{g}^{k},\tilde{Q}_{r} + \tilde{\nu}_{k}\tilde{Q}_{g})\leq \epsilon '(1 + \mathcal{V} + H).$
C Additional Figures
D.1 Experiment Details
Device and Packages. We run all the experiment on a single 1080Ti GPU. We implement the R-TD3 agent using StableBaseline3 (Raffin et al., 2021). The neural network is implemented using Pytorch (Paszke et al., 2019). Figures were generated using code included in the supplementary material in less than 24 hours on a single Nvidia RTX A5000.
Using a Neural Network to Learn  $\phi$ . We use a multi-layer perceptron to learn  $\phi$ . Specifically, we sample 100000 data points using a random policy, storing  $s$ ,  $a$ ,  $r$  and  $g$ . The inputs of the network are state and action, passing through fully connected (fc) layers with size 256, 128, 64, 64. ReLU is used as activation function between fc layers, while a SoftMax layer is applied after the last fc layer. We treat the outcome of this network as  $\phi$ . To learn  $\phi$ , we apply two separated fc layers (without bias) with size 1 to  $\hat{\phi}$  and treat the outputs as predicted  $r$  and predicted  $g$ . A combination of MSE losses of  $r$  and  $g$  are adopted. We use Adam as the optimizer. Weight decay is set to 1e-4 and learning rate is set to 1e-3, while batch size is 128.
Note that,  $\hat{\phi}$  is linear regarding  $r$  and  $g$ , but the linearity of transition kernel cannot be captured using such a schema. Therefore, equivalently we made an assumption that there always exists measure  $\mu_h$  such that for given  $\hat{\phi}$ , the linearity of transition kernel holds. It's a stronger assumption than Assumption 3.1.
D.2 Maximize True-Positives; Synthetic Data
D.3 Maximize True-Positives; "Adult" Data
Table 3: A comparison of baseline and RL policies in a semi-synthetic environment incorporating data synthesized from the UCI "Adult Data Set" (Dua and Graff, 2017), as detailed in section Appendix A.2. Policies share the objective of maximizing the cumulative true positive fraction of a binary classifications (Section 4), subject to three fairness constraints (Section 5.1) (columns). The qualitative difference between the synthetic environment (Table 2) and this, modified environment can be largely explained by the fact that  $\operatorname{Pr}(Y = 1 | X = x)$ ) is not actually monotonically increasing in  $x$ , as stipulated by Assumption 4.1. (Appendix A.2)