SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} = \max  \left\{\sum_ {t \in \mathcal {T}, r \in \{0, 1 \}} \mathbb {E} [ \{c _ {t} \beta_ {t | r} \} \pi_ {r} (X) \mathbb {E} [ Y \mid T = t, X ] \mathbb {I} [ X \in \mathcal {X} ^ {\mathrm {n o}} ] ] ]: \underline {{B}} \leq c _ {1} \leq \overline {{B}}, c _ {0} = 1 - c _ {1} \right\} \\ = \max  \left\{\sum_ {t \in \mathcal {T}, r \in \{0, 1 \}} \mathbb {E} [ \left\{c _ {t} \beta_ {t | r} \right\} \mathbb {E} [ Y \pi_ {r} (X) \mid T = t ] \mathbb {I} [ X \in \mathcal {X} ^ {\mathrm {n o}} ] ] ]: \underline {{B}} \leq c _ {1} \leq \bar {B}, c _ {0} = 1 - c _ {1} \right\} \quad \text {L O T E} \\ = \sum_ {t \in \mathcal {T}, r \in \{0, 1 \}} \mathbb {E} \left[ c _ {r t} ^ {*} \beta_ {t | r} \mathbb {E} \left[ Y \pi_ {r} (X) \mid T = t \right] \mathbb {I} \left[ X \in \mathcal {X} ^ {\mathrm {n o}} \right] \right] \\ \end{array}
$$
$$
\text {w h e r e} c _ {r t} ^ {*} = \left\{ \begin{array}{l l} \overline {{B}}   \mathbb {I}   [ t = 1 ] + \underline {{B}}   \mathbb {I}   [ t = 0 ] & \text {i f}   \mathbb {E} [ Y \pi_ {r} (X)   |   T = t ] \geq 0 \\ \overline {{B}}   \mathbb {I}   [ t = 0 ] + \underline {{B}}   \mathbb {I}   [ t = 1 ] & \text {i f}   \mathbb {E} [ Y \pi_ {r} (X)   |   T = t ] <   0 \end{array} \right.
$$
Proof of proposition 6.
$$
\begin{array}{l} \max  _ {\pi} \mathbb {E} [ c (\pi , T (\pi), Y (\pi)) \mathbb {I} [ X \notin \mathcal {X} ^ {\mathrm {n o}} ] ] + \mathbb {E} [ c (\pi , T (\pi), Y (\pi)) \mathbb {I} [ X \in \mathcal {X} ^ {\mathrm {n o}} ] ] (8) \\ \mathbb {E} [ T (\pi) \mathbb {I} [ X \notin \mathcal {X} ^ {\mathrm {n o}} ] \mid A = a ] - \mathbb {E} [ T (\pi) \mathbb {I} [ X \notin \mathcal {X} ^ {\mathrm {n o}} ] \mid A = b ] (9) \\ + \mathbb {E} [ T (\pi) \mathbb {I} [ X \in \mathcal {X} ^ {\mathrm {n o}} ] \mid A = a ] - \mathbb {E} [ T (\pi) \mathbb {I} [ X \in \mathcal {X} ^ {\mathrm {n o}} ] \mid A = b ] \leq \epsilon , \forall q _ {r 1} \in \mathcal {U} (10) \\ \end{array}
$$
$$
g _ {r} (x, u) = \left(\mu_ {r 1} (x, u) - \mu_ {r 0} (x, u)\right)
$$
then we can rewrite this further and apply the standard epigraph transformation:
max  $t$
$$
\begin{array}{l} t - \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \sum_ {u \in \{a, b \}} \sum_ {r \in \{0, 1 \}} \left\{g _ {r} (x, u) \pi_ {r} (x, u) f (x, u) \right\} q _ {r 1} (x, u) \} d x \leq V _ {o v} (\pi) + \mathbb {E} [ \mu_ {0} ], \forall q _ {r 1} \in \mathcal {U} \\ \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \{f (x \mid a) (\sum_ {r} \pi_ {r} (x, a) q _ {r 1} (x, a)) - f (x \mid b) (\sum_ {r} \pi_ {r} (x, b) q _ {r 1} (x, b)) \} + \mathbb {E} [ \Delta_ {o v} T (\pi) ] \leq \epsilon , \forall q _ {r 1} \in \mathcal {U} \\ \end{array}
$$
Project the uncertainty set onto the direct product of uncertainty sets:
$$
\begin{array}{l} t - \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \sum_ {u \in \{a, b \}} \sum_ {r \in \{0, 1 \}} \left\{g _ {r} (x, u) \pi_ {r} (x, u) f (x, u) \right\} q _ {r 1} (x, u) \} d x \leq V _ {o v} (\pi) + \mathbb {E} [ \mu_ {0} ], \forall q _ {r 1} \in \mathcal {U} _ {\infty} \\ \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \left\{f (x \mid a) \left(\sum_ {r} \pi_ {r} (x, a) q _ {r 1} (x, a)\right) - f (x \mid b) \left(\sum_ {r} \pi_ {r} (x, b) q _ {r 1} (x, b)\right) \right\} + \mathbb {E} \left[ \Delta_ {o v} T (\pi) \right] \leq \epsilon , \forall q _ {r 1} \in \mathcal {U} _ {\epsilon} \\ \end{array}
$$
Clearly robust feasibility of the resource parity constraint over the interval is obtained by the highest/lowest bounds for groups  $a, b$ , respectively:
$$
\begin{array}{l} t - \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \sum_ {u \in \{a, b \}} \sum_ {r \in \{0, 1 \}} \left\{g _ {r} (x, u) \pi_ {r} (x, u) f (x, u) \right\} q _ {r 1} (x, u) \} d x \leq V _ {o v} (\pi) + \mathbb {E} [ \mu_ {0} ], \forall q _ {r 1} \in \mathcal {U} _ {\infty} \\ \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \{f (x \mid a) (\sum_ {r} \pi_ {r} (x, a) \overline {{B}} _ {r} (x, a)) - f (x \mid b) (\sum_ {r} \pi_ {r} (x, b) \underline {{B}} _ {r} (x, u)) \} + \mathbb {E} [ \Delta_ {o v} T (\pi) ] \leq \epsilon \\ \end{array}
$$
$$
\delta_ {r 1} (x, u) = \frac {2 \left(q _ {r 1} (x , u) - \underline {{B}} _ {r} (x , u)\right)}{\overline {{B}} _ {r} (x , u) - \underline {{B}} _ {r} (x , u)} - \left(\overline {{B}} _ {r} (x, u) - \underline {{B}} _ {r} (x, u)\right),
$$
$$
\{\underline {{B}} _ {r} (x, u) \leq q _ {r 1} (x, u) \leq \bar {B} _ {r} (x, u) \} \Longrightarrow \{\| \delta_ {r 1} (x, u) \| _ {\infty} \leq 1 \}
$$
$$
q _ {r 1} (x, u) = \underline {{B}} _ {r} (x, u) + \frac {1}{2} (\bar {B} _ {r} (x, u) - \underline {{B}} _ {r} (x, u)) (\delta_ {r 1} (x, u) + 1).
$$
For brevity we denote  $\Delta B = (\overline{B}_r(x,u) - \underline{B}_r(x,u))$ , so
$$
\begin{array}{l} t + \max_{\substack{\| \delta_{r1}(x,u)\|_{\infty}\leq 1\\ r\in \{0,1\} ,u\in \{a,b\}}}\left\{-\int_{x\in \mathcal{X}^{\mathrm{no}}}\sum_{u\in \{a,b\}}\sum_{r\in \{0,1\}}\{g_{r}(x,u)\pi_{r}(x,u)f(x,u)\} \frac{1}{2}\Delta B(x,u)\delta_{r1}(x,u)dx\right\} -c_{1}(\pi)\leq V_{ov}(\pi) + \mathbb{E}_{\pi} \\ \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \left\{f (x \mid a) \left(\sum_ {r} \pi_ {r} (x, a) \bar {B} _ {r} (x, a)\right) - f (x \mid b) \left(\sum_ {r} \pi_ {r} (x, b) \underline {{B}} _ {r} (x, u)\right) \right\} + \mathbb {E} \left[ \Delta_ {o v} T (\pi) \right] \leq \epsilon , \\ \end{array}
$$
$$
c _ {1} (\pi) = \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \sum_ {u \in \{a, b \}} \sum_ {r \in \{0, 1 \}} \left\{g _ {r} (x, u) \pi_ {r} (x, u) f (x, u) \right\} \left(\underline {{B}} _ {r} (x, u) + \frac {1}{2} \left(\bar {B} _ {r} (x, u) - \underline {{B}} _ {r} (x, u)\right)\right) d x
$$
This is equivalent to:
$$
\begin{array}{l} t + \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \sum_ {u \in \{a, b \}} \sum_ {r \in \{0, 1 \}} | - g _ {r} (x, u) \pi_ {r} (x, u) f (x, u) | \frac {1}{2} \Delta B (x, u) d x - c _ {1} (\pi) \leq V _ {o v} (\pi) + \mathbb {E} [ \mu_ {0} ] \\ \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \{f (x \mid a) (\sum_ {r} \pi_ {r} (x, a) \overline {{B}} _ {r} (x, a)) - f (x \mid b) (\sum_ {r} \pi_ {r} (x, b) \underline {{B}} _ {r} (x, u)) \} + \mathbb {E} [ \Delta_ {o v} T (\pi) ] \leq \epsilon \\ \end{array}
$$
Undoing the epigraph transformation, we obtain:
$$
\begin{array}{l} \max  V _ {o v} (\pi) + \mathbb {E} [ \mu_ {0} ] + c _ {1} (\pi) - \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \sum_ {u \in \{a, b \}} \sum_ {r \in \{0, 1 \}} | - g _ {r} (x, u) \pi_ {r} (x, u) f (x, u) | \frac {1}{2} \Delta B (x, u) d x \\ \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \left\{f (x \mid a) \left(\sum_ {r} \pi_ {r} (x, a) \bar {B} _ {r} (x, a)\right) - f (x \mid b) \left(\sum_ {r} \pi_ {r} (x, b) \underline {{B}} _ {r} (x, u)\right) \right\} + \mathbb {E} \left[ \Delta_ {o v} T (\pi) \right] \leq \epsilon \\ \end{array}
$$
and simplifying the absolute value:
$$
\begin{array}{l} \max V _ {o v} (\pi) + \mathbb {E} [ \mu_ {0} ] + c _ {1} (\pi) - \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \sum_ {u \in \{a, b \}} \sum_ {r \in \{0, 1 \}} | g _ {r} (x, u) \pi_ {r} (x, u) f (x, u) | \frac {1}{2} \Delta B (x, u) d x \\ \int_ {x \in \mathcal {X} ^ {\mathrm {n o}}} \left\{f (x \mid a) \left(\sum_ {r} \pi_ {r} (x, a) \bar {B} _ {r} (x, a)\right) - f (x \mid b) \left(\sum_ {r} \pi_ {r} (x, b) \underline {{B}} _ {r} (x, u)\right) \right\} + \mathbb {E} \left[ \Delta_ {o v} T (\pi) \right] \leq \epsilon \\ \end{array}
$$
C.3 Proofs for general fairness-constrained policy optimization algorithm and analysis
We begin with some notation that will simplify some statemets. Define, for observation tuples  $O \sim (X, A, R, T, Y)$ , the value estimate  $v(Q; \eta)$  given some pseudo-outcome  $\psi(O; \eta)$  dependent on observation information and nuisance functions  $\eta$ . (We often suppress notation of  $\eta$  for brevity). We let estimators sub/super-scripted by 1 denote estimators from the first dataset.
$$
v ^ {(\cdot)} (Q) = \mathbb {E} _ {\pi \sim Q} [ \pi \psi_ {(\cdot)} \mid O ], \text {f o r} (\cdot) \in \{\emptyset , \mathrm {D R} \}
$$
$$
V ^ {(\cdot)} (Q) = \mathbb {E} [ v ^ {(\cdot)} (Q) ]
$$
$$
\hat {V} _ {1} ^ {(\cdot)} (Q) = \mathbb {E} _ {n _ {1}} [ v ^ {(\cdot)} (Q) ]
$$
$$
g _ {j} (O; Q) = \mathbb {E} _ {\pi \sim Q} \left[ g _ {j} (O; \pi) \mid O, \mathcal {E} _ {j} \right]
$$
$$
h _ {j} (Q) = \mathbb {E} [ g _ {j} (O; Q) \mid \mathcal {E} _ {j} ]
$$
$$
\hat {h} _ {j} ^ {1} (Q) = \mathbb {E} _ {n _ {1}} \left[ g _ {j} (O; Q) \mid \mathcal {E} _ {j} \right]
$$
C.3.1 Preliminaries: results from other works used without proof
Theorem 3 (Thm. 3, [1] (saddle point generalization bound (non-localized))). Let  $\rho \coloneqq \max_h \| M\hat{\mu}(h) - \hat{c} \|_{\infty}$ . Let Assumption 1 hold for  $C' \geq 2C + 2 + \sqrt{\ln(4/\delta)/2}$ , where  $\delta > 0$ . Let  $Q^{\star}$  minimize  $V(Q)$  subject to  $M\mu(Q) \leq c$ . Then Algorithm 1 with  $\nu \propto n^{-\alpha}$ ,  $B \propto n^{\alpha}$  and  $\eta \propto \rho^{-2}n^{-2\alpha}$  terminates in  $O\left(\rho^{2}n^{4\alpha}\ln |\mathcal{K}|\right)$  iterations and returns  $\hat{Q}$ . If  $np_j^{\star} \geq 8\log(2/\delta)$  for all  $j$ , then with probability at least  $1 - (|\mathcal{J}| + 1)\delta$  then for all  $k$ $\hat{Q}$  satisfies:
$$
V (\hat {Q}) \leq V (Q ^ {\star}) + \widetilde {O} \left(n ^ {- \alpha}\right)
$$
$$
\gamma_ {k} (\widehat {Q}) \leq c _ {k} + \frac {1 + 2 \nu}{B} + \sum_ {j \in \mathcal {J}} | M _ {k, j} | \widetilde {O} \left(\left(n p _ {j} ^ {\star}\right) ^ {- \alpha}\right)
$$
The proof of [1, Thm. 3] is modular in invoking Rademacher complexity bounds on the objective function and constraint moments, so that invoking standard Rademacher complexity bounds for off-policy evaluation/learning [5, 45] yields the above statement for  $V(\pi)$  (and analogously, randomized policies by [7, Thm. 12.2] giving stability for convex hulls of policy classes).
Lemma 2 (Lemma 4, [17]). Consider a function class  $\mathcal{F}:\mathcal{X}\to \mathbb{R}^d$ , with  $\sup_{f\in \mathcal{F}}\| f\|_{\infty ,2}\leq 1$  and pick any  $f^{*}\in \mathcal{F}$ . Assume that  $v(\pi)$  is L-Lipschitz in its first argument with respect to the  $\ell_2$  norm and let:
$$
Z _ {n} (r) = \sup  _ {Q \in \mathcal {Q}} \left\{\left| \mathbb {E} _ {n} [ \hat {v} (Q) - \hat {v} (Q ^ {*}) ] - \mathbb {E} [ v (Q) - v (Q ^ {*}) ] \right|: \mathbb {E} \left[ \left(\mathbb {E} _ {\pi \sim Q} [ v (\pi) ] - \mathbb {E} _ {\pi \sim Q ^ {*}} [ v (\pi) ]\right) ^ {2} \right] ^ {\frac {1}{2}} \leq r \right\}
$$
Then for some universal constants  $c_{1}, c_{2}$ :
$$
\Pr \left[ Z _ {n} (r) \geq 1 6 L \sum_ {t = 1} ^ {d} \mathcal {R} \left(r, \operatorname {c o n v} \left(\Pi_ {t}\right) - Q _ {t} ^ {*}\right) + u \right] \leq c _ {1} \exp \left\{- \frac {c _ {2} n u ^ {2}}{L ^ {2} r ^ {2} + 2 L u} \right\}
$$
Moreover, if  $\delta_{n}$  is any solution to the inequalities:
$$
\forall t \in \{1, \dots , d \}: \mathcal {R} \left(\delta ; \operatorname {s t a r} \left(\operatorname {c o n v} \left(\Pi_ {t}\right) - Q _ {t} ^ {*}\right)\right) \leq \delta^ {2}
$$
then for each  $r \geq \delta_n$ :
$$
P \left(Z _ {n} (r) \geq 1 6 L d r \delta_ {n} + u\right) \leq c _ {1} \exp \left\{- \frac {c _ {2} n u ^ {2}}{L ^ {2} r ^ {2} + 2 L u} \right\}
$$
Lemma 3 (Concentration of conditional moments ([1, 47]). For any  $j \in \mathcal{J}$ , with probability at least  $1 - \delta$ , for all  $Q$ ,
$$
\left| \widehat {h} _ {j} (Q) - h _ {j} (Q) \right| \leq 2 \mathcal {R} _ {n _ {j}} (\mathcal {H}) + \frac {2}{\sqrt {n _ {j}}} + \sqrt {\frac {\ln (2 / \delta)}{2 n _ {j}}}
$$
If  $np_j^* \geq 8\log(2/\delta)$ , then with probability at least  $1 - \delta$ , for all  $Q$ ,
$$
\left| \widehat {h} _ {j} (Q) - h _ {j} (Q) \right| \leq 2 \mathcal {R} _ {n p _ {j} ^ {*} / 2} (\mathcal {H}) + 2 \sqrt {\frac {2}{n p _ {j} ^ {*}}} + \sqrt {\frac {\ln (4 / \delta)}{n p _ {j} ^ {*}}}
$$
Lemma 4 (Orthogonality (analogous to [12] (Lemma 8), others)). Suppose the nuisance estimates satisfy a mean-squared-error bound
$$
\max _ {l} \{\mathbb {E} [ (\hat {\eta} _ {l} - \eta_ {l}) ^ {2} ] \} _ {l \in [ L ]} := \chi_ {n} ^ {2}
$$
Then w.p.  $1 - \delta$  over the randomness of the policy sample,
$$
V (Q _ {0}) - V (\hat {Q}) \leq O \left(R _ {n, \delta} + \chi_ {n} ^ {2}\right)
$$
C.4 Adapted lemmas
In this subsection we collect results similar to those that have appeared previously, but that require substantial additional argumentation in our specific saddle point setting.
Lemma 5 (Feasible vs. oracle nuisances in low-variance regret slices ([12], Lemma 9)). Consider the setting of Corollary 7. Suppose that the mean squared error of the nuisance estimates is upper bounded w.p.  $1 - \delta$  by  $h_{n,\delta}^2$  and suppose  $h_{n,\delta}^2 \leq \epsilon_n$ . Then:
$$
V_{2}^{0} = \sup_{\pi ,\pi^{\prime}\in \Pi_{*}\left(\epsilon_{n} + 2h_{n,\delta}^{2}\right)}\operatorname {Var}\left(v_{DR}^{0}(x;\pi) - v_{DR}^{0}\left(x;\pi^{\prime}\right)\right)
$$
Then  $V_{2}\leq V_{2}^{0} + O(h_{n,\delta})$
Proof of Theorem 1. We first study the meta-algorithm with "oracle" nuisance functions  $\eta = \eta_0$ .
$$
\Pi_ {2} \left(\epsilon_ {n}\right) = \left\{\pi \in \Pi : \mathbb {E} _ {n _ {1}} \left[ v (Q; \eta_ {0}) - v (\hat {Q} _ {1}; \eta_ {0}) \right] \leq \epsilon_ {n}, \mathbb {E} _ {n _ {1}} \left[ g _ {j} (O; \pi , \eta_ {0}) - g _ {j} \left(O; \hat {\pi} _ {1}, \eta_ {0}\right) \mid \mathcal {E} _ {j} \right] \leq \epsilon_ {n}, j \in \hat {\mathcal {I}} _ {1} \right\}
$$
$$
\mathcal {Q} _ {2} \left(\epsilon_ {n}\right) = \left\{Q \in \Delta \left(\Pi_ {2} \left(\epsilon_ {n}\right)\right) \right\}
$$
$$
\mathcal {Q} ^ {*} \left(\epsilon_ {n}\right) = \left\{ \right.Q \in \Delta (\Pi): \mathbb {E} \left[ \right.\left( \right.v (Q; \eta_ {0}) - v (Q ^ {*}; \eta_ {0}) \left. \right] \leq \epsilon_ {n}, \mathbb {E} \left[ g _ {j} (O; Q, \eta_ {0}) \mid \mathcal {E} _ {j} \right] - \mathbb {E} \left[ g _ {j} (O; Q ^ {*}, \eta_ {0}) \mid \mathcal {E} _ {j} \right] \leq \epsilon_ {n} \left. \right\}
$$
In the following, we suppress notational dependence on  $\eta_0$ .
Note that  $\hat{Q}_1\in \mathcal{Q}_2(\epsilon_n)$
Step 1: First we argue that w.p.  $1 - \delta /6$ ,  $Q^{*}\in \mathcal{Q}_{2}$
Invoking Theorem 3 on the output of the first stage of the algorithm, yields that with probability  $1 - \frac{\delta}{6}$  over the randomness in  $\mathcal{D}_1$ , by choice of  $\epsilon_n = \bar{O}(n^{-\alpha})$ ,
$$
V (\hat {Q} _ {1}) \leq V (Q ^ {*}) + \epsilon_ {n} / 2
$$
$$
\gamma_ {k} (\hat {Q} _ {1}) \leq d _ {k} + \sum_ {j \in \mathcal {J}} | M _ {k, j} | \widetilde {O} \left(\left(n p _ {j} ^ {*}\right) ^ {- \alpha}\right) \leq d _ {k} + \epsilon_ {n} / 2 \quad \text {f o r a l l} k
$$
Further, by Lemma 2,
$$
\sup  _ {Q \in \mathcal {Q}} | \mathbb {E} _ {n _ {1}} [ (v (Q) - v (Q ^ {*})) ] - \mathbb {E} [ (v (Q) - v (Q ^ {*})) ] | \leq \epsilon_ {n} / 2
$$
$$
\sup  _ {Q \in \mathcal {Q}} | \mathbb {E} _ {n _ {1}} [ (g (O; Q) - g (O; Q ^ {*})) ] - \mathbb {E} [ (g (O; Q) - g (O; Q ^ {*})) ] | \leq \epsilon_ {n} / 2
$$
Therefore, with high probability on the good event,  $Q^{*}\in \mathcal{Q}_{2}$
Step 2: Again invoking Theorem 3, this time on the output of the second stage of the algorithm with function space  $\Pi_2$  (hence implicitly  $Q_{2}$ ), and conditioning on the "good event" that  $Q^{*} \in Q_{2}$ , we obtain the bound that with probability  $\geq 1 - \delta / 3$  over the randomness of the second sample  $\mathcal{D}_2$ ,
$$
V (\hat {Q} _ {2}) \leq V (Q ^ {*}) + \epsilon_ {n} / 2
$$
$$
\gamma_ {k} \left(\hat {Q} _ {2}\right) \leq \gamma_ {k} \left(Q ^ {*}\right) + \epsilon_ {n} / 2
$$
Step 3: empirical small-regret slices relate to population small-regret slices, and variance bounds
We show that if  $Q \in \mathcal{Q}_2$ , then with high probability  $Q \in \mathcal{Q}_2^0$  (defined on small population value- and constraint-regret slices relative to  $\hat{Q}_1$  rather than small empirical regret slices)
$$
\mathcal {Q} _ {2} ^ {0} = \{Q \in \operatorname {c o n v} (\Pi) \colon \left| V (Q) - V (\hat {Q} _ {1}) \right| \leq \epsilon_ {n} / 2, \mathbb {E} [ g _ {j} (O; Q) - g _ {j} (O; \hat {Q} _ {1})) \mid \mathcal {E} _ {j} ] \leq \epsilon_ {n}, \forall j \}
$$
so that w.h.p.  $\mathcal{Q}_2\subseteq \mathcal{Q}_2^0$
Note that for  $Q \in \mathcal{Q}$ , w.h.p.  $1 - \delta / 6$  over the first sample, we have that
$$
\begin{array}{l} \sup  _ {Q \in \mathcal {Q}} \left| \mathbb {E} _ {n} [ v (Q) - v (\hat {Q} _ {1}) ] - \mathbb {E} [ v (Q) - v (\hat {Q} _ {1}) ] \right| \leq 2 \sup  _ {Q \in \mathcal {Q}} | \mathbb {E} _ {n} [ v (Q) ] - \mathbb {E} [ v (Q) ] | \leq \epsilon , \\ \sup  _ {Q \in \mathcal {Q}} \left| \mathbb {E} _ {n _ {1}} \left[ g _ {j} (O; Q) - g _ {j} (O; \hat {Q} _ {1}) \mid \mathcal {E} _ {j} \right] - \mathbb {E} \left[ g _ {j} (O; Q) - g _ {j} (O; \hat {Q} _ {1}) \mid \mathcal {E} _ {j} \right] \right| \\ \leq 2 \sup  _ {Q \in \mathcal {Q}} | \mathbb {E} _ {n _ {1}} [ g _ {j} (O; Q) \mid \mathcal {E} _ {j} ] - \mathbb {E} [ g _ {j} (O; Q) \mid \mathcal {E} _ {j} ] | \leq \epsilon , \forall j \\ \end{array}
$$
The second bound follows from [7, Theorem 12.2] (equivalence of Rademacher complexity over convex hull of the policy class) and linearity of the policy value and constraint estimators in  $\pi$ , and hence  $Q$ .
On the other hand since  $Q_{1}$  achieves low policy regret, the triangle inequality implies that we can contain the true policy by increasing the error radius. That is, for all  $Q \in \mathcal{Q}_2$ , with high probability  $\geq 1 - \delta / 3$ :
$$
\left. \right.\left| \mathbb {E} \left[\left(v (Q) - v \left(Q ^ {*}\right)\right)\right]\right| \leq \left| \mathbb {E} \left[\left(v (Q) - v \left(\hat {Q} _ {1}\right)\right)\right]\right| + \left| \mathbb {E} \left[\left(v \left(\hat {Q} _ {1}\right) - v \left(Q ^ {*}\right)\right)\right]\right| \leq \epsilon_ {n}
$$
$$
\left| \mathbb {E} \left[ g _ {j} (O; Q) - g _ {j} (O; Q ^ {*}) \mid \mathcal {E} _ {j} \right] \right| \leq \left| \mathbb {E} \left[ g _ {j} (O; Q) - g _ {j} (O; \hat {Q} _ {1}) \mid \mathcal {E} _ {j} \right] \right| + \left| \mathbb {E} \left[ g _ {j} (O; \hat {Q} _ {1}) - g _ {j} (O; Q ^ {*}) \mid \mathcal {E} _ {j} \right] \right| \leq \epsilon_ {n}
$$
Define the space of distributions over policies that achieve value and constraint regret in the population of at most  $\epsilon_{n}$ :
$$
\mathcal {Q} _ {*} (\epsilon_ {n}) = \{Q \in \mathcal {Q}: V (Q) - V (Q ^ {*}) \leq \epsilon_ {n}, \mathbb {E} \left[ g _ {j} (O; Q) - g _ {j} (O; Q ^ {*}) \mid \mathcal {E} _ {j} \right] \leq \epsilon_ {n}, \forall j \},
$$
so that on that high-probability event,
$$
\mathcal {Q} _ {2} ^ {0} \left(\epsilon_ {n}\right) \subseteq \mathcal {Q} _ {*} \left(\epsilon_ {n}\right). \tag {11}
$$
Then on that event with probability  $\geq 1 - \delta /3$
$$
\begin{array}{l} r _ {2} ^ {2} = \sup  _ {Q \in \mathcal {Q} _ {2}} \mathbb {E} [ (v (Q) - v (Q ^ {*})) ^ {2} ] \leq \sup  _ {Q \in \mathcal {Q} ^ {*} (\epsilon_ {n})} \mathbb {E} [ (v (Q) - v (Q ^ {*})) ^ {2} ] \\ = \sup  _ {Q \in \mathcal {Q} ^ {*} \left(\epsilon_ {n}\right)} \operatorname {V a r} (v (Q) - v \left(Q ^ {*}\right)) + \mathbb {E} [ (v (Q) - v \left(Q ^ {*}\right)) ] ^ {2} \\ \leq \sup  _ {Q \in \mathcal {Q} ^ {*} (\epsilon_ {n})} \operatorname {V a r} (v (Q) - v (Q ^ {*})) + \epsilon_ {n} ^ {2} \\ \end{array}
$$
$$
r _ {2} \leq \sqrt {\sup  _ {Q \in \mathcal {Q} _ {*} (\epsilon_ {n})} \operatorname {V a r} (v (Q) - v (Q ^ {*}))} + 2 \epsilon_ {n} = \sqrt {V _ {2}} + 2 \epsilon_ {n}
$$
Combining this with the local Rademacher complexity bound, we obtain that:
$$
\mathbb {E} [ v (\hat {Q} _ {2}) - v (Q ^ {*}) ] = O \left(\kappa \left(\sqrt {V _ {2}} + 2 \epsilon_ {n}, \mathcal {Q} _ {*} (\epsilon_ {n})\right) + \sqrt {\frac {V _ {2} \log (3 / \delta)}{n}}\right)
$$
These same arguments apply for the variance of the constraints
$$
V _ {2} ^ {j} = \sup  \left\{\operatorname {V a r} \left(g _ {j} (O; Q) - g _ {j} \left(O; Q ^ {\prime}\right)\right): Q, Q ^ {\prime} \in \mathcal {Q} _ {*} \left(\tilde {\epsilon} _ {n}\right) \right\}
$$
C.6 Proofs of auxiliary/adapted lemmas
Proof of Lemma 5. The proof is analogous to that of [12, Lemma 9] except for the step of establishing that  $\pi_{*}\in \mathcal{Q}_{\epsilon_{n} + O(\chi_{n,\delta}^{2})}^{0}$ : in our case we must establish relationships between saddlepoints under estimated vs. true nuisances. We show an analogous version below.
Define the saddle points to the following problems (with estimated vs. true nuisances):
$$
(Q _ {0, 0} ^ {*}, \lambda_ {0, 0} ^ {*}) \in \arg \min  _ {Q} \max  _ {\lambda} \mathbb {E} [ v _ {D R} (Q; \eta_ {0}) ] + \lambda^ {\top} (\gamma_ {D R} (Q; \eta_ {0}) - d) := L (Q, \lambda ; \eta_ {0}, \eta_ {0}) := L (Q, \lambda),
$$
$$
(Q _ {\eta , 0} ^ {*}, \lambda_ {\eta , 0} ^ {*}) \in \arg \min  _ {Q} \max  _ {\lambda} \mathbb {E} [ v _ {D R} (Q; \eta) ] + \lambda^ {\top} (\gamma_ {D R} (Q; \eta_ {0}) - d),
$$
$$
\left(Q ^ {*}, \lambda^ {*}\right) \in \arg \min  _ {Q} \max  _ {\lambda} \mathbb {E} \left[ v _ {D R} (Q; \eta) \right] + \lambda^ {\top} \left(\gamma_ {D R} (Q; \eta) - d\right).
$$
We have that:
$$
\begin{array}{l} \mathbb {E} \left[ v _ {D R} \left(Q ^ {*}\right) \right] \leq L \left(Q ^ {*}, \lambda^ {*}; \eta , \eta\right) + \nu \\ \leq L \left(Q ^ {*}, \lambda^ {*}; \eta , \eta_ {0}\right) + \nu + \chi_ {n, \delta} ^ {2} \\ \leq L \left(Q ^ {*}, \lambda^ {*}; \eta , \eta_ {0}\right) + \nu + \chi_ {n, \delta} ^ {2} \\ \leq L \left(Q ^ {*}, \lambda_ {\eta , 0} ^ {*}; \eta , \eta_ {0}\right) + \nu + \chi_ {n, \delta} ^ {2} \\ \leq L \left(Q _ {\eta , 0} ^ {*}, \lambda_ {\eta , 0} ^ {*}; \eta , \eta_ {0}\right) + \left| L \left(Q _ {\eta , 0} ^ {*}, \lambda_ {\eta , 0} ^ {*}; \eta , \eta_ {0}\right) - L \left(Q ^ {*}, \lambda_ {\eta , 0} ^ {*}; \eta , \eta_ {0}\right) \right| + \nu + \chi_ {n, \delta} ^ {2} \\ \leq L \left(Q _ {\eta , 0} ^ {*}, \lambda_ {\eta , 0} ^ {*}; \eta , \eta_ {0}\right) + \epsilon_ {n} + \nu + \chi_ {n, \delta} ^ {2} \\ \leq \mathbb {E} \left[ v _ {D R} \left(Q _ {\eta , 0} ^ {*}; \eta\right) \right] + \epsilon_ {n} + 2 \nu + \chi_ {n, \delta} ^ {2} \\ \leq \mathbb {E} [ v _ {D R} (Q _ {0, 0} ^ {*}; \eta) ] + \epsilon_ {n} + 2 \nu + \chi_ {n, \delta} ^ {2} \\ \end{array}
$$
by Lemma 4
by saddlepoint prop.
triangle ineq.
assuming  $\epsilon_{n}\geq \chi_{n,\delta}^{2}$
apx. complementary slackness
suboptimality
$$
\mathbb {E} \left[ v _ {D R} \left(Q ^ {*}; \eta\right) \right] - \mathbb {E} \left[ v _ {D R} \left(Q _ {0, 0} ^ {*}; \eta\right) \right] \leq \epsilon_ {n} + 2 \nu + \chi_ {n, \delta} ^ {2}.
$$
We generally assume that the saddlepoint suboptimality  $\nu$  is of lower order than  $\epsilon_{n}$  (since it is under our computational control).
Applying Lemma 4 gives;
$$
V \left(Q ^ {*}\right) - V \left(Q _ {0, 0} ^ {*}\right) \leq \epsilon_ {n} + 2 \nu + 2 \chi_ {n, \delta} ^ {2}.
$$
Define policy classes with respect to small-population regret slices (with a nuisance-estimation enlarged radius):
$$
\mathcal {Q} ^ {0} (\epsilon) = \left\{Q \in \Delta (\Pi): V \left(Q _ {0} ^ {*}\right) - V (Q) \leq \epsilon , \gamma \left(Q _ {0} ^ {*}\right) - \gamma (Q) \leq \epsilon \right\}
$$
$$
V _ {2} ^ {o b j} \leq \sup  _ {Q \in \mathcal {Q} ^ {0} (\epsilon_ {n})} \operatorname {V a r} (v _ {D R} (O; \pi) - v _ {D R} (O; \pi^ {*})),
$$
where we have shown that  $\pi^{*}\in \mathcal{Q}^{0}(\epsilon +2\nu +2\chi_{n,\delta}^{2})$
Following the result of the argumentation in [12, Lemma 9] from here on out gives the result.
D.1 Oregon Health Insurance Study
The Oregon Health Insurance Study [16] is an important study on the causal effect of expanding public health insurance on healthcare utilization, outcomes, and other outcomes. It is based on a randomized controlled trial made possible by resource limitations, which enabled the use of a randomized lottery to expand Medicaid eligibility for low-income uninsured adults. Outcomes of interest included health care utilization, financial hardship, health, and labor market outcomes and political participation.