SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 9. For any  $(C_{\mathbf{R}}^{*},H,\varepsilon ,\mu_{0})$  satisfying  $C_\mathbf{R}^*\geq 2$ ,  $H\geq 2$  and  $\varepsilon \le H\mu_0 / 64$ , there exists an RDP with horizon  $H$ ,  $L_{1}^{\mathrm{p}}$ -distinguishability  $\mu_0$  and a regular behavior policy  $\pi^{\mathrm{b}}$  with RDP single-policy concentrability  $C_{\mathbf{R}}^{*}$ , such that if  $\mathcal{D}$  has been generated using  $\pi^{\mathrm{b}}$  and  $\mathbf{R}$ , and
then, for any algorithm  $\mathfrak{A}:\mathcal{D}\mapsto \widehat{\pi}$  returning non-Markov deterministic policies, the probability that  $\widehat{\pi}$  is not  $\varepsilon$ -optimal is at least  $1 / 4$ .
Proof. Denote with  $\pi^{\mathrm{b}}$  a regular policy in  $\mathbf{R}$  and  $\mathcal{D} \in \mathbb{D}$  a dataset of episodes of length  $H$ , collected from  $\mathbf{R}$  and the behavior policy  $\pi^{\mathrm{b}}$ . For an RDP  $\mathbf{R}$ , let  $\Pi_{\mathrm{d}} = \mathcal{A}^{\mathcal{H}}$  be the set of deterministic non-Markov policies and  $\mathfrak{A} = \mathbb{D} \to \Pi_{\mathrm{d}}$  an offline RL algorithm. For  $\delta < 0.5$ , we say that an algorithm  $\mathfrak{A}$  is  $(\varepsilon, \delta)$ -PAC for the class of RDPs  $\mathbb{R}$  under  $\varphi$ , if, for every  $\mathbf{R} \in \mathbb{R}$  and  $\mathcal{D} \in \mathbb{D}$ , if the condition  $\varphi(\mathcal{D}, \pi^{\mathrm{b}})$  is verified, then the output policy  $\mathfrak{A}(\mathcal{D})$  is  $\varepsilon$ -optimal in  $\mathbf{R}$ , with probability  $1 - \delta$ . One notable case is that of  $\varphi$  requiring a minimum dataset size.
Since the output of a generic algorithm might be any generic non-Markov deterministic policy, we cannot restrict our attention to regular policies. We expand the value of a policy  $\pi : \mathcal{H} \to \mathcal{A}$  in a RDP  $\mathbf{R}_{x,c} \in \mathbb{R}(L, H, \xi, \eta)$  as follows:
$$
\begin{array}{l} \mathbb {E} \left[ V _ {0} ^ {\pi} \left(h _ {0}\right) \right] = \mathbb {E} \left[ \sum_ {i = 1} ^ {H} r _ {i} \mid \pi \right] (47) \\ = (H - L) \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid \pi\right) (48) \\ = (H - L) \sum_ {q \in \mathcal {Q} _ {L}} \mathbb {P} \left(q _ {L} = q \mid \pi\right) \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q, \pi\right) (49) \\ = (H - L) \left(\mathbb {P} \left(q _ {L} = q _ {0 L} \mid \pi\right) \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {0 L}, \pi\right) \right. \\ + \mathbb {P} \left(q _ {L} = q _ {1 L} \mid \pi\right) \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {1 L}, \pi\right)) \\ + (H - L) \left(\mathbb {P} \left(q _ {L} = q _ {b L} \mid \pi\right) \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {b L}, \pi\right)\right) (50) \\ = \frac {H - L}{2} \left(\mathbb {P} \left(q _ {L} = q _ {0 L} \mid o _ {0} \in \{0, 1 \}, \pi\right) \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {0 L}, \pi\right) \right. \\ + \mathbb {P} \left(q _ {L} = q _ {1 L} \mid o _ {0} \in \{0, 1 \}, \pi\right) \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {1 L}, \pi\right)) \\ + \frac {H - L}{2} \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {b L}, \pi\right) (51) \\ = \frac {H - L}{4} \left(\mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {0 L}, \pi\right) + \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {1 L}, \pi\right)\right) \\ + \frac {H - L}{2} \mathbb {P} \left(q _ {L + 1} = q _ {+, L + 1} \mid q _ {L} = q _ {b L}, \pi\right) (52) \\ = \frac {H - L}{4} \left(\mathbb {P} (a _ {L} = a _ {0} ^ {\prime} \mid q _ {L} = q _ {0 L}, \pi\right) \mathbb {P} (o _ {L} = + \mid q _ {L} = q _ {0 L}, a _ {L} = a _ {0} ^ {\prime}) \\ + \left(1 - \mathbb {P} \left(a _ {L} = a _ {0} ^ {\prime} \mid q _ {L} = q _ {0 L}, \pi\right)\right) \mathbb {P} \left(o _ {L} = + \mid q _ {L} = q _ {0 L}, a _ {L} = a _ {1} ^ {\prime}\right) \\ + \left(1 - \mathbb {P} \left(a _ {L} = a _ {1} ^ {\prime} \mid q _ {L} = q _ {1 L}, \pi\right)\right) \mathbb {P} \left(o _ {L} = + \mid q _ {L} = q _ {1 L}, a _ {L} = a _ {0} ^ {\prime}\right) \\ + \mathbb {P} \left(a _ {L} = a _ {1} ^ {\prime} \mid q _ {L} = q _ {1 L}, \pi\right) \mathbb {P} \left(o _ {L} = + \mid q _ {L} = q _ {1 L}, a _ {L} = a _ {1} ^ {\prime}\right)) \\ + \frac {H - L}{2} \left(\mathbb {P} \left(a _ {L} = a _ {0} ^ {\prime} \mid q _ {L} = q _ {b L}, \pi\right) \mathbb {P} \left(o _ {L} = + \mid q _ {L} = q _ {b L}, a _ {L} = a _ {0} ^ {\prime}\right) \right. \\ \end{array}
$$
$$
+ \mathbb {P} \left(a _ {L} = a _ {1} ^ {\prime} \mid q _ {L} = q _ {b L}, \pi\right) \mathbb {P} \left(o _ {L} = + \mid q _ {L} = q _ {b L}, a _ {L} = a _ {1} ^ {\prime}\right) \tag {53}
$$
where in Eq. (52) we have used the uniform probability over  $x$ . Now, for any history-dependent deterministic policy  $\pi$  in episodic RDPs, it is possible to identify an associated regular stochastic policy  $\pi_{\mathrm{r}}: \mathcal{Q}' \to \Delta(\mathcal{A})$ , where  $\mathcal{Q}' \coloneqq \mathcal{Q} \setminus \{q_{o_{\perp}}\}$  and:
$$
\pi_ {r} (q, a) := \mathbb {P} (\pi (h) = a \mid \bar {\tau} (h) = q) \tag {54}
$$
$$
= \sum_ {h ^ {\prime} \in \bar {\tau} ^ {- 1} (q)} \mathbb {I} \left(\pi \left(h ^ {\prime}\right) = a\right) \frac {\mathbb {P} \left(h = h ^ {\prime} \mid \pi\right)}{\mathbb {P} (q \mid \pi)} \tag {55}
$$
In other words,  $\pi_{\mathrm{r}}$  encodes the probability that  $\pi$  takes action  $a$ , given that some history has led to state  $q$ . With this convention, we resume from Eq. (53)
$$
\begin{array}{l} \mathbb {E} \left[ V _ {0} ^ {\pi} \left(h _ {0}\right) \right] = \frac {H - L}{4} \left(\pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right) v _ {\xi} (+) + \left(1 - \pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right)\right) v _ {\xi} (-) \right. \\ + \left(1 - \pi_ {r} \left(q _ {1 L}, a _ {1} ^ {\prime}\right)\right) v _ {\xi} (-) + \pi_ {r} \left(q _ {1 L}, a _ {1} ^ {\prime}\right) v _ {\xi} (+)) \\ + \frac {H - L}{2} \left(\pi_ {\mathrm {r}} \left(q _ {b L}, a _ {0} ^ {\prime}\right) u (+) + \pi_ {\mathrm {r}} \left(q _ {b L}, a _ {1} ^ {\prime}\right) \left(\mathbb {I} \left(c = a _ {1} ^ {\prime}\right) v _ {\eta} (+) + \mathbb {I} \left(c = a _ {0} ^ {\prime}\right) v _ {\eta} (-)\right) \right. (56) \\ = \frac {H - L}{8} \left(\pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right) (1 + \xi) + \left(1 - \pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right)\right) (1 - \xi) \right. \\ + \left(1 - \pi_ {\mathrm {r}} \left(q _ {1 L}, a _ {1} ^ {\prime}\right)\right) (1 - \xi) + \pi_ {\mathrm {r}} \left(q _ {1 L}, a _ {1} ^ {\prime}\right) (1 + \xi)) \\ + \frac {H - L}{4} \left(\pi_ {\mathrm {r}} \left(q _ {b L}, a _ {0} ^ {\prime}\right) + \pi_ {\mathrm {r}} \left(q _ {b L}, a _ {1} ^ {\prime}\right) \left(\mathbb {I} \left(c = a _ {1} ^ {\prime}\right) (1 + \eta) + \mathbb {I} \left(c = a _ {0} ^ {\prime}\right) (1 - \eta)\right)\right) (57) \\ = \frac {H - L}{4} \left(1 - \xi + \xi \pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right) + \xi \pi_ {\mathrm {r}} \left(q _ {1 L}, a _ {1} ^ {\prime}\right) \right. \\ + \pi_ {\mathrm {r}} \left(q _ {b L}, a _ {0} ^ {\prime}\right) + \pi_ {\mathrm {r}} \left(q _ {b L}, a _ {1} ^ {\prime}\right) \left(1 + \eta \mathbb {I} \left(c = a _ {1} ^ {\prime}\right) - \eta \mathbb {I} \left(c = a _ {0} ^ {\prime}\right)\right) (58) \\ \end{array}
$$
For the optimal policy in particular, this becomes:
$$
V ^ {*} = \frac {H - L}{4} \left(1 + \xi + \mathbb {I} \left(c = a _ {0} ^ {\prime}\right) + (1 + \eta) \mathbb {I} \left(c = a _ {1} ^ {\prime}\right)\right) \tag {59}
$$
From the  $\varepsilon$ -optimality of  $\pi = \mathfrak{A}(\mathcal{D})$ , then,
$$
\varepsilon \geq \mathbb {E} \left[ V _ {0} ^ {*} \left(h _ {0}\right) - V _ {0} ^ {\pi} \left(h _ {0}\right) \right] \tag {60}
$$
$$
\begin{array}{l} = \frac {H - L}{4} \left(2 \xi - \xi \pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right) - \xi \pi_ {\mathrm {r}} \left(q _ {1 L}, a _ {1} ^ {\prime}\right) \right. \\ + \eta \mathbb {I} \left(c = a _ {1} ^ {\prime}\right) \left(1 - \pi_ {\mathrm {r}} \left(q _ {b L}, a _ {1} ^ {\prime}\right)\right) + \eta \mathbb {I} \left(c = a _ {0} ^ {\prime}\right) \pi_ {\mathrm {r}} \left(q _ {b L}, a _ {1} ^ {\prime}\right)) \tag {61} \\ \end{array}
$$
$$
\begin{array}{l} = \frac {H - L}{4} \left(\xi \left(2 - \pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right) - \pi_ {\mathrm {r}} \left(q _ {1 L}, a _ {1} ^ {\prime}\right)\right) + \eta \left(1 - \pi_ {\mathrm {r}} \left(q _ {b L}, c\right)\right)\right) (62) \\ \geq \frac {H - L}{4} \max  \left\{\xi \left(1 - \pi_ {\mathrm {r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right)\right), \xi \left(1 - \pi_ {\mathrm {r}} \left(q _ {1 L}, a _ {1} ^ {\prime}\right)\right), \eta \left(1 - \pi_ {\mathrm {r}} \left(q _ {b L}, c\right)\right) \right\} (63) \\ \end{array}
$$
Now, assume that
$$
\min  \{\xi , \eta \} \geq \frac {1 6 \varepsilon}{H - L} \tag {64}
$$
Then, all of the following is true:  $\pi_{\mathrm{r}}(q_{0L},a_0^{\prime})\geq 3 / 4$ $\pi_{\mathrm{r}}(q_{1L},a_1^{\prime})\geq 3 / 4$ $\pi_{\mathrm{r}}(q_{bL},c)\geq 3 / 4$  . This means that, for small  $\varepsilon$  , any  $\varepsilon$  -optimal policy must frequently select the optimal action for both the parity problem and the bandit. Let us represent the first two events with  $B_{\mathfrak{p}}$  and the third with  $B_{\mathrm{b}}$  Since  $\mathfrak{A}$  is  $(\varepsilon ,\delta)$  -PAC for  $\mathbb{R}(L,H,\xi ,\eta)$  under  $\varphi$  , the probability of  $B_{\mathfrak{p}}\wedge B_{\mathfrak{b}}$  is at least  $1 - \delta$  , for any  $\mathcal{D}$  and  $\pi^{\mathrm{b}}$  satisfying  $\varphi (\mathcal{D},\pi^{\mathrm{b}})$
We proceed to compute the necessary data to satisfy both events with high probability. The dataset  $\mathcal{D}$  can be partitioned in two subsets  $\mathcal{D}_{\mathfrak{p}}$  and  $\mathcal{D}_{\mathrm{b}}$ , containing any episode from  $\mathcal{D}$  whose initial observation is  $\{0,1\}$  and  $+$ , respectively. The two datasets share no information and  $\mathcal{D}_{\mathfrak{p}}$  and  $\mathcal{D}_{\mathrm{b}}$  are mutually independent. To see this, we observe that the sequence  $a_{L + 1}r_{L + 1}o_{L + 1}\dots o_{H}$  is independent of
$a_0r_0o_0\ldots a_L$  given  $o_{L}$ , since  $+$  or  $-$  determines at step  $L$  determines the rest of the episode. Also, for any two episodes  $e_H,e_H^\prime$ , the sequence  $a_1r_1o_1\ldots o_L$  is independent of  $a_1^{\prime}r_1^{\prime}o_1^{\prime}\ldots o_L^{\prime}$  given  $o_0$ . Since,  $o_0\sim s$ , that is the starting distribution, the two datasets are independent. Let  $\mathcal{Q}_{\mathfrak{p}} = \{q_s,q_{o_\perp}\} \cup \{q_{0i},q_{1i}\}_{i = 1,\dots ,L}\cup \{q_{+,i},q_{-,i}\}_{i = L + 1,\dots ,H}$  and  $\mathcal{Q}_{\mathrm{b}} = \{q_s,q_{o_\perp}\} \cup \{q_{bi}\}_{i = 1,\dots ,L}\cup$ $\{q_{+,i},q_{-,i}\}_{i = L + 1,\dots ,H}$  be the reachable states in the two datasets. Then, we consider two separate classes  $\mathbb{R}(L,H,\xi)$  and  $\mathbb{R}(L,H,\eta)$  as the sets of RDPs in  $\mathbb{R}(L,H,\xi ,\eta)$ , restricted to  $\mathcal{Q}_{\mathfrak{p}}$  and  $\mathcal{Q}_{\mathrm{b}}$ , respectively. To do so we construct  $\mathbf{R}_r\in \mathbb{R}(L,H,\xi)$  and  $\mathbf{R}_c\in \mathbb{R}(L,H,\eta)$  such that the initial observation follows unif(\{0,1\}) in  $\mathbf{R}_r$  and  $\mathbb{1}_{+}$  in  $\mathbf{R}_c$ . Now, from the independence of the two datasets and the fact that  $\mathfrak{A}$  is  $(\varepsilon ,\delta)$ -PAC in  $\mathcal{D}$ , there must exist an algorithm  $\mathfrak{A}_{\mathfrak{p}}:\mathcal{D}_{\mathfrak{p}}\mapsto \pi_{\mathfrak{p}}$  that is  $(2\varepsilon ,\delta)$ -PAC in  $\mathbb{R}(L,H,\xi)$  under some  $\varphi_{\mathfrak{p}}$ , and  $\mathfrak{A}_{\mathfrak{b}}:\mathcal{D}_{\mathfrak{b}}\mapsto \pi_{\mathfrak{b}}$  that is  $(2\varepsilon ,\delta)$ -PAC in  $\mathbb{R}(L,H,\eta)$  under some  $\varphi_{\mathfrak{b}}$ . If this was not the case,  $B_{\mathfrak{p}}\wedge B_{\mathfrak{b}}$  could not be verified in one of the two terms.
We analyze  $\mathfrak{A}_{\mathfrak{p}}$  first and we show that its requirement  $\varphi_{\mathfrak{p}}$  is  $|\mathcal{D}_{\mathfrak{p}}| \in \Omega(L/\xi) \cup 2^{\Omega(L)}$ . For a contradiction, assume this is not the case and that  $|\mathcal{D}_{\mathfrak{p}}| = g(L,\xi) \notin \Omega(L/\xi) \cup 2^{\Omega(L)}$  is allowed. Then, we can use  $\mathfrak{A}_{\mathfrak{p}}$  to solve the noisy parity problem under the streaming setting with  $g(L,\xi)$  samples (this setting has been introduced in Appendix E.1). We proceed as follows. Consider any noisy parity function  $f_{x,\xi}$  with unknown  $x$ . Sample a sequence of strings  $\{y_i\}_i \in 2^L$  from the uniform distribution and collect  $g(L,\xi)$  pairs  $(y_i,p_i)$ , sampling  $p_i \sim f_{x,\xi}(y_i)$ . Then, for  $H > L$ , compose a dataset of episodes  $\{e_i\}_i$ . All actions of  $e_i$  are selected uniformly in  $\{a_0',a_1'\}$ . The observations  $o_{0:L-1}$  are  $y_i$  and  $o_L$  equals  $p_i$  if  $a_L = a_0'$ ,  $1 - p_i$ , otherwise (0 and 1 take roles of + and - symbols here). Rewards  $r_{L+1:H}$  are equal to one if  $o_L = 1$ , null otherwise. We obtain that dataset so constructed is equally likely under this procedure than under the uniform policy and the RDP  $\mathbf{R}_x \in \mathbb{R}(L,H,\xi)$ . Since  $\mathfrak{A}_{\mathfrak{p}}$  is  $(2\varepsilon,\delta)$ -PAC for  $\mathbb{R}(L,H,\xi)$ , with probability  $1 - \delta$ , the output policy  $\pi_{\mathfrak{p}}$  satisfies:
$$
\min  \left\{\pi_ {\mathrm {p r}} \left(q _ {0 L}, a _ {0} ^ {\prime}\right), \pi_ {\mathrm {p r}} \left(q _ {1 L}, a _ {1} ^ {\prime}\right) \right\} \geq 3 / 4 \tag {65}
$$
where  $\pi_{\mathrm{pr}}$  is the stochastic regular policy for  $\pi_{\mathfrak{p}}$ . This can be seen by our assumption in Eq. (64) and doubling both  $\varepsilon$  and the sub-optimality gap of Eq. (63), due to the updated probability for the initial observation. Then, for any sequence  $y \in 2^{L}$  and associated history  $h_{L - 1}$  with  $o_{0:L - 1} = y$ ,
$$
f _ {x} (y) = \underset {i = 0, 1} {\arg \max } \pi_ {\mathfrak {p}} \left(h _ {L - 1}, a _ {i} ^ {\prime}\right) \tag {66}
$$
which is the noiseless parity function based on  $x$ . This means that it is possible to reconstruct  $x$  solely by interacting with  $\pi_{\mathfrak{p}}$ , without collecting further samples. The solution we have described is a streaming algorithm with sample complexity  $g(L, \xi)$ . Since this contradicts Lemma 18, we have proven  $|\mathcal{D}_{\mathfrak{p}}| \in \Omega(L / \xi) \cup 2^{\Omega(L)}$ .
We now consider the bandit problem, which is solved by  $\mathfrak{A}_{\mathrm{b}}$ . Similarly to the previous case, from the  $\varepsilon$ -optimality of  $\mathfrak{A}_{\mathrm{b}}(\mathcal{D}_{\mathrm{b}})$ , we obtain the necessary condition:  $\pi_{\mathrm{br}}(q_{bL},c) \geq 3/4$  from Eq. (63). This condition is expressed for the stochastic policy  $\pi_{\mathrm{br}}$ , However, we notice that for  $q_{bL}$  in particular, the only possible history is  $h_{L-1} = a_0 + a_1 \ldots+$ , where all actions must also be deterministic. Then,
$$
\pi_ {\mathrm {b r}} \left(q _ {b L}, c\right) = \mathbb {P} \left(\pi_ {\mathrm {b}} (h) = c \mid \bar {\tau} (h) = q _ {b L}\right) = \mathbb {I} \left(\pi_ {\mathrm {b}} \left(h _ {L - 1}\right) = c\right) \tag {67}
$$
implying that  $\pi_{\mathrm{br}}$  can only be deterministic for  $q_{bL}$ . This means that  $\mathfrak{A}_{\mathrm{b}}$  must solve best-arm identification in the two arm bandit at  $q_{bL}$ . We can compose a simplified dataset that is relevant for the bandit as:
$$
\mathcal {D} _ {\mathrm {b}} ^ {\prime} = \left\{a _ {L} o _ {L}: e _ {H} \in \mathcal {D} _ {\mathrm {b}} \right\} \tag {68}
$$
Since  $\mathcal{D}_{\mathrm{b}}$  can be deterministically reconstructed from  $\mathcal{D}_{\mathrm{b}}^{\prime}$ , we have the following conditional independence:  $\pi_{\mathrm{b}} \perp c \mid \mathcal{D}_{\mathrm{b}}^{\prime}$ , where  $c \in \{a_0^{\prime}, a_1^{\prime}\}$  is the optimal arm, and  $\pi_{\mathrm{b}} = \mathfrak{A}_{\mathrm{b}}(\mathcal{D}_{\mathrm{b}})$  is the output of the algorithm. Denoting with  $\hat{c} = \pi_{\mathrm{b}}(h_{L-1})$  the selected arm, the error probability is  $P_e := \mathbb{P}(\hat{c} \neq c)$ . Applying Fano's inequality from Theorem 12 to the variables  $c \to \mathcal{D}_{\mathrm{b}}^{\prime} \to \hat{c}$  gives:
$$
\begin{array}{l} H _ {2} \left(P _ {e}\right) \geq H \left(c \mid \mathcal {D} _ {\mathrm {b}} ^ {\prime}\right) (69) \\ = H (c) - I \left(c; \mathcal {D} _ {\mathrm {b}} ^ {\prime}\right) = \log 2 - I \left(c; \mathcal {D} _ {\mathrm {b}} ^ {\prime}\right) (70) \\ \end{array}
$$
where we have used the fact that  $\hat{c}$  is a Bernoulli variable and the uniform prior over  $c$ . Now, assuming  $C \geq 2$ , we construct the following behavior policy:  $\pi^{\mathsf{b}}(q_{bL},a_0) = 1 - 1 / C$  and  $\pi^{\mathsf{b}}(q_{bL},a_1) = 1 / C$ . In the following, we write  $N_{\mathsf{b}} \coloneqq |\mathcal{D}_{\mathsf{b}}|$  and omit the implicit dependency on  $\pi^{\mathsf{b}}$ .
$$
\begin{array}{l} I (c; \mathcal {D} _ {\mathrm {b}} ^ {\prime}) = H \left(\mathcal {D} _ {\mathrm {b}} ^ {\prime}\right) - H \left(\mathcal {D} _ {\mathrm {b}} ^ {\prime} \mid c\right) (71) \\ = N _ {\mathrm {b}} \left(H \left(a _ {L} o _ {L}\right) - H \left(a _ {L} o _ {L} \mid c\right)\right) (72) \\ \end{array}
$$
$$
= N _ {\mathrm {b}} D _ {\mathrm {K L}} \left(\mathbb {P} \left(a _ {L} o _ {L}, c\right) \parallel \mathbb {P} \left(a _ {L} o _ {L}\right) \mathbb {P} (c)\right) \tag {73}
$$
$$
= \frac {N _ {\mathrm {b}}}{2} \sum_ {a, c ^ {\prime} \in \mathcal {A}, o \in \mathcal {O}} \mathbb {P} (a, o \mid c ^ {\prime}) \log \frac {\mathbb {P} (a , o \mid c ^ {\prime})}{\mathbb {P} (a , o)} \tag {74}
$$
$$
= \frac {N _ {\mathrm {b}}}{2} \sum_ {a, c ^ {\prime} \in \mathcal {A}, o \in \mathcal {O}} \mathbb {P} (a, o \mid c ^ {\prime}) \log \frac {\mathbb {P} (a \mid c ^ {\prime}) \mathbb {P} (o \mid c ^ {\prime} , a)}{\sum_ {c ^ {\prime \prime}} \mathbb {P} (a \mid c ^ {\prime \prime}) \mathbb {P} (o \mid c ^ {\prime \prime} , a) / 2} \tag {75}
$$
$$
= \frac {N _ {\mathrm {b}}}{2} \sum_ {a, c ^ {\prime} \in \mathcal {A}, o \in \mathcal {O}} \mathbb {P} (a, o \mid c ^ {\prime}) \log \frac {2 \mathbb {P} (o \mid c ^ {\prime} , a)}{\sum_ {c ^ {\prime \prime}} \mathbb {P} (o \mid c ^ {\prime \prime} , a)} \tag {76}
$$
$$
= \frac {N _ {\mathrm {b}}}{2} \sum_ {a, c ^ {\prime} \in \mathcal {A}, o \in \mathcal {O}} \mathbb {P} (a, o \mid c ^ {\prime}) \log \left(2 \mathbb {P} (o \mid c ^ {\prime}, a)\right) \tag {77}
$$
$$
= \frac {N _ {\mathrm {b}}}{2} \sum_ {c ^ {\prime} \in \mathcal {A}, o \in \mathcal {O}} \mathbb {P} \left(a _ {1} ^ {\prime}, o \mid c ^ {\prime}\right) \log \left(2 \mathbb {P} \left(o \mid c ^ {\prime}, a _ {1} ^ {\prime}\right)\right) \tag {78}
$$
$$
= \frac {N _ {\mathrm {b}}}{2} \sum_ {o \in \mathcal {O}} \left(\mathbb {P} \left(a _ {1} ^ {\prime}, o \mid c = a _ {0} ^ {\prime}\right) \log \left(2 \mathbb {P} \left(o \mid c = a _ {0} ^ {\prime}, a _ {1} ^ {\prime}\right)\right) + \mathbb {P} \left(a _ {1} ^ {\prime}, o \mid c = a _ {1} ^ {\prime}\right) \log \left(2 \mathbb {P} \left(o \mid c = a _ {1} ^ {\prime}, a _ {1} ^ {\prime}\right)\right)\right) \tag {79}
$$
$$
= N _ {\mathrm {b}} \left(\mathbb {P} \left(a _ {1} ^ {\prime}, + \mid c = a _ {0} ^ {\prime}\right) \log \left(2 \mathbb {P} (+ \mid c = a _ {0} ^ {\prime}, a _ {1} ^ {\prime})\right) + \mathbb {P} \left(a _ {1} ^ {\prime}, + \mid c = a _ {1} ^ {\prime}\right) \log \left(2 \mathbb {P} (+ \mid c = a _ {1} ^ {\prime}, a _ {1} ^ {\prime})\right)\right) \tag {80}
$$
$$
= N _ {\mathrm {b}} \left(\frac {1 - \eta}{2 C} \log (1 - \eta) + \frac {1 + \eta}{2 C} \log (1 + \eta)\right) \tag {81}
$$
$$
= \frac {N _ {\mathrm {b}}}{C} D _ {\mathrm {K L}} \left(v _ {\eta} \| v _ {0}\right) \tag {82}
$$
$$
\leq \frac {N _ {\mathrm {b}} \eta^ {2}}{C} \tag {83}
$$
Then from Eq. (70), and the fact that  $\mathfrak{A}_{\mathrm{b}}$  is  $(2\varepsilon, \delta)$ -PAC,
$$
H (\delta) \geq H _ {2} \left(P _ {e}\right) \geq \log 2 - \frac {N _ {\mathrm {b}} \eta^ {2}}{C} \tag {84}
$$
$$
\Rightarrow N _ {\mathrm {b}} \geq \frac {C}{\eta^ {2}} (\log 2 - H (\delta)) \tag {85}
$$
Which means that this must be  $\varphi_{\mathrm{b}}$  , the requirement for  $\mathfrak{A}_{\mathrm{b}}$
Finally, to compose the results from both branches, we observe that  $|\mathcal{D}| = |\mathcal{D}_{\mathfrak{p}}| + |\mathcal{D}_{\mathfrak{b}}|$ . Also, for any  $\delta \in (0, 0.5)$ , say  $1/4$ ,  $(\log 2 - H(\delta))$  becomes a positive constant, and we can add both sizes asymptotically:
$$
| \mathcal {D} | \in \Omega \left(\frac {H}{\xi} + \frac {C}{\eta^ {2}}\right) \tag {86}
$$
To relate the parameters to features of the RDP, we observe that the number of states of any RDP in  $\mathbb{R}(L,H,\xi ,\eta)$  is  $Q\leq 3H$  . Also, the behavior policy is uniform everywhere except in  $q_{bL}$  Assuming  $C\geq 2$  , the computation of the single-policy concentrability coefficient yields  $C_{\mathbf{R}}^{*} = C$  , for any  $c\in \{a_0^{\prime},a_1^{\prime}\}$  . Next, we compute the  $L_1^p$  -distinguishability of any RDP in this class. The  $L_{1}^{p}$  distinguishability of a set of states  $\mathcal{Q}$  is the minimum  $L_{1}$  distance in distribution between episodes prefixes that are generated starting from any two states in  $\mathcal{Q}$  . Let us consider the  $L_{1}$  norm for the pair  $q_{01}$  and  $q_{11}$
$$
\begin{array}{l} \| \mathbb {P} \left(e _ {1: H} \mid q _ {0 1}, \pi^ {\mathrm {b}}\right) - \mathbb {P} \left(e _ {1: H} \mid q _ {0 1}, \pi^ {\mathrm {b}}\right) \| _ {1} = (87) \\ = \sum_ {e \in \mathcal {E} _ {H - 1}} | \mathbb {P} \left(e _ {1: H} = e \mid q _ {0 1}\right) - \mathbb {P} \left(e _ {1: H} = e \mid q _ {1 1}\right) | (88) \\ = \sum_ {e a r o \in \mathcal {E} _ {L}} \mathbb {P} \left(e _ {1: L - 1} = e\right) \left| \mathbb {P} \left(a _ {L} = a, r _ {L} = r, o _ {L} = o \mid q _ {0 1}, e\right) - \mathbb {P} \left(a _ {L} = a, r _ {L} = r, o _ {L} = o \mid q _ {1 1}, e\right) \right| (89) \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {a o \in \mathcal {A O}} | \mathbb {P} (a _ {L} = a, o _ {L} = o \mid q _ {0 L}) - \mathbb {P} (a _ {L} = a, o _ {L} = o \mid q _ {1 L}) | (90) \\ = (1 / 2) \sum_ {o \in \mathcal {O}} | \mathbb {P} \left(o _ {L} = o \mid a _ {L} = a _ {0} ^ {\prime}, q _ {0 L}\right) - \mathbb {P} \left(o _ {L} = o \mid a _ {L} = a _ {0} ^ {\prime}, q _ {1 L}\right) | (91) \\ = + (1 / 2) \sum_ {o \in \mathcal {O}} | \mathbb {P} \left(o _ {L} = o \mid a _ {L} = a _ {1} ^ {\prime}, q _ {0 L}\right) - \mathbb {P} \left(o _ {L} = o \mid a _ {L} = a _ {1} ^ {\prime}, q _ {1 L}\right) | (92) \\ = \sum_ {o \in \mathcal {O}} | \mathbb {P} \left(o _ {L} = o \mid a _ {L} = a _ {0} ^ {\prime}, q _ {0 L}\right) - \mathbb {P} \left(o _ {L} = o \mid a _ {L} = a _ {0} ^ {\prime}, q _ {1 L}\right) | (93) \\ = 2 \left| \mathbb {P} \left(o _ {L} = + \mid a _ {L} = a _ {0} ^ {\prime}, q _ {0 L}\right) - \mathbb {P} \left(o _ {L} = + \mid a _ {L} = a _ {0} ^ {\prime}, q _ {1 L}\right) \right| (94) \\ = 2 \xi (95) \\ \end{array}
$$