SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is easy to verify that the bound  $N_4$  in (24) is less than the bound  $N_2$  in (23). Hence a worst-case bound is obtained by assuming that  $|\mathcal{X}(qao)| / N \geq \varepsilon /(4\overline{Q} AOH\overline{C})$  for each  $t \in [H]$  and each candidate state  $qao \in \mathcal{Q}_{c,t + 1}^{\prime}$ , which yields an upper bound  $N_2$ . Note that ADACT-H-A takes as input an upper bound  $\overline{Q}$  on the number of RDP states  $|\mathcal{Q}^{\prime}|$  of  $\mathbf{R}^{\prime}$ , as well as an upper bound  $\overline{C}$  of the concentrability coefficient  $C_{\mathbf{R}^{\prime}}^{*}$ . If the learning agent has no prior knowledge of  $\overline{Q}$  and  $\overline{C}$ , it could start with small estimates of  $\overline{Q}$  and  $\overline{C}$ , and in the case that ADACT-H-A returns Failure or the resulting policy has larger concentrability than  $\overline{C}$  for  $\mathbf{R}^{\prime}$ , iteratively double the estimates  $\overline{Q}$  and/or  $\overline{C}$  and call the algorithm again. This only increases the computational complexity of ADACT-H-A by a factor  $O(\log QC_{\mathbf{R}^{\prime}}^{*})$ , and the resulting upper bounds  $\overline{Q}$  and  $\overline{C}$  do not exceed  $2Q$  and  $2C_{\mathbf{R}^{\prime}}^{*}$ . Since we already have an estimate  $\overline{Q}$ , in each iteration we can call ADACT-H-A with  $\delta = \delta_1 / (2\overline{Q} AO)$  to ensure that the bound  $N_2$  holds for each candidate state simultaneously with probability  $1 - \delta_1$ . Substituting this value of  $\delta$  in the bound  $N_2$  in (23) and using  $\overline{Q} < 2Q$  and  $\overline{C} < 2C_{\mathbf{R}^{\prime}}^{*}$  yields the sample complexity bound stated in the theorem.
Lemma 15. The concentrability  $C_{\mathbf{R}'}^*$  of the approximate RDP  $\mathbf{R}'$  satisfies
$$
C _ {\mathbf {R} ^ {\prime}} ^ {*} \leq C _ {\mathbf {R}} ^ {*} (1 + 3 \bar {\mathcal {Q}} A O).
$$
Proof. For each  $t > 0$ , let  $d_t'(q_t^{\mathbf{e}})$  be the occupancy of the side state  $q_t^{\mathbf{e}}$  in the approximate RDP  $\mathbf{R}'$ . We prove by induction on  $t$  that  $d_t'(q_t^{\mathbf{e}})$  satisfies
$$
d _ {t} ^ {\prime} \left(q _ {t} ^ {\mathbf {e}}\right) <   \frac {\varepsilon \sum_ {u = 0} ^ {t - 1} \left| \mathcal {Q} _ {u} \right|}{2 \bar {Q} H \bar {C}} \leq \frac {\varepsilon}{2 H \bar {C}}.
$$
The base case is given by  $t = 1$ . In this case, a candidate state  $(q_0, ao)$  is mapped to  $q_1^{\mathrm{e}}$  if  $d_t^{\mathrm{b}}(q_0, ao) = d_t'(q_0, ao) < \varepsilon / (2\overline{Q}AOH\overline{C})$ . Since there can be at most  $AO = |\mathcal{Q}_0|AO$  such candidate states, we have
$$
d _ {t} ^ {\prime} \left(q _ {t} ^ {\mathrm {e}}\right) <   \frac {\varepsilon | \mathcal {Q} _ {0} | A O}{2 \overline {{Q}} A O H \overline {{C}}} = \frac {\varepsilon | \mathcal {Q} _ {0} |}{2 \overline {{Q}} H \overline {{C}}}.
$$
For  $t > 1$ , a candidate state  $(q_{t-1}, ao)$  is mapped to  $q_t^{\mathrm{e}}$  if  $d_t'(q_{t-1}, ao) < \varepsilon / (2\overline{Q}AOH\overline{C})$ . Again, there can be at most  $|Q_{t-1}|AO$  such candidate states. Since all occupancy of  $q_{t-1}^{\mathrm{e}}$  is also mapped to  $q_t^{\mathrm{e}}$ , we have
$$
d _ {t} ^ {\prime} (q _ {t} ^ {\mathbf {e}}) <   d _ {t - 1} ^ {\prime} (q _ {t - 1} ^ {\mathbf {e}}) + \frac {\varepsilon | \mathcal {Q} _ {t - 1} | A O}{2 \overline {{Q}} A O H \overline {{C}}} <   \frac {\varepsilon \sum_ {u = 0} ^ {t - 2} | \mathcal {Q} _ {u} |}{2 \overline {{Q}} H \overline {{C}}} + \frac {\varepsilon | \mathcal {Q} _ {t - 1} |}{2 \overline {{Q}} H \overline {{C}}} = \frac {\varepsilon \sum_ {u = 0} ^ {t - 1} | \mathcal {Q} _ {u} |}{2 \overline {{Q}} H \overline {{C}}},
$$
where we have used the induction hypothesis.
Consider a candidate state  $(q,ao)$  of  $\mathbf{R}$  at time  $t$ . Due to approximation, some histories in  $\bar{\tau}^{-1}(q)$  are mapped to side states in  $\mathbf{R}'$  instead of  $q$ , and we can therefore write  $d_t^{\mathrm{b}}(q,ao) = d_t'(q,ao) + \xi \leq d_t'(q,ao) + d_t'(q_{t}^{\mathrm{e}})$ , where  $\xi$  is the total occupancy of histories in  $\bar{\tau}^{-1}(q)$  mapped to side states. In turn, this implies
$$
d _ {t} ^ {*} (q, a o) \leq d _ {t} ^ {\flat} (q, a o) C _ {\mathbf {R}} ^ {*} \leq (d _ {t} ^ {\prime} (q, a o) + d _ {t} ^ {\prime} (q _ {t} ^ {\mathbf {e}})) C _ {\mathbf {R}} ^ {*} <   (d _ {t} ^ {\prime} (q, a o) + \frac {\varepsilon}{2 H \overline {{C}}}) C _ {\mathbf {R}} ^ {*}.
$$
The concentrability of a candidate state  $(q,ao)$  in the approximate RDP  $\mathbf{R}'$  that is not mapped to a side state (i.e.  $d_t'(q,ao) \geq \varepsilon/(6\overline{Q}AOH\overline{C}))$  can now be bounded as
$$
\frac {d _ {t} ^ {*} (q , a o)}{d _ {t} ^ {\prime} (q , a o)} <   \frac {d _ {t} ^ {\prime} (q , a o) + \varepsilon / (2 H \overline {{C}})}{d _ {t} ^ {\prime} (q , a o)} C _ {\mathbf {R}} ^ {*} = \left(1 + \frac {\varepsilon}{2 H \overline {{C}} d _ {t} ^ {\prime} (q , a o)}\right) C _ {\mathbf {R}} ^ {*} \leq C _ {\mathbf {R}} ^ {*} (1 + 3 \overline {{Q}} A O).
$$
This concludes the proof of the lemma.
C.4 Distinguishability Parameters
As defined in the main text, for  $t \in [H]$ , we consider a metric  $L$  over distributions on the remaining part of the episode  $\Delta(\mathcal{E}_{\ell})$ , for  $\ell = H - t$ . Then, the  $L$ -distinguishability of an RDP  $\mathbf{R}$  and a policy  $\pi$  is the maximum  $\mu_0$  such that, for any  $t \in [H]$  and any two distinct  $q, q' \in \mathcal{Q}_t$ , the probability distributions over suffix traces  $e_{t:H} \in \mathcal{E}_{\ell}$  from the two states satisfy
$$
L \left(\mathbb {P} \left(e _ {t: H} \mid q _ {t} = q, \pi\right), \mathbb {P} \left(e _ {t: H} \mid q _ {t} = q ^ {\prime}, \pi\right)\right) \geq \mu_ {0} \tag {25}
$$
So,  $\mu_0$  is a feature of the RDP and the policy combined and it quantifies the distance between any two distinct states of the RDP with respect to the distributions they induce over the observable quantities. Distinguishability parameters have been first introduced in Ron et al. [56] and later generalized for other metrics. They can be also found in Balle [59] for PDFA learning and in Ronca and De Giacomo [26], Ronca et al. [27] for RDP learning.
According to the definition we adopt, there exists an  $L$ -distinguishability for any RDP and policy. However, as stated in Assumption 2, we require  $\mu_0$  to be strictly positive. This does not constitute a restriction for the RDP, since it can be always minimized while preserving all conditional probabilities. Though it implies that, in any state, the behavior policy takes with positive probability all actions that are needed to observe episode suffixes that have different probability under the two states. Clearly if this was not the case for two distinct  $q, q' \in \mathcal{Q}_t$  at some  $t \in [H]$ ,  $\mathbb{P}(e_{t:H} \mid q_t = q, \pi) = \mathbb{P}(e_{t:H} \mid q_t = q', \pi)$  and no information would be available for the algorithm to distinguish  $q$  and  $q'$ . Assumption 2 is implied by Assumption 1. However, it becomes necessary for Theorem 8, since this does not rely on Assumption 1.
The metric selected also influences the actual value of the distinguishability parameter. In this paper, we adopt  $L_{\infty}^{\mathrm{p}}$ , as it can be seen from the TESTDISTINCT function in the two algorithms. A more standard distance would be  $L_{\infty}$ . According to Eq. (25), an  $L_{\infty}$ -distinguishability of  $\mu_0$  implies that for any  $t \in [H]$  and two distinct  $q, q' \in \mathcal{Q}_t$ ,
$$
\max  _ {e \in \mathcal {E} _ {H - t}} | \mathbb {P} \left(e _ {t: H} = e \mid q _ {t} = q\right) - \mathbb {P} \left(e _ {t: H} = e \mid q _ {t} = q ^ {\prime}\right) | \geq \mu_ {0}. \tag {26}
$$
This means that some sequence until the end of the episode has a different probability of being generated from the two states. Although similar, the  $L_{\infty}^{\mathsf{p}}$  distance, maximizes for the full trace as well as any of its prefixes as
$$
\max  _ {u \in [ H - t ], e \in \mathcal {E} _ {u}} | \mathbb {P} \left(e _ {t: H} = e * \mid q _ {t} = q\right) - \mathbb {P} \left(e _ {t: H} = e * \mid q _ {t} = q ^ {\prime}\right) | \geq \mu_ {0} \tag {27}
$$
As it has been discussed in Balle [59], Appendix A.5, the prefix  $L_{\infty}^{\mathsf{p}}$  metric always upper bounds the  $L_{\infty}$  metric, up to a multiplicative factor, while there are pairs of distributions in which  $L_{\infty}$  is exponentially smaller than  $L_{\infty}^{\mathsf{p}}$  with respect to the expected suffix length. This motivates our choice. Moreover, in the specific case of our fixed horizon setting, we have that the  $L_{\infty}^{\mathsf{p}}$ -distinguishability is never lower than  $L_{\infty}$ -distinguishability. Note that in the hard instance of Appendix E.2, the two coincide.
The lower bound is stated in terms of the  $L_1^{\mathsf{p}}$ -distinguishability of the RDP. While  $L_{\infty}^{\mathsf{p}}$  is achieved for one specific trace prefix maximizing the difference in probability,  $L_1^{\mathsf{p}}$  takes all traces into account as  $\sum_{u\in [H - t],e\in \mathcal{E}_u}|\mathbb{P}(e_{t:H} = e*|q_t = q) - \mathbb{P}(e_{t:H} = e*|q_t = q')|$ . Due to this relation, the  $L_{\infty}^{\mathsf{p}}$ -distinguishability always lower bounds the  $L_1^{\mathsf{p}}$ -distinguishability in the fixed horizon setting.
D RegORL with Subsampled VI-LCB
In this section we demonstrate the composition of our proposed algorithm with a specific Offline Reinforcement Learning algorithm for MDPs. Specifically, we adopt Subsampled VI-LCB, from Algorithm 3 of Li et al. [16] and report the combined sample complexity of this choice, through a simple application of Theorem 7.
First, we introduce the occupancy distribution and the single-policy concentrability coefficient for MDPs. Let  $\mathbf{M} = \langle \mathcal{Q},\mathcal{A},\mathcal{R},T,R,H\rangle$  be an MDP with states  $\mathcal{Q}$ , horizon  $H$ , transition function  $T:\mathcal{Q}\times \mathcal{A}\to \mathcal{Q}$  and reward function  $R:\mathcal{Q}\times \mathcal{A}\rightarrow \Delta (\mathcal{R})$ . The state-action occupancy distribution of a policy  $\pi :\mathcal{Q}\to \Delta (\mathcal{A})$  in  $\mathbf{M}$  at step  $t\in [H]$  is  $d_{\mathfrak{m},t}^{\pi}(q,a) = \mathbb{P}(q_t = q,a_t = a\mid \mathbf{M},\pi)$ . For our
purposes, it suffices to consider a fixed initial state  $q_{0}$ . Finally, the MDP single-policy concentrability of a policy  $\pi^{\mathrm{b}}$  is [15]:
$$
C ^ {*} = \max  _ {t \in [ H ], q \in \mathcal {Q}, a \in \mathcal {A}} \frac {d _ {\mathrm {m} , t} ^ {\pi^ {*}} (q , a)}{d _ {\mathrm {m} , t} ^ {\pi^ {\mathrm {b}}} (q , a)} \tag {28}
$$
We can now express the sample complexity of Subsampled VI-LCB.
Theorem 16 (Li et al. [16]). Let  $\mathcal{D}$  be a dataset of  $N_{\mathfrak{m}}$  episodes, sampled from an MDP  $\mathbf{M}$  with a Markov policy  $\pi^{\mathrm{b}}$ . For any  $\varepsilon \in (0,H]$  and  $0 < \delta < 1/12$ , with probability exceeding  $1 - \delta$ , the policy  $\widehat{\pi}$  returned by Subsampled VI-LCB obeys  $V_0^*(q_0) - V_0^{\widehat{\pi}}(q_0) \leq \varepsilon$ , as long as:
$$
N _ {\mathrm {m}} \geq \frac {c H ^ {3} Q C ^ {*} \log \frac {N _ {\mathrm {m}} H}{\delta}}{\varepsilon^ {2}} \tag {29}
$$
for a positive constant  $c$ .
The analysis in Li et al. [16] of Subsampled VI-LCB assumes that the reward function is deterministic and known. Thus, restricting our attention to this setting, we consider any episodic RDP with history-dependent, deterministic rewards. The reward function can be regarded as known, since it may be easily extracted from the dataset resulting from the Markov transformation of Definition 2.
Corollary 17. Let  $\mathcal{D}$  be a dataset of  $N$  episodes, sampled with a regular policy  $\pi^{\mathrm{b}}\in \Pi_{\mathbf{R}}$  from an RDP  $\mathbf{R}$  with deterministic rewards. If Subsampled VI-LCB is the OFFLINERL algorithm in Algorithm 1, then, for any  $\varepsilon \in (0,H]$  and  $0 < \delta < 1 / 12$ , with probability exceeding  $1 - \delta$ , the output of RegORL  $(\mathcal{D},\varepsilon ,\delta)$  is an  $\varepsilon$ -optimal policy of  $\mathbf{R}$ , as long as
$$
N \geq 2 \max  \left\{\frac {2 1 \log (8 Q A O / \delta)}{d _ {\operatorname* {m i n}} ^ {\mathrm {b}} \mu_ {0}} \sqrt {H \log (2 A R O)}, \frac {c H ^ {3} Q C _ {\mathbf {R}} ^ {*} \log \frac {2 N H}{\delta}}{\varepsilon^ {2}} \right\} \tag {30}
$$
Proof. This corollary follows as a direct application of Theorem 16 to Theorem 6. It only remains to verify that the single-policy concentrability of the MDP underlying the dataset  $\mathcal{D}'$  that Subsampled VI-LCB receives is  $C_{\mathbf{R}}^*$ . The dataset  $\mathcal{D}'$  is generated according to the Markov transformation  $\bar{\tau}$  from Definition 2. We only consider the cases in which ADACT-H succeeds. Let  $\pi \in \Pi_{\mathbf{R}}$  be any regular policy and  $q_t, q_t'$  the states reached at step  $t$  by  $\mathbf{R}$  and  $\mathbf{M}_{\mathbf{R}}$ , respectively. Then for  $t > 0$ ,
$$
\begin{array}{l} d _ {t} ^ {\pi} (q, a) := \mathbb {P} \left(q _ {t} = q \mid \mathbf {R}, \pi\right) \pi_ {\mathrm {r}} (q, a) (31) \\ = \mathbb {P} (\bar {\tau} (h _ {t - 1}) = q \mid \mathbf {R}, \pi) \pi_ {\mathrm {r}} (q, a) (32) \\ = \mathbb {P} \left(q _ {t} ^ {\prime} = q \mid \mathbf {M} _ {\mathbf {R}}, \pi_ {\mathrm {r}}\right) \pi_ {\mathrm {r}} (q, a) (33) \\ = d _ {\mathbf {m}, t} ^ {\pi_ {\mathrm {r}}} (q, a) (34) \\ \end{array}
$$
This is valid for any regular policy, and for the optimal and behavior policies in particular. Then,
$$
\begin{array}{l} C _ {\mathbf {R}} ^ {*} = \max  _ {t \in [ H ], q \in \mathcal {Q} _ {t}, a o \in \mathcal {A O}} \frac {d _ {t} ^ {*} (q , a o)}{d _ {t} ^ {\mathrm {b}} (q , a o)} (35) \\ = \max  _ {t \in [ H ], q \in \mathcal {Q} _ {t}, a o \in \mathcal {A O}} \frac {d _ {t} ^ {\pi^ {*}} (q , a) \theta_ {\mathrm {o}} (q , a , o)}{d _ {t} ^ {\pi^ {\mathrm {b}}} (q , a) \theta_ {\mathrm {o}} (q , a , o)} (36) \\ = \max  _ {t \in [ H ], q \in \mathcal {Q}, a \in \mathcal {A}} \frac {d _ {\mathrm {m} , t} ^ {\pi_ {\mathrm {r}} ^ {*}} (q , a)}{d _ {\mathrm {m} , t} ^ {\pi_ {\mathrm {r}} ^ {\mathrm {b}}} (q , a)} (37) \\ = C ^ {*} (38) \\ \end{array}
$$
Similarly to the previous corollary, it is also possible to combine Theorem 16 with Theorem 8. In this case, the sample complexity of Subsampled VI-LCB for learning an  $\varepsilon /2$ -accurate policy with probability  $1 - \delta /2$  would be combined with  $N_{\delta /2}^{\prime}$  of Theorem 8.
E Sample Complexity Lower Bound: Proof of Theorem 9
In this section, we prove the sample complexity lower bound in Theorem 9. The proof is based on a suitable composition of a two-armed bandit and an instance of the learning problem associated to noisy parity functions. We first describe this latter problem and its sample-complexity lower bound in Appendix E.1. Then, we compose a hard class of RDP instances in Appendix E.2, and prove the final statement in Appendix E.3.
E.1 Learning parity with noise
Let  $\mathbb{B} = \{0,1\}$  and  $L\in \mathbb{N}$ . For any string  $x\in \mathbb{B}^L$ , the parity function  $f_{x}:\mathbb{B}^{L}\to \mathbb{B}$  is  $f_{x}(y) = \oplus_{i\in [L - 1]}x_{i}y_{i}$ , where  $\oplus$  is addition modulo 2. For noise parameter  $\xi \in (0,0.5)$ , a noisy parity function  $f_{x,\xi}$  returns  $f_{x}(y)$  with probability  $0.5 + \xi$  and  $1 - f_{x}(y)$  otherwise. Consider the class of parity functions  $\mathbb{F}(L) = \{f_x\}_{x\in \mathbb{B}^L}$  and the class of noisy parity functions  $\mathbb{F}(L,\xi) = \{f_{x,\xi}\}_{x\in \mathbb{B}^L}$ . Assume that  $x,y_{1},y_{2},\ldots \sim \mathrm{unif}(\mathbb{B}^{L})$  are uniformly sampled. The success probability of a streaming algorithm  $\mathfrak{A}$  for  $\mathbb{F}(L,\xi)$  is the probability that  $\mathfrak{A}$  recovers  $x$ , given in input a sequence of observations  $(y_i,f_{x,\xi}(y_i))_i$ .
Lemma 18. Any streaming algorithm for  $\mathbb{F}(L,\xi)$  with a success probability higher than  $O(2^{-L})$  requires at least  $\Omega (L / \xi)$  or  $2^{\Omega (L)}$  input samples  $(y_{i},f_{x,\xi}(y_{i}))_{i}$ .
Proof. Learning in  $\mathbb{F}(L, \xi)$  is the problem of recovering  $x \in 2^{\mathbb{B}}$  from noisy data  $(y_i, b_i)$ , where  $b_i = f_x(y_i)$  with probability  $0.5 + \xi$ , and  $b_i = 1 - f_x(y_i)$  otherwise. This is the problem of learning in  $\mathbb{F}(L)$  with corruption rate  $0.5 - \xi$ . Hence, we focus on the problem of learning noiseless parity first.
The Statistical Query dimension  $\mathrm{SQDIM}(\mathcal{C},d)$ , characterizes the complexity of learning in the class  $\mathcal{C}$  with respect to the prior distribution  $d\in \Delta (\mathcal{C})$ . As defined in [73],  $\mathrm{SQDIM}(\mathcal{C},d)$  is the maximum  $n\in \mathbb{N}$  such that there exist distinct  $f_{1},\ldots ,f_{n}\in \mathcal{C}$ , such that their pairwise correlations with respect to  $d$  are between  $-1 / n$  and  $1 / n$ . For the class of parity functions, under the uniform distribution over  $x$ ,  $\mathrm{SQDIM}(\mathbb{F}(L),\mathrm{unif}) = 2^L$ . This was already observed in [74], for a slightly different notion of SQ dimension. However, to verify this, we can consider a natural ordering over binary strings in  $\mathcal{X}$ , and represent the problem of learning  $\mathbb{F}(L)$  as a matrix  $M = (m_{ij})\in \{1, - 1\}^{2^L\times 2^L}$ , defined as  $m_{ij} = (-1)^{f_{x_j}(y_i)} = (-1)^{y_i\cdot x_j}$ , where scalar product is modulo 2. We have that  $M$  is a Hadamard matrix. Then, since every row is orthogonal to the others, and the same is true for columns, every couple of parity functions are uncorrelated under the uniform distribution over  $x$ .
Regarding the noisy parity problem, since  $\mathrm{SQDIM}(\mathbb{F}(L),\mathrm{unif}) = 2^L$ , we can apply Corollary 8 of [75] with  $m = 2^L$ , to have that the matrix  $M$  corresponding to the parity problem is a  $(k,l) - L_{2}$ -extractor with error  $2^{-r}$ , for  $k,l,r\in \Omega (L)$ . Since  $M$  is a suitable extractor, we can apply Theorem 1 of [76], which considers the problem of learning  $M$  with the additional noise parameter  $\xi$ . We obtain that, in the streaming setting, any branching program  $B$  for  $\mathbb{F}(L,\xi)$  whose depth is at most  $2^{f_1(k,l,r)}$  and width is at most  $2^{ckl / \xi}$  has a success probability of at most  $O(2^{-f_1(k,l,r)})$ , where  $c$  is a suitable constant and  $f_{1}$  is equation (1) of [76].
Then, if the success probability is not in  $O(2^{-f_1(k,l,r)})$ , meaning it is higher, we have that the depth of  $\mathfrak{A}$  exceeds  $2^{f_1(k,l,r)}$  or the width of  $\mathfrak{A}$  exceeds  $2^{ckl / \xi}$ . Since  $k,l,r\in \Omega (L)$ , then, if the success probability is not in  $O(2^{-L})$ , the depth of  $\mathfrak{A}$  is  $2^{\Omega (L)}$  or the width of  $\mathfrak{A}$  is  $2^{\Omega (L^2) / \xi}$ . Width and depth refer to the computational model that represents  $\mathfrak{A}$  as a branching program. A branching program is a directed acyclic graph in which internal nodes have one outgoing edge for each possible input sample, that is  $|\mathbb{B}^L\times \mathbb{B}| = 2^{L + 1}$  in our problem, and leaves correspond to algorithm decisions. From the required width and depth we know that  $\mathfrak{A}$  has a leaf in layer  $2^{\Omega (L)}$  or in a layer that contains  $2^{\Omega (L^2) / \xi}$  nodes. The former case implies a worst case sample complexity requirement that is exponential in  $L$ . For the latter, we observe that in order to reach that width, at least  $\log_{2^{L + 1}}2^{\Omega (L^2 /\xi)}$  transitions and input samples, are required. This is  $\Omega (L / \xi)$ .
Figure 3: One episodic RDP instance  $\mathbf{R}_{101,1} \in \mathbb{R}(3,5,\xi,\eta)$ , associated to the parity function  $f_{101}$  and optimal arm  $a_1'$ . The transition function is represented by the arcs, labelled by observations (transitions do not depend on actions). The star denotes any symbol. If the label of a state  $q$  is  $a.d$ , then the observation function is  $\theta_{\mathrm{o}}(q,a) = d$ , where  $d \in \Delta(\mathcal{O})$  (some irrelevant outputs are omitted).
E.2 Class of hard RDP instances
For our main lower bound, we define a class of hard RDP instances. Figure 3 shows one possible instance in this class. We will soon define it formally, but we can observe that its structure is organized in two main paths. The two branches in the top part encode a parity computation according to some hidden code  $x \in \mathbb{B}^L$ , so that behaving optimally in that region requires to solve a parity problem (the one of Lemma 18). The bottom part reaches a two-armed bandit whose optimal action is  $c$ . The right-most states are winning or losing states that provide a positive and null reward accordingly.
Formally, we define a class of hard RDP instances as  $\mathbb{R}(L,H,\xi ,\eta) = \{\mathbf{R}_{x,c}\}_{x\in \mathbb{B}^L,c\in \{0,1\}}$  where  $\mathbf{R}_{x,c} = \langle \mathcal{Q},\mathcal{A}\mathcal{O},\Omega ,\tau ,\theta ,q_s,H\rangle$  for  $\mathcal{Q} = \{q_{s},q_{o_{\perp}}\} \cup \{q_{0i},q_{1i},q_{bi}\}_{i = 1,\dots,L}\cup \{q_{+i},q_{-i}\}_{i = L + 1,\dots ,H}$ ,  $\mathcal{A} = \{a_0',a_1'\}$ ,  $\mathcal{O} = \{0,1, + , - \}$ . Assume  $L\geq 1$  and  $H > L$ . Rewards are zero everywhere, except in the winning states
$$
\theta_ {r} (q, a, r) = \mathbb {1} _ {1} \text {i f} q = q _ {+ i} \text {w i t h} i > L, \mathbb {1} _ {0}, \text {o t h e r w i s e} \tag {39}
$$
where we recall that  $\mathbb{1}_x$  represents the deterministic distribution on  $x$ . For observation probabilities, we denote the distributions  $u(o) \coloneqq \mathrm{unif}\{0,1\}$  and
$$
v _ {\alpha} (o) := \left\{ \begin{array}{l l} \frac {1 + \alpha}{2} & \text {i f} o = + \\ \frac {1 - \alpha}{2} & \text {i f} o = - \\ 0 & \text {o t h e r w i s e} \end{array} \quad s (o) := \left\{ \begin{array}{l l} 1 / 4 & \text {i f} o = 0 \\ 1 / 4 & \text {i f} o = 1 \\ 1 / 2 & \text {i f} o = + \end{array} \right. \right. \tag {40}
$$
Now define observations as
$$
\theta_ {\circ} (q, a, o) = \left\{ \begin{array}{l l} s (o) & \text {i f} q = q _ {s} \\ u (o) & \text {i f} q \in \left\{q _ {0 i}, q _ {1 i} \right\} _ {i = 1, \dots , L - 1} \\ v _ {\xi} (o) & \text {i f} q = q _ {0 L} \wedge a = a _ {0} ^ {\prime} \text {o r} q = q _ {1 L} \wedge a = a _ {1} ^ {\prime} \\ v _ {- \xi} (o) & \text {i f} q = q _ {0 L} \wedge a = a _ {1} ^ {\prime} \text {o r} q = q _ {0 L} \wedge a = a _ {1} ^ {\prime} \\ \mathbb {1} _ {+} (o) & \text {i f} q = q _ {b i} \text {w i t h} i <   L \\ v _ {0} (o) & \text {i f} q = q _ {b L} \wedge a = a _ {0} ^ {\prime} \\ v _ {\eta} (o) & \text {i f} q = q _ {b L} \wedge a = a _ {1} ^ {\prime} \wedge c = 1 \\ v _ {- \eta} (o) & \text {i f} q = q _ {b L} \wedge a = a _ {1} ^ {\prime} \wedge c = 0 \\ \mathbb {1} _ {o _ {\perp}} (o) & \text {i f} q = q _ {o _ {\perp}} \end{array} \right. \tag {41}
$$
Finally, the transition function is defined such that  $\bar{\tau} (q_s,h_{L - 1}) = q_{iL}$  with  $i = f_{x}(o_{0:L - 1})$ , and
$$
\tau \left(q _ {k L}, a +\right) = q _ {+, L + 1} \quad \tau \left(q _ {k L}, a -\right) = q _ {-, L + 1} \quad \text {f o r} k = 1, 2 \tag {42}
$$
$$
\tau \left(q _ {b L}, a +\right) = q _ {+, L + 1} \quad \tau \left(q _ {b L}, a -\right) = q _ {-, L + 1} \tag {43}
$$
$$
\tau \left(q _ {+ i}, a o\right) = q _ {+, i + 1} \quad \tau \left(q _ {- i}, a o\right) = q _ {-, i + 1} \tag {44}
$$
$$
\tau \left(q _ {s}, a +\right) = q _ {b 1} \quad \tau \left(q _ {b i}, a o\right) = q _ {b, i + 1} \quad \text {f o r} i <   L \tag {45}
$$
$$
\tau \left(q _ {+ H}, a o\right) = q _ {o _ {\perp}} \quad \tau \left(q _ {- H}, a o\right) = q _ {o _ {\perp}} \quad \tau \left(q _ {o _ {\perp}}, a, o\right) = q _ {o _ {\perp}} \tag {46}
$$
In addition, the trnasitions  $\tau(q_{0i}, ao)$  and  $\tau(q_{1i}, ao)$ , for  $i < L$ , are defined according to  $o \in \{0, 1\}$  and the parity code  $x$ . Namely,  $\tau(q_{0i}, ao)$  equals  $q_{1,i+1}$  iff  $o \oplus x(i) = 1$ , and  $q_{0,i+1}$ , otherwise.  $\tau(q_{1i}, ao)$  is defined analogously.