SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the MDP  $M^{\star}$ , there is a unique state  $s_{h_1}$  which  $\tau$  must visit. In the layers between  $h_1$  and  $h_2$ , all trajectories are only allowed take 1 on states with index  $\mathrm{idx}(s_{h_1})$ , but they are not required to. Thus we can compute that the contribution to  $C_{h_2}(m)$  from trajectories with the fixed layer decomposition to be at most  $2H$ . The reasoning is as follows. At  $h_1$ , there is exactly one  $(s,a)$  pair which is reachable by trajectories with this fixed layer decomposition, since any  $\tau$  must take  $a = 1$  at  $s_{h_1}$ . Subsequently we can add at most two reachable pairs in every layer  $h \in \{h_1 + 1,\dots,h_2 - 1\}$  due to encountering a state  $j[h]$  or  $j'[h]$  where  $j = \mathrm{idx}(s_{h_1})$ , and at layer  $h_2$  we must play  $a = 1$ , for a total of  $1 + 2(h_2 - h_1 - 1) \leq 2H$ . Using similar reasoning the contribution to  $C_{h_3}(m)$  from trajectories with this fixed layer decomposition is at most  $(2H) \cdot (4H)$ , and so on. Continuing in this way, we have the final bound of  $\prod_{k=1}^{m}(2kH)$ . Since this holds for a fixed choice of  $(h_1,\dots,h_m)$  in total we have  $C_H(m) \leq H^m \cdot \prod_{k=1}^{m}(2kH) = \mathcal{O}(H^{4m})$ .
When  $m \geq \ell + 1$ , observe that the block-free property on  $B$  implies that for any  $J \subseteq [2^H]$  with  $|J| = \ell$  we have  $|\cap_{j \in J} \Pi_j| \leq \ell \log 2^{2H}$ . So for any trajectory  $\tau$  with layer decomposition such that  $m \geq \ell$  we can redo the previous analysis and argue that there is at most  $\ell \log 2^{2H}$  multiplicative factor contribution to the value  $C_H(m)$  due to all trajectories which have layer decompositions longer than  $\ell$ . Thus we arrive at the bound  $\sum_{m \geq \ell + 1} C_H(m) \leq \mathcal{O}(H^{4\ell}) \cdot \ell \log 2^{2H} \leq \mathcal{O}(\ell \cdot H^{4\ell + 1})$ .
E.5 Proof of Lemma 3
The proof of this stopping time lemma follows standard machinery for PAC lower bounds (Garivier et al., 2019; Domingues et al., 2021; Sekhari et al., 2021). In the following we use  $\mathrm{KL}(P\| Q)$  to denote the Kullback-Leibler divergence between two distributions  $P$  and  $Q$  and  $\mathrm{kl}(p\| q)$  to denote the Kullback-Leibler divergence between two Bernoulli distributions with parameters  $p, q \in [0,1]$ .
For any  $\pi^{\star} \in \Pi^{(\ell)}$  we denote the random variable
$$
N ^ {\pi^ {\star}} = \sum_ {t = 1} ^ {\eta \wedge T _ {\max }} \mathbb {1} \left\{\mathbb {A} _ {t} (\operatorname {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\operatorname {i d x} (s _ {1}) _ {1: H}) \text {a n d} \operatorname {i d x} (s _ {1}) \in \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}} \right\},
$$
the number of episodes for which the algorithm's policy at round  $t \in [\eta \wedge T_{\max}]$  matches that of  $\pi^{\star}$  on a certain relevant state of  $\pi^{\star}$ .
In the sequel we will prove upper and lower bounds on the intermediate quantity  $\sum_{\pi^{\star} \in \Pi} \mathbb{E}_0\left[N^{\pi^{\star}}\right]$  and relate these quantities to  $\mathbb{E}_0[\eta]$ .
Step 1: Upper Bound. First we prove an upper bound. We can compute that
$$
\begin{array}{l} \sum_ {\pi^ {*} \in \Pi} \mathbb {E} _ {0} \left[ N ^ {\pi^ {*}} \right] \\ = \sum_ {t = 1} ^ {T _ {\max }} \sum_ {\pi^ {\star} \in \Pi} \mathbb {E} _ {0} \left[ \mathbb {1} \left\{\eta > t - 1 \right\} \mathbb {1} \left\{\mathbb {A} _ {t} (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \text {a n d} \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\operatorname {r e l}} ^ {\pi^ {\star}} \right\} \right] \\ = \sum_ {t = 1} ^ {T _ {\max }} \mathbb {E} _ {0} \left[ \mathbb {1} \left\{\eta > t - 1 \right\} \sum_ {\pi^ {\star} \in \Pi} \mathbb {1} \left\{\mathbb {A} _ {t} (\operatorname {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\operatorname {i d x} (s _ {1}) _ {1: H}) \text {a n d} \operatorname {i d x} (s _ {1}) \in \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}} \right\} \right] \\ \stackrel {(i)} {\leq} \sum_ {t = 1} ^ {T _ {\max }} \mathbb {E} _ {0} [ \mathbb {1} \{\eta > t - 1 \} ] \leq \mathbb {E} _ {0} [ \eta \wedge T _ {\max } ] \leq \mathbb {E} _ {0} [ \eta ]. \tag {9} \\ \end{array}
$$
Here, the first inequality follows because for every index  $j$  and every  $\pi^{\star} \in \Pi_{j}^{(\ell)}$ , each  $\pi^{\star}$  admits a unique sequence of actions (by Property (3) of Lemma 2), so any policy  $\mathbb{A}_t$  can completely match with at most one of the  $\pi^{\star}$ .
Step 2: Lower Bound. Now we turn to the lower bound. We use a change of measure argument.
$$
\begin{array}{l} \mathbb {E} _ {0} \left[ N ^ {\pi^ {\star}} \right] \stackrel {(i)} {\geq} \mathbb {E} _ {0, \pi^ {\star}} \left[ N ^ {\pi^ {\star}} \right] - T _ {\max } \Delta (T _ {\max }) \\ = \frac {1}{| \Phi |} \sum_ {\phi \in \Phi} \mathbb {E} _ {0, \pi^ {\star}, \phi} \left[ N ^ {\pi^ {\star}} \right] - T _ {\max} \Delta (T _ {\max}) \\ \stackrel {(i i)} {\geq} \frac {1}{7} \cdot \frac {1}{| \Phi |} \sum_ {\phi \in \Phi} \mathrm {K L} \left(\mathbb {P} _ {0, \pi^ {\star}, \phi} ^ {\mathcal {F} _ {\eta \wedge T _ {\max }}} \| \mathbb {P} _ {\pi^ {\star}, \phi} ^ {\mathcal {F} _ {\eta \wedge T _ {\max }}}\right) - T _ {\max } \Delta (T _ {\max }) \\ \stackrel {(i i i)} {\geq} \frac {1}{7} \cdot \operatorname {K L} \left(\mathbb {P} _ {0, \pi^ {\star}} ^ {\mathcal {F} _ {\eta \wedge T _ {\max }}} \| \mathbb {P} _ {\pi^ {\star}} ^ {\mathcal {F} _ {\eta \wedge T _ {\max }}}\right) - T _ {\max } \Delta (T _ {\max }) \\ \end{array}
$$
The inequality  $(i)$  follows from a change of measure argument using Lemma 6, with  $\Delta (T_{\mathrm{max}}):= T_{\mathrm{max}}^{2} / 2^{H + 3}$ . Here,  $\mathcal{F}_{\eta \wedge T_{\mathrm{max}}}$  denotes the natural filtration generated by the first  $\eta \wedge T_{\mathrm{max}}$  episodes. The inequality  $(ii)$  follows from Lemma 7, using the fact that  $M_{0,\pi^{\star},\phi}$  and  $M_{\pi^{\star},\phi}$  have identical transitions and only differ in rewards at layer  $H$  for the trajectories which reach the end of a relevant combination lock. The number of times this occurs is exactly  $N^{\pi^{\star}}$ . The factor  $1 / 7$  is a lower bound on  $\mathrm{kl}(1 / 2\| 3 / 4)$ . The inequality  $(iii)$  follows by the convexity of KL divergence.
Now we apply Lemma 8 to lower bound the expectation for any  $\mathcal{F}_{\eta \wedge T_{\max}}$  -measurable random variable  $Z\in [0,1]$  as
$$
\begin{array}{l} \mathbb {E} _ {0} \left[ N ^ {\pi^ {\star}} \right] \geq \frac {1}{7} \cdot \operatorname {k l} \left(\mathbb {E} _ {0, \pi^ {\star}} [ Z ] \| \mathbb {E} _ {\pi^ {\star}} [ Z ]\right) - T _ {\max } \Delta \left(T _ {\max }\right) \\ \geq \frac {1}{7} \cdot (1 - \mathbb {E} _ {0, \pi^ {\star}} [ Z ]) \log \left(\frac {1}{1 - \mathbb {E} _ {\pi^ {\star}} [ Z ]}\right) - \frac {\log (2)}{7} - T _ {\max} \Delta (T _ {\max}), \\ \end{array}
$$
where the second inequality follows from the bound  $\mathrm{kl}(p\| q)\geq (1 - p)\log (1 / (1 - q)) - \log (2)$  (see, e.g., Domingues et al., 2021, Lemma 15).
Now we pick  $Z = Z_{\pi^{\star}} \coloneqq \mathbb{1}\{\eta < T_{\max} \text{ and } \mathcal{E}(\pi^{\star}, \phi, \mathbb{A}_f(\mathcal{D}))\}$  and note that  $\mathbb{E}_{\pi^{\star}}[Z_{\pi^{\star}}] \geq 1 - 2\delta$  by assumption. This implies that
$$
\mathbb {E} _ {0} \left[ N ^ {\pi^ {\star}} \right] \geq \left(1 - \mathbb {E} _ {0, \pi^ {\star}} \left[ Z _ {\pi^ {\star}} \right]\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - \frac {\log (2)}{7} - T _ {\max } \Delta \left(T _ {\max }\right).
$$
Another application of Lemma 6 gives
$$
\mathbb {E} _ {0} \left[ N ^ {\pi^ {\star}} \right] \geq \left(1 - \mathbb {E} _ {0} \left[ Z _ {\pi^ {\star}} \right]\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - \frac {\log (2)}{7} - \Delta \left(T _ {\max }\right) \left(T _ {\max } + \frac {1}{7} \log \left(\frac {1}{2 \delta}\right)\right).
$$
Summing the above over  $\pi^{\star}\in \Pi^{(\ell)}$ , we get
$$
\sum_ {\pi^ {\star}} \mathbb {E} _ {0} \left[ N ^ {\pi^ {\star}} \right] \geq \left(| \Pi^ {(\ell)} | - \sum_ {\pi^ {\star}} \mathbb {E} _ {0} \left[ Z _ {\pi^ {\star}} \right]\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - | \Pi^ {(\ell)} | \cdot \frac {\log (2)}{7} - | \Pi^ {(\ell)} | \cdot \Delta \left(T _ {\max }\right) \left(T _ {\max } + \frac {1}{7} \log \left(\frac {1}{2 \delta}\right)\right). \tag {10}
$$
It remains to prove an upper bound on  $\sum_{\pi^{\star}}\mathbb{E}_0[Z_{\pi^{\star}}]$ . We calculate that
$$
\begin{array}{l} \sum_ {\pi^ {\star}} \mathbb {E} _ {0} \left[ Z _ {\pi^ {\star}} \right] = \sum_ {\pi^ {\star}} \mathbb {E} _ {0} \left[ \mathbb {1} \left\{\eta <   T _ {\max } \text {a n d} \mathcal {E} \left(\pi^ {\star}, \phi , \mathbb {A} _ {f} (\mathcal {D})\right) \right\} \right] \\ \leq \sum_ {\pi^ {\star}} \mathbb {E} _ {0} \left[ \mathbb {1} \left\{\mathbb {P} _ {\pi^ {\star}} \left[ \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \right] \geq \frac {\varepsilon}{4} \right\} \right] \\ \leq \frac {4}{\varepsilon} \cdot \mathbb {E} _ {0} \left[ \sum_ {\pi^ {\star}} \mathbb {P} _ {\pi^ {\star}} \left[ \operatorname {i d x} \left(s _ {1}\right) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) \left(\operatorname {i d x} \left(s _ {1}\right) _ {1: H}\right) = \pi^ {\star} \left(\operatorname {i d x} \left(s _ {1}\right) _ {1: H}\right) \right] \right] \tag {11} \\ \end{array}
$$
The last inequality is an application of Markov's inequality.
Now we carefully investigate the sum. For any  $\phi \in \Phi$ , the sum can be rewritten as
$$
\begin{array}{l} \sum_ {\pi^ {\star}} \mathbb {P} _ {\pi^ {\star}, \phi} \left[ \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \right] \\ = \sum_ {\pi^ {\star}} \sum_ {s _ {1} \in S _ {1}} \mathbb {P} _ {\pi^ {\star}, \phi} [ s _ {1} ] \mathbb {P} _ {\pi^ {\star}, \phi} \left[ \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \mid s _ {1} \right] \\ \end{array}
$$
$$
\stackrel {(i)} {=} \frac {1}{| \mathcal {S} _ {1} |} \sum_ {s _ {1} \in \mathcal {S} _ {1}} \sum_ {\pi^ {\star}} \mathbb {P} _ {\pi^ {\star}, \phi} \left[ \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \mid s _ {1} \right]
$$
$$
\stackrel {(i i)} {=} \frac {1}{| \mathcal {S} _ {1} |} \sum_ {s _ {1} \in \mathcal {S} _ {1}} \sum_ {\pi^ {\star}} \mathbb {1} \left\{\mathrm {i d x} \left(s _ {1}\right) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) \left(\mathrm {i d x} \left(s _ {1}\right) _ {1: H}\right) = \pi^ {\star} \left(\mathrm {i d x} \left(s _ {1}\right) _ {1: H}\right) \right\}. \tag {12}
$$
The equality  $(i)$  follows because regardless of which MDP  $M_{\pi^{\star}}$  we are in, the first state is distributed uniformly over  $\mathcal{S}_1$ . The equality  $(ii)$  follows because once we condition on the first state  $s_1$ , the probability is either 0 or 1.
Fix any start state  $s_1$ . We can write
$$
\begin{array}{l} \sum_ {\pi^ {\star}} \mathbb {1} \left\{\operatorname {i d x} \left(s _ {1}\right) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) \left(\operatorname {i d x} \left(s _ {1}\right) _ {1: H}\right) \pi^ {\star} \left(\operatorname {i d x} \left(s _ {1}\right) _ {1: H}\right) \right\} \\ = \sum_ {\pi^ {\star} \in \Pi_ {\mathrm {i d x} (s _ {1})} ^ {(\ell)}} \mathbb {1} \{\mathbb {A} _ {f} (\mathcal {D}) (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \} = 1, \\ \end{array}
$$
where the second equality uses the fact that on any index  $j$ , each  $\pi^{\star} \in \Pi_{j}^{(\ell)}$  behaves differently (Property (3) of Lemma 2), so  $\mathbb{A}_f(\mathcal{D})$  can match at most one of these behaviors. Plugging this back into Eq. (12), averaging over  $\phi \in \Phi$ , and combining with Eq. (11), we arrive at the bound
$$
\sum_ {\pi^ {\star}} \mathbb {E} _ {0} \left[ Z _ {\pi^ {\star}} \right] \leq \frac {4}{\varepsilon}.
$$
We now use this in conjunction with Eq. (10) to arrive at the final lower bound
$$
\sum_ {\pi^ {\star}} \mathbb {E} _ {0} \left[ N ^ {\pi^ {\star}} \right] \geq \left(| \Pi^ {(\ell)} | - \frac {4}{\varepsilon}\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - | \Pi^ {(\ell)} | \cdot \frac {\log (2)}{7} - | \Pi^ {(\ell)} | \cdot \Delta \left(T _ {\max }\right) \left(T _ {\max } + \frac {1}{7} \log \left(\frac {1}{2 \delta}\right)\right). \tag {13}
$$
Step 3: Putting it All Together. Combining Eqs. (9) and (13), plugging in our choice of  $\Delta(T_{\mathrm{max}})$  and simplifying we get
$$
\mathbb {E} _ {0} [ \eta ] \geq \left(| \Pi^ {(\ell)} | - \frac {4}{\varepsilon}\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - | \Pi^ {(\ell)} | \cdot \frac {\log (2)}{7} - | \Pi^ {(\ell)} | \cdot \Delta (T _ {\max }) \left(T _ {\max } + \frac {1}{7} \log \left(\frac {1}{2 \delta}\right)\right).
$$
$$
\geq \left(\frac {| \Pi^ {(\ell)} |}{2} - \frac {4}{\varepsilon}\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - | \Pi^ {(\ell)} | \cdot \frac {T _ {\max } ^ {2}}{2 ^ {H + 3}} \left(T _ {\max } + \frac {1}{7} \log \left(\frac {1}{2 \delta}\right)\right).
$$
The last inequality follows since  $\delta \leq 1 / 8$  implies  $\log (1 / (2\delta))\geq 2\log (2)$
This concludes the proof of Lemma 3.
E.6 Change of Measure Lemma
Lemma 6. Let  $Z\in [0,1]$  be a  $\mathcal{F}_{T_{\mathrm{max}}}$ -measurable random variable. Then, for every  $\pi^{\star}\in \Pi^{(\ell)}$
$$
| \mathbb {E} _ {0} [ Z ] - \mathbb {E} _ {0, \pi^ {\star}} [ Z ] | \leq \Delta (T _ {\max}) := \frac {T _ {\max} ^ {2}}{2 ^ {H + 3}}
$$
$$
| \mathbb {E} _ {0} [ Z ] - \mathbb {E} _ {0, \pi^ {\star}} [ Z ] | \leq \operatorname {T V} \left(\mathbb {P} _ {0} ^ {\mathcal {F} _ {T _ {\max }}}, \mathbb {P} _ {0, \pi^ {\star}} ^ {\mathcal {F} _ {T _ {\max }}}\right) \leq \sum_ {t = 1} ^ {T _ {\max }} \mathbb {E} _ {0} [ \operatorname {T V} (\mathbb {P} _ {0} [ \cdot | \mathcal {F} _ {t - 1} ], \mathbb {P} _ {0, \pi^ {\star}} [ \cdot | \mathcal {F} _ {t - 1} ]) ].
$$
Here  $\mathbb{P}_0[\cdot |\mathcal{F}_t]$  denotes the conditional distribution of the  $t$ -th trajectory given the first  $t - 1$  trajectories. Similarly  $\mathbb{P}_{0,\pi^{\star}}[\cdot |\mathcal{F}_t]$  is the averaged over decoders condition distribution of the  $t$ -th trajectory given the first  $t - 1$  trajectories. The second inequality follows by chain rule of TV distance (see, e.g., Polyanskiy and Wu, 2022, pg. 152).
Now we examine each term  $\mathrm{TV}(\mathbb{P}_0[\cdot |\mathcal{F}_{t - 1}],\mathbb{P}_{0,\pi^*}[\cdot |\mathcal{F}_{t - 1}])$ . Fix a history  $\mathcal{F}_{t - 1}$  and sequence  $s_{1:H}$  where all  $s_i$  have the same index. We want to bound the quantity
$$
\left| \mathbb {P} _ {0, \pi^ {\star}} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] - \mathbb {P} _ {0} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] \right|,
$$
where it is understood that the random variable  $S_{1:H}^{(t)}$  is drawn according to the MDP dynamics and algorithm's policy  $\mathbb{A}_t$  (which is in turn a measurable function of  $\mathcal{F}_{t-1}$ ).
We observe that the second term is exactly
$$
\mathbb {P} _ {0} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] = \frac {1}{| \mathcal {S} _ {1} |} \cdot \frac {1}{2 ^ {H - 1}},
$$
since the state  $s_1$  appears with probability  $1 / |S_1|$  and the transitions in  $M_0$  are uniform to the next state in the combination lock, so each sequence is equally as likely.
For the first term, again the state  $s_1$  appears with probability  $1 / |S_1|$ . Suppose that  $\mathrm{idx}(s_1) \notin \mathcal{J}_{\mathrm{rel}}^{\pi^\star}$ . Then the dynamics of  $\mathbb{P}_{0,\pi^\star,\phi}$  for all  $\phi \in \Phi$  are exactly the same as  $M_0$ , so again the probability in this case is  $1 / (|S_1|2^{H - 1})$ . Now consider when  $\mathrm{idx}(s_1) \in \mathcal{J}_{\mathrm{rel}}^{\pi^\star}$ . At some point  $\widehat{h} \in [H + 1]$ , the policy  $\mathbb{A}_t$  will deviate from  $\pi^\star$  for the first time (if  $\mathbb{A}_t$  never deviates from  $\pi^\star$  we set  $\widehat{h} = H + 1$ ). The layer  $\widehat{h}$  is only a function of  $s_1$  and  $\mathbb{A}_t$  and does not depend on the MDP dynamics. The correct decoder must assign  $\phi(s_{1:\widehat{h} -1}) = \mathrm{GOOD}$  and  $\phi(s_{\widehat{h}:H}) = \mathrm{BAD}$ , so therefore we have
$$
\mathbb {P} _ {0, \pi^ {\star}} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] = \mathbb {P} _ {0, \pi^ {\star}} \left[ \phi (s _ {1: \widehat {h} - 1}) = \text {G O O D a n d} \phi (s _ {\widehat {h}: H}) = \text {B A D} \mid \mathcal {F} _ {t - 1} \right]
$$
If  $s_1 \notin \mathcal{F}_{t-1}$ , i.e., we are seeing  $s_1$  for the first time, then the conditional distribution over the labels given by  $\phi$  is the same as the unconditioned distribution:
$$
\mathbb {P} _ {0, \pi^ {\star}} \left[ \phi \left(s _ {1: \widehat {h} - 1}\right) = \text {G O O D a n d} \phi \left(s _ {\widehat {h}: H}\right) = \text {B A D} \mid \mathcal {F} _ {t - 1} \right] = \frac {1}{| \mathcal {S} _ {1} |} \cdot \frac {1}{2 ^ {H - 1}}.
$$
Otherwise, if  $s_1 \in \mathcal{F}_{t-1}$  then we bound the conditional probability by 1.
$$
\mathbb {P} _ {0, \pi^ {\star}} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] \leq \frac {1}{| \mathcal {S} _ {1} |}.
$$
Putting this all together we can compute
$$
\mathbb {P} _ {0, \pi^ {\star}} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] \quad \left\{ \begin{array}{l l} = \frac {1}{| \mathcal {S} _ {1} |} \cdot \frac {1}{2 ^ {H - 1}} & \text {i f} \quad \mathrm {i d x} (s _ {1}) \notin \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}}, \\ = \frac {1}{| \mathcal {S} _ {1} |} \cdot \frac {1}{2 ^ {H - 1}} & \text {i f} \quad \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}} \text {a n d} s _ {1} \notin \mathcal {F} _ {t - 1}, \\ \leq \frac {1}{| \mathcal {S} _ {1} |} & \text {i f} \quad \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}} \text {a n d} s _ {1} \in \mathcal {F} _ {t - 1}, \\ = 0 & \text {o t h e r w i s e}. \end{array} \right.
$$
Therefore we have the bound
$$
\left| \mathbb {P} _ {0, \pi^ {\star}} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] - \mathbb {P} _ {0} \left[ S _ {1: H} ^ {(t)} = s _ {1: H} \mid \mathcal {F} _ {t - 1} \right] \right| \leq \frac {1}{| \mathcal {S} _ {1} |} \mathbb {1} \left\{\mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}}, s _ {1} \in \mathcal {F} _ {t - 1} \right\}.
$$
Summing over all possible sequences  $s_{1:H}$  we have
$$
\operatorname {T V} \left(\mathbb {P} _ {0} [ \cdot | \mathcal {F} _ {t - 1} ], \mathbb {P} _ {0, \pi^ {\star}} [ \cdot | \mathcal {F} _ {t - 1} ]\right) \leq \frac {1}{2} \cdot \frac {(t - 1) \cdot 2 ^ {H - 1}}{| \mathcal {S} _ {1} |},
$$
since the only sequences  $s_{1:H}$  for which the difference in the two measures are nonzero are the ones for which  $s_1\in \mathcal{F}_{t - 1}$ , of which there are  $(t - 1)\cdot 2^{H - 1}$  of them.
Lastly, taking expectations and summing over  $t = 1$  to  $T_{\mathrm{max}}$  and plugging in the value of  $|\mathcal{S}_1| = 2^{2H}$  we have the final bound.
The next lemma is a straightforward modification of (Domingues et al., 2021, Lemma 5), with varying rewards instead of varying transitions.
Lemma 7. Let  $M$  and  $M'$  be two MDPs that are identical in transition and differ in the reward distributions, denote  $r_h(s,a)$  and  $r_h'(s,a)$ . Assume that for all  $(s,a)$  we have  $r_h(s,a) \ll r_h'(s,a)$ . Then for any stopping time  $\eta$  with respect to  $(\mathcal{F}^t)_{t \geq 1}$  that satisfies  $\mathbb{P}_M[\eta < \infty] = 1$ ,
$$
\mathrm {K L} \Big (\mathbb {P} _ {M} ^ {I _ {\eta}} \| \mathbb {P} _ {M ^ {\prime}} ^ {I _ {\eta}} \Big) = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}, h \in [ H ]} \mathbb {E} _ {M} [ N _ {s, a, h} ^ {\eta} ] \cdot \mathrm {K L} (r _ {h} (s, a) \| r _ {h} ^ {\prime} (s, a)),
$$
where  $N_{s,a,h}^{\eta} \coloneqq \sum_{t=1}^{\eta} \mathbb{1}\left\{(S_h^{(t)}, A_h^{(t)}) = (s,a)\right\}$  and  $I_{\eta}: \Omega \mapsto \bigcup_{t \geq 1} \mathcal{I}_t: \omega \mapsto I_{\eta(\omega)}(\omega)$  is the random vector representing the history up to episode  $\eta$ .
Lemma 8 (Lemma 1, Garivier et al. (2019)). Consider a measurable space  $(\Omega, \mathcal{F})$  equipped with two distributions  $\mathbb{P}_1$  and  $\mathbb{P}_2$ . For any  $\mathcal{F}$ -measurable function  $Z: \Omega \mapsto [0,1]$  we have
$$
\mathrm {K L} \left(\mathbb {P} _ {1} \| \mathbb {P} _ {2}\right) \geq \mathrm {k l} \left(\mathbb {E} _ {1} [ Z ] \| \mathbb {E} _ {2} [ Z ]\right).
$$
F Proofs for Section 6
F.1 Algorithm Sketch
In this section we provide a high level sketch of Algorithm 1. We will describe the key algorithmic ideas, as well as the empirical estimation of policy-specific MRPs, in the two phases of POPLER.
State Identification Phase
The goal of the state identification phase is to discover all such petal states that are reachable with probability  $\Omega(\varepsilon/D)$ . The algorithm proceeds in a loop and sequentially grows the set  $\mathcal{T}$ , which contains tuples of the form  $(s, \pi_s)$ , where  $s \in \bigcup_{\pi \in \Pi} S_\pi$  is a sufficiently reachable petal state (for some policy) and  $\pi_s$  denotes a policy that reaches  $s$  with probability  $\Omega(\varepsilon/D)$ . We also denote  $\mathcal{S}^{\mathrm{rch}} := \{s : (s, \pi_s) \in \mathcal{T}\}$  to denote the set of reachable states in  $\mathcal{T}$ . Initially,  $\mathcal{T}$  only contains a dummy start state  $s_\top$  and a null policy. We will collect data using the DataCollector subroutine that: for a given  $(s, \pi_s) \in \mathcal{T}$ , first run  $\pi_s$  to reach state  $s$ , and if we succeed in reaching  $s$ , restart exploration by sampling a policy from  $\mathrm{Uniform}(\Pi_{\mathrm{exp}})$ . Note that DataCollector will be sample-efficient for any  $(s, \pi_s)$  since  $\Omega(\varepsilon/D)$  fraction of the trajectories obtained via  $\pi_s$  are guaranteed to reach  $s$  (by definition of  $\pi_s$  and construction of set  $\mathcal{T}$ ). Initially, we run DataCollector using  $\mathrm{Uniform}(\Pi_{\mathrm{exp}})$  from the start, where we slightly abuse the notation and assume that all trajectories in the MDP start at the dummy state  $s_\top$  at time step  $h = 0$ .
In every loop, the algorithm attempts to find a new petal state  $\bar{s}$  for some  $\pi \in \Pi$  that is guaranteed to be  $\Omega(\varepsilon/D)$ -reachable by  $\pi$ . This is accomplished by constructing a (estimated and partial) version of the policy-specific MRP using the datasets collected up until that loop (line 9). In particular given a policy  $\pi$  and a set  $S_{\pi}^{\mathrm{rch}} = S_{\pi}^{+} \cap S^{\mathrm{rch}}$ , we construct  $\widehat{\mathfrak{M}}_{S^{\mathrm{rch}}}^{\pi} = \mathrm{MRP}(S_{\pi}^{+}, \widehat{P}^{\pi}, \widehat{R}^{\pi}, H, s_{\top}, s_{\bot})$  which essentially compresses our empirical knowledge of the original MDP relevant to the policy  $\pi$ . In particular, for any states  $s \in S_{\pi} \cup \{s_{\top}\}$  and  $s' \in S_{\pi} \cup \{s_{\bot}\}$  residing in different layers  $h < h'$  in the underlying MDP, we define:
Transition  $\widehat{P}_{s\rightarrow s^{\prime}}^{\pi}$  as: