SlowGuess/NeurIPS_2023_OCR huggingface dataset:
that is,  $\Pi_j^{(\ell)}$  are the policies which take an action  $a = 1$  on at least one state with index  $j$ .
We also define the set of relevant state indices for a given policy  $\pi \in \Pi^{(\ell)}$  as
$$
\mathcal {J} _ {\mathrm {r e l}} ^ {\pi} := \left\{j \in [ 2 ^ {2 H} ]: \pi \in \Pi_ {j} ^ {(\ell)} \right\}.
$$
For any policy  $\pi$  we denote  $\pi(j_{1:H}) := (\pi(j[1]), \ldots, \pi(j[H])) \in \{0, 1\}^H$  to be the vector that represents the actions that  $\pi$  takes on the states in index  $j$ . The vector  $\pi(j_{1:H}')$  is defined similarly.
Lemma 2. Let  $H, \varepsilon,$  and  $\ell$  satisfy the assumptions of Theorem 5. There exists a policy class  $\Pi^{(\ell)}$  of size  $N = 1 / (6\varepsilon^{\ell})$  which satisfies the following properties.
(1) For every  $j\in [2^{2H}]$  we have  $|\Pi_j^{(\ell)}|\in [\varepsilon N / 2,2\varepsilon N]$
(2) For every  $\pi \in \Pi$  we have  $|\mathcal{J}_{\mathrm{rel}}^{\pi}|\geq \varepsilon /2\cdot 2^{2H}$
(3) For every  $\pi \in \Pi_j^{(\ell)}$ , the vector  $\pi(j_{1:H})$  is unique and always equal to  $\pi(j_{1:H}^{\prime})$ .
(4) Bounded spanning capacity:  $\mathfrak{C}(\Pi^{(\ell)})\leq c\cdot H^{4\ell +2}$  for some universal constant  $c > 0$
E.2 Construction of MDP Family
The family  $\mathcal{M} = \{M_{\pi^{\star},\phi}\}_{\pi^{\star}\in \Pi^{(\ell)},\phi \in \Phi}$  will be a family of MDPs which are indexed by a policy  $\pi^{\star}$  as well as a decoder function  $\phi :S\mapsto \{\mathrm{GOOD,BAD}\}$ , which assigns each state to be "good" or "bad" in a sense that will be described later on. An example construction of an MDP  $M_{\pi^{\star},\phi}$  is illustrated in Figure 1. For brevity, the bracket notation used to denote the layer that each state lies in has been omitted in the figure.
Figure 1: Illustration of the lower bound from Theorem 3. Blue arrows represent taking the action  $\pi^{\star}(s)$ , while red arrows represent taking the action  $1 - \pi^{\star}(s)$ . Purple arrows denote uniform transition to the states in the next layer, regardless of action. The MDP  $M_{\pi^{\star},\phi}$  is a uniform distribution of  $2^{2H}$  combination locks of two types. In the relevant combination locks (such as Lock 1 in the figure), following  $\pi^{\star}$  keeps one in the "good" chain and gives reward of  $\mathrm{Ber}(3/4)$  in the last layer, while deviating from  $\pi^{\star}$  leads one to the "bad" chain and gives reward of  $\mathrm{Ber}(1/2)$ . In irrelevant combination locks (such as Lock 50 in the figure), the next state is uniform regardless of action, and all rewards at the last layer are  $\mathrm{Ber}(1/2)$ .
Decoder Function Class. The decoder function class  $\Phi$  will be the set of all possible mappings which for every  $j\in [2^{2H}]$  and  $h\geq 2$  assign exactly one of  $j[h]$  or  $j^{\prime}[h]$  to the label GOOD (where the other is assigned to the label BAD). There are  $(2^{H - 1})^{2^{2H}}$  such functions. The label of a state will be used to describe the transition dynamics. Intuitively, a learner who does not know the decoder function  $\phi$  will not be able to tell if a certain state has the label GOOD or BAD when visiting that state for the first time.
Transition Dynamics. The MDP  $M_{\pi^*, \phi}$  will be a uniform distribution over  $2^{2H}$  combination locks  $\{\mathsf{CL}_j\}_{j \in [2^{2H}]}$  with disjoint states. More formally,  $s_1 \sim \mathrm{Uniform}(\{j[1]\}_{j \in [2^{2H}]})$ . From each start state  $j[1]$ , only the  $2H - 2$  states corresponding to index  $j$  at layers  $h \geq 2$  will be reachable in the combination lock  $\mathsf{CL}_j$ .
In the following, we will describe each combination lock  $\mathrm{CL}_j$ , which forms the basic building block of the MDP construction.
- Good/Bad Set. At every layer  $h \in [H]$ , for each  $j[h]$  and  $j'[h]$ , the decoder function  $\phi$  assigns one of them to be GOOD and one of them to be BAD. We will henceforth denote  $j_g[h]$  to be the good state and  $j_b[h]$  to be the bad state. Observe that by construction in Eq. (7), for every  $\pi \in \Pi^{(\ell)}$  and  $h \in [H]$  we have  $\pi(j_g[h]) = \pi(j_b[h])$ .
- Dynamics of  $\mathsf{CL}_j$ , if  $j \in \mathcal{J}_{\mathrm{rel}}^{\pi^*}$ . Here, the transition dynamics of the combination locks are deterministic. For every  $h \in [H]$ ,
- On good states  $j_{g}[h]$  we transit to the next good state iff the action is  $\pi^{\star}$ :
$$
P (s ^ {\prime} \mid j _ {g} [ h ], a) = \left\{ \begin{array}{l l} \mathbb {1} \{s ^ {\prime} = j _ {g} [ h + 1 ] \}, & \text {i f} a = \pi^ {\star} (j _ {g} [ h ]) \\ \mathbb {1} \{s ^ {\prime} = j _ {b} [ h + 1 ] \}, & \text {i f} a \neq \pi^ {\star} (j _ {g} [ h ]). \end{array} \right.
$$
- On bad states  $j_b[h]$  we always transit to the next bad state:
$$
P \left(s ^ {\prime} \mid j _ {b} [ h ], a\right) = \mathbb {1} \left\{s ^ {\prime} = j _ {b} [ h + 1 ] \right\}, \quad \text {f o r a l l} a \in \mathcal {A}.
$$
- Dynamics of  $\mathsf{CL}_j$ , if  $j \notin \mathcal{I}_{\mathrm{rel}}^{\pi^*}$ . If  $j$  is not a relevant index for  $\pi^*$ , then the transitions are uniformly random regardless of the current state/action. For every  $h \in [H]$ ,
$$
P (\cdot \mid j _ {g} [ h ], a) = P (\cdot \mid j _ {b} [ h ], a) = \mathrm {U n i f o r m} (\{j _ {g} [ h + 1 ], j _ {b} [ h + 1 ] \}), \quad \mathrm {f o r a l l} a \in \mathcal {A}.
$$
- Reward Structure. The reward function is nonzero only at layer  $H$ , and is defined as
$$
R (s, a) = \operatorname {B e r} \left(\frac {1}{2} + \frac {1}{4} \cdot \mathbb {1} \left\{\pi^ {\star} \in \Pi_ {j} ^ {(\ell)} \right\} \cdot \mathbb {1} \left\{s = j _ {g} [ H ], a = \pi^ {\star} \left(j _ {g} [ H ]\right) \right\}\right)
$$
That is, we get  $3/4$  whenever we reach the  $H$ -th good state for an index  $j$  which is relevant for  $\pi^{\star}$ , and  $1/2$  reward otherwise.
Reference MDPs. We define several reference MDPs.
- In the reference MDP  $M_0$ , the initial start state is again taken to be the uniform distribution, i.e.,  $s_1 \sim \mathrm{Uniform}(\{j[1]\}_{j \in [2^{2H}]})$ , and all the combination locks behave the same and have uniform transitions to the next state along the chain: for every  $h \in [H]$  and  $j \in [2^{2H}]$ ,
$$
P (\cdot \mid j [ h ], a) = P (\cdot \mid j ^ {\prime} [ h ], a) = \operatorname {U n i f o r m} (\{j [ h + 1 ], j ^ {\prime} [ h + 1 ] \}), \quad \text {f o r a l l} a \in \mathcal {A}.
$$
The rewards for  $M_0$  are  $\mathrm{Ber}(1 / 2)$  for every  $(s,a)\in S_H\times \mathcal{A}$
- For any decoder  $\phi \in \Phi$ , the reference MDP  $M_{0,\pi^{\star},\phi}$  has the same transitions as  $M_{\pi^{\star},\phi}$  but the rewards are  $\mathrm{Ber}(1/2)$  for every  $(s,a) \in S_H \times \mathcal{A}$ .
E.3 Proof of Theorem 5
We are now ready to prove the lower bound using the construction of the MDP family  $\mathcal{M}$ .
Value Calculation. Consider any  $M_{\pi^{\star},\phi} \in \mathcal{M}$ . For any policy  $\pi \in \mathcal{A}^S$  we use  $V_{\pi^{\star},\phi}(\pi)$  to denote the value of running  $\pi$  in MDP  $M_{\pi^{\star},\phi}$ . By construction we can see that
$$
V _ {\pi^ {\star}, \phi} (\pi) = \frac {1}{2} + \frac {1}{4} \cdot \mathbb {P} _ {\pi^ {\star}, \phi} \left[ \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \pi (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \right], \tag {4}
$$
where in the above, we defined for any  $s_1$ ,  $\pi (\mathrm{idx}(s_1)_{1:H}) = \pi (j_{1:H}) = (\pi (j[1]),\ldots ,\pi (j[H]))$  where  $j$  denotes  $\mathrm{idx}(s_1)$ . Informally speaking, the second term counts the additional reward that  $\pi$  gets for solving a combination lock rooted at a relevant state index  $\mathrm{idx}(s_1)\in \mathcal{J}_{\mathrm{rel}}^{\pi^{\star}}$ . By Property (2) and (3) of Lemma 2, we additionally have  $V_{\pi^{\star},\phi}(\pi^{\star})\geq 1 / 2 + \varepsilon /8$ , as well as  $V_{\pi^{\star},\phi}(\pi) = 1 / 2$  for all other  $\pi \neq \pi^{\star}\in \Pi^{(\ell)}$ .
By Eq. (4), if  $\pi$  is an  $\varepsilon / 16$ -optimal policy on  $M_{\pi^*, \phi}$  it must satisfy
$$
\mathbb {P} _ {\pi^ {\star}, \phi} \Big [ \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\mathrm {r e l}} ^ {\pi^ {\star}} \text {a n d} \pi (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \Big ] \geq \frac {\varepsilon}{4}.
$$
Averaged Measures. We define the following measures which will be used in the analysis.
- Define  $\mathbb{P}_{\pi^{\star}}[\cdot] = \frac{1}{|\Phi|} \sum_{\phi \in \Phi} \mathbb{P}_{\pi^{\star}, \phi}[\cdot]$  to be the averaged measure where we first pick  $\phi$  uniformly among all decoders and then consider the distribution induced by  $M_{\pi^{\star}, \phi}$ .
- Define the averaged measure  $\mathbb{P}_{0,\pi^{\star}}[\cdot] = \frac{1}{|\Phi|}\sum_{\phi \in \Phi}\mathbb{P}_{0,\pi^{\star},\phi}[\cdot]$  where we pick  $\phi$  uniformly and then consider the distribution induced by  $M_{0,\pi^{\star},\phi}$ .
For both averaged measures the expectations  $\mathbb{E}_{\pi^{\star}}$  and  $\mathbb{E}_{0,\pi^{\star}}$  are defined analogously.
Algorithm and Stopping Time. Recall that an algorithm  $\mathbb{A}$  is comprised of two phases. In the first phase, it collects some number of trajectories by interacting with the MDP in episodes. We use  $\eta$  to denote the (random) number of episodes after which  $\mathbb{A}$  terminates. We also use  $\mathbb{A}_t$  to denote the intermediate policy that the algorithm runs in round  $t$  for  $t \in [\eta]$ . In the second phase,  $\mathbb{A}$  outputs<sup>6</sup> a
<sup>6</sup>We present the lower bound for the class of deterministic algorithms that output a deterministic policy. However, all the arguments could be extended to stochastic algorithms.
policy  $\widehat{\pi}$ . We use the notation  $\mathbb{A}_f: \{\tau^{(t)}\}_{t \in [\eta]} \mapsto \mathcal{A}^S$  to denote the second phase of  $\mathbb{A}$  which outputs  $\widehat{\pi}$  as a measurable function of collected data.
For any policy  $\pi^{\star}$ , decoder  $\phi$ , and dataset  $\mathcal{D}$  we define the event
$$
\mathcal {E} \left(\pi^ {\star}, \phi , \mathbb {A} _ {f} (\mathcal {D})\right) := \left\{\mathbb {P} _ {\pi^ {\star}, \phi} \left[ \mathrm {i d x} (s _ {1}) \in \mathcal {J} _ {\text {r e l}} ^ {\pi^ {\star}} \text {a n d} \mathbb {A} _ {f} (\mathcal {D}) (\mathrm {i d x} (s _ {1}) _ {1: H}) = \pi^ {\star} (\mathrm {i d x} (s _ {1}) _ {1: H}) \right] \geq \frac {\varepsilon}{4} \right\}.
$$
The event  $\mathcal{E}(\pi^{\star},\phi ,\mathbb{A}_f(\mathcal{D}))$  is measurable with respect to the random variable  $\mathcal{D}$ , which denotes the collected data.
Under this notation, the PAC learning guarantee on  $\mathbb{A}$  implies that for every  $\pi^{\star} \in \Pi^{(\ell)}$ ,  $\phi \in \Phi$  we have
$$
\mathbb {P} _ {\pi^ {\star}, \phi} [ \mathcal {E} (\pi^ {\star}, \phi , \mathbb {A} _ {f} (\mathcal {D})) ] \geq 7 / 8.
$$
Moreover via an averaging argument we also have
$$
\mathbb {P} _ {\pi^ {\star}} \left[ \mathcal {E} \left(\pi^ {\star}, \phi , \mathbb {A} _ {f} (\mathcal {D})\right) \right] \geq 7 / 8. \tag {5}
$$
Lower Bound Argument. We apply a truncation to the stopping time  $\eta$ . Define  $T_{\mathrm{max}} \coloneqq 2^{H/3}$ . Observe that if  $\mathbb{P}_{\pi^{\star}}[\eta > T_{\mathrm{max}}] > 1/8$  for some  $\pi^{\star} \in \Pi^{(\ell)}$  then the lower bound immediately follows, since
$$
\max _ {\phi \in \Phi} \mathbb {E} _ {\pi^ {\star}, \phi} [ \eta ] > \mathbb {E} _ {\pi^ {\star}} [ \eta ] \geq \mathbb {P} _ {\pi^ {\star}} [ \eta > T _ {\max } ] \cdot T _ {\max } \geq T _ {\max } / 8,
$$
so there must exist an MDP  $M_{\pi^*, \phi}$  for which  $\mathbb{A}$  collects at least  $T_{\max} / 8 = 2^{H / 3 - 3}$  samples in expectation.
Otherwise we have  $\mathbb{P}_{\pi^{\star}}[\eta > T_{\max}] \leq 1/8$  for all  $\pi^{\star} \in \Pi^{(\ell)}$ . This further implies that for all  $\pi^{\star} \in \Pi^{(\ell)}$ ,
$$
\begin{array}{l} \mathbb {P} _ {\pi^ {*}} [ \eta <   T _ {\max } \text {a n d} \mathcal {E} (\pi^ {\star}, \phi , \mathbb {A} _ {f} (\mathcal {D})) ] \\ = \mathbb {P} _ {\pi^ {\star}} [ \mathcal {E} (\pi^ {\star}, \phi , \mathbb {A} _ {f} (\mathcal {D})) ] - \mathbb {P} _ {\pi^ {\star}} [ \eta > T _ {\max } \text {a n d} \mathcal {E} (\pi^ {\star}, \phi , \mathbb {A} _ {f} (\mathcal {D})) ] \geq 3 / 4. \tag {6} \\ \end{array}
$$
However, in the following, we will argue that if Eq. (6) holds then  $\mathbb{A}$  must query a significant number of samples in  $M_0$ .
Lemma 3 (Stopping Time Lemma). Let  $\delta \in (0,1/8]$ . Let  $\mathbb{A}$  be an  $(\varepsilon/16,\delta)$ -PAC algorithm. Let  $T_{\max} \in \mathbb{N}$ . Suppose that  $\mathbb{P}_{\pi^{\star}}[\eta < T_{\max}$  and  $\mathcal{E}(\pi^{\star},\phi ,\mathbb{A}_f(\mathcal{D}))] \geq 1 - 2\delta$  for all  $\pi^{\star} \in \Pi^{(\ell)}$ . The expected stopping time for  $\mathbb{A}$  on  $M_0$  is at least
$$
\mathbb {E} _ {0} [ \eta ] \geq \left(\frac {| \Pi^ {(\ell)} |}{2} - \frac {4}{\varepsilon}\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - | \Pi^ {(\ell)} | \cdot \frac {T _ {\max} {} ^ {2}}{2 ^ {H + 3}} \left(T _ {\max} + \frac {1}{7} \log \left(\frac {1}{2 \delta}\right)\right).
$$
Using Lemma 3 with  $\delta = 1 / 8$  and plugging in the value of  $|\Pi^{(\ell)}|$  and  $T_{\mathrm{max}}$ , we see that
$$
\mathbb {E} _ {0} [ \eta ] \geq \left(\frac {| \Pi^ {(\ell)} |}{2} - \frac {4}{\varepsilon}\right) \cdot \frac {1}{7} \log \left(\frac {1}{2 \delta}\right) - | \Pi^ {(\ell)} | \cdot \frac {T _ {\max} {} ^ {2}}{2 ^ {H + 3}} \left(T _ {\max} + \frac {1}{7} \log \left(\frac {1}{2 \delta}\right)\right) \geq \frac {| \Pi^ {(\ell)} |}{2 0}.
$$
For the second inequality, we used the fact that  $\ell \geq 2$ ,  $H \geq 10^5$ , and  $\varepsilon < 1 / 10^7$ .
We have shown that either there exists some MDP  $M_{\pi^{\star},\phi}$  for which  $\mathbb{A}$  collects at least  $T_{\max} / 8 = 2^{H / 3 - 3}$  samples in expectation, or  $\mathbb{A}$  must query at least  $|\Pi^{(\ell)}| / 20 = 1 / (120\varepsilon^{\ell})$  trajectories in expectation in  $M_0$ . Putting it all together, the lower bound on the sample complexity is at least
$$
\min  \left\{\frac {1}{1 2 0 \varepsilon^ {\ell}}, 2 ^ {H / 3 - 3} \right\}.
$$
This concludes the proof of Theorem 5.
E.4 Proof of Lemma 2
To prove Lemma 2, we first use a probabilistic argument to construct a certain binary matrix  $B$  which satisfies several properties, and then construct  $\Pi^{(\ell)}$  using  $B$  and verify it satisfies Properties (1)-(4).
Binary Matrix Construction. First, we define a block-free property of binary matrices.
Definition 5 (Block-free Matrices). Fix parameters  $k, \ell, N, d \in \mathbb{N}$  where  $k \leq N$  and  $l \leq d$ . We say a binary matrix  $B \in \{0,1\}^{N \times d}$  is  $(k, \ell)$ -block-free if the following holds: for every  $I \subseteq [N]$  with  $|I| = k$ , and  $J \subseteq [d]$  with  $|J| = \ell$  there exists some  $(i, j) \in I \times J$  with  $B_{ij} = 0$ .
In words, matrices which are  $(k,\ell)$ -block-free do not contain a  $k\times \ell$  block of all 1s.
Lemma 4. Fix any  $\varepsilon \in (0,1 / 10)$  and  $\ell \in \mathbb{N}$ . For any
$$
d \in \left[ \frac {1 6 \ell \cdot \log (1 / \varepsilon)}{\varepsilon}, \frac {1}{2 0} \cdot \exp \left(\frac {1}{4 8 \varepsilon^ {\ell - 1}}\right) \right],
$$
there exists a binary matrix  $B \in \{0,1\}^{N \times d}$  with  $N = 1 / (6 \cdot \varepsilon^{\ell})$  such that:
(1) (Row sum): for every row  $i \in [N]$ , we have  $\sum_{j} B_{ij} \geq \varepsilon d / 2$ .
(2) (Column sum): for every column  $j \in [d]$ , we have  $\sum_{i} B_{ij} \in [\varepsilon N / 2, 2\varepsilon N]$ .
(3) The matrix  $B$  is  $(\ell \log d, \ell)$ -block-free.
Proof of Lemma 4. The existence of  $B$  is proven using the probabilistic method. Let  $\widetilde{B} \in \{0,1\}^{N \times d}$  be a random matrix where each entry is i.i.d. chosen to be 1 with probability  $\varepsilon$ .
By Chernoff bounds (Lemma 18), for every row  $i \in [N]$ , we have  $\mathbb{P}[\sum_j B_{ij} \leq \frac{\varepsilon d}{2}] \leq \exp(-\varepsilon d/8)$ ; likewise for every column  $j \in [d]$ , we have  $\mathbb{P}[\sum_j B_{ij} \notin [\frac{\varepsilon N}{2}, 2\varepsilon N]] \leq 2 \exp(-\varepsilon N/8)$ . By union bound, the matrix  $\widetilde{B}$  satisfies the first two properties with probability at least 0.8 as long as
$$
d \geq (8 \log 1 0 N) / \varepsilon , \quad \text {a n d} \quad N \geq (8 \log 2 0 d) / \varepsilon .
$$
One can check that under the choice of  $N = 1 / (6\cdot \varepsilon^{\ell})$  and the assumption on  $d$ , both constraints are met.
Now we examine the probability of  $\widetilde{B}$  satisfies the block-free property with parameters  $(k := \ell \log d, \ell)$ . Let  $X$  be the random variable which denotes the number of submatrices which violate the block-free property in  $\widetilde{B}$ , i.e.,
$$
X = \left| \left\{I \times J: I \subset [ N ], | I | = k, J \subset [ d ], | J | = \ell , \widetilde {B} _ {i j} = 1 \forall (i, j) \in I \times J \right\} \right|.
$$
By linearity of expectation, we have
$$
\mathbb {E} [ X ] \leq N ^ {k} d ^ {\ell} \varepsilon^ {k \ell}.
$$
We now plug in the choice  $k = \ell \log d$  and observe that as long as  $N \leq 1 / (2e \cdot \varepsilon^{\ell})$  we have  $\mathbb{E}[X] \leq 1/2$ . By Markov's inequality,  $\mathbb{P}[X = 0] \geq 1/2$ .
Therefore with positive probability,  $\widetilde{B}$  satisfies all 3 properties (otherwise we would have a contradiction via inclusion-exclusion principle). Thus, there exists a matrix  $B$  which satisfies all of the above three properties, proving the result of Lemma 4.
Policy Class Construction. For the given  $\varepsilon$  and  $\ell \in \{2,\dots ,H\}$  we will use Lemma 4 to construct a policy class  $\Pi^{(\ell)}$  which has bounded spanning capacity but is hard to explore. We instantiate Lemma 4 with the given  $\ell$  and  $d = 2^{2H}$ , and use the resulting matrix  $B$  to construct  $\Pi^{(\ell)} = \{\pi_i\}_{i\in [N]}$  with  $|\Pi^{(\ell)}| = N = 1 / (6\varepsilon^{\ell})$ .
Recall that we assume that
$$
H \geq h _ {0}, \quad \text {a n d} \quad \varepsilon \in \left[ \frac {1}{2 ^ {c H}}, \frac {1}{1 0 0 H} \right].
$$
We claim that under these assumptions, the requirement of Lemma 4 is met:
$$
d = 2 ^ {2 H} \in \left[ \frac {1 6 \ell \cdot \log (1 / \varepsilon)}{\varepsilon}, \frac {1}{2 0} \cdot \exp \left(\frac {1}{4 8 \varepsilon^ {\ell - 1}}\right) \right].
$$
For the lower bound, we can check that:
$$
\frac {1 6 \ell \cdot \log (1 / \varepsilon)}{\varepsilon} \leq 1 6 H \cdot c H \cdot 2 ^ {c H} \leq 2 ^ {2 H},
$$
where we use the bound  $\ell \leq H$  and  $\varepsilon \geq 2^{-cH}$ . The last inequality holds for sufficiently small universal constant  $c \in (0,1)$  and sufficiently large  $H \geq h_0$ .
For the upper bound, we can also check that
$$
\frac {1}{2 0} \cdot \exp \left(\frac {1}{4 8 \varepsilon^ {\ell - 1}}\right) \geq \frac {1}{2 0} \cdot \exp \left(\frac {1 0 0 H}{4 8}\right) \geq 2 ^ {2 H},
$$
where we use the bound  $\ell \geq 2$  and  $\varepsilon \leq 1 / (100H)$ . The last inequality holds for sufficiently large  $H$ . We define the policies as follows: for every  $\pi_i \in \Pi^{(\ell)}$  we set
$$
\text {f o r e v e r y} j \in [ 2 ^ {2 H} ]: \quad \pi_ {i} (j [ h ]) = \pi_ {i} (j ^ {\prime} [ h ]) = \left\{ \begin{array}{l l} \operatorname {b i t} _ {h} \left(\sum_ {a \leq i} B _ {a j}\right) & \text {i f} B _ {i j} = 1, \\ 0 & \text {i f} B _ {i j} = 0. \end{array} \right. \tag {7}
$$
The function  $\mathrm{bit}_h:[2^H -1]\mapsto \{0,1\}$  selects the  $h$  -th bit in the binary representation of the input.
Verifying Properties (1) - (4) of Lemma 2. Properties (1) - (3) are straightforward from the construction of  $B$  and  $\Pi^{(\ell)}$ , since  $\pi_i \in \Pi_j^{(\ell)}$  if and only if  $B_{ij} = 1$ . The only detail which requires some care is that we require that  $2\varepsilon N < 2^H$  in order for Property (3) to hold, since otherwise we cannot assign the behaviors of the policies according to Eq. (7). However, by assumption, this always holds, since  $2\varepsilon N = 1 / (3\varepsilon^{\ell -1}) \leq 2^H$ .
We now prove Property (4) that  $\Pi^{(\ell)}$  has bounded spanning capacity. To prove this we will use the block-free property of the underlying binary matrix  $B$ .
Fix any deterministic MDP  $M^{\star}$  which witnesses  $\mathfrak{C}(\Pi^{(\ell)})$  at layer  $h^\star$ . To bound  $\mathfrak{C}(\Pi^{(\ell)})$ , we need to count the contribution to  $C_{h^{\star}}^{\mathrm{reach}}(\Pi;M^{\star})$  from trajectories  $\tau$  which are produced by some  $\pi \in \Pi^{(\ell)}$  on  $M$ . We first define a layer decomposition for a trajectory  $\tau = (s_1,a_1,s_2,a_2,\ldots ,s_H,a_H)$  as the unique tuple of indices  $(h_1,h_2,\dots h_m)$ , where each  $h_k\in [H]$ , that satisfies the following properties:
- The layers satisfy  $h_1 < h_2 < \dots < h_m$ .
- The layer  $h_1$  represents the first layer where  $a_{h_1} = 1$ .
- The layer  $h_2$  represents the first layer where  $a_{h_2} = 1$  on some state  $s_{h_2}$  such that
$$
\operatorname {i d x} \left(s _ {h _ {2}}\right) \notin \left\{\operatorname {i d x} \left(s _ {h _ {1}}\right) \right\}.
$$
- The layer  $h_3$  represents the first layer where  $a_{h_3} = 1$  on some state  $s_{h_3}$  such that
$$
\operatorname {i d x} \left(s _ {h _ {3}}\right) \notin \left\{\operatorname {i d x} \left(s _ {h _ {1}}\right), \operatorname {i d x} \left(s _ {h _ {2}}\right) \right\}.
$$
- More generally the layer  $h_k$ ,  $k \in [m]$  represents the first layer where  $a_{h_k} = 1$  on some state  $s_{h_k}$  such that
$$
\operatorname {i d x} \left(s _ {h _ {k}}\right) \notin \left\{\operatorname {i d x} \left(s _ {h _ {1}}\right), \dots , \operatorname {i d x} \left(s _ {h _ {k - 1}}\right) \right\}.
$$
In other words, the layer  $h_k$  represents the  $k$ -th layer for where action is  $a = 1$  on a new state index which  $\tau$  has never played  $a = 1$  on before.
We will count the contribution to  $C_{h^{\star}}^{\mathrm{reach}}(\Pi; M^{\star})$  by doing casework on the length of the layer decomposition for any  $\tau$ . That is, for every length  $m \in \{0, \dots, H\}$ , we will bound  $C_{h^{\star}}(m)$ , which is defined to be the total number of  $(s, a)$  at layer  $h^{\star}$  which, for some  $\pi \in \Pi^{(\ell)}$ , a trajectory  $\pi \rightsquigarrow \tau$  that has a  $m$ -length layer decomposition visits. Then we apply the bound
$$
C _ {h ^ {\star}} ^ {\text {r e a c h}} (\Pi ; M ^ {\star}) \leq \sum_ {m = 0} ^ {H} C _ {h ^ {\star}} (m). \tag {8}
$$
Note that this will overcount, since the same  $(s,a)$  pair can belong to multiple different trajectories with different length layer decompositions.
Lemma 5. The following bounds hold:
- For any  $m \leq \ell$ ,  $C_{h^{\star}}(m) \leq H^{m} \cdot \prod_{k=1}^{m} (2kH) = \mathcal{O}(H^{4m})$ .
- We have  $\sum_{m\geq \ell +1}C_{h^{\star}}(m)\leq \mathcal{O}(\ell \cdot H^{4\ell +1})$
Therefore, applying Lemma 5 to Eq. (8), we have the bound that
$$
\mathfrak {C} (\Pi^ {(\ell)}) \leq \left(\sum_ {m \leq \ell} O (H ^ {4 m})\right) + O (\ell \cdot H ^ {4 \ell + 1}) \leq O (H ^ {4 \ell + 2}).
$$
This concludes the proof of Lemma 2.
Proof of Lemma 5. All of our upper bounds will be monotone in the value of  $h^\star$ , so we will prove the bounds for  $C_H(m)$ . In the following, fix any deterministic MDP  $M^\star$ .
First we start with the case where  $m = 0$ . The trajectory  $\tau$  must play  $a = 0$  at all times; since there is only one such  $\tau$ , we have  $C_H(0) = 1$ .
Now we will bound  $C_H(m)$ , for any  $m \in \{1, \dots, \ell\}$ . Observe that there are  $\binom{H}{m} \leq H^m$  ways to pick the tuple  $(h_1, \ldots, h_m)$ . Now we will fix  $(h_1, \ldots, h_m)$  and count the contributions to  $C_H(m)$  for trajectories  $\tau$  which have this fixed layer decomposition, and then sum up over all possible choices of  $(h_1, \ldots, h_m)$ .