SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\ell$ -tons. In the following, we will denote  $\Pi_{\ell} \coloneqq \Pi_{\ell - \text{ton}}$ . We will first prove that  $\mathfrak{C}(\Pi_{\ell}) \leq 2H^{\ell}$ . To show this, we will prove that  $\mathfrak{C}_h(\Pi_{\ell}) \leq 2h^\ell$  by induction on  $H$ . When  $H = 1$ , the class is a subclass of the above contextual bandit class, hence we have  $\mathfrak{C}_1(\Pi_{\ell}) \leq 2$ . Next, suppose  $\mathfrak{C}_{h-1}(\Pi_{\ell}) \leq 2(h-1)^\ell$ . Fix any deterministic MDP and call the first state  $s_1$ . Policies taking  $a = 1$  at  $s_1$  can only take  $a = 1$  on at most  $\ell - 1$  states in the following layers. Such policies reach at most  $\mathfrak{C}_{h-1}(\Pi_{\ell-1})$  states in layer  $h$ . Policies taking  $a = 0$  at  $s_1$  can only take  $a = 1$  on at most  $\ell$  states in the following layers. Such policies reach at most  $\mathfrak{C}_{h-1}(\Pi_{\ell})$  states in layer  $h$ . Hence we get
$$
\mathfrak {C} _ {h} \left(\Pi_ {\ell}\right) \leq \mathfrak {C} _ {h - 1} \left(\Pi_ {\ell - 1}\right) + \mathfrak {C} _ {h - 1} \left(\Pi_ {\ell}\right) \leq 2 (h - 1) ^ {\ell - 1} + 2 (h - 1) ^ {\ell} \leq 2 h ^ {\ell}.
$$
This finishes the proof of the induction hypothesis. Based on the induction argument, we get
$$
\mathfrak {C} (\Pi_ {\ell}) = \max  _ {h \in [ H ]} \mathfrak {C} _ {h} (\Pi_ {\ell}) \leq 2 H ^ {\ell}.
$$
Additionally, choose
$$
\Pi_ {\text {c o r e}} = \left\{\pi_ {0} \right\} \cup \left\{\pi_ {h}: 1 \leq h \leq H \right\},
$$
where  $\pi_0(s) \equiv 0$ , and  $\pi_h$  chooses the action 1 on all the states at layer  $h$ , i.e.,  $\pi_h(s) \coloneqq \mathbb{1}\{s \in S_h\}$ . For every  $\pi \in \Pi_\ell$ , we choose  $S_\pi$  to be the states for which  $\pi(s) = 1$  (there are at most  $\ell$  such states). Fix any partial trajectory  $\tau = (s_h, a_h, \dots, s_{h'}, a_{h'})$  which satisfies  $\pi \rightsquigarrow \tau$ . Suppose that for all  $i \in (h, h']$ ,  $s_i \notin S_\pi$ . Then we must have  $a_i = 0$  for all  $i \in (h, h']$ . Hence  $\pi_h \rightsquigarrow \tau$  (if  $a_h = 1$ ) or  $\pi_0 \rightsquigarrow \tau$  (if  $a_h = 0$ ), and  $\tau$  is consistent with some policy in  $\Pi_{\mathrm{core}}$ . Therefore,  $\Pi_\ell$  is an  $(H + 1, \ell)$ -sunflower.
1-Active Policies. We will first prove that  $\mathfrak{C}(\Pi_{1 - \mathrm{act}})\leq 2H$ . For any deterministic MDP, we use  $\bar{S}_h$  to denote the set of states reachable by  $\Pi_{1 - \mathrm{act}}$  at layer  $h$ . We will show that  $\bar{S}_h\leq 2h$  by induction on  $h$ . For  $h = 1$ , this holds since any deterministic MDP has only one state in the first layer. Suppose it holds at layer  $h$ . Then, we have
$$
\left. \left| \bar {S} _ {h + 1} \right| \leq \left| \left\{\left(s, \pi (s)\right): s \in \bar {S} _ {h}, \pi \in \Pi \right\} \right|. \right.
$$
Note that policies in  $\Pi_{1\text{-act}}$  must take  $a = 0$  on every  $s \notin \{s_{(1,1)}, s_{(1,2)}, \dots, s_{(1,H)}\}$ . Hence  $|\{(s, \pi(s)) \mid s \in \bar{S}_h, \pi \in \Pi\}| \leq |\bar{S}_h| + 1 \leq h + 1$ . Thus, the induction argument is complete. As a consequence we have  $\mathfrak{C}_h(\Pi) \leq 2h$  for all  $h$ , so
$$
\mathfrak {C} (\Pi_ {1 - \text {a c t}}) = \max  _ {h \in [ H ]} \mathfrak {C} _ {h} (\Pi_ {1 - \text {a c t}}) \leq 2 H.
$$
Additionally, if we choose  $S_{\pi} = \{s_{(1,1)}, s_{(1,2)}, \dots, s_{(1,H)}\}$  for all  $\pi \in \Pi$  as well as
where  $\pi_0(s) \equiv 0$  and  $\pi_h(s) \coloneqq \mathbb{1}\{s \in S_h\}$ . Now fix any partial trajectory  $\tau = (s_h, a_h, \dots, s_{h'}, a_{h'})$  which satisfies  $\pi \rightsquigarrow \tau$ . If we have  $i \in (h, h']$ ,  $s_i \notin S_\pi$ , then we must have  $a_i = 0$ . Thus,  $\pi_h \rightsquigarrow \tau$  (if  $a_h = 1$ ) or  $\pi_0 \rightsquigarrow \tau$  (if  $a_h = 0$ ), so  $\tau$  is consistent with some policy in  $\Pi_{\mathrm{core}}$ . Therefore,  $\Pi_{1-\mathrm{act}}$  is a  $(H + 1, H)$ -sunflower.
All-Active Policies. For any deterministic MDP, there is a single state  $s_{(j,1)}$  in the first layer. Any policy which takes  $a = 1$  at state  $s_{(j,1)}$  must belong to  $\Pi_{j\text{-act}}$ . Hence such policies can reach at most  $\mathfrak{C}_{h - 1}(\Pi_{j\text{-act}})$  states in layer  $h$ . For polices which take action 0 at state  $h$ , all these policies will transit to a fixed state in layer 2. Hence such policies can reach at most  $\mathfrak{C}_{h - 1}(\Pi_{\mathrm{act}})$  states at layer  $h$ . Therefore, we get
$$
\mathfrak {C} _ {h} (\Pi_ {\mathrm {a c t}}) \leq \mathfrak {C} _ {h - 1} (\Pi_ {\mathrm {a c t}}) + \max  _ {j} \mathfrak {C} _ {h - 1} (\Pi_ {j - \mathrm {a c t}}) \leq \mathfrak {C} _ {h - 1} (\Pi_ {\mathrm {a c t}}) + 2 (h - 1).
$$
By telescoping, we get
$$
\mathfrak {C} _ {h} \left(\Pi_ {\text {a c t}}\right) \leq h (h - 1),
$$
$$
\mathfrak {C} (\Pi_ {\mathrm {a c t}}) = \max  _ {h \in [ H ]} \mathfrak {C} _ {h} (\Pi_ {\mathrm {a c t}}) \leq H (H - 1).
$$
Additionally, if we choose  $S_{\pi} = \{s_{(j,1)},\dots ,s_{(j,H)}\}$  for all  $\pi \in \Pi_{j\text{-act}}$ , as well as
where  $\pi_0(s) \coloneqq 0$  and  $\pi_h(s) \coloneqq \mathbb{1}\{s \in S_h\}$ . Now fix any partial trajectory  $\tau = (s_h, a_h, \dots, s_{h'}, a_{h'})$  which satisfies  $\pi \rightsquigarrow \tau$ . If we have  $i \in (h, h']$ ,  $s_i \notin S_\pi$ , then we must have  $a_i = 0$ . Thus,  $\pi_h \rightsquigarrow \tau$  (if  $a_h = 1$ ) or  $\pi_0 \rightsquigarrow \tau$  (if  $a_h = 0$ ), so  $\tau$  is consistent with some policy in  $\Pi_{\mathrm{core}}$ . Therefore,  $\Pi_{\mathrm{act}}$  is a  $(H + 1, H)$ -sunflower.
Policy Classes for Continuous State Spaces. In some cases, it is possible to construct policy classes over continuous state spaces that have bounded spanning capacity. For example, consider  $\Pi_{\mathrm{sing}}$ , which is defined over a discrete (but large) state space. We can extend this to continuous state space by defining new state spaces  $S_{h} = \{s_{(x,h)} : x \in \mathbb{R}\}$  for all  $h \in [H]$ , action space  $\mathcal{A} = \{0,1\}$ , and policy class
$$
\widetilde {\Pi_ {\mathrm {s i n g}}} := \left\{\pi_ {(i, h ^ {\prime}}): \pi_ {(i, h ^ {\prime})} (s _ {(x, h)}) = \mathbb {1} \{x \in [ i, i + 1) \text {a n d} h = h ^ {\prime} \}, i \in \mathbb {N}, h ^ {\prime} \in [ H ] \right\}.
$$
Essentially, we have expanded each state to be an interval on the real line. Using the same reasoning, we have the bound  $\mathfrak{C}(\widetilde{\Pi_{\mathrm{sing}}}) = H + 1$ . One can also generalize this construction to the policy class  $\widetilde{\Pi_{\ell - \mathrm{ton}}}$  and preserve the same value of  $\mathfrak{C}$ .<sup>5</sup>
However, in general, this expansion to continuous state spaces may blow up the spanning capacity. Consider a similar modification to  $\Pi_{1\text{-act}}$  (again, with the same new state space and action space  $\mathcal{A} = \{0,1\}$ ):
$$
\widetilde {\Pi_ {1 - \text {a c t}}} := \left\{\pi : \pi \left(s _ {(x, h)}\right) = 0 \text {i f} x \notin [ 0, 1) ] \right\}.
$$
While  $\mathfrak{C}(\Pi_{1 - \mathrm{act}}) = \Theta (H)$ , it is easy to see that  $\mathfrak{C}(\widetilde{\Pi_{1 - \mathrm{act}}}) = 2^H$  since one can construct a  $H$ -layer deterministic tree using states in  $[0,1)$  as every  $(s,a)$  pair at layer  $H$  will be reachable by  $\widetilde{\Pi_{1 - \mathrm{act}}}$ .
Fix any  $M \in \mathcal{M}^{\mathrm{sto}}$ , as well as  $h \in [H]$ . We claim that
$$
\Gamma_ {h} := \sum_ {s _ {h} \in \mathcal {S} _ {h}, a _ {h} \in \mathcal {A} _ {h}} \sup  _ {\pi \in \Pi} d _ {h} ^ {\pi} \left(s _ {h}, a _ {h}; M\right) \leq \max  _ {M ^ {\prime} \in \mathcal {M} ^ {\det }} C _ {h} ^ {\text {r e a c h}} \left(\Pi ; M ^ {\prime}\right). \tag {3}
$$
Here,  $d_h^\pi (s_h,a_h;M)$  is the state-action visitation distribution of the policy  $\pi$  on MDP  $M$ .
We first set up additional notation. Let us define a prefix as any tuple of pairs of the form
$$
\left(s _ {1}, a _ {1}, s _ {2}, a _ {2}, \dots , s _ {k}, a _ {k}\right) \quad \text {o r} \quad \left(s _ {1}, a _ {1}, s _ {2}, a _ {2}, \dots , s _ {k}, a _ {k}, s _ {k + 1}\right).
$$
To compute the  $(K, D)$  values of  $\widetilde{\Pi_{\ell - \mathrm{ton}}}$ , the previous arguments do not go through, since the sets  $S_{\pi}$  are infinite. With a suitable extension of Definition 4 to allow for non-Markovian  $\Pi_{\mathrm{core}}$ , it is possible to show that  $\widetilde{\Pi_{\ell - \mathrm{ton}}}$  is an  $(\mathcal{O}(H^{\ell}), 0)$ -sunflower; Furthermore, the proof of Theorem 4 can be easily adapted to work under this extension.
We will denote prefix sequences as  $(s_{1:k}, a_{1:k})$  or  $(s_{1:k+1}, a_{1:k})$  respectively. For any prefix  $(s_{1:k}, a_{1:k})$  (similarly prefixes of the type  $(s_{1:k+1}, a_{1:k}))$  we let  $d_h^\pi(s_h, a_h \mid (s_{1:k}, a_{1:k}); M)$  denote the conditional probability of reaching  $(s_h, a_h)$  under policy  $\pi$  given one observed the prefix  $(s_{1:k}, a_{1:k})$  in MDP  $M$ , with  $d_h^\pi(s_h, a_h \mid (s_{1:k}, a_{1:k}); M) = 0$  if  $\pi \not\to (s_{1:k}, a_{1:k})$  or  $\pi \not\to (s_h, a_h)$ .
In the following proof, we assume that the start state  $s_1$  is fixed, but this is without any loss of generality, and the proof can easily be adapted to hold for stochastic start states.
Our strategy will be to explicitly compute the quantity  $\Gamma_h$  in terms of the dynamics of  $M$  and show that we can upper bound it by a "derandomized" MDP  $M'$  which maximizes reachability at layer  $h$ . Let us unroll one step of the dynamics:
$$
\begin{array}{l} \Gamma_ {h} := \sum_ {s _ {h} \in \mathcal {S} _ {h}, a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} d _ {h} ^ {\pi} (s _ {h}, a _ {h}; M) \\ \stackrel {(i)} {=} \sum_ {s _ {h} \in \mathcal {S} _ {h}, a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} d _ {h} ^ {\pi} (s _ {h}, a _ {h} \mid s _ {1}; M), \\ \stackrel {(i i)} {=} \sum_ {s _ {h} \in S _ {h}, a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} \left\{\sum_ {a _ {1} \in \mathcal {A}} d _ {h} ^ {\pi} \left(s _ {h}, a _ {h} \mid s _ {1}, a _ {1}; M\right) \right\} \\ \stackrel {(i i i)} {\leq} \sum_ {a _ {1} \in \mathcal {A}} \sum_ {s _ {h} \in \mathcal {S} _ {h}, a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} d _ {h} ^ {\pi} (s _ {h}, a _ {h} \mid s _ {1}, a _ {1}; M). \\ \end{array}
$$
The equality  $(i)$  follows from the fact that  $M$  always starts at  $s_1$ . The equality  $(ii)$  follows from the fact that  $\pi$  is deterministic, so there exists exactly one  $a' = \pi(s_1)$  for which  $d_h^\pi(s_h, a_h \mid s_1, a'; M) = d_h^\pi(s_h, a_h \mid s_1; M)$ , with all other  $a'' \neq a'$  satisfying  $d_h^\pi(s_h, a_h \mid s_1, a''; M) = 0$ . The inequality  $(iii)$  follows by swapping the supremum and the sum.
Continuing in this way, we can show that
$$
\begin{array}{l} \Gamma_ {h} = \sum_ {a _ {1} \in \mathcal {A}} \sum_ {s _ {h} \in \mathcal {S} _ {h}, a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} \left\{\sum_ {s _ {2} \in \mathcal {S} _ {2}} P (s _ {2} | s _ {1}, a _ {1}) \sum_ {a _ {2} \in \mathcal {A}} d _ {h} ^ {\pi} (s _ {h}, a _ {h} \mid (s _ {1: 2}, a _ {1: 2}); M) \right\} \\ \leq \sum_ {a _ {1} \in \mathcal {A}} \sum_ {s _ {2} \in \mathcal {S} _ {2}} P (s _ {2} | s _ {1}, a _ {1}) \sum_ {a _ {2} \in \mathcal {A}} \sum_ {s _ {h} \in \mathcal {S} _ {h}, a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} d _ {h} ^ {\pi} (s _ {h}, a _ {h} \mid (s _ {1: 2}, a _ {1: 2}); M) \\ \end{array}
$$
$$
\begin{array}{l} \leq \sum_ {a _ {1} \in \mathcal {A}} \sum_ {s _ {2} \in \mathcal {S} _ {2}} P (s _ {2} | s _ {1}, a _ {1}) \sum_ {a _ {2} \in \mathcal {A}} \dots \sum_ {s _ {h - 1} \in \mathcal {S} _ {h - 1}} P (s _ {h - 1} | s _ {h - 2}, a _ {h - 2}) \\ \times \sum_ {a _ {h - 1} \in \mathcal {A}} \sum_ {s _ {h} \in \mathcal {S} _ {h}, a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} d _ {h} ^ {\pi} (s _ {h}, a _ {h} \mid (s _ {1: h - 1}, a _ {1: h - 1}); M). \\ \end{array}
$$
Now we examine the conditional visitation  $d_h^\pi(s_h, a_h \mid (s_{1:h-1}, a_{1:h-1}); M)$ . Observe that it can be rewritten as
$$
d _ {h} ^ {\pi} \left(s _ {h}, a _ {h} \mid \left(s _ {1: h - 1}, a _ {1: h - 1}\right); M\right) = P \left(s _ {h} \mid s _ {h - 1}, a _ {h - 1}\right) \cdot \mathbb {1} \left\{\pi \rightsquigarrow \left(s _ {1: h}, a _ {1: h}\right)\right\}.
$$
Plugging this back into the previous display, and again swapping the supremum and the sum, we get that
$$
\begin{array}{l} \Gamma_ {h} \leq \sum_ {a _ {1} \in \mathcal {A}} \dots \sum_ {s _ {h} \in \mathcal {S} _ {h}} \mathrm {P} (s _ {h} | s _ {h - 1}, a _ {h - 1}) \sum_ {a _ {h} \in \mathcal {A}} \sup  _ {\pi \in \Pi} \mathbb {1} \left\{\pi \rightsquigarrow \left(s _ {1: h}, a _ {1: h}\right)\right\} \\ = \sum_ {a _ {1} \in \mathcal {A}} \dots \sum_ {s _ {h} \in \mathcal {S} _ {h}} P (s _ {h} | s _ {h - 1}, a _ {h - 1}) \sum_ {a _ {h} \in \mathcal {A}} \mathbb {1} \left\{\exists \pi \in \Pi : \pi \rightsquigarrow (s _ {1: h}, a _ {1: h}) \right\} \\ \end{array}
$$
Our last step is to derandomize the stochastic transitions in the above stochastic MDP, simply by taking the sup over the transition probabilities:
$$
\Gamma_ {h} \leq \sum_ {a _ {1} \in \mathcal {A}} \sup  _ {s _ {2} \in \mathcal {S} _ {2}} \sum_ {a _ {2} \in \mathcal {A}} \dots \sup  _ {s _ {h} \in \mathcal {S} _ {h}} \sum_ {a _ {h} \in \mathcal {A}} \mathbb {1} \{\exists \pi \in \Pi : \pi \rightsquigarrow (s _ {1: h}, a _ {1: h}) \} = \max  _ {M ^ {\prime} \in \mathcal {M} ^ {\mathrm {d e t}}} C _ {h} ^ {\mathrm {r e a c h}} (\Pi ; M ^ {\prime}).
$$
The right hand side of the inequality is exactly the definition of  $\max_{M' \in \mathcal{M}^{\mathrm{det}}} C_h^{\mathrm{reach}}(\Pi; M')$ , thus proving Eq. (3). In particular, the above process defines the deterministic MDP which maximizes the reachability at level  $h$ . Taking the maximum over  $h$  as well as supremum over  $M$ , we see that  $\sup_{M \in \mathcal{M}^{\mathrm{sto}}} C^{\mathrm{cov}}(\Pi; M) \leq \mathfrak{C}(\Pi)$ . Furthermore, from the definitions we have
$$
\mathfrak {C} (\Pi) = \sup  _ {M \in \mathcal {M} ^ {\mathrm {d e t}}} C ^ {\mathrm {c o v}} (\Pi ; M) \leq \sup  _ {M \in \mathcal {M} ^ {\mathrm {s t o}}} C ^ {\mathrm {c o v}} (\Pi ; M).
$$
This concludes the proof of Lemma 1.
C.2 Coverability is Not Sufficient for Online RL
In this section, we observe that bounded coverability by itself is not sufficient to ensure sample efficient agnostic PAC RL in the online interactive model. First note that Theorem 3 already shows this indirectly. In particular, in Theorem 3, we show that there exists a policy class with bounded spanning capacity that is hard to learn in online RL. However, recall Lemma 1 which implies that any policy class with bounded spanning capacity must also have bounded coverability; and thus the lower bound in Theorem 3 can be trivially extended to argue that bounded coverability by itself does not suffice for statistically efficient agnostic online RL.
However, we can also show the insufficiency of coverability through a much simpler route by directly invoking the lower bound construction in Sekhari et al. (2021). In particular, Sekhari et al. (2021) provides a construction for a low rank MDP with rich observations which satisfies  $C^{\mathrm{cov}}(\Pi;M) = \mathcal{O}(1)$  for every  $M \in \mathcal{M}$ , but still needs  $2^{\Omega(H)}$  many samples for any  $(\Theta(1),\Theta(1))$ -PAC learner (see Theorem 2 in their paper for more details; we simply set  $d = \Theta(H)$  to get our lower bound).
We do not know if coverability is also insufficient for the generative model setting; we conjecture that one may be able to show, using a similar construction, that coverability is insufficient when  $\mathfrak{C}(\Pi)$  is large, showing that one cannot adapt to benign problem instances.
Algorithm 2 TrajectoryTree (Kearns et al., 1999)
Require: Policy class  $\Pi$ , generative access to the underlying MDP  $M$ , number of samples  $n$   
1: Initialize dataset of trajectory trees  $\mathcal{D} = \emptyset$ .  
2: for  $i = 1, \ldots, n$  do  
3: Initialize trajectory tree  $\widehat{T}_i = \emptyset$ .  
4: Sample initial state  $s_1^{(i)} \sim \mu$ .  
5: while True do // Sample transitions and rewards for a trajectory tree  
6: Find any unsampled  $(s, a)$  s.t.  $(s, a)$  is reachable in  $\widehat{T}_i$  by some  $\pi \in \Pi$ .  
7: if no such  $(s, a)$  exists then break  
8: end if  
9: Sample  $s' \sim P(\cdot | s, a)$  and  $r \sim R(s, a)$   
10: Add transition  $(s, a, r, s')$  to  $\widehat{T}_i$ .  
11: end while  
12:  $\mathcal{D} \gets \mathcal{D} \cup \widehat{T}_i$ .  
13: end for  
14: for  $\pi \in \Pi$  do // Policy evaluation  
15: Set  $\widehat{V}^{\pi} \gets \frac{1}{n} \sum_{i=1}^{n} \widehat{v}_i^{\pi}$ , where  $\widehat{v}_i^{\pi}$  is the cumulative reward of  $\pi$  on  $\widehat{T}_i$ .  
16: end for  
17: Return  $\widehat{\pi} \gets \arg \max_{\pi \in \Pi} \widehat{V}^{\pi}$ .
We show that, with minor changes, the TrajectoryTree algorithm of Kearns et al. (1999) attains the guarantee in Theorem 1. The pseudocode can be found in Algorithm 2. The key modification is line 6: we simply observe that only  $(s,a)$  pairs which are reachable by some  $\pi \in \Pi$  in the current
tree  $\widehat{T}_i$  need to be sampled (in contrast, in the original algorithm of Kearns et al. (1999), they sample all  $A^H$  transitions).
Fix any  $\pi \in \Pi$ . For every trajectory tree  $i \in [n]$ , the algorithm has collected enough transitions so that  $\widehat{v}_i^\pi$  is well-defined, by line 6 of the algorithm. By the sampling process, it is clear that the values  $\{\widehat{v}_i^\pi\}_{i \in [n]}$  are i.i.d. generated. We claim that they are unbiased estimates of  $V^\pi$ . Observe that one way of defining  $V^\pi$  is the expected value of the following process:
(1) For every  $(s,a)\in S\times \mathcal{A}$ , independently sample a next state  $s^{\prime}\sim P(\cdot |s,a)$  and a reward  $r\sim R(s,a)$  to define a deterministic MDP  $\widehat{M}$
(2) Return the value  $\widehat{v}^{\pi}$  to be the value of  $\pi$  run on  $\widehat{M}$ .
Define the law of this process as  $\overline{\mathcal{Q}}$ . The sampling process of TrajectoryTree (call the law of this process  $\mathcal{Q}$ ) can be viewed as sampling the subset of  $\widehat{M}^{\mathrm{det}}$  which is reachable by some  $\pi \in \Pi$ . Thus, we have
$$
V ^ {\pi} = \mathbb {E} _ {\widehat {M} \sim \overline {{\mathcal {Q}}} [ \widehat {v} ^ {\pi} ]} = \mathbb {E} _ {\widehat {T} \sim \mathcal {Q}} \left[ \mathbb {E} \left[ \widehat {v} ^ {\pi} \mid \widehat {T} \right] \right] = \mathbb {E} _ {\widehat {T} \sim \mathcal {Q}} [ \widehat {v} ^ {\pi} ],
$$
where the second equality is due to the law of total probability, and the third equality is due to the fact that  $\widehat{v}^{\pi}$  is measurable with respect to the trajectory tree  $\widehat{T}$ . Thus,  $\{\widehat{v}_i^\pi\}_{i\in [n]}$  are unbiased estimates of  $V^{\pi}$ .
Therefore, by Hoeffding's inequality (Lemma 17) we see that  $|V^{\pi} - \widehat{V}^{\pi}| \leq \sqrt{\frac{\log(2 / \delta)}{2n}}$ . Applying union bound we see that when the number of trajectory trees exceeds  $n \gtrsim \frac{\log(|\Pi| / \delta)}{\varepsilon^2}$ , with probability at least  $1 - \delta$ , for all  $\pi \in \Pi$ , the estimates satisfy  $|V^{\pi} - \widehat{V}^{\pi}| \leq \varepsilon / 2$ . Thus the TrajectoryTree algorithm returns an  $\varepsilon$ -optimal policy. Since each trajectory tree uses at most  $H \cdot \mathfrak{C}(\Pi)$  queries to the generative model, we have the claimed sample complexity bound.
Fix any worst-case deterministic MDP  $M^{\star}$  which witnesses  $\mathfrak{C}(\Pi)$  at layer  $h^\star$ . Since  $\mathfrak{C}(\Pi)$  is a property depending on the dynamics of  $M^{\star}$ , we can assume that  $M^{\star}$  has zero rewards. We can also assume that the algorithm knows  $M^{\star}$  and  $h^\star$  (this only makes the lower bound stronger). We construct a family of instances  $\mathcal{M}^{\star}$  where all the MDPs in  $\mathcal{M}^{\star}$  have the same dynamics as  $M^{\star}$  but different nonzero rewards at the reachable  $(s, a)$  pairs at layer  $h^\star$ .
Observe that we can embed a multi-armed bandit instance with  $\mathfrak{C}(\Pi)$  arms using the class  $\mathcal{M}^{\star}$ . The value of any policy  $\pi \in \Pi$  is exactly the reward that it receives at the unique  $(s, a)$  pair in layer  $h^{\star}$  that it reaches. Any  $(\varepsilon, \delta)$ -PAC algorithm that works over the family of instances  $\mathcal{M}^{\star}$  must return a policy  $\hat{\pi}$  that reaches an  $(s, a)$  pair in layer  $h^{\star}$  with near-optimal reward. Furthermore, in the generative model setting, the algorithm can only receive information about a single  $(s, a)$  pair. Thus, such a PAC algorithm must also be able to PAC learn the best arm for multi-armed bandits with  $\mathfrak{C}(\Pi)$  arms. Therefore, we can directly apply existing PAC lower bounds which show that the sample complexity of  $(\varepsilon, \delta)$ -PAC learning the best arm for  $K$ -armed multi-armed bandits is at least  $\Omega\left(\frac{K}{\varepsilon^{2}} \cdot \log \frac{1}{\delta}\right)$  (see, e.g., Mannor and Tsitsiklis, 2004).
The upper bound is obtained by a simple modification of the argument in the proof of Theorem 1. In terms of data collection, the trajectory tree collected every time is the same fixed deterministic MDP (with different rewards); furthermore, one can always execute line 6 and line 9 for a deterministic MDP since the algorithm can execute a sequence of actions to get to any new  $(s,a)$  pair required by line 9. Thus in every episode of online interaction we are guaranteed to add the new  $(s,a)$  pair to the trajectory tree.
The lower bound trivially extends because the proof of Appendix D.2 uses a family of MDPs with deterministic transitions (that are even known to the algorithm beforehand).
E Proofs for Section 5
In this section, we prove Theorem 3, which shows a superpolynomial lower bound on the sample complexity required to learn bounded spanning capacity classes, ruling out poly  $(\mathfrak{C}(\Pi), H, \log |\Pi|)$  sample complexity for online RL. We restate the theorem below with the precise constants:
Theorem 5 (Lower bound for online RL). Let  $h_0 \in \mathbb{N}$  and  $c \in (0,1)$  be universal constants. Fix any  $H \geq h_0$ . Let  $\varepsilon \in (1/2^{cH}, 1/(100H))$  and  $\ell \in \{2, \ldots, H\}$  such that  $1 / \varepsilon^\ell \leq 2^H$ . There exists a policy class  $\Pi^{(\ell)}$  of size  $1/(6\varepsilon^\ell)$  with  $\mathfrak{C}(\Pi^{(\ell)}) \leq O(H^{4\ell + 2})$  and a family of MDPs  $\mathcal{M}$  with state space  $S$  of size  $H \cdot 2^{2H + 1}$ , binary action space, horizon  $H$  such that: for any  $(\varepsilon / 16, 1/8)$ -PAC algorithm, there exists an  $M \in \mathcal{M}$  in which the algorithm has to collect at least
$$
\min \left\{\frac {1}{1 2 0 \varepsilon^ {\ell}}, 2 ^ {H / 3 - 3} \right\} \quad o n l i n e t r a j e c t o r i e s i n e x p e c t a t i o n.
$$
E.1 Construction of State Space, Action Space, and Policy Class
State and Action Spaces. We define the state space  $\mathcal{S}$ . In every layer  $h \in [H]$ , there will be  $2^{2H + 1}$  states. The states will be paired up, and each state will be denoted by either  $j[h]$  or  $j'[h]$ , so  $S_{h} = \{j[h]: j \in [2^{2H}]\} \cup \{j'[h]: j \in [2^{2H}]\}$ . For any state  $s \in \mathcal{S}$ , we define the index of  $s$ , denoted  $\mathrm{idx}(s)$  as the unique  $j \in [2^{2H}]$  such that  $s \in \{j[h]\}_{h \in [H]} \cup \{j'[h]\}_{h \in [H]}$ . In total there are  $H \cdot 2^{2H + 1}$  states. The action space is  $\mathcal{A} = \{0,1\}$ .
Policy Class. For the given  $\varepsilon$  and  $\ell \in \{2,\dots ,H\}$ , we show via a probabilistic argument the existence of a large policy class  $\Pi^{(\ell)}$  which has bounded spanning capacity but is hard to explore. We state several properties in Lemma 2 which will be exploited in the lower bound.
We introduce some additional notation. For any  $j \in [2^{2H}]$  we denote
$$
\Pi_ {j} ^ {(\ell)} := \{\pi \in \Pi^ {(\ell)}: \exists h \in [ H ], \pi (j [ h ]) = 1 \},
$$