SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} V ^ {\pi} - V _ {\mathrm {M R P}} ^ {\pi} = \mathbb {E} ^ {\pi} [ R [ \tau ] \mathbb {1} \{\tau \cap S _ {\pi} ^ {\mathrm {r e m}} \neq \emptyset \} - \mathbb {1} \{\exists k \in [ H ]: s _ {h _ {k}} \in S _ {\pi} ^ {\mathrm {r e m}} \text {a n d} \forall h > h _ {k}: s _ {h} \notin S _ {\pi} \} ] \\ \leq \mathbb {E} ^ {\pi} [ R [ \tau ] \mathbb {1} \{\tau \cap \mathcal {S} _ {\pi} ^ {\mathrm {r e m}} \neq \emptyset \} ] \\ \leq D \cdot \frac {\varepsilon}{4 D} = \frac {\varepsilon}{4}, \\ \end{array}
$$
where the first inequality follows by just ignoring the second indicator term, and the second inequality follows by taking a union bound over all possible values of  $S_{\pi}^{\mathrm{rem}}$  as well as the conclusion of Lemma 13.
Putting it all together, we get that
$$
\left| \widehat {V} ^ {\pi} - V ^ {\pi} \right| \leq \left| V ^ {\pi} - V _ {\mathrm {M R P}} ^ {\pi} \right| + \left| \widehat {V} ^ {\pi} - V _ {\mathrm {M R P}} ^ {\pi} \right| \leq \frac {\varepsilon}{4} + \frac {\varepsilon}{2} <   \varepsilon .
$$
This concludes the proof of Lemma 14.
F.4 Proof of Theorem 4
We assume the events defined in Lemmas 10, 12 and 13 hold (which happens with probability at least  $1 - 2\delta$ ). With our choices of  $n_1, n_2$  in Eq. (21), the total number of samples used in our algorithm is at most
$$
n _ {1} + n _ {2} \cdot \frac {1 2 H D \mathfrak {C} (\Pi)}{\varepsilon} = \widetilde {\mathcal {O}} \left(\left(\frac {1}{\varepsilon^ {2}} + \frac {H D ^ {6} \mathfrak {C} (\Pi)}{\varepsilon^ {4}}\right) \cdot K ^ {2} \log \frac {| \Pi |}{\delta}\right).
$$
After the termination of the while loop, we know that for any policy  $\pi \in \Pi$  and  $s\in S_{\pi}^{\mathrm{rem}}$  we have
$$
\bar {d} ^ {\pi} (s; \neg \mathcal {S} _ {\pi} ^ {\mathrm {r e m}}) \leq \frac {\varepsilon}{4 D}.
$$
Therefore, by Lemma 14, we know for every  $\pi \in \Pi$ ,  $|\widehat{V}^{\pi} - V^{\pi}| \leq \varepsilon$ . Hence the output policy  $\widehat{\pi} \in \arg \max_{\pi} \widehat{V}^{\pi}$  satisfies
$$
\max  _ {\pi \in \Pi} V ^ {\pi} - V ^ {\widehat {\pi}} \leq 2 \varepsilon + \widehat {V} ^ {\pi} - \widehat {V} ^ {\widehat {\pi}} \leq 2 \varepsilon .
$$
Rescaling  $\varepsilon$  by  $2\varepsilon$  and  $\delta$  by  $2\delta$  concludes the proof of Theorem 4.
F.5 Sunflower Property is Insufficient By Itself
We give an example of a policy class  $\Pi$  for which the sunflower property holds for  $K$ ,  $D = \mathrm{poly}(H)$  but  $\mathfrak{C}(\Pi) = 2^H$ . Therefore, in light of Theorem 2, the sunflower property by itself cannot ensure statistically efficient agnostic PAC RL in the online access model.
The example is as follows: Consider a binary tree MDP with  $2^{H} - 1$  states and action space  $\mathcal{A} = \{0,1\}$ . The policy class  $\Pi$  will be able to get to every  $(s,a)$  pair in layer  $H$ . To define the policies, we consider each possible trajectory  $\tau = (s_{1},a_{1},\dots ,s_{H},a_{H})$  and let:
$$
\Pi := \left\{\pi_ {\tau}: \pi_ {\tau} (s) = \left\{ \begin{array}{l l} a _ {i} & \text {i f} s _ {i} \in \tau , \\ 0 & \text {o t h e r w i s e}, \end{array} \right. \right\}.
$$
Thus it is clear that  $\mathfrak{C}(\Pi) = 2^H$ , but the sunflower property holds with  $K = 1$ ,  $D = H$  by taking  $\Pi_{\mathrm{core}} = \{\pi_0\}$  (the policy which always picks  $a = 0$ ).
G Infinite Policy Classes
In this section we discuss the extensions of our results to infinite policy classes.
G.1 Definitions and Preliminary Lemmas
We will state our results in terms of the Natarajan dimension, which is a generalization of the VC dimension used to study multiclass learning. We note that the results in this section could be stated in terms of other complexity measures from multiclass learning such as the graph dimension and DS dimension (see, e.g., Natarajan, 1989; Shalev-Shwartz and Ben-David, 2014; Daniely and Shalev-Shwartz, 2014; Brukhim et al., 2022a); for simplicity we analyze guarantees in terms of the Natarajan dimension.
Definition 7 (Natarajan Dimension (Natarajan, 1989)). Let  $\mathcal{X}$  be an instance space and  $\mathcal{Y}$  be a finite label space. Given a class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$ , we define its Natarajan dimension, denoted  $\mathrm{Ndim}(\mathcal{H})$ , to be the maximum cardinality of a set  $C \subseteq \mathcal{X}$  that satisfies the following: there exists  $h_0, h_1 : C \to \mathcal{Y}$  such that (1) for all  $x \in C$ ,  $h_0(x) \neq h_1(x)$ , and (2) for all  $B \subseteq C$ , there exists  $h \in \mathcal{H}$  such that for all  $x \in B$ ,  $h(x) = h_0(x)$  and for all  $x \in C \setminus B$ ,  $h(x) = h_1(x)$ .
A notation we will use throughout is the projection operator. For a hypothesis class  $\mathcal{H} \subseteq \mathcal{V}^{\mathcal{X}}$  and a finite set  $X = (x_{1},\dots ,x_{n}) \in \mathcal{X}^{n}$ , we define the projection of  $\mathcal{H}$  on to  $X$  as
$$
\left. \mathcal {H} \right| _ {X} := \left\{\left(h \left(x _ {1}\right), \dots , h \left(x _ {n}\right)\right): h \in \mathcal {H} \right\}.
$$
Lemma 15 (Sauer's Lemma for Natarajan Classes (Haussler and Long, 1995)). Given a hypothesis class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  with  $|\mathcal{Y}| = K$  and  $\mathrm{Ndim}(\mathcal{H}) \leq d$ , we have for every  $X = (x_{1}, \dots, x_{n}) \in \mathcal{X}^{n}$
$$
\left| \mathcal {H} \right| _ {X} \Big | \leq \left(\frac {n e (K + 1) ^ {2}}{2 d}\right) ^ {d}.
$$
Theorem 6 (Multiclass Fundamental Theorem (Shalev-Shwartz and Ben-David, 2014)). For any class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  with  $\mathrm{Ndim}(\mathcal{H}) = d$  and  $|\mathcal{Y}| = K$ , the minimax sample complexity of  $(\varepsilon, \delta)$  agnostic PAC learning  $\mathcal{H}$  can be bounded as
$$
\Omega \left(\frac {d + \log (1 / \delta)}{\varepsilon^ {2}}\right) \leq n (\Pi ; \varepsilon , \delta) \leq \mathcal {O} \left(\frac {d \log K + \log (1 / \delta)}{\varepsilon^ {2}}\right).
$$
Definition 8 (Pseudodimension). Let  $\mathcal{X}$  be an instance space. Given a hypothesis class  $\mathcal{H} \subseteq \mathbb{R}^{\mathcal{X}}$ , its pseudodimension, denoted  $\mathrm{Pdim}(\mathcal{H})$ , is defined as  $\mathrm{Pdim}(\mathcal{H}) := \mathrm{VC}(\mathcal{H}^{+})$ , where  $\mathcal{H}^{+} := \{(x,\theta) \mapsto \mathbb{1}\{h(x) \leq \theta\} : h \in \mathcal{H}\}$ .
Definition 9 (Covering Numbers). Given a hypothesis class  $\mathcal{H} \subseteq \mathbb{R}^{\mathcal{X}}$ ,  $\alpha > 0$ , and  $X = (x_{1}, \dots, x_{n}) \in \mathcal{X}^{n}$ , the covering number  $\mathcal{N}_{1}(\mathcal{H}, \alpha, X)$  is the minimum cardinality of a set  $C \subset \mathbb{R}^{n}$  such that for any  $h \in \mathcal{H}$  there exists a  $c \in C$  such that  $\frac{1}{n} \sum_{i=1}^{n} |h(x_{i}) - c_{i}| \leq \alpha$ .
Lemma 16 (Jiang et al. (2017), see also Pollard (2012); Luc et al. (1996)). Let  $\mathcal{H} \subset [0,1]^{\mathcal{X}}$  be a real-valued hypothesis class, and let  $X = (x_{1},\dots ,x_{n})$  be i.i.d. samples drawn from some distribution  $\mathcal{D}$  on  $\mathcal{X}$ . Then for any  $\alpha > 0$
$$
\mathbb {P} \left[ \sup  _ {h \in \mathcal {H}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} h \left(x _ {i}\right) - \mathbb {E} [ h (x) ] \right| > \alpha \right] \leq 8 \mathbb {E} \left[ \mathcal {N} _ {1} (\mathcal {H}, \alpha / 8, X) \right] \cdot \exp \left(- \frac {n \alpha^ {2}}{1 2 8}\right).
$$
Furthermore if  $\mathrm{Pdim}(\mathcal{H})\leq d$  then we have the bound
$$
\mathbb {P} \left[ \sup  _ {h \in \mathcal {H}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} h \left(x _ {i}\right) - \mathbb {E} [ h (x) ] \right| > \alpha \right] \leq 8 e (d + 1) \left(\frac {1 6 e}{\alpha}\right) ^ {d} \cdot \exp \left(- \frac {n \alpha^ {2}}{1 2 8}\right),
$$
which is at most  $\delta$  as long as  $n \geq \frac{128}{\alpha^2} \left( d \log \frac{16e}{\alpha} + \log (8e(d + 1)) + \log \frac{1}{\delta} \right)$ .
G.2 Generative Model Lower Bound
First we address the lower bound. Observe that it is possible to achieve a lower bound that depends on  $\mathrm{Ndim}(\Pi)$  with the following construction. First, identify the layer  $h\in [H]$  such that the witnessing
set  $C$  contains the maximal number of states in  $\mathcal{S}_h$ ; by pigeonhole principle there must be at least  $\mathrm{Ndim}(\Pi) / H$  such states in layer  $h$ . Then, we construct an MDP which "embeds" a hard multiclass learning problem at layer  $h$  over these states. A lower bound of  $\Omega \left( \frac{\mathrm{Ndim}(\Pi)}{H \varepsilon^2} \cdot \log \frac{1}{\delta} \right)$  follows from Theorem 6.
By combining Theorem 2 with the above we get the following corollary.
Corollary 3 (Lower Bound for Generative Model with Infinite Policy Classes). For any policy class  $\Pi$ , the minimax sample complexity  $(\varepsilon, \delta)$ -PAC learning  $\Pi$  is at least
$$
n _ {\mathrm {g e n}} (\Pi ; \varepsilon , \delta) \geq \Omega \biggl (\frac {\mathfrak {C} (\Pi) + \mathrm {N d i m} (\Pi) / H}{\varepsilon^ {2}} \cdot \log \frac {1}{\delta} \biggr).
$$
Again, since the generative model setting is easier than online RL, this lower bound also extends to the online RL setting.
Our bound is additive in  $\mathfrak{C}(\Pi)$  and  $\mathrm{Ndim}(\Pi)$ ; we do not know if it is possible to strengthen this to be a product of the two factors, as we will achieve in the upper bound in the next section.
G.3 Generative Model Upper Bound
For the upper bounds, we can replace the dependence on  $\log |\Pi|$  with  $\mathrm{Ndim}(\Pi)$  (and additional log factors). In particular, we can modify the analysis of the TrajectoryTree to account for infinite policy classes. Recall that our analysis of TrajectoryTree required us to prove a uniform convergence guarantee for the estimate  $\widehat{V}^{\pi}$ : with probability at least  $1 - \delta$ , for all  $\pi \in \Pi$ , we have  $|\widehat{V}^{\pi} - V^{\pi}| \lesssim \varepsilon$ . We previously used a union bound over  $|\Pi|$ , which gave us the  $\log |\Pi|$  dependence. Now we sketch an argument to replace it with  $\mathrm{Ndim}(\Pi)$ .
Let  $\mathcal{T}$  be the set of all possible trajectory trees. We introduce the notation  $v^{\pi}:\mathcal{T}\to \mathbb{R}$  to denote the function that takes as input a trajectory tree  $\widehat{T}$  (for example, as sampled by TrajectoryTree) and returns the value of running  $\pi$  on it. Then we can rewrite the desired uniform convergence guarantee:
$$
\text {w . p .} 1 - \delta , \quad \sup  _ {\pi \in \Pi} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} v ^ {\pi} (\widehat {T} _ {i}) - \mathbb {E} \left[ v ^ {\pi} (\widehat {T}) \right] \right| \leq \varepsilon . \tag {28}
$$
In light of Lemma 16, we will compute the pseudodimension for the function class  $\mathcal{V}^{\Pi} = \{v^{\pi} : \pi \in \Pi\}$ . Define the subgraph class
$$
\mathcal {V} ^ {\Pi , +} := \left\{\left(\widehat {T}, \theta\right) \mapsto \mathbb {1} \left\{v ^ {\pi} (\widehat {T}) \leq \theta \right\}: \pi \in \Pi \right\} \subseteq \left\{0, 1 \right\} ^ {\mathcal {T} \times \mathbb {R}}
$$
By definition,  $\mathrm{Pdim}(\mathcal{V}^{\Pi}) = \mathrm{VC}(\mathcal{V}^{\Pi, + })$ . Fix any  $X = \left\{(\widehat{T}_1,\theta_1),\dots ,(\widehat{T}_d,\theta_d)\right\} \in (\mathcal{T}\times \mathbb{R})^d$ . In order to show that  $\mathrm{VC}(\mathcal{V}^{\Pi, + })\leq d$  for some value of  $d$  it suffices to prove that  $|\mathcal{V}^{\Pi, + }|_{X}| < 2^{d}$ .
For any index  $t \in [d]$ , we also denote  $\pi(\vec{s_i}) \in \mathcal{A}^{\leq H\mathfrak{C}(\Pi)}$  to be the vector of actions selected by  $\pi$  on all  $\Pi$ -reachable states in  $\widehat{T_i}$  (of which there are at most  $H \cdot \mathfrak{C}(\Pi)$ ). We claim that
$$
\left| \mathcal {V} ^ {\Pi , +} \right| _ {X} \leq | \left\{\left(\pi (\vec {s} _ {1}), \dots , \pi (\vec {s} _ {d})\right): \pi \in \Pi \right\} | =: \left| \Pi \right| _ {X}. \tag {29}
$$
This is true because once the  $d$  trajectory trees are fixed, for every  $\pi \in \Pi$ , the value of the vector  $\mathcal{V}^{\{\pi\}, +}\big|_{X} \in \{0,1\}^{d}$  is determined by the trajectory that  $\pi$  takes in every trajectory tree. This in turn is determined by the assignment of actions to every reachable state in all the  $d$  trajectory trees, of which there are at most  $\mathfrak{C}(\Pi) \cdot H \cdot d$  of. Therefore, we can upper bound the size of  $\mathcal{V}^{\Pi, +}\big|_{X}$  by the number of ways any  $\pi \in \Pi$  assign actions to every state in  $\widehat{T}_1, \dots, \widehat{T}_d$ .
Applying Lemma 15 to Eq. (29), we get that
$$
\left| \mathcal {V} ^ {\Pi , +} \right| _ {X} \Big | \leq \left(\frac {H \mathfrak {C} (\Pi) d \cdot e \cdot (A + 1) ^ {2}}{2 N \dim (\Pi)}\right) ^ {N \dim (\Pi)}.
$$
For the choice of  $d = \tilde{\mathcal{O}}(\mathrm{Ndim}(\Pi))$ , the previous display is at most  $2^d$ , thus proving the bound on  $\mathrm{Pdim}(\mathcal{V}^{\Pi})$ . Lastly, the bound can be plugged back into Lemma 16 to get a bound on the error of TrajectoryTree: the statement in Eq. (28) holds using
$$
n = \widetilde {\mathcal {O}} \left(H \mathfrak {C} (\Pi) \cdot \frac {\mathrm {N d i m} (\Pi) + \log \frac {1}{\delta}}{\varepsilon^ {2}}\right) \quad \text {s a m p l e s}.
$$
This in turn yields a guarantee on  $\widehat{\pi}$  returned by TrajectoryTree.
G.4 Online RL Upper Bound
The modified analysis for the online RL upper bound (Theorem 4) proceeds similarly; we sketch the ideas below.
There are two places in the proof of Theorem 4 which require a union bound over  $|\Pi|$ : the event  $\mathcal{E}_{\mathrm{est}}$  (defined by Lemma 10) that the estimated transitions and rewards of the MRPs are close to their population versions, and the event  $\mathcal{E}_{\mathrm{data}}$  (defined by Lemma 12) that the datasets collected are large enough. The latter is easy to address, since we can simply modify the algorithm's while loop to break after  $\mathcal{O}\left(\frac{HD\mathfrak{C}(\Pi)}{\varepsilon}\right)$  iterations and union bound over the size of the set  $|\mathcal{T}|$  instead of the worst-case bound on the size  $D|\Pi|$ . For  $\mathcal{E}_{\mathrm{data}}$ , we follow a similar strategy as the analysis for the generative model upper bound.
Fix a state  $s$ . Recall that the estimate for the probability transition kernel in the MDP in Eq. (19) takes the form
$$
\widehat {P} _ {s \rightarrow s ^ {\prime}} ^ {\pi} = \frac {1}{| \mathcal {D} _ {s} |} \sum_ {\tau \in \mathcal {D} _ {s}} \frac {\mathbb {1} \left\{\pi \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}}{\frac {1}{| \Pi_ {\mathrm {c o r e}} |} \sum_ {\pi^ {\prime} \in \Pi_ {\mathrm {c o r e}}} \mathbb {1} \left\{\pi_ {e} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}} \mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \right\}.
$$
(The analysis for the rewards is similar, so we omit it from this proof sketch.)
We set up some notation. Define the function  $p_{s\rightarrow s^{\prime}}^{\pi}:(S\times \mathcal{A}\times \mathbb{R})^{H}\to [0,|\Pi_{\mathrm{core}}|]$  as
$$
p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau) := \frac {\mathbb {1} \left\{\pi \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}}{\frac {1}{| \Pi_ {\text {c o r e}} |} \sum_ {\pi^ {\prime} \in \Pi_ {\text {c o r e}}} \mathbb {1} \left\{\pi_ {e} \rightsquigarrow \tau_ {h : h ^ {\prime}} \right\}} \mathbb {1} \left\{\tau \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \right\}, \tag {30}
$$
with the implicit restriction of the domain to trajectories  $\tau$  for which the denominator is nonzero. We have  $\mathbb{E}[p_{s\to s'}^\pi (\tau)] = P_{s\to s'}^\pi$ . Also let  $\Pi_s = \{\pi \in \Pi : s \in S_\pi\}$ .
Restated in this notation, our objective is to show the uniform convergence guarantee
$$
\text {w . p .} 1 - \delta , \quad \sup  _ {\pi \in \Pi_ {s}, s ^ {\prime} \in \mathcal {S} _ {\pi}} \left| \frac {1}{\left| \mathcal {D} _ {s} \right|} \sum_ {\tau \in \mathcal {D} _ {s}} p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau) - \mathbb {E} \left[ p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau) \right]\right| \leq \varepsilon . \tag {31}
$$
Again, in light of Lemma 16, we need to compute the pseudodimension for the function class  $\mathcal{P}^{\Pi_s} = \{p_{s\to s'}^\pi :\pi \in \Pi_s,s'\in S_\pi \}$ , since these are all possible transitions that we might use the dataset  $\mathcal{D}_s$  to evaluate. Define the subgraph class
$$
\mathcal {P} ^ {\Pi_ {s}, +} := \left\{\left(\tau , \theta\right) \mapsto \mathbb {1} \left\{p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau) \leq \theta \right\}: \pi \in \Pi_ {s}, s ^ {\prime} \in \mathcal {S} _ {\pi} \right\}.
$$
Fix the set  $X = \{(\tau_1,\theta_1),\dots ,(\tau_d,\theta_d)\} \in ((\mathcal{S}\times \mathcal{A}\times \mathbb{R})^H\times \mathbb{R})^d$  where the trajectories  $\tau_{1},\dots ,\tau_{d}$  pass through  $s$ . We also denote  $S_{X}$  to be the union of all states which appear in  $\tau_{1},\dots ,\tau_{d}$ . In order to show a bound that  $\mathrm{Pdim}(\mathcal{P}^{\Pi_s})\leq d$  it suffices to prove that  $|\mathcal{P}^{\Pi_s, + }|_X| < 2^d$ .
We first observe that
$$
\left| \mathcal {P} ^ {\Pi_ {s}, +} \right| _ {X} | \leq 1 + \sum_ {s ^ {\prime} \in \mathcal {S} _ {X}} | \left\{\left(\mathbb {1} \left\{p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau_ {1}) \leq \theta_ {1} \right\}, \dots , \mathbb {1} \left\{p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau_ {d}) \leq \theta_ {d} \right\}\right): \pi \in \Pi_ {s} \right\}.
$$
The inequality follows because for any choice  $s' \notin S_X$ , we have
$$
\left(\mathbb {1} \left\{p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau_ {1}) \leq \theta_ {1} \right\}, \dots , \mathbb {1} \left\{p _ {s \rightarrow s ^ {\prime}} ^ {\pi} (\tau_ {d}) \leq \theta_ {d} \right\}\right) = \vec {0},
$$
no matter what  $\pi$  is, contributing at most 1 to the count. Furthermore, once we have fixed  $s'$  and the  $\{\tau_1,\dots ,\tau_d\}$  the quantities  $\frac{1}{|\Pi_{\mathrm{core}}|}\sum_{\pi '\in \Pi_{\mathrm{core}}}\mathbb{1}\bigl {\{}\pi_e\rightsquigarrow \tau_{i,h:h'}\bigr \}$  for every  $i\in [d]$  are constant (do not depend on  $\pi$ ), so we can reparameterize  $\theta_i^{\prime}:= \theta_i\cdot \frac{1}{|\Pi_{\mathrm{core}}|}\sum_{\pi '\in \Pi_{\mathrm{core}}}\mathbb{1}\bigl {\{}\pi_e\rightsquigarrow \tau_{i,h:h'}\bigr \}$  to get:
$$
\left| \mathcal {P} ^ {\Pi_ {s, +}} \right| _ {X} | \leq 1 + \sum_ {s ^ {\prime} \in \mathcal {S} _ {X}} \left| \left\{\left(b _ {1} (\pi), \dots , b _ {d} (\pi)\right): \pi \in \Pi \right\} \right|, \tag {32}
$$
$$
\text {w h e r e} \quad b _ {i} (\pi) := \mathbb {1} \left\{\mathbb {1} \left\{\pi \rightsquigarrow \tau_ {i, h: h ^ {\prime}} \right\} \mathbb {1} \left\{\tau_ {i} \in \mathfrak {T} _ {\pi} (s \rightarrow s ^ {\prime}) \right\} \leq \theta_ {i} ^ {\prime} \right\}.
$$
Now we count how many values the vector  $(b_{1}(\pi),\dots ,b_{d}(\pi))$  can take for different  $\pi \in \Pi_s$ . Without loss of generality, we can (1) assume that the  $\theta_i^\prime = 0$  (since a product of indicators can only take values in  $\{0,1\}$ , and if  $\theta^{\prime}\geq 1$  then we must have  $b_{i}(\pi) = 1$  for every  $\pi$ ), and (2)  $s^\prime \in \tau_i$  for each  $i\in [d]$  (otherwise  $b_{i}(\pi) = 0$  for every  $\pi \in \Pi$ ). So we can rewrite  $b_{i}(\pi) = \mathbb{1}\{\pi \rightsquigarrow \tau_{i,h:h^{\prime}}\} \mathbb{1}\{\tau_i\in \mathfrak{T}_\pi (s\to s')\}$ . For every fixed choice of  $s^\prime$  we upper bound the size of the set as:
$$
\begin{array}{l} \left. \left| \left\{\left(b _ {1} (\pi), \dots , b _ {d} (\pi)\right): \pi \in \Pi \right\} \right| \right. \\ \stackrel {(i)} {\leq} \left|\left\{\left(\mathbb {1} \left\{\pi \rightsquigarrow \tau_ {1, h: h ^ {\prime}} \right\}, \dots , \mathbb {1} \left\{\pi \rightsquigarrow \tau_ {d, h: h ^ {\prime}} \right\}\right): \pi \in \Pi \right\}\right| \\ \times \left|\left\{\left(\mathbb {1} \left\{\tau_ {1} \in \mathfrak {T} _ {\pi} \left(s \rightarrow s ^ {\prime}\right)\right\}, \dots , \mathbb {1} \left\{\tau_ {d} \in \mathfrak {T} _ {\pi} \left(s \rightarrow s ^ {\prime}\right)\right\}\right): \pi \in \Pi \right\}\right| \\ \stackrel {(i i)} {\leq} \left| \left\{\left(\pi (s ^ {(1)}), \pi (s ^ {(2)}), \dots , \pi (s ^ {(d H)})\right): \pi \in \Pi \right\} \right| \times \left| \left\{\left(\mathbb {1} \left\{s ^ {(1)} \in \mathcal {S} _ {\pi} \right\}, \dots , \mathbb {1} \left\{s ^ {(d H)} \in \mathcal {S} _ {\pi} \right\}\right): \pi \in \Pi \right\} \right| \\ \stackrel {(i i i)} {\leq} \left(\frac {d H \cdot e (A + 1) ^ {2}}{2 \mathrm {N d i m} (\Pi)}\right) ^ {\mathrm {N d i m} (\Pi)} \times (d H) ^ {D}. \tag {33} \\ \end{array}
$$
The inequality  $(i)$  follows by upper bounding by the Cartesian product. The inequality  $(ii)$  follows because (1) for the first term, the vector  $(\mathbb{1}\{\pi \rightsquigarrow \tau_{1,h:h'}\}, \dots, \mathbb{1}\{\pi \rightsquigarrow \tau_{d,h:h'}\})$  is determined by the number of possible behaviors  $\pi$  has over all  $dH$  states in the trajectories, and (2) for the second term, the vector  $(\mathbb{1}\{\tau_1 \in \mathfrak{T}_{\pi}(s \to s')\}, \dots, \mathbb{1}\{\tau_d \in \mathfrak{T}_{\pi}(s \to s')\})$  is determined by which of the  $dH$  states lie in the petal set for  $S_{\pi}$ . The inequality  $(iii)$  follows by applying Lemma 15 to the first term and Sauer's Lemma to the second term, further noting that every petal  $S_{\pi}$  set has cardinality at most  $D$ .
Combining Eqs. (32) and (33) we get the final bound that
$$
\left| \mathcal {P} ^ {\Pi_ {s, +}} \right| _ {X} | \leq 1 + (d H) ^ {D + 1} \cdot \left(\frac {d H \cdot e (A + 1) ^ {2}}{2 \mathrm {N d i m} (\Pi)}\right) ^ {\mathrm {N d i m} (\Pi)}.
$$
To conclude the calculation, we observe that this bound is  $< 2^d$  whenever  $d = \widetilde{\mathcal{O}}(D + \mathrm{Ndim}(\Pi))$ , which we can again use in conjunction with Lemma 16 to prove the desired uniform convergence statement found in Eq. (31). Ultimately this allows us to replace the  $\log |\Pi|$  with  $\widetilde{\mathcal{O}}(D + \mathrm{Ndim}(\Pi))$  in the upper bound of Theorem 4; the precise details are omitted.
H Connections to Other Complexity Measures
We show relationships between the spanning capacity and several other combinatorial measures of complexity.
For every  $h \in [H]$  we denote the state space at layer  $h$  as  $S_{h} := \{s_{(j,h)} : j \in [K]\}$  for some  $K \in \mathbb{N}$ . We will restrict ourselves to binary action spaces  $\mathcal{A} = \{0,1\}$ , but the definitions and results can be extended to larger (but finite) action spaces. In addition, we will henceforth assume that all policy classes  $\Pi$  under consideration satisfy the following stationarity assumption.
Assumption 1. The policy class  $\Pi$  satisfies stationarity: for every  $\pi \in \Pi$  we have
$$
\pi \left(s _ {(j, 1)}\right) = \pi \left(s _ {(j, 2)}\right) = \dots = \pi \left(s _ {(j, H)}\right) \quad f o r e v e r y j \in [ K ].
$$
For any  $\pi \in \Pi$  and  $j\in [K]$ , we use  $\pi (j)$  as a shorthand to denote the value of  $\pi (s_{(j,h)})$  for every  $h$ .
The stationarity assumption is not required but is useful for simplifying the definitions and results.
H.1 Definitions and Relationships
First, we state several complexity measures in learning theory. For further discussion on these quantities, see (Foster et al., 2021c; Li et al., 2022).
Definition 10 (Combinatorial Eluder Dimension). Fix any stationary base policy  $\bar{\pi}$ . The combinatorial eluder dimension of  $\Pi$  w.r.t.  $\bar{\pi}$ , denoted  $\dim_{\mathsf{E}}(\Pi ;\bar{\pi})$ , is the length of the longest sequence  $(j_1,\pi_1),\ldots ,(j_N,\pi_N)$  such that for every  $\ell \in [N]$ :
$$
\pi_ {\ell} (j _ {\ell}) \neq \bar {\pi} (j _ {\ell}), \quad a n d \quad \forall k <   \ell , \pi_ {\ell} (j _ {k}) = \bar {\pi} (j _ {k}).
$$
We define the combinatorial eluder dimension of  $\Pi$  as  $\dim_{\mathsf{E}}(\Pi) \coloneqq \sup_{\pi \in \Pi} \dim_{\mathsf{E}}(\Pi; \bar{\pi})$ .
Definition 11 (Star Number (Hanneke and Yang, 2015)). Fix any stationary base policy  $\bar{\pi}$ . The star number of  $\Pi$  w.r.t.  $\bar{\pi}$ , denoted  $\dim_{\mathsf{S}}(\Pi; \bar{\pi})$ , is the length of the longest sequence  $(j_1, \pi_1), \ldots, (j_N, \pi_N)$  such that for every  $\ell \in [N]$ :