SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When is Agnostic Reinforcement Learning Statistically Tractable?*
Zeyu Jia
Gene Li2
$^{1}$ MIT,  $^{2}$ TTIC
We study the problem of agnostic PAC reinforcement learning (RL): given a policy class II, how many rounds of interaction with an unknown MDP (with a potentially large state and action space) are required to learn an  $\varepsilon$ -suboptimal policy with respect to II? Towards that end, we introduce a new complexity measure, called the spanning capacity, that depends solely on the set II and is independent of the MDP dynamics. With a generative model, we show that for any policy class II, bounded spanning capacity characterizes PAC learnability. However, for online RL, the situation is more subtle. We show there exists a policy class II with a bounded spanning capacity that requires a superpolynomial number of samples to learn. This reveals a surprising separation for agnostic learnability between generative access and online access models (as well as between deterministic/stochastic MDPs under online access). On the positive side, we identify an additional sunflower structure, which in conjunction with bounded spanning capacity enables statistically efficient online RL via a new algorithm called POPLER, which takes inspiration from classical importance sampling methods as well as techniques for reachable-state identification and policy evaluation in reward-free exploration.
Reinforcement Learning (RL) has emerged as a powerful paradigm for solving complex decision-making problems, demonstrating impressive empirical successes in a wide array of challenging tasks, from achieving superhuman performance in the game of Go (Silver et al., 2017) to solving intricate robotic manipulation tasks (Lillicrap et al., 2016; Akkaya et al., 2019; Ji et al., 2023). Many practical domains in RL often involve rich observations such as images, text, or audio (Mnih et al., 2015; Li et al., 2016; Ouyang et al., 2022). Since these state spaces can be vast and complex, traditional tabular RL approaches (Kearns and Singh, 2002; Brafman and Tennenholtz, 2002; Azar et al., 2017; Jin et al., 2018) cannot scale. This has led to a need to develop provable and efficient approaches for RL that utilize function approximation to generalize observational data to unknown states/actions.
The goal of this paper is to study the sample complexity of policy-based RL, which is arguably the simplest setting for RL with function approximation (Kearns et al., 1999; Kakade, 2003). In policy-based RL, an abstract function class  $\Pi$  of policies (mappings from states to actions) is given to the learner. For example,  $\Pi$  can be the set of all the policies represented by a certain deep neural network architecture. The objective of the learner is to interact with an unknown MDP to find a policy  $\hat{\pi}$  that competes with the best policy in  $\Pi$ , i.e., for some prespecified  $\varepsilon$ , the policy  $\hat{\pi}$  satisfies
$$
V ^ {\widehat {\pi}} \geq \max  _ {\pi \in \Pi} V ^ {\pi} - \varepsilon , \tag {1}
$$
where  $V^{\pi}$  denotes the value of policy  $\pi$  on the underlying MDP. We henceforth call Eq. (1) the "agnostic PAC reinforcement learning" objective. Our paper addresses the following question:
What structural assumptions on  $\Pi$  enable statistically efficient agnostic PAC reinforcement learning?
Characterizing (agnostic) learnability for various problem settings is perhaps the most fundamental question in statistical learning theory. For the simpler setting of supervised learning (which is RL with binary actions, horizon 1, and binary rewards), the story is complete: a hypothesis class  $\Pi$  is agnostically learnable if and only if its VC dimension is bounded (Vapnik and Chervonenkis, 1971, 1974; Blumer et al., 1989; Ehrenfeucht et al., 1989), and the ERM algorithm—which returns the hypothesis with the smallest training loss—is statistically optimal (up to log factors). However, RL (with  $H > 1$ ) is significantly more challenging, and we are still far from a rigorous understanding of when agnostic RL is statistically tractable, or what algorithms to use in large-scale RL problems.
While significant effort has been invested over the past decade in both theory and practice to develop algorithms that utilize function approximation, existing theoretical guarantees require additional assumptions on the MDP. The most commonly adopted assumption is realizability: the learner can precisely model the value function or the dynamics of the underlying MDP (see, e.g., Russo and Van Roy, 2013; Jiang et al., 2017; Sun et al., 2019; Wang et al., 2020c; Du et al., 2021; Jin et al., 2021a; Foster et al., 2021a). Unfortunately, realizability is a fragile assumption that rarely holds in practice. Moreover, even mild misspecification can cause catastrophic breakdown of theoretical guarantees (Du et al., 2019b; Lattimore et al., 2020). Furthermore, in various applications, the optimal policy  $\pi^{\star} \coloneqq \arg \max_{\pi \in \Pi} V^{\pi}$  may have a succinct representation, but the optimal value function  $V^{\star}$  can be highly complex, rendering accurate approximation of dynamics/value functions infeasible without substantial domain knowledge (Dong et al., 2020). Thus, we desire algorithms for agnostic RL that can work with no modeling assumptions on the underlying MDP. On the other hand, it is also well known without any assumptions on  $\Pi$ , when  $\Pi$  is large and the MDP has a large state and action space, agnostic RL may be intractable with sample complexity scaling exponentially in the horizon (Agarwal et al., 2019). Thus, some structural assumptions on  $\Pi$  are needed, and towards that end, the goal of our paper is to understand what assumptions are sufficient or necessary for statistically efficient agnostic RL, and to develop provable algorithms for learning. Our main contributions are:
- We introduce a new complexity measure called the spanning capacity, which solely depends on the policy class  $\Pi$  and is independent of the underlying MDP. We illustrate the spanning capacity with examples, and show why it is a natural complexity measure for agnostic PAC RL (Section 3).
- We show that the spanning capacity is both necessary and sufficient for agnostic PAC RL with a generative model, with upper and lower bounds matching up to  $\log |\Pi|$  and  $\mathrm{poly}(H)$  factors (Section 4). Thus, bounded spanning capacity characterizes agnostic PAC learnability in RL with a generative model.
- Moving to the online setting, we first show that the bounded spanning capacity by itself is insufficient for agnostic PAC RL by proving a superpolynomial lower bound on the sample complexity required to learn a specific  $\Pi$ , thus demonstrating a separation between generative and online interaction models for agnostic PAC RL (Section 5).
- Given the previous lower bound, we propose an additional property of the policy class called the sunflower property, that allows for efficient exploration and is satisfied by many policy classes of interest. We provide a new agnostic PAC RL algorithm called POPLER that is statistically efficient whenever the given policy class has both bounded spanning capacity and the sunflower property (Section 6). POPLER leverages importance sampling as well as reachable state identification techniques to estimate the values of policies. Our algorithm and analysis utilize a new tool called the policy-specific Markov reward process, which may be of independent interest.
2 Setup and Motivation
We begin by introducing our setup for reinforcement learning (RL), the relevant notation, and the goal of agnostic RL.
2.1 RL Preliminaries
We consider reinforcement learning in an episodic Markov decision process (MDP) with horizon  $H$ .
Markov Decision Processes. Denote the MDP as  $M = \mathrm{MDP}(\mathcal{S},\mathcal{A},P,R,H,\mu)$ , which consists of a state space  $\mathcal{S}$ , action space  $\mathcal{A}$ , horizon  $H$ , probability transition kernel  $P:S\times \mathcal{A}\to \Delta (\mathcal{S})$ , reward function  $R:S\times \mathcal{A}\rightarrow \Delta ([0,1])$ , and initial distribution  $\mu \in \Delta (\mathcal{S})$ . For ease of exposition, we assume that  $\mathcal{S}$  and  $\mathcal{A}$  are finite (but possibly large) with cardinality  $S$  and  $A$  respectively. We assume a layered state space, i.e.,  $\mathcal{S} = \mathcal{S}_1\cup \mathcal{S}_2\cup \dots \cup \mathcal{S}_H$  where  $\mathcal{S}_i\cap \mathcal{S}_j = \emptyset$  for all  $i\neq j$ . Thus, given a state  $s\in S$ , it can be inferred which layer  $\mathcal{S}_h$  in the MDP it belongs to. We denote a trajectory  $\tau = (s_1,a_1,r_1,\ldots ,s_H,a_H,r_H)$ , where at each step  $h\in [H]$ , an action  $a_{h}\in \mathcal{A}$  is played, a reward  $r_h$  is drawn independently from the distribution  $R(s_{h},a_{h})$ , and each subsequent state  $s_{h + 1}$  is drawn from  $P(\cdot |s_h,a_h)$ . Lastly, we assume that the cumulative reward of any trajectory is bounded by 1.
Policy-based Reinforcement Learning. We assume that the learner is given a policy class  $\Pi \subseteq \mathcal{A}^S$ . For any policy  $\pi \in \mathcal{A}^S$ , we denote  $\pi(s)$  as the action that  $\pi$  takes when presented a state  $s$ . We use  $\mathbb{E}^\pi[\cdot]$  and  $\mathbb{P}^\pi[\cdot]$  to denote the expectation and probability under the process of a trajectory drawn from the MDP  $M$  by policy  $\pi$ . Additionally, for any  $h, h' \leq H$ , we say that a partial trajectory  $\tau = (s_h, a_h, s_{h+1}, a_{h+1}, \ldots, s_{h'}, a_{h'})$  is consistent with  $\pi$  if for all  $h \leq i \leq h'$ , we have  $\pi(s_i) = a_i$ . We use the notation  $\pi \rightsquigarrow \tau$  to denote that  $\tau$  is consistent with  $\pi$ .
The state-value function (also called  $V$ -function) and state-action-value function (also called  $Q$ -function) are defined such that for any  $\pi$ , and  $s, a$ ,
$$
V _ {h} ^ {\pi} (s) = \mathbb {E} ^ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} R \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s \right], Q _ {h} ^ {\pi} (s, a) = \mathbb {E} ^ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} R \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s, a _ {h} = a \right].
$$
Furthermore, whenever clear from the context, we denote  $V^{\pi} \coloneqq \mathbb{E}_{s_1 \sim \mu} V_1^{\pi}(s_1)$ . Finally, for any policy  $\pi \in \mathcal{A}^S$ , we also define the occupancy measure as  $d_h^\pi(s, a) \coloneqq \mathbb{P}^\pi[s_h = s, a_h = a]$  and  $d_h^\pi(s) \coloneqq \mathbb{P}^\pi[s_h = s]$ .
Models of Interaction. We consider two standard models of interaction in the RL literature:
- Generative Model. The learner has access to a simulator which it can query for any  $(s, a)$ , and observe a sample  $(s', r)$  drawn as  $s' \sim P(\cdot | s, a)$  and  $r \sim R(s, a)$ .<sup>3</sup>
- Online Interaction Model. The learner can submit a (potentially non-Markovian) policy  $\widetilde{\pi}$  and receive back a trajectory sampled by running  $\widetilde{\pi}$  on the MDP. Since online access can be simulated via generative access, learning under online access is only more challenging than learning under generative access (up to a factor of  $H$ ). Adhering to commonly used terminology, we will refer to RL under the online interaction model as "online RL".
We define  $\mathcal{M}^{\mathrm{sto}}$  as the set of all (stochastic and deterministic) MDPs of horizon  $H$  over the state space  $S$  and action space  $\mathcal{A}$ . Additionally, we define  $\mathcal{M}^{\mathrm{detP}} \subset \mathcal{M}^{\mathrm{sto}}$  and  $\mathcal{M}^{\mathrm{det}} \subset \mathcal{M}^{\mathrm{detP}}$  to denote the set of all MDPs with deterministic transitions but stochastic rewards, and of all MDPs with both deterministic transitions and deterministic rewards, respectively.
2.2 Agnostic PAC RL
Our goal is to understand the sample complexity of agnostic PAC RL, i.e., the number of interactions required to find a policy that can compete with the best policy within the given class  $\Pi$  for the underlying MDP. An algorithm  $\mathbb{A}$  is an  $(\varepsilon, \delta)$ -PAC RL algorithm for an MDP  $M$ , if after interacting with  $M$  (either in the generative model or online RL),  $\mathbb{A}$  returns a policy  $\widehat{\pi}$  that satisfies the guarantee<sup>4</sup>
$$
V ^ {\widehat {\pi}} \geq \max  _ {\pi \in \Pi} V ^ {\pi} - \varepsilon ,
$$
2Throughout the paper, we assume that the policy classes  $\Pi$  under consideration consist of deterministic policies. Extending our work to stochastic policy classes is an interesting direction for future research.
<sup>3</sup>Within the generative model, one can further distinguish between a more restrictive "local" access model (also called the "reset" model), where the learner can query  $(s,a)$  for any  $s\in S$  that it has seen already, or "global" access, where the learner can query for any  $(s,a)$  without restriction. For the generative model, our upper bounds hold in the local access model, while our lower bounds hold for the global access model.
Our results are agnostic in the sense that we do not make the assumption that the optimal policy for the underlying MDP is in  $\Pi$ , but instead, only wish to complete with the best policy in  $\Pi$ . We also do not assume that the learner has a value function class or a model class that captures the optimal value functions or dynamics.
with probability at least  $1 - \delta$ . For a policy class  $\Pi$  and a MDP class  $\mathcal{M}$ , we say that  $\mathbb{A}$  has sample complexity  $n_{\mathrm{on}}^{\mathbb{A}}(\Pi, \mathcal{M}; \varepsilon, \delta)$  (resp.  $n_{\mathrm{gen}}^{\mathbb{A}}(\Pi, \mathcal{M}; \varepsilon, \delta)$ ) if for every MDP  $M \in \mathcal{M}$ ,  $\mathbb{A}$  is an  $(\varepsilon, \delta)$ -PAC RL algorithm and collects at most  $n_{\mathrm{on}}^{\mathbb{A}}(\Pi, \mathcal{M}; \varepsilon, \delta)$  trajectories in the online interaction model (resp. generative model) in order to return  $\widehat{\pi}$ .
We define the minimax sample complexity for agnostically learning  $\Pi$  over  $\mathcal{M}$  as the minimum sample complexity of any  $(\varepsilon, \delta)$ -PAC RL algorithm, i.e.
$$
n _ {\mathrm {o n}} (\Pi , \mathcal {M}; \varepsilon , \delta) := \min  _ {\mathbb {A}} n _ {\mathrm {o n}} ^ {\mathbb {A}} (\Pi , \mathcal {M}; \varepsilon , \delta), \quad \text {a n d} \quad n _ {\mathrm {g e n}} (\Pi , \mathcal {M}; \varepsilon , \delta) := \min  _ {\mathbb {A}} n _ {\mathrm {g e n}} ^ {\mathbb {A}} (\Pi , \mathcal {M}; \varepsilon , \delta).
$$
For brevity, when  $\mathcal{M} = \mathcal{M}^{\mathrm{sto}}$ , we will drop the dependence on  $\mathcal{M}$  in our notation, e.g., we will write  $n_{\mathrm{on}}(\Pi ;\varepsilon ,\delta)$  and  $n_{\mathrm{gen}}(\Pi ;\varepsilon ,\delta)$  to denote  $n_{\mathrm{on}}(\Pi ,\mathcal{M};\varepsilon ,\delta)$  and  $n_{\mathrm{gen}}(\Pi ,\mathcal{M};\varepsilon ,\delta)$  respectively.
Known Results in Agnostic RL. We first note the following classical result which shows that agnostic PAC RL is statistically intractable in the worst case.
Proposition 1 (No Free Lunch Theorem for RL; Krishnamurthy et al. (2016)). There exists a policy class  $\Pi$  for which the minimax sample complexity under a generative model is at least  $n_{\mathrm{gen}}(\Pi ;\varepsilon ,\delta) = \Omega (\min \{A^H,|\Pi |,SA\} /\varepsilon^2)$ .
Since online RL is only harder than learning with a generative model, the lower bound in Proposition 1 extends to online RL. Proposition 1 is the analogue of the classical No Free Lunch results in statistical learning theory (Shalev-Shwartz and Ben-David, 2014); it indicates that without placing further assumptions on the MDP or the policy class  $\Pi$  (e.g., by introducing additional structure or constraining the state/action space sizes, policy class size, or the horizon), sample efficient agnostic PAC RL is not possible.
Indeed, an almost matching upper bound of  $n_{\mathrm{on}}(\Pi; \varepsilon, \delta) = \widetilde{\mathcal{O}}(\min\{A^H, |\Pi|, HSA\}/\varepsilon^2)$  is quite easy to obtain. The  $|\Pi|/\varepsilon^2$  guarantee can simply be obtained by iterating over  $\pi \in \Pi$ , collecting  $\widetilde{\mathcal{O}}(1/\varepsilon^2)$  trajectories per policy, and then picking the policy with highest empirical value. The  $HSA/\varepsilon^2$  guarantee can be obtained by running known algorithms for tabular RL (Zhang et al., 2021b). Finally, the  $A^H/\varepsilon^2$  guarantee is achieved by the classical importance sampling (IS) algorithm (Kearns et al., 1999; Agarwal et al., 2019). Since importance sampling will be an important technique that we repeatedly use and build upon in this paper, we give a formal description of the algorithm below:
ImportanceSampling:
- Collect  $n = \mathcal{O}(A^H\log |\Pi | / \varepsilon^2)$  trajectories by executing  $(a_{1},\ldots ,a_{H})\sim \mathrm{Uniform}(\mathcal{A}^{H}).$
- Return  $\widehat{\pi} = \arg \max_{\pi \in \Pi} \widehat{v}_{\mathrm{IS}}^{\pi}$ , where  $\widehat{v}_{\mathrm{IS}}^{\pi} \coloneqq \frac{A^{H}}{n} \sum_{i=1}^{n} \mathbb{1}\left\{\pi \rightsquigarrow \tau^{(i)}\right\} \left(\sum_{h=1}^{H} r_{h}^{(i)}\right)$ .
For every  $\pi \in \Pi$ , the quantity  $\widehat{v}_{\mathrm{IS}}^{\pi}$  is an unbiased estimate of  $V^{\pi}$  with variance  $A^{H}$ ; the sample complexity result follows by standard concentration guarantees (see, e.g., Agarwal et al., 2019).
Towards Structural Assumptions for Statistically Efficient Agnostic PAC RL. Of course, No Free Lunch results do not necessarily spell doom—for example, in supervised learning, various structural assumptions have been studied that enable statistically efficient learning. Furthermore, there has been a substantial effort in developing complexity measures like VC dimension, fat-shattering dimension, covering numbers, etc. that characterize agnostic PAC learnability under different scenarios (Shalev-Shwartz and Ben-David, 2014). In this paper, we consider the agnostic reinforcement learning setting, and explore whether there exists a complexity measure that characterizes learnability for every policy class II. Formally, can we establish a complexity measure  $\mathfrak{C}$  (a function that maps policy classes to real numbers), such that for any II, the minimax sample complexity satisfies
$$
n _ {\mathrm {o n}} (\Pi ; \varepsilon , \delta) = \widetilde {\Theta} \bigl (\operatorname {p o l y} \bigl (\mathfrak {C} (\Pi), H, \varepsilon^ {- 1}, \log \delta^ {- 1} \bigr) \bigr),
$$
where  $\mathfrak{C}(\Pi)$  denotes the complexity of  $\Pi$ . We mainly focus on finite (but large) policy classes and assume that the  $\log |\Pi|$  factors in our upper bounds are mild. In Appendix G, we discuss how our results can be extended to infinite policy classes.
Is Proposition 1 Tight for Every  $\Pi$ ? In light of Proposition 1, one obvious candidate is  $\overline{\mathfrak{C}}(\Pi) = \min \{A^H, |\Pi|, SA\}$ . While  $\overline{\mathfrak{C}}(\Pi)$  is definitely sufficient to upper bound the minimax
sample complexity for any policy class  $\Pi$  up to log factors, a priori it is not clear if it is also necessary for every policy class  $\Pi$ . In fact, our next proposition implies that  $\overline{\mathfrak{C}}(\Pi)$  is indeed not the right measure of complexity by giving an example of a policy class for which  $\overline{\mathfrak{C}}(\Pi) := \min \{A^H, |\Pi|, SA\}$  is exponentially larger than the minimax sample complexity for agnostic learning for that policy class, even when  $\varepsilon$  is constant.
Proposition 2. Let  $H \in \mathbb{N}$ ,  $K \in \mathbb{N}$ ,  $S_{h} = \{s_{(i,h)} : i \in [K]\}$  for all  $h \in [H]$ , and  $\mathcal{A} = \{0,1\}$ . Consider the singleton policy class:  $\Pi_{\mathrm{sing}} := \{\pi_{(i',h')} : i' \in [K], h' \in [H]\}$ , where  $\pi_{(i',h')}$  takes the action 1 on state  $s_{(i',h')}$ , and 0 everywhere else. Then  $\min \{A^H, |\Pi_{\mathrm{sing}}|, SA\} = 2^H$  but  $n_{\mathrm{on}}(\Pi_{\mathrm{sing}}; \varepsilon, \delta) \leq \widetilde{\mathcal{O}}(H^3 \cdot \log(1/\delta)/\varepsilon^2)$ .
The above upper bound on minimax sample complexity holds arbitrarily large values of  $K$ , and can be obtained as a corollary of our more general upper bound in Section 6. The key intuition for why  $\Pi_{\mathrm{sing}}$  can be learned in  $\mathrm{poly}(H)$  samples is that even though the policy class and the state space are large when  $K$  is large, the set of possible trajectories obtained by running any  $\pi \in \Pi_{\mathrm{sing}}$  has low complexity. In particular, every trajectory  $\tau$  has at most one  $a_h = 1$ . This observation enables us to employ the straightforward modification of the classical IS algorithm: draw  $\mathrm{poly}(H) \cdot \log(1/\delta)/\varepsilon^2$  samples from the uniform distribution over  $\Pi_{\mathrm{core}} = \{\pi_h : h \in [H]\}$  where the policy  $\pi_h$  takes the action 1 on every state at layer  $h$  and 0 everywhere else. The variance of the resulting estimator  $\widehat{v}_{\mathrm{IS}}^\pi$  is  $1/H$ , so the sample complexity of this modified variant of IS has only  $\mathrm{poly}(H)$  dependence by standard concentration bounds.
In the sequel, we present a new complexity measure that formalizes this intuition that a policy class  $\Pi$  is efficiently learnable if the set of trajectories induced by policies in  $\Pi$  is small.
3 Spanning Capacity
The spanning capacity precisely captures the intuition that trajectories obtained by running any  $\pi \in \Pi$  have "low complexity." We first define a notion of reachability: in deterministic MDP  $M \in \mathcal{M}^{\mathrm{det}}$ , we say  $(s, a)$  is reachable by  $\pi \in \Pi$  if  $(s, a)$  lies on the trajectory obtained by running  $\pi$  on  $M$ . Roughly speaking, the spanning capacity measures "complexity" of  $\Pi$  as the maximum number of state-action pairs which are reachable by some  $\pi \in \Pi$  in any deterministic MDP.
Definition 1 (spanning capacity). Fix a deterministic MDP  $M \in \mathcal{M}^{\mathrm{det}}$ . We define the cumulative reachability at layer  $h \in [H]$ , denoted by  $C_h^{\mathrm{reach}}(\Pi; M)$ , as
$$
C _ {h} ^ {\text {r e a c h}} (\Pi ; M) := | \{(s, a): (s, a) \text {i s r e a c h a b l e b y} \Pi \text {a t l a y e r} h \} |.
$$
We define the spanning capacity of  $\Pi$  as
$$
\mathfrak{C}(\Pi):= \max_{h\in [H]}\max_{M\in \mathcal{M}^{\det}}C_{h}^{\text{reach}}(\Pi ;M).
$$
To build intuition, we first look at some simple examples with small spanning capacity:
- Contextual Bandits: Consider the standard formulation of contextual bandits (i.e., RL with  $H = 1$ ). For any policy class  $\Pi_{\mathrm{cb}}$ , since  $H = 1$ , the largest deterministic MDP we can construct has a single state  $s_1$  and at most  $A$  actions available on  $s_1$ , so  $\mathfrak{C}(\Pi_{\mathrm{cb}}) \leq A$ .
- Tabular MDPs: Consider tabular RL with the policy class  $\Pi_{\mathrm{tab}} = \mathcal{A}^S$  consisting of all deterministic policies on the underlying state space. Depending on the relationship between  $S, A$  and  $H$ , we have two possible bounds on  $\mathfrak{C}(\Pi_{\mathrm{tab}}) \leq \min \{A^H, SA\}$ . If the state space is exponentially large in  $H$ , then it is possible to construct a full  $A$ -ary "tree" such that every  $(s, a)$  pair at layer  $H$  is visited, giving us the  $A^H$  bound. However, if the state space is small, then the number of  $(s, a)$  pairs available at any layer  $H$  is trivially bounded by  $SA$ .
- Bounded Cardinality Policy Classes: For any policy class  $\Pi$ , we always have that  $\mathfrak{C}(\Pi) \leq |\Pi|$ , since in any deterministic MDP, in any layer  $h \in [H]$ , each  $\pi \in \Pi$  can visit at most one new  $(s, a)$  pair. Thus, for policy classes  $|\Pi_{\text{small}}|$  with small cardinality (e.g.  $|\Pi_{\text{small}}| = O(\text{poly}(H, A))$ ), the spanning capacity is also small; Note that in this case, we allow our sample complexity bounds to depend on  $|\Pi_{\text{small}}|$ .
- Singletons: For the singleton class we have  $\mathfrak{C}(\Pi_{\mathrm{sing}}) = H + 1$ , since once we fix a deterministic MDP, there are at most  $H$  states where we can split from the trajectory taken by the policy which