Title: When is Agnostic Reinforcement Learning Statistically Tractable?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Agnostic Reinforcement Learning

Sample Complexity

Learning Theory

Complexity Measure
Abstract: We study the problem of agnostic PAC reinforcement learning (RL): given a policy class $\Pi$, how many rounds of interaction with an unknown MDP (with a potentially large state and action space) are required to learn an $\epsilon$-suboptimal policy with respect to \(\Pi\)? Towards that end, we introduce a new complexity measure, called the \emph{spanning capacity}, that depends solely on the set \(\Pi\) and is independent of the MDP dynamics. With a generative model, we show that the spanning capacity characterizes PAC learnability for every policy class $\Pi$. However, for online RL, the situation is more subtle. We show there exists a policy class $\Pi$ with a bounded spanning capacity that requires a superpolynomial number of samples to learn. This reveals a surprising separation for agnostic learnability between generative access and online access models (as well as between deterministic/stochastic MDPs under online access). On the positive side, we identify an additional \emph{sunflower} structure which in conjunction with bounded spanning capacity enables statistically efficient online RL via a new algorithm called POPLER, which takes inspiration from classical importance sampling methods as well as recent developments for reachable-state identification and policy evaluation in reward-free exploration.

When is Agnostic Reinforcement Learning Statistically Tractable? ∗

Abstract

We study the problem of agnostic PAC reinforcement learning (RL): given a policy class Π , how many rounds of interaction with an unknown MDP (with a potentially large state and action space) are required to learn an ε -suboptimal policy with respect to Π ? Towards that end, we introduce a new complexity measure, called the spanning capacity , that depends solely on the set Π and is independent of the MDP dynamics. With a generative model, we show that for any policy class Π , bounded spanning capacity characterizes PAC learnability. However, for online RL, the situation is more subtle. We show there exists a policy class Π with a bounded spanning capacity that requires a superpolynomial number of samples to learn. This reveals a surprising separation for agnostic learnability between generative access and online access models (as well as between deterministic/stochastic MDPs under online access). On the positive side, we identify an additional sunflower structure, which in conjunction with bounded spanning capacity enables statistically efficient online RL via a new algorithm called POPLER, which takes inspiration from classical importance sampling methods as well as techniques for reachable-state identification and policy evaluation in reward-free exploration.

1 Introduction

Reinforcement Learning (RL) has emerged as a powerful paradigm for solving complex decisionmaking problems, demonstrating impressive empirical successes in a wide array of challenging tasks, from achieving superhuman performance in the game of Go (Silver et al., 2017) to solving intricate robotic manipulation tasks (Lillicrap et al., 2016; Akkaya et al., 2019; Ji et al., 2023). Many practical domains in RL often involve rich observations such as images, text, or audio (Mnih et al., 2015; Li et al., 2016; Ouyang et al., 2022). Since these state spaces can be vast and complex, traditional tabular RL approaches (Kearns and Singh, 2002; Brafman and Tennenholtz, 2002; Azar et al., 2017; Jin et al., 2018) cannot scale. This has led to a need to develop provable and efficient approaches for RL that utilize function approximation to generalize observational data to unknown states/actions.

The goal of this paper is to study the sample complexity of policy-based RL, which is arguably the simplest setting for RL with function approximation (Kearns et al., 1999; Kakade, 2003). In policy-based RL, an abstract function class Π of policies (mappings from states to actions) is given to the learner. For example, Π can be the set of all the policies represented by a certain deep neural network architecture. The objective of the learner is to interact with an unknown MDP to find a policy ̂ π that competes with the best policy in Π , i.e., for some prespecified ε , the policy ̂ π satisfies

where V π denotes the value of policy π on the underlying MDP. We henceforth call Eq. (1) the 'agnostic PAC reinforcement learning' objective. Our paper addresses the following question:

What structural assumptions on Π enable statistically efficient agnostic PAC reinforcement learning?

Characterizing (agnostic) learnability for various problem settings is perhaps the most fundamental question in statistical learning theory. For the simpler setting of supervised learning (which is RL with binary actions, horizon 1, and binary rewards), the story is complete: a hypothesis class Π is agnostically learnable if and only if its VC dimension is bounded (Vapnik and Chervonenkis, 1971, 1974; Blumer et al., 1989; Ehrenfeucht et al., 1989), and the ERM algorithm-which returns the hypothesis with the smallest training loss-is statistically optimal (up to log factors). However, RL (with H > 1 ) is significantly more challenging, and we are still far from a rigorous understanding of when agnostic RL is statistically tractable, or what algorithms to use in large-scale RL problems.

While significant effort has been invested over the past decade in both theory and practice to develop algorithms that utilize function approximation, existing theoretical guarantees require additional assumptions on the MDP. The most commonly adopted assumption is realizability : the learner can precisely model the value function or the dynamics of the underlying MDP (see, e.g., Russo and Van Roy, 2013; Jiang et al., 2017; Sun et al., 2019; Wang et al., 2020c; Du et al., 2021; Jin et al., 2021a; Foster et al., 2021a). Unfortunately, realizability is a fragile assumption that rarely holds in practice. Moreover, even mild misspecification can cause catastrophic breakdown of theoretical guarantees (Du et al., 2019b; Lattimore et al., 2020). Furthermore, in various applications, the optimal policy π ⋆ := arg max π ∈ Π V π may have a succinct representation, but the optimal value function V ⋆ can be highly complex, rendering accurate approximation of dynamics/value functions infeasible without substantial domain knowledge (Dong et al., 2020). Thus, we desire algorithms for agnostic RL that can work with no modeling assumptions on the underlying MDP . On the other hand, it is also well known without any assumptions on Π , when Π is large and the MDP has a large state and action space, agnostic RL may be intractable with sample complexity scaling exponentially in the horizon (Agarwal et al., 2019). Thus, some structural assumptions on Π are needed, and towards that end, the goal of our paper is to understand what assumptions are sufficient or necessary for statistically efficient agnostic RL, and to develop provable algorithms for learning. Our main contributions are:

We introduce a new complexity measure called the spanning capacity , which solely depends on the policy class Π and is independent of the underlying MDP. We illustrate the spanning capacity with examples, and show why it is a natural complexity measure for agnostic PAC RL (Section 3).

We show that the spanning capacity is both necessary and sufficient for agnostic PAC RL with a generative model, with upper and lower bounds matching up to log | Π | and poly( H ) factors (Section 4). Thus, bounded spanning capacity characterizes agnostic PAC learnability in RL with a generative model.

Moving to the online setting, we first show that the bounded spanning capacity by itself is insufficient for agnostic PAC RL by proving a superpolynomial lower bound on the sample complexity required to learn a specific Π , thus demonstrating a separation between generative and online interaction models for agnostic PAC RL (Section 5).

Given the previous lower bound, we propose an additional property of the policy class called the sunflower property, that allows for efficient exploration and is satisfied by many policy classes of interest. We provide a new agnostic PAC RL algorithm called POPLER that is statistically efficient whenever the given policy class has both bounded spanning capacity and the sunflower property (Section 6). POPLER leverages importance sampling as well as reachable state identification techniques to estimate the values of policies. Our algorithm and analysis utilize a new tool called the policy-specific Markov reward process , which may be of independent interest.

2 Setup and Motivation

We begin by introducing our setup for reinforcement learning (RL), the relevant notation, and the goal of agnostic RL.

2.1 RL Preliminaries

We consider reinforcement learning in an episodic Markov decision process (MDP) with horizon H .

̸

Markov Decision Processes. Denote the MDP as M = MDP( S , A , P, R, H, µ ) , which consists of a state space S , action space A , horizon H , probability transition kernel P : S × A → ∆( S ) , reward function R : S × A → ∆([0 , 1]) , and initial distribution µ ∈ ∆( S ) . For ease of exposition, we assume that S and A are finite (but possibly large) with cardinality S and A respectively. We assume a layered state space, i.e., S = S 1 ∪ S 2 ∪ · · · ∪ S H where S i ∩ S j = ∅ for all i = j . Thus, given a state s ∈ S , it can be inferred which layer S h in the MDP it belongs to. We denote a trajectory τ = ( s 1 , a 1 , r 1 , . . . , s H , a H , r H ) , where at each step h ∈ [ H ] , an action a h ∈ A is played, a reward r h is drawn independently from the distribution R ( s h , a h ) , and each subsequent state s h +1 is drawn from P ( ·| s h , a h ) . Lastly, we assume that the cumulative reward of any trajectory is bounded by 1.

Policy-based Reinforcement Learning. We assume that the learner is given a policy class Π ⊆ A S . 2 For any policy π ∈ A S , we denote π ( s ) as the action that π takes when presented a state s . We use E π [ · ] and P π [ · ] to denote the expectation and probability under the process of a trajectory drawn from the MDP M by policy π . Additionally, for any h, h ′ ≤ H , we say that a partial trajectory τ = ( s h , a h , s h +1 , a h +1 , . . . , s h ′ , a h ′ ) is consistent with π if for all h ≤ i ≤ h ′ , we have π ( s i ) = a i . We use the notation π ⇝ τ to denote that τ is consistent with π .

The state-value function (also called V -function ) and state-action-value function (also called Q -function ) are defined such that for any π , and s, a ,

Furthermore, whenever clear from the context, we denote V π := E s 1 ∼ µ V π 1 ( s 1 ) . Finally, for any policy π ∈ A S , we also define the occupancy measure as d π h ( s, a ) := P π [ s h = s, a h = a ] and d π h ( s ) := P π [ s h = s ] .

Models of Interaction. We consider two standard models of interaction in the RL literature:

Generative Model. The learner has access to a simulator which it can query for any ( s, a ) , and observe a sample ( s ′ , r ) drawn as s ′ ∼ P ( ·| s, a ) and r ∼ R ( s, a ) . 3

Online Interaction Model. The learner can submit a (potentially non-Markovian) policy ˜ π and receive back a trajectory sampled by running ˜ π on the MDP. Since online access can be simulated via generative access, learning under online access is only more challenging than learning under generative access (up to a factor of H ). Adhering to commonly used terminology, we will refer to RL under the online interaction model as 'online RL'.

We define M sto as the set of all (stochastic and deterministic) MDPs of horizon H over the state space S and action space A . Additionally, we define M detP ⊂ M sto and M det ⊂ M detP to denote the set of all MDPs with deterministic transitions but stochastic rewards, and of all MDPs with both deterministic transitions and deterministic rewards, respectively.

2.2 Agnostic PAC RL

Our goal is to understand the sample complexity of agnostic PAC RL, i.e., the number of interactions required to find a policy that can compete with the best policy within the given class Π for the underlying MDP. An algorithm A is an ( ε, δ ) -PAC RL algorithm for an MDP M , if after interacting with M (either in the generative model or online RL), A returns a policy ̂ π that satisfies the guarantee 4

with probability at least 1 -δ . For a policy class Π and a MDP class M , we say that A has sample complexity n A on (Π , M ; ε, δ ) (resp. n A gen (Π , M ; ε, δ ) ) if for every MDP M ∈ M , A is an ( ε, δ ) -PAC RL algorithm and collects at most n A on (Π , M ; ε, δ ) trajectories in the online interaction model (resp. generative model) in order to return ̂ π .

We define the minimax sample complexity for agnostically learning Π over M as the minimum sample complexity of any ( ε, δ ) -PAC RL algorithm, i.e.

For brevity, when M = M sto , we will drop the dependence on M in our notation, e.g., we will write n on (Π; ε, δ ) and n gen (Π; ε, δ ) to denote n on (Π , M ; ε, δ ) and n gen (Π , M ; ε, δ ) respectively.

Known Results in Agnostic RL. We first note the following classical result which shows that agnostic PAC RL is statistically intractable in the worst case.

Proposition 1 (No Free Lunch Theorem for RL; Krishnamurthy et al. (2016)) . There exists a policy class Π for which the minimax sample complexity under a generative model is at least n gen (Π; ε, δ ) = Ω(min { A H , | Π | , SA } /ε 2 ) .

Since online RL is only harder than learning with a generative model, the lower bound in Proposition 1 extends to online RL. Proposition 1 is the analogue of the classical No Free Lunch results in statistical learning theory (Shalev-Shwartz and Ben-David, 2014); it indicates that without placing further assumptions on the MDP or the policy class Π (e.g., by introducing additional structure or constraining the state/action space sizes, policy class size, or the horizon), sample efficient agnostic PAC RL is not possible.

Indeed, an almost matching upper bound of n on (Π; ε, δ ) = ˜ O (min { A H , | Π | , HSA } /ε 2 ) is quite easy to obtain. The | Π | /ε 2 guarantee can simply be obtained by iterating over π ∈ Π , collecting ˜ O (1 /ε 2 ) trajectories per policy, and then picking the policy with highest empirical value. The HSA/ε 2 guarantee can be obtained by running known algorithms for tabular RL (Zhang et al., 2021b). Finally, the A H /ε 2 guarantee is achieved by the classical importance sampling (IS) algorithm (Kearns et al., 1999; Agarwal et al., 2019). Since importance sampling will be an important technique that we repeatedly use and build upon in this paper, we give a formal description of the algorithm below:

ImportanceSampling : · Collect n = O ( A H log | Π | /ε 2 ) trajectories by executing ( a 1 , . . . , a H ) ∼ Uniform( A H ) . · Return ̂ π = arg max π ∈ Π ̂ v π IS , where ̂ v π IS := A H n ∑ n i =1 { π ⇝ τ ( i ) } ( ∑ H h =1 r ( i ) h ) .

✶ For every π ∈ Π , the quantity ̂ v π IS is an unbiased estimate of V π with variance A H ; the sample complexity result follows by standard concentration guarantees (see, e.g., Agarwal et al., 2019).

Towards Structural Assumptions for Statistically Efficient Agnostic PAC RL. Of course, No Free Lunch results do not necessarily spell doom-for example, in supervised learning, various structural assumptions have been studied that enable statistically efficient learning. Furthermore, there has been a substantial effort in developing complexity measures like VC dimension, fat-shattering dimension, covering numbers, etc. that characterize agnostic PAC learnability under different scenarios (Shalev-Shwartz and Ben-David, 2014). In this paper, we consider the agnostic reinforcement learning setting, and explore whether there exists a complexity measure that characterizes learnability for every policy class Π . Formally, can we establish a complexity measure C (a function that maps policy classes to real numbers), such that for any Π , the minimax sample complexity satisfies

where C (Π) denotes the complexity of Π . We mainly focus on finite (but large) policy classes and assume that the log | Π | factors in our upper bounds are mild. In Appendix G, we discuss how our results can be extended to infinite policy classes.

Is Proposition 1 Tight for Every Π ? In light of Proposition 1, one obvious candidate is C (Π) = min { A H , | Π | , SA } . While C (Π) is definitely sufficient to upper bound the minimax

sample complexity for any policy class Π up to log factors, a priori it is not clear if it is also necessary for every policy class Π . In fact, our next proposition implies that C (Π) is indeed not the right measure of complexity by giving an example of a policy class for which C (Π) := min { A H , | Π | , SA } is exponentially larger than the minimax sample complexity for agnostic learning for that policy class, even when ε is constant.

Proposition 2. Let H ∈ N , K ∈ N , S h = { s ( i,h ) : i ∈ [ K ] } for all h ∈ [ H ] , and A = { 0 , 1 } . Consider the singleton policy class: Π sing := { π ( i ′ ,h ′ ) : i ′ ∈ [ K ] , h ′ ∈ [ H ] } , where π ( i ′ ,h ′ ) takes the action 1 on state s ( i ′ ,h ′ ) , and 0 everywhere else. Then min { A H , | Π sing | , SA } = 2 H but n on (Π sing ; ε, δ ) ≤ ˜ O ( H 3 · log(1 /δ ) /ε 2 ) .

The above upper bound on minimax sample complexity holds arbitrarily large values of K , and can be obtained as a corollary of our more general upper bound in Section 6. The key intuition for why Π sing can be learned in poly( H ) samples is that even though the policy class and the state space are large when K is large, the set of possible trajectories obtained by running any π ∈ Π sing has low complexity. In particular, every trajectory τ has at most one a h = 1 . This observation enables us to employ the straightforward modification of the classical IS algorithm: draw poly( H ) · log(1 /δ ) /ε 2 samples from the uniform distribution over Π core = { π h : h ∈ [ H ] } where the policy π h takes the action 1 on every state at layer h and 0 everywhere else. The variance of the resulting estimator ̂ v π IS is 1 /H , so the sample complexity of this modified variant of IS has only poly( H ) dependence by standard concentration bounds.

In the sequel, we present a new complexity measure that formalizes this intuition that a policy class Π is efficiently learnable if the set of trajectories induced by policies in Π is small.

3 Spanning Capacity

The spanning capacity precisely captures the intuition that trajectories obtained by running any π ∈ Π have 'low complexity.' We first define a notion of reachability: in deterministic MDP M ∈ M det , we say ( s, a ) is reachable by π ∈ Π if ( s, a ) lies on the trajectory obtained by running π on M . Roughly speaking, the spanning capacity measures 'complexity' of Π as the maximum number of state-action pairs which are reachable by some π ∈ Π in any deterministic MDP.

Definition 1 (spanning capacity) . Fix a deterministic MDP M ∈ M det . We define the cumulative reachability at layer h ∈ [ H ] , denoted by C reach h (Π; M ) , as

We define the spanning capacity of Π as

To build intuition, we first look at some simple examples with small spanning capacity:

Contextual Bandits: Consider the standard formulation of contextual bandits (i.e., RL with H = 1 ). For any policy class Π cb , since H = 1 , the largest deterministic MDP we can construct has a single state s 1 and at most A actions available on s 1 , so C (Π cb ) ≤ A .

Tabular MDPs: Consider tabular RL with the policy class Π tab = A S consisting of all deterministic policies on the underlying state space. Depending on the relationship between S, A and H , we have two possible bounds on C (Π tab ) ≤ min { A H , SA } . If the state space is exponentially large in H , then it is possible to construct a full A -ary 'tree' such that every ( s, a ) pair at layer H is visited, giving us the A H bound. However, if the state space is small, then the number of ( s, a ) pairs available at any layer H is trivially bounded by SA .

Bounded Cardinality Policy Classes: For any policy class Π , we always have that C (Π) ≤ | Π | , since in any deterministic MDP, in any layer h ∈ [ H ] , each π ∈ Π can visit at most one new ( s, a ) pair. Thus, for policy classes | Π small | with small cardinality (e.g. | Π small | = O (poly( H,A )) , the spanning capacity is also small; Note that in this case, we allow our sample complexity bounds to depend on | Π small | .

Singletons: For the singleton class we have C (Π sing ) = H +1 , since once we fix a deterministic MDP, there are at most H states where we can split from the trajectory taken by the policy which

always plays a = 0 , so the maximum number of ( s, a ) pairs reachable at layer h ∈ [ H ] is h +1 . Observe that in light of Proposition 2, the spanning capacity for Π sing is 'on the right order' for characterizing the minimax sample complexity for agnostic PAC RL.

Before proceeding, we note that for any policy class Π , the spanning capacity is always bounded. Proposition 3. For any policy class Π , we have C (Π) ≤ min { A H , | Π | , SA } .

Proposition 3 recovers the worst-case upper and lower bound from Section 2.2. However, for many policy classes, spanning capacity is substantially smaller than upper bound of Proposition 3. In addition to the examples we provided above, we list several additional policy classes with small spanning capacity. For these policy classes we set the state/action spaces to be S h = { s ( i,h ) : i ∈ [ K ] } for all h ∈ [ H ] and A = { 0 , 1 } , respectively. All proofs are deferred to Appendix B.

ℓ -tons : A natural generalization of singletons. We define Π ℓ -ton := { π I : I ⊂ S , | I | ≤ ℓ } , where the policy π I is defined s.t. π I ( s ) = { s ∈ I } for any s ∈ S . Here, C (Π ℓ -ton ) = Θ( H ℓ ) .

̸

✶ · 1-Active Policies : We define Π 1 -act to be the class of policies which can take both possible actions on a single state s (1 ,h ) in each layer h , but on other states s ( i,h ) for i = 1 must take action 0. Formally, Π 1 -act := { π b | b ∈ { 0 , 1 } H } , where for any b ∈ { 0 , 1 } H the policy π b is defined such that π b ( s ) = b [ h ] if s = s (1 ,h ) , and π b ( s ) = 0 otherwise.

All-Active Policies : We define Π j -act := { π b | b ∈ { 0 , 1 } H } , where for any b ∈ { 0 , 1 } H the policy π b is defined such that π b ( s ) = b [ h ] if s = s ( j,h ) , and π b ( s ) = 0 otherwise. We let Π act := ⋃ K j =1 Π j -act . Here, C (Π act ) = Θ( H 2 ) .

A natural interpretation of the spanning capacity is that it represents the largest 'needle in a haystack' that can be embedded in a deterministic MDP using the policy class Π . To see this, let ( M ⋆ , h ⋆ ) be the MDP and layer which witnesses C (Π) , and let { ( s i , a i ) } C (Π) i =1 be the set of state-action pairs reachable by Π in M ⋆ at layer h ⋆ . Then one can hide a reward of 1 on one of these state-action pairs; since every trajectory visits a single ( s i , a i ) at layer h ⋆ , we need at least C (Π) samples in order to discover which state-action pair has the hidden reward. Note that in this agnostic learning setup, we only need to care about the states that are reachable using Π , even though the h ⋆ layer may have other non-reachable states and actions.

3.1 Connection to Coverability

The spanning capacity has another interpretation as the worst-case coverability , a structural parameter defined in a recent work by Xie et al. (2022).

Definition 2 (Coverability, Xie et al. (2022)) . For any MDP M and policy class Π , the coverability coefficient C cov is denoted

The last equality is shown in Lemma 3 of Xie et al. (2022), and it says that the coverability coefficient is equivalent to a notion of cumulative reachability (one can check that their definition coincides with ours for deterministic MDPs).

Coverage conditions date back to the analysis of the classic Fitted Q-Iteration (FQI) algorithm (Munos, 2007; Munos and Szepesvári, 2008), and have extensively been studied in offline RL. Various models like tabular MDPs, linear MDPs, low-rank MDPs, and exogenous MDPs satisfy the above coverage condition (Antos et al., 2008; Chen and Jiang, 2019; Jin et al., 2021b; Rashidinejad et al., 2021; Zhan et al., 2022; Xie et al., 2022), and recently, Xie et al. showed that coverability can be used to prove regret guarantees for online RL, albeit under the additional assumption of value function realizability.

It is straightforward from Definition 2 that our notion of spanning capacity is worst-case coverability when we maximize over deterministic MDPs, since for any deterministic MDP, sup π ∈ Π d π h ( s, a ) = ✶ { ( s, a ) is reachable by Π at layer h } . The next lemma shows that our notion of spanning capacity is exactly worst-case coverability even when we maximize over the larger class of stochastic MDPs. As a consequence, there always exists a deterministic MDP that witnesses worst-case coverability.

Lemma 1. For any policy class Π , we have sup M ∈M sto C cov (Π; M ) = C (Π) .

While spanning capacity is equal to the worst-case coverability, we remark that the two definitions have different origins. The notion of coverability bridges offline and online RL, and was introduced in Xie et al. (2022) to characterize when sample efficient learning is possible in value-based RL, where the learner has access to a realizable value function class. On the other hand, spanning capacity is developed for the much weaker agnostic RL setting, where the learner only has access to a policy class (and does not have access to a realizable value function class). Note that a realizable value function class can be used to construct a policy class that contains the optimal policy, but the converse is not true. Furthermore, note that the above equivalence only holds in a worst-case sense (over MDPs). In fact, as we show in Appendix C, coverability alone is not sufficient for sample efficient agnostic PAC RL in the online interaction model.

4 Generative Model: Spanning Capacity is Necessary and Sufficient

In this section, we show that for any policy class, the spanning capacity characterizes the minimax sample complexity for agnostic PAC RL under generative model.

Theorem 1 (Upper Bound for Generative Model) . For any Π , the minimax sample complexity ( ε, δ ) -PAC learning Π is at most n gen (Π; ε, δ ) ≤ O ( H · C (Π) ε 2 · log | Π | δ ) .

The proof can be found in Appendix D.1, and is a straightforward modification of the classic trajectory tree method from Kearns et al. (1999): using generative access, sample O (log | Π | /ε 2 ) deterministic trajectory trees from the MDP to get unbiased evaluations for every π ∈ Π ; the number of generative queries made is bounded since the size of the maximum deterministic tree is at most H · C (Π) .

Theorem 2 (Lower Bound for Generative Model) . For any Π , the minimax sample complexity ( ε, δ ) -PAC learning Π is at least n gen (Π; ε, δ ) ≥ Ω ( C (Π) ε 2 · log 1 δ ) .

The proof can be found in Appendix D.2. Intuitively, given an MDP M ⋆ which witnesses C (Π) , one can embed a bandit instance on the relevant ( s, a ) pairs spanned by Π in M ⋆ . The lower bound follows by a reduction to the lower bound for ( ε, δ ) -PAC learning in multi-armed bandits.

Together, Theorem 1 and Theorem 2 paint a relatively complete picture for the minimax sample complexity of learning any policy class Π , in the generative model, up to an H · log | Π | factor.

Deterministic MDPs. A similar guarantee holds for online RL over deterministic MDPs.

Corollary 1. Over the class of MDPs with deterministic transitions, the minimax sample complexity of ( ε, δ ) -PAC learning any Π is

The upper bound follows because the trajectory tree algorithm for deterministic transitions samples the same tree over and over again (with different stochastic rewards). The lower bound trivially extends because the lower bound of Theorem 2 actually uses an MDP M ∈ M detP (in fact, the transitions of M are also known to the learner).

5 Online RL: Spanning Capacity is Not Sufficient

Given that fact that spanning capacity characterizes the minimax sample complexity of agnostic PAC RL in the generative model, one might be tempted to conjecture that spanning capacity is also the right characterization in online RL. The lower bound is clear since online RL is at least as hard as learning with a generative model, so Theorem 2 already shows that spanning capacity is necessary . But is it also sufficient?

In this section, we prove a surprising negative result showing that bounded spanning capacity by itself is not sufficient to characterize the minimax sample complexity in online RL. In particular, we provide an example for which we have a superpolynomial (in H ) lower bound on the number of trajectories needed for learning in online RL, that is not captured by any polynomial function of spanning capacity. This implies that, contrary to RL with a generative model, one can not hope for n on (Π; ε, δ ) = ˜ Θ(poly( C (Π) , H, ε -1 , log δ -1 )) in online RL.

Theorem 3 (Lower Bound for Online RL) . Fix any sufficiently large H . Let ε ∈ (1 / 2 O ( H ) , O (1 /H )) and ℓ ∈ { 2 , . . . , H } such that 1 /ε ℓ ≤ 2 H . There exists a policy class Π ( ℓ ) of size O (1 /ε ℓ ) with C (Π ( ℓ ) ) ≤ O ( H 4 ℓ +2 ) and a family of MDPs M with state space S of size 2 O ( H ) , binary action space, and horizon H such that: for any ( ε, 1 / 8) -PAC algorithm, there exists an MDP M ∈ M for which the algorithm must collect at least Ω(min { 1 ε ℓ , 2 H/ 3 } ) online trajectories in expectation.

Informally speaking, the lower bound shows that there exists a policy class Π for which n on (Π; ε, δ ) = 1 /ε Ω(log H C (Π)) . In order to interpret this theorem, we can instantiate choices of ε = 1 / 2 √ H and ℓ = √ H to show an explicit separation.

Corollary 2. For any sufficiently large H , there exists a policy class Π with C (Π) = 2 O ( √ H log H ) such that for any (1 / 2 √ H , 1 / 8) -PAC algorithm, there exists an MDP for which the algorithm must collect at least 2 Ω( H ) online trajectories in expectation.

In conjunction with the results of Section 4, Theorem 3 shows that (1) online RL is strictly harder than RL with generative access, and (2) online RL for stochastic MDPs is strictly harder than online RL for MDPs with deterministic transitions. We defer the proof of Theorem 3 to Appendix E. Our lower bound introduces several technical novelties: the family M utilizes a contextual variant of the combination lock, and the policy class Π is constructed via a careful probabilistic argument such that it is hard to explore despite having small spanning capacity.

6 Statistically Efficient Agnostic Learning in Online RL

The lower bound in Theorem 3 suggests that further structural assumptions on Π are needed for statistically efficient agnostic RL under the online interaction model. Essentially, the lower bound example provided in Theorem 3 is hard to agnostically learn because any two distinct policies π, π ′ ∈ Π can differ substantially on a large subset of states (of size at least ε · 2 2 H ). Thus, we cannot hope to learn 'in parallel' via a low variance IS strategy that utilizes extrapolation to evaluate all policies π ∈ Π , as we did for singletons.

In the sequel, we consider the following sunflower property to rule out such problematic scenarios, and show how bounded spanning capacity along with the sunflower property enable sample-efficient agnostic RL in the online interaction model. The sunflower property only depends on the state space, action space, and policy class, and is independent of the transition dynamics and rewards of the underlying MDP. We first define a petal, a key ingredient of a sunflower.

Definition 3 (Petal) . For a policy set ¯ Π , and states ¯ S ⊆ S , a policy π is said to be a ¯ S -petal on ¯ Π if for all h ≤ h ′ ≤ H , and partial trajectories τ = ( s h , a h , · · · , s h ′ , a h ′ ) that are consistent with π : either τ is also consistent with some π ′ ∈ ¯ Π , or there exists i ∈ ( h, h ′ ] s.t. s i ∈ ¯ S .

Informally, π is a ¯ S -petal on ¯ Π if any trajectory that can be obtained using π can either also be obtained using a policy in ¯ Π or must pass through ¯ S . Thus, any policy that is a ¯ S -petal on ¯ Π can only differentiate from ¯ Π in a structured way. A policy class is said to be a sunflower if it is a union of petals as defined below:

Definition 4 (Sunflower) . A policy class Π is said to be a ( K,D ) -sunflower if there exists a set Π core of Markovian policies with | Π core | ≤ K such that for every policy π ∈ Π there exists a set S π ⊆ S , of size at most D , so that π is an S π -petal on Π core .

Our next theorem provides a sample complexity bound for Agnostic PAC RL for policy classes that have ( K,D ) -sunflower structure. This bound is obtained via a new exploration algorithm called POPLER that takes as input the set Π core and corresponding petals {S π } π ∈ Π and leverages importance sampling as well as reachable state identification techniques to simultaneously estimate the value of every policy in Π .

Theorem 4. Let ε, δ > 0 . Suppose the policy class Π satisfies Definition 1 with spanning capacity C (Π) , and is a ( K,D ) -sunflower. Then, for any MDP M , with probability at least 1 -δ , POPLER (Algorithm 1) succeeds in returning a policy ̂ π that satisfies V ̂ π ≥ max π ∈ Π V π -ε , after collecting

The proof of Theorem 4, and the corresponding hyperparameters in POPLER needed to obtain the above bound, can be found in Appendix F. Before diving into the algorithm and proof details, let us highlight several key aspects of the above sample complexity bound:

Note that a class Π may be a ( K,D ) -sunflower for many different choices of K and D . Barring computational issues, one can enumerate over all choices of Π core and {S π } π ∈ Π , and check if Π is a ( K,D ) -sunflower for that choice of K = | Π core | and D = max π ∈ Π |S π | . Since our bound in Theorem 4 scales with K and D , we are free to choose K and D to minimize the corresponding sample complexity bound.

In order to get a polynomial sample complexity in Theorem 4, both C (Π) and ( K,D ) are required to be poly( H, log | Π | ) . All of the policy classes considered in Section 3 have the sunflower property, with both K,D = poly( H ) , and thus our sample complexity bound extends for all these classes. See Appendix B for details.

Notice that for Theorem 4 to hold, we need both bounded spanning capacity as well as the sunflower structure on the policy class with bounded ( K,D ) . Thus, one may wonder if we can obtain a similar polynomial sample complexity guarantee in online RL under weaker assumptions. In Theorem 3, we already showed that bounded C (Π) alone is not sufficient to obtain polynomial sample complexity in online RL. Likewise, as we show in Appendix F, sunflower property with bounded ( K,D ) alone is also not sufficient for polynomial sample complexity, and hence both assumptions cannot be individually removed. However, it is an interesting question if there is some other structural assumption that combines both spanning capacity and the sunflower property, and is both sufficient and necessary for agnostic PAC learning in online RL. See Section 7 for further discussions on this.

WhyDoes the Sunflower Property Enable Sample-Efficient Learning? Intuitively, the sunflower property captures the intuition of simultaneous estimation of all policies π ∈ Π via importance sampling (IS), and allows control of both bias and variance. Let π be a S π -petal on Π core . Any trajectory τ ⇝ π that avoids S π must be consistent with some policy in Π core , and will thus be covered by the data collected using π ′ ∼ Uniform(Π core ) . Thus, using IS with variance scaling with K , one can create a biased estimator for V π , where the bias is only due to trajectories that pass through S π . There are two cases. If every state in S π has small reachability under π , i.e. d π ( s ) ≪ ε for every s ∈ S π , then the IS estimate will have a low bias (linear in |S π | ), so we can compute V π up to error at most ε |S π | . On the other hand, if d π ( s ) is large for some s ∈ S π , it is possible to explicitly control the bias that arises from trajectories passing through them since there are at most D of them.

6.1 Algorithm and Proof Ideas

POPLER (Algorithm 1) takes as input a policy class Π , as well as sets Π core and {S π } π ∈ Π , which can be computed beforehand by enumeration. POPLER has two phases: a state identification phase , where it finds 'petal' states s ∈ ⋃ π ∈ Π S π that are reachable with sufficiently large probability; and an evaluation phase where it computes estimates ̂ V π for every π ∈ Π . It uses three subroutines DataCollector , EstReachability , and Evaluate , whose pseudocodes are stated in Appendix F.2.

The structure of the algorithm is reminiscent of reward-free exploration algorithms in tabular RL, e.g. Jin et al. (2020), where we first identify (petal) states that are reachable with probability at least Ω( ε/D ) and build a policy cover for these states, and then use dynamic programming to estimate the values. However, contrary to the classical tabular RL setting, because the state space can be large, our setting is much more challenging and necessitates technical innovations. In particular, we can no longer enumerate over all petal states and check if they are sufficiently reachable by some policy π ∈ Π (since the total number of petal states ∑ π ∈ Π |S π | could scale linearly in | Π | , a factor that we do not want to appear in our sample complexity). Instead, the key observation that we rely on is that if spanning capacity is bounded, then by the equivalence of spanning capacity and worst-case coverability (Lemma 1) and due to (2), the number of highly reachable (and thus relevant) petal states is also bounded. Thus, we only need to build a policy cover to reach these relevant petal states. Our algorithm does this in a sample-efficient sequential manner. For both state identification as well as evaluation, we interleave importance sampling estimates with the construction of a policy-specific Markov Reward Processes (MRPs), which are defined for every π ∈ Π . The challenge is doing all of this 'in parallel' for every π ∈ Π through extensive sample reuse to avoid a blowup of | Π | or S in the sample complexity. For the exact construction of these policy-specific MRPs and how they are used in the algorithm, please see Appendix F.2.

Algorithm 1 P olicy OP timization by L earning ε -R eachable States ( POPLER )

7 Conclusion and Discussion

In this paper, we investigated when agnostic RL is statistically tractable in large state and action spaces, and introduced spanning capacity as a natural measure of complexity that only depends on the policy class, and is independent of the MDP rewards and transitions. We first showed that bounded spanning capacity is both necessary and sufficient for agnostic PAC RL under the generative access model. However, we also provided a negative result showing that bounded spanning capacity does not suffice for online RL, thus showing a surprising separation between agnostic RL with a generative model and online interaction. We then provided an additional structural assumption, called the sunflower property, that allows for statistically efficient learning in online RL. Our sample complexity bound for online RL is obtained using a novel exploration algorithm called POPLER that relies on certain policy-specific Markov Reward Processes to guide exploration, and takes inspiration from the classical importance sampling method and reward-free exploration algorithms for tabular MDPs.

Our results pave the way for several future lines of inquiry. The most important direction is exploring complexity measures to tightly characterize the minimax sample complexity for online RL. On the upper bound side, Theorem 4 shows that bounded spanning capacity along with an additional sunflower property is sufficient for online RL. On the lower bound side, while bounded spanning capacity is necessary (by Theorem 2), we do not know if the sunflower property is also necessary. Another direction is understanding if one can improve the statistical complexity of policy-based learning using stronger feedback models; in Appendix I we explore whether receiving expert feedback in the form of the optimal value function { Q ⋆ ( s, a ) } a ∈A on the visited states can help. Surprisingly, the answer depends on realizability of the optimal policy π ⋆ ∈ Π : when π ⋆ ∈ Π , Q ⋆ feedback can be utilized to achieve an O (poly(log | Π | , H, 1 / ε )) sample complexity bound, with no dependence on C (Π) ; if π ⋆ / ∈ Π , then one can not hope to learn with less than Ω( C (Π)) samples in the worst case. Understanding the role of π ⋆ -realizability and exploring the benefits of other feedback models in agnostic RL are interesting future research directions. Other research directions include sharpening the sample complexity bound in Theorem 4, extending POPLER for regret minimization, and developing computationally efficient algorithms.

Acknowledgments and Disclosure of Funding

We thank Dylan Foster, Pritish Kamath, Jason D. Lee, Wen Sun, and Cong Ma for helpful discussions. GL and NS are partially supported by the NSF and the Simons Foundation. Part of this work was completed while GL was visiting Princeton University. AR acknowledges support from the ONR through award N00014-20-1-2336, ARO through award W911NF-21-1-0328, and from the DOE through award DE-SC0022199.

References

Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning , pages 3692-3702. PMLR, 2019.

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. PC-PG: Policy cover directed exploration for provable policy gradient learning. Advances in Neural Information Processing Systems , 2020.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Naman Agarwal, Brian Bullins, and Karan Singh. Variance-reduced conservative policy iteration. In International Conference on Algorithmic Learning Theory , pages 3-33. PMLR, 2023.

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Aymen Al-Marjani, Andrea Tirinzoni, and Emilie Kaufmann. Active coverage for pac reinforcement learning. arXiv preprint arXiv:2306.13601 , 2023.

Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private pac learning implies finite littlestone dimension. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pages 852-860, 2019.

Philip Amortila, Nan Jiang, Dhruv Madeka, and Dean P Foster. A few expert queries suffices for sample-efficient rl with resets and linear value approximation. arXiv preprint arXiv:2207.08342 , 2022.

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellmanresidual minimization based fitted policy iteration and a single sample path. Machine Learning , 2008.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

James Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic programming. Advances in neural information processing systems , 16, 2003.

Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786 , 2019.

Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM) , 36(4):929-965, 1989.

Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 2013.

Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for nearoptimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213-231, 2002.

Nataly Brukhim, Daniel Carmon, Irit Dinur, Shay Moran, and Amir Yehudayoff. A characterization of multiclass learnability. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 943-955. IEEE, 2022a.

Nataly Brukhim, Elad Hazan, and Karan Singh. A boosting approach to reinforcement learning. Advances in Neural Information Processing Systems , 35:33806-33817, 2022b.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , 2019.

Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms . MIT press, 2022.

Amit Daniely and Shai Shalev-Shwartz. Optimal learners for multiclass problems. In Conference on Learning Theory , pages 287-316. PMLR, 2014.

Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. On oracle-efficient PAC RL with rich observations. In Advances in Neural Information Processing Systems , 2018.

Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory , pages 578-598. PMLR, 2021.

Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. In International conference on machine learning , pages 2627-2637. PMLR, 2020.

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dud'ik, and John Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning , 2019a.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR, 2021.

Simon S Du, Sham M Kakade, Ruosong Wang, and Lin F Yang. Is a good representation sufficient for sample efficient reinforcement learning? arXiv preprint arXiv:1910.03016 , 2019b.

Yonathan Efroni, Dipendra Misra, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Provably filtering exogenous distractors using multistep inverse dynamics. In International Conference on Learning Representations , 2021.

Yonathan Efroni, Dylan J Foster, Dipendra Misra, Akshay Krishnamurthy, and John Langford. Sample-efficient reinforcement learning in the presence of exogenous information. In Conference on Learning Theory , pages 5062-5127. PMLR, 2022.

Andrzej Ehrenfeucht, David Haussler, Michael Kearns, and Leslie Valiant. A general lower bound on the number of examples needed for learning. Information and Computation , 82(3):247-261, 1989.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021a.

Dylan J Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: Fundamental barriers for value function approximation. In Conference on Learning Theory , 2021b.

Dylan J Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. Conference on Learning Theory , 2021c.

Dylan J Foster, Noah Golowich, and Yanjun Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. arXiv preprint arXiv:2301.08215 , 2023.

Aurélien Garivier, Pierre Ménard, and Gilles Stoltz. Explore first, exploit next: The true shape of regret in bandit problems. Mathematics of Operations Research , 44(2):377-399, 2019.

Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning , 91: 325-349, 2013.

Noah Golowich and Ankur Moitra. Can q-learning be improved with advice? In Conference on Learning Theory , pages 4548-4619. PMLR, 2022.

Omer Gottesman, Yao Liu, Scott Sussex, Emma Brunskill, and Finale Doshi-Velez. Combining parametric and nonparametric models for off-policy evaluation. In International Conference on Machine Learning , pages 2366-2375. PMLR, 2019.

Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems , 35:15281-15295, 2022.

Steve Hanneke and Liu Yang. Minimax analysis of active learning. J. Mach. Learn. Res. , 16(1): 3487-3602, 2015.

David Haussler and Philip M Long. A generalization of sauer's lemma. Journal of Combinatorial Theory, Series A , 71(2):219-240, 1995.

Audrey Huang, Jinglin Chen, and Nan Jiang. Reinforcement learning in low-rank mdps with density features. arXiv preprint arXiv:2302.02252 , 2023.

Yandong Ji, Gabriel B Margolis, and Pulkit Agrawal. Dribblebot: Dynamic legged manipulation in the wild. arXiv preprint arXiv:2304.01159 , 2023.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR, 2017.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR, 2020.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418, 2021a.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , 2021b.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , pages 267-274, 2002.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Sham Machandranath Kakade. On the sample complexity of reinforcement learning . University of London, University College London (United Kingdom), 2003.

Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning , 49:209-232, 2002.

Michael Kearns, Yishay Mansour, and Andrew Ng. Approximate planning in large pomdps via reusable trajectories. Advances in Neural Information Processing Systems , 12, 1999.

Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Pac reinforcement learning with rich observations. Advances in Neural Information Processing Systems , 29, 2016.

Tor Lattimore, Csaba Szepesvari, and Gellert Weisz. Learning with good feature representations in bandits and in rl with a generative model. In International Conference on Machine Learning , pages 5662-5670. PMLR, 2020.

Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning , pages 1-9. PMLR, 2013.

Gen Li, Yuling Yan, Yuxin Chen, and Jianqing Fan. Minimax-optimal reward-agnostic exploration in reinforcement learning. arXiv preprint arXiv:2304.07278 , 2023.

Gene Li, Pritish Kamath, Dylan J Foster, and Nati Srebro. Understanding the eluder dimension. Advances in Neural Information Processing Systems , 35:23737-23750, 2022.

Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541 , 2016.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations , 2016.

Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems , 33:7624-7636, 2020.

Devroye Luc, Györfi László, and Lugosi Gábor. A probabilistic theory of pattern recognition, volume 31 of applications of mathematics, 1996.

Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research , 5(Jun):623-648, 2004.

Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning , pages 7599-7608. PMLR, 2021.

Zakaria Mhammedi, Dylan J Foster, and Alexander Rakhlin. Representation learning with multi-step inverse kinematics: An efficient and optimal approach to rich-observation rl. arXiv preprint arXiv:2304.05889 , 2023.

Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning , 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 2015.

Wenlong Mou, Zheng Wen, and Xi Chen. On the sample complexity of reinforcement learning with policy space generalization. arXiv preprint arXiv:2008.07353 , 2020.

Rémi Munos. Performance bounds in l_p-norm for approximate value iteration. SIAM journal on control and optimization , 46(2):541-561, 2007.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 2008.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems , 2019.

Balas K Natarajan. On learning sets and functions. Machine Learning , 4:67-97, 1989.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35: 27730-27744, 2022.

David Pollard. Convergence of stochastic processes . Springer Science & Business Media, 2012.

Yury Polyanskiy and Yihong Wu. Information theory: From coding to learning, 2022.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 2021.

Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 661-668. JMLR Workshop and Conference Proceedings, 2010.

Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979 , 2014.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347 , 2017.

Ayush Sekhari, Christoph Dann, Mehryar Mohri, Yishay Mansour, and Karthik Sridharan. Agnostic reinforcement learning with low-rank MDPs and rich observations. Advances in Neural Information Processing Systems , 2021.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Conference on learning theory , 2019.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

Andrea Tirinzoni, Aymen Al-Marjani, and Emilie Kaufmann. Optimistic pac reinforcement learning: the instance-dependent view. In International Conference on Algorithmic Learning Theory , pages 1460-1480. PMLR, 2023.

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954 , 2018.

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline RL in low-rank MDPs. arXiv:2110.04652 , 2021.

Vladimir Vapnik and Alexey Chervonenkis. Theory of pattern recognition, 1974.

VN Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications , 16(2):264, 1971.

Andrew Wagenmaker and Kevin G Jamieson. Instance-dependent near-optimal policy identification in linear mdps via online experiment design. Advances in Neural Information Processing Systems , 35:5968-5981, 2022.

Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Rewardfree rl is no harder than reward-aware rl in linear markov decision processes. In International Conference on Machine Learning , pages 22430-22456. PMLR, 2022.

Ruosong Wang, Simon S Du, Lin Yang, and Russ R Salakhutdinov. On reward-free reinforcement learning with linear function approximation. Advances in neural information processing systems , 33:17816-17826, 2020a.

Ruosong Wang, Dean P. Foster, and Sham M. Kakade. What are the statistical limits of offline RL with linear function approximation?, 2020b.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020c.

Gellért Weisz, Philip Amortila, and Csaba Szepesvári. Exponential lower bounds for planning in mdps with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory , pages 1237-1264. PMLR, 2021.

Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research , 23(282):1-36, 2022.

Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in Neural Information Processing Systems , 32, 2019.

Tengyang Xie, Dylan J Foster, Yu Bai, Nan Jiang, and Sham M Kakade. The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157 , 2022.

Ming Yin and Yu-Xiang Wang. Asymptotically efficient off-policy evaluation for tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3948-3958. PMLR, 2020.

Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning , pages 12287-12297. PMLR, 2021.

Andrea Zanette, Alessandro Lazaric, Mykel J Kochenderfer, and Emma Brunskill. Provably efficient reward-agnostic navigation with linear value iteration. Advances in Neural Information Processing Systems , 33:11756-11766, 2020.

Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. arXiv preprint arXiv:2105.11066 , 2021.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Reward-free model-based reinforcement learning with linear function approximation. Advances in Neural Information Processing Systems , 34: 1582-1593, 2021a.

Zihan Zhang, Xiangyang Ji, and Simon Du. Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon. In Conference on Learning Theory , pages 4528-4531. PMLR, 2021b.

Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. CoRR , 2022.

Contents of Appendix

A Detailed Comparison to Related Works

Reinforcement Learning (RL) has seen substantial progress over the past few years, with several different directions of work being pursued for efficiently solving RL problems that occur in practice. The classical approach to solving an RL problem is to model it as a tabular MDP. A long line of work (Sutton and Barto, 2018; Agarwal et al., 2019; Kearns and Singh, 2002; Brafman and Tennenholtz, 2002; Auer et al., 2008; Azar et al., 2017; Gheshlaghi Azar et al., 2013; Jin et al., 2018) has studied provably sample-efficient learning algorithms that can find the optimal policy in tabular RL. Unfortunately, the sample complexity of such algorithms unavoidably scales with the size of the state/action spaces, so they fail to be efficient in practical RL problems with large state/action spaces. In order to develop algorithms for the more practical large state/action RL settings, various assumptions have been considered in the prior works. In the following, we provide a detailed comparison of our setup and assumptions with the existing literature.

RL with Function Approximation. A popular paradigm for developing algorithms for MDPs with large state/action spaces is to use function approximation to either model the MDP dynamics or optimal value functions. Over the last decade, there has been a long line of work (Jiang et al., 2017; Dann et al., 2018; Sun et al., 2019; Du et al., 2019a; Wang et al., 2020c; Du et al., 2021; Foster et al., 2021a; Jin et al., 2021a; Zhong et al., 2022; Foster et al., 2023) in understanding structural conditions on the function class and the underlying MDP that enable statistically efficient RL. However, all of these works rely crucially on the realizability assumption, namely that the true model / value function belong to the chosen class. Unfortunately, such an assumption is too strong to hold in practice. Furthermore, the prior works using function approximation make additional assumptions like Bellman Completeness that are difficult to verify for the underlying task.

In our work, we study the problem of agnostic RL to sidestep these challenges. In particular, instead of modeling the value/dynamics, the learner now models 'good policies" for the underlying task, and the learning objective is to find a policy that can perform as well as the best in the chosen policy class. We note that while a realizable value class/dynamics class F can be converted into a realizable policy class Π F by choosing the greedy policies for each value function/dynamics, the converse is not true. Thus, our agnostic RL objective relies on a strictly weaker modeling assumption.

Connections to Decision-Estimation Coefficient (DEC). The seminal work of Foster et al. (2021a) provides a unified complexity measure called Decision-Estimation Coefficient (DEC) that characterizes the complexity of model-based RL. Given the generality of the E2D algorithm of Foster et al. (2021a), one may be wondering if our results can be recovered using their framework via a model-based approach. In particular, can we recover the sample complexity bound in Theorems 1 or 4 by considering the model class M det or M sto along with the decision set Π . To the best of our knowledge, the framework of Foster et al. (2021a) do not directly recover our results, however, the complexity measures are closely related. Note that one can upper-bound the DEC by the coverability coefficient Xie et al. (2022); furthermore we show that C (Π) is worst-case coverability (Lemma 1) so it follows that DEC is upper-bounded by C (Π) . However, the algorithm in Foster et al. (2021a) achieves regret bounds which scale with DEC · log |M sto | , which can be vacuous in the large statespace setting since log |M sto | ∝ |S| ; In contrast, our upper bounds have no explicit dependence on |S| .

RLwith Rich Observations. Various settings have been studied where the dynamics are determined by a simple latent state space, but instead of observing the latent states directly, the learner receives rich observations corresponding to the underlying latent states. These include the Block MDP (Krishnamurthy et al., 2016; Du et al., 2019a; Misra et al., 2020; Mhammedi et al., 2023), LowRank MDPs (Uehara et al., 2021; Huang et al., 2023), Exogenous MDPs (Efroni et al., 2021; Xie et al., 2022; Efroni et al., 2022), etc. However, all of these prior works assume that the learner is given a realizable decoder class (consisting of functions that map observations to latent states) that contains the true decoder for the underlying MDP. Additionally, they require strong assumptions on the underlying latent state space dynamics, e.g. it is tabular or low-rank, in order to make learning tractable. Thus, their guarantees are not agnostic. In fact, given a realizable decoder class and additional structure on the latent state dynamics, one can construct a policy class that contains the optimal policy for the MDP, but the converse is not true. Thus, our agnostic RL setting is strictly more general.

Relation to Exponential Lower Bounds for RL with Function Approximation. Recently, many statistical lower bounds have been developed in RL with function approximation under only realizability. A line of work including (Wang et al., 2020b; Zanette, 2021; Weisz et al., 2021; Foster et al., 2021b) showed that the sample complexity scales exponentially in the horizon H for learning the optimal policy for RL problems where only the optimal value function Q ⋆ is linear w.r.t. the given features. Similarly, Du et al. (2019b) showed that one may need exponentially in H even if the optimal policy is linear w.r.t. the true features. These lower bounds can be extended to our agnostic RL setting, giving similar exponential in H lower bounds for agnostic RL, thus supplementing the well-known lower bounds (Krishnamurthy et al., 2016) which show that agnostic RL is intractable without additional structural assumptions on the policy class. However, to recall, the focus of this paper is to propose assumptions, like Definition 1 or 4, that circumvent these lower bounds and allow for sample efficient agnostic RL.

Importance Sampling for RL. Various importance sampling based estimators (Xie et al., 2019; Jiang and Li, 2016; Gottesman et al., 2019; Yin and Wang, 2020; Thomas and Brunskill, 2016; Nachum et al., 2019) have been developed in RL literature to provide reliable off-policy evaluation in offline RL. However, these methods also require realizable value function approximation and rely on additional assumptions on the off-policy/offline data, in particular, that the offline data covers the state/action space that is explored by the comparator policy. We note that this line of work does not directly overlap with our current approach but provides a valuable tool for dealing with off-policy data.

Agnostic RL in Low-Rank MDPs. Sekhari et al. (2021) explored agnostic PAC RL in low-rank MDPs, and showed that one can perform agnostic learning w.r.t. any policy class for MDPs that have a small rank. While their guarantees are similar to ours, i.e., they compete with the best policy in the given class and do not assume access to a realizable dynamics / value-function class, the key objectives of the two works are complementary. Sekhari et al. (2021) explore assumptions on the underlying MDP dynamics which suffice for agnostic learning for any given policy class, whereas we ask what assumptions on the given policy class suffice for agnostic learning for any underlying dynamics. Exploring the benefits of structure in both the policy class and the underlying MDP in agnostic RL is an interesting direction for future research.

Policy Gradient Methods. A significant body of work in RL, in both theory (Agarwal et al., 2021; Abbasi-Yadkori et al., 2019; Bhandari and Russo, 2019; Liu et al., 2020; Agarwal et al., 2020; Zhan et al., 2021; Xiao, 2022) and practice (Kakade, 2001; Kakade and Langford, 2002; Levine and Koltun, 2013; Schulman et al., 2015, 2017), studies policy-gradient based methods that directly search for the best policy in a given policy class. These approaches often leverage mirror-descent style analysis, and can deliver guarantees that are similar to ours, i.e. the returned policy can compete with any policy in the given class, which is an agnostic guarantee in some sense. However, these works primarily study smooth and parametric policy classes, e.g. tabular and linear policy classes, which limits their applicability for a broader range of problem instances. Furthermore, they require strong additional assumptions to work: for instance, that the learner is given a good reset distribution that can cover the occupancy measure of the policy that we wish to compare to, and that the policy class satisfies a certain 'policy completeness assumption"; both of which are difficult to verify in practice. In contrast, our work makes no such assumptions but instead studies what kind of policy classes are learnable for any MDP.

CPI, PSDP, and Other Reductions to Supervised Learning. Various RL methods have been developed that return a policy that performs as well as the best policy in the given policy class, by reducing the RL problem from supervised learning. The key difference from policy-gradient based methods (discussed previously) is that these approaches do not require a smoothly parameterized policy class, but instead rely on access to a supervised learning oracle w.r.t. the given policy class. Popular approaches include Conservative Policy Iteration (CPI) (Kakade and Langford, 2002; Kakade, 2003; Brukhim et al., 2022b; Agarwal et al., 2023), PSDP (Bagnell et al., 2003), Behavior Cloning (Ross and Bagnell, 2010; Torabi et al., 2018), etc. We note that these algorithms rely on additional assumptions, including 'policy completeness assumption" and a good sampling / reset distribution that covers the policies that we wish to compare to; in comparison, we do not make any such assumptions in our work.

Efficient RL via reductions to online regression oracles w.r.t. the given policy class has also been studied, see, e.g., DAgger (Ross et al., 2011), AggreVaTe (Ross and Bagnell, 2014), etc. However, these algorithms rely on much stronger feedback. In particular the learner, on the states which it visits, can query an expert policy (that we wish to complete with) for its actions or the value function. On the other hand, in this paper, we restrict ourselves to the standard RL setting where the learner only gets instantaneous reward signal. In Appendix I we investigate whether such stronger feedback can be used for agnostic RL.

Reward-Free RL. From a technical viewpoint, our algorithm (Algorithm 1) share similaries to algorithms developed in the reward-free RL literature (Jin et al., 2020). In reward-free RL, the goal of the learner is to output a dataset or a set of policies, after interacting with the underlying MDP, that can be later used for planning (with no further interaction with the MDP) for downstream reward functions. The key ideas in our Algorithm 1, in particular, that the learner first finds states I that are Ω( ε ) -reachable and corresponding policies that can reach them, and then outputs datasets {D s } s ∈I that can be later used for evaluating any policy π ∈ Π , share similarities to algorithmic ideas used in reward-free RL. However, our algorithm strictly generalizes prior works in reward-free RL, and in particular can work with large state-action spaces where the notion of reachability as well as the offline RL objective, is defined w.r.t. the given policy class. In comparison, prior reward-free RL works compete with the best policy for the underlying MDP, and make structure assumptions on the dynamics, e.g. tabular structure (Jin et al., 2020; Ménard et al., 2021; Li et al., 2023) or linear dynamics (Wang et al., 2020a; Zanette et al., 2020; Zhang et al., 2021a; Wagenmaker et al., 2022), to make the problem tractable.

Instance Optimal Measures. Several recent works including Wagenmaker and Jamieson (2022); Tirinzoni et al. (2023); Bottou et al. (2013); Al-Marjani et al. (2023) have explored instance-dependent complexity measures for PAC RL. At a high level, these instance-dependent bounds are obtained via similar algorithmic ideas to ours that combine reward-free exploration with policy elimination. However, there are major differences. Firstly, these prior works in instance-dependent PAC RL operate under additional modeling assumptions on the MDP dynamics, e.g., that it is a tabular or linear MDP. Secondly, they require additional reachability assumptions on the state space, which is restrictive for MDPs with large states/actions; in fact, their sample complexity bounds typically have a dependence on the number of states/actions in the lower order terms. Finally, they implicitly assume that the optimal policy π ⋆ ∈ Π , and thus the provided algorithms do not transfer cleanly to the agnostic PAC RL setting considered in our paper.

Other Complexity Measures for RL. A recent work by Mou et al. (2020) proposed a new notion of eluder dimension for the policy class, and provide upper bounds for policy-based RL when the class Π has bounded eluder dimension. However, they require various additional assumptions: that the policy class contains the optimal policy, the learner has access to a generative model, and that the optimal value function has a gap. On the other hand, we do not make any such assumptions and characterize learnability in terms of spanning capacity or the size of the minimal sunflower in Π . We discuss connections to the eluder dimension, as well as other classical complexity measures in learning theory in Appendix H.

B Examples of Policy Classes

In this section, we will prove that examples in Section 3 have both bounded spanning capacity and the sunflower property with small K and D . To facilitate our discussion, we define the following notation: for any policy class Π we let

where C reach h (Π; M ) is defined in Definition 1. That is, C h (Π) is the per-layer spanning capacity of Π . Then as defined in Definition 1, we have

Tabular MDP. Since there are at most |S h | states in layer h , it is obvious that C h (Π) ≤ |S h | A , so therefore C (Π) ≤ SA . Additionally, if we choose Π core = { π a : π a ( s ) = a, a ∈ A} to be the set

of policies which play the constant a for each a ∈ A and S π = S for every π ∈ Π , then any partial trajectory which satisfies the condition in Definition 4 is of the form ( s h , a h ) , which is consistent with π a h ∈ Π core . Hence Π is a ( A,S ) -sunflower.

Contextual Bandit. Since there is only one layer, any deterministic MDP has a single state with at most A actions possible, so C (Π) ≤ A . Additionally, if we choose Π core = { π a : π a ( s ) ≡ a, a ∈ A} , and S π = ∅ for every π ∈ Π , then any partial trajectory which satisfies the condition in Definition 4 is in the form ( s, a ) , which is consistent with π a ∈ Π core . Hence Π is a ( A, 0) -sunflower.

H -Layer Contextual Bandit. By induction, it is easy to see that any deterministic MDP spans at most A h -1 states in layer h , each of which has at most A actions. Hence C (Π) ≤ A H . Additionally, if we choose

and S π = ∅ for every π ∈ Π , then any partial trajectory which satisfies the condition in Definition 4 is in the form ( s 1 , a 1 , · · · , s H , a H ) , which is consistent with π a 1 ,a 2 , ··· ,a H ∈ Π core . Hence Π is a ( A H , 0) -sunflower.

ℓ -tons. In the following, we will denote Π ℓ := Π ℓ -ton . We will first prove that C (Π ℓ ) ≤ 2 H ℓ . To show this, we will prove that C h (Π ℓ ) ≤ 2 h ℓ by induction on H . When H = 1 , the class is a subclass of the above contextual bandit class, hence we have C 1 (Π ℓ ) ≤ 2 . Next, suppose C h -1 (Π ℓ ) ≤ 2( h -1) ℓ . Fix any deterministic MDP and call the first state s 1 . Policies taking a = 1 at s 1 can only take a = 1 on at most ℓ -1 states in the following layers. Such policies reach at most C h -1 (Π ℓ -1 ) states in layer h . Policies taking a = 0 at s 1 can only take a = 1 on at most ℓ states in the following layers. Such policies reach at most C h -1 (Π ℓ ) states in layer h . Hence we get

This finishes the proof of the induction hypothesis. Based on the induction argument, we get

where π 0 ( s ) ≡ 0 , and π h chooses the action 1 on all the states at layer h , i.e., π h ( s ) := ✶ { s ∈ S h } . For every π ∈ Π ℓ , we choose S π to be the states for which π ( s ) = 1 (there are at most ℓ such states). Fix any partial trajectory τ = ( s h , a h · · · , s h ′ , a h ′ ) which satisfies π ⇝ τ . Suppose that for all i ∈ ( h, h ′ ] , s i ̸∈ S π . Then we must have a i = 0 for all i ∈ ( h, h ′ ] . Hence π h ⇝ τ (if a h = 1 ) or π 0 ⇝ τ (if a h = 0 ), and τ is consistent with some policy in Π core . Therefore, Π ℓ is an ( H +1 , ℓ ) -sunflower.

1 -Active Policies. We will first prove that C (Π 1 -act ) ≤ 2 H . For any deterministic MDP, we use ¯ S h to denote the set of states reachable by Π 1 -act at layer h . We will show that ¯ S h ≤ 2 h by induction on h . For h = 1 , this holds since any deterministic MDP has only one state in the first layer. Suppose it holds at layer h . Then, we have

Note that policies in Π 1 -act must take a = 0 on every s / ∈ { s (1 , 1) , s (1 , 2) , · · · , s (1 ,H ) } . Hence |{ ( s, π ( s )) | s ∈ ¯ S h , π ∈ Π }| ≤ | ¯ S h | +1 ≤ h +1 . Thus, the induction argument is complete. As a consequence we have C h (Π) ≤ 2 h for all h , so

Additionally, if we choose S π = { s (1 , 1) , s (1 , 2) , · · · , s (1 ,H ) } for all π ∈ Π as well as

where π 0 ( s ) ≡ 0 and π h ( s ) := ✶ { s ∈ S h } . Now fix any partial trajectory τ = ( s h , a h · · · , s h ′ , a h ′ ) which satisfies π ⇝ τ . If we have i ∈ ( h, h ′ ] , s i ̸∈ S π , then we must have a i = 0 . Thus, π h ⇝ τ (if a h = 1 ) or π 0 ⇝ τ (if a h = 0 ), so τ is consistent with some policy in Π core . Therefore, Π 1 -act is a ( H +1 , H ) -sunflower.

Additionally, choose

All-Active Policies. For any deterministic MDP, there is a single state s ( j, 1) in the first layer. Any policy which takes a = 1 at state s ( j, 1) must belong to Π j -act . Hence such policies can reach at most C h -1 (Π j -act ) states in layer h . For polices which take action 0 at state h , all these policies will transit to a fixed state in layer 2 . Hence such policies can reach at most C h -1 (Π act ) states at layer h . Therefore, we get

By telescoping, we get which indicates that

Additionally, if we choose S π = { s ( j, 1) , · · · , s ( j,H ) } for all π ∈ Π j -act , as well as

where π 0 ( s ) := 0 and π h ( s ) := ✶ { s ∈ S h } . Now fix any partial trajectory τ = ( s h , a h · · · , s h ′ , a h ′ ) which satisfies π ⇝ τ . If we have i ∈ ( h, h ′ ] , s i ̸∈ S π , then we must have a i = 0 . Thus, π h ⇝ τ (if a h = 1 ) or π 0 ⇝ τ (if a h = 0 ), so τ is consistent with some policy in Π core . Therefore, Π act is a ( H +1 , H ) -sunflower.

Policy Classes for Continuous State Spaces. In some cases, it is possible to construct policy classes over continuous state spaces that have bounded spanning capacity. For example, consider Π sing , which is defined over a discrete (but large) state space. We can extend this to continuous state space by defining new state spaces S h = { s ( x,h ) : x ∈ R } for all h ∈ [ H ] , action space A = { 0 , 1 } , and policy class

˜ Π sing := { π ( i,h ′ ) : π ( i,h ′ ) ( s ( x,h ) ) = ✶ { x ∈ [ i, i +1) and h = h ′ } , i ∈ N , h ′ ∈ [ H ] } . Essentially, we have expanded each state to be an interval on the real line. Using the same reasoning, we have the bound C ( ˜ Π sing ) = H +1 . One can also generalize this construction to the policy class ˜ Π ℓ -ton and preserve the same value of C . 5

However, in general, this expansion to continuous state spaces may blow up the spanning capacity. Consider a similar modification to Π 1 -act (again, with the same new state space and action space A = { 0 , 1 } ):

While C (Π 1 -act ) = Θ( H ) , it is easy to see that C ( ˜ Π 1 -act ) = 2 H since one can construct a H -layer deterministic tree using states in [0 , 1) as every ( s, a ) pair at layer H will be reachable by ˜ Π 1 -act .

C Proofs for Section 3

C.1 Proof of Lemma 1

Fix any M ∈ M sto , as well as h ∈ [ H ] . We claim that

Here, d π h ( s h , a h ; M ) is the state-action visitation distribution of the policy π on MDP M .

We first set up additional notation. Let us define a prefix as any tuple of pairs of the form

̸

We will denote prefix sequences as ( s 1: k , a 1: k ) or ( s 1: k +1 , a 1: k ) respectively. For any prefix ( s 1: k , a 1: k ) (similarly prefixes of the type ( s 1: k +1 , a 1: k ) ) we let d π h ( s h , a h | ( s 1: k , a 1: k ); M ) denote the conditional probability of reaching ( s h , a h ) under policy π given one observed the prefix ( s 1: k , a 1: k ) in MDP M , with d π h ( s h , a h | ( s 1: k , a 1: k ); M ) = 0 if π ⇝ ( s 1: k , a 1: k ) or π ⇝ ( s h , a h ) .

̸

In the following proof, we assume that the start state s 1 is fixed, but this is without any loss of generality, and the proof can easily be adapted to hold for stochastic start states.

Our strategy will be to explicitly compute the quantity Γ h in terms of the dynamics of M and show that we can upper bound it by a 'derandomized' MDP M ′ which maximizes reachability at layer h . Let us unroll one step of the dynamics:

̸

The equality ( i ) follows from the fact that M always starts at s 1 . The equality ( ii ) follows from the fact that π is deterministic, so there exists exactly one a ′ = π ( s 1 ) for which d π h ( s h , a h | s 1 , a ′ ; M ) = d π h ( s h , a h | s 1 ; M ) , with all other a ′′ = a ′ satisfying d π h ( s h , a h | s 1 , a ′′ ; M ) = 0 . The inequality ( iii ) follows by swapping the supremum and the sum.

Continuing in this way, we can show that

Now we examine the conditional visitation d π h ( s h , a h | ( s 1: h -1 , a 1: h -1 ); M ) . Observe that it can be rewritten as d π h ( s h , a h | ( s 1: h -1 , a 1: h -1 ); M ) = P ( s h | s h -1 , a h -1 ) · ✶ { π ⇝ ( s 1: h , a 1: h ) } . Plugging this back into the previous display, and again swapping the supremum and the sum, we get that

Our last step is to derandomize the stochastic transitions in the above stochastic MDP, simply by taking the sup over the transition probabilities:

The right hand side of the inequality is exactly the definition of max M ′ ∈M det C reach h (Π; M ′ ) , thus proving Eq. (3). In particular, the above process defines the deterministic MDP which maximizes the reachability at level h . Taking the maximum over h as well as supremum over M , we see that sup M ∈M sto C cov (Π; M ) ≤ C (Π) . Furthermore, from the definitions we have

This concludes the proof of Lemma 1.

C.2 Coverability is Not Sufficient for Online RL

In this section, we observe that bounded coverability by itself is not sufficient to ensure sample efficient agnostic PAC RL in the online interactive model. First note that Theorem 3 already shows this indirectly. In particular, in Theorem 3, we show that there exists a policy class with bounded spanning capacity that is hard to learn in online RL. However, recall Lemma 1 which implies that any policy class with bounded spanning capacity must also have bounded coverability; and thus the lower bound in Theorem 3 can be trivially extended to argue that bounded coverability by itself does not suffice for statistically efficient agnostic online RL.

However, we can also show the insufficiency of coverability through a much simpler route by directly invoking the lower bound construction in Sekhari et al. (2021). In particular, Sekhari et al. (2021) provides a construction for a low rank MDP with rich observations which satisfies C cov (Π; M ) = O (1) for every M ∈ M , but still needs 2 Ω( H ) many samples for any (Θ(1) , Θ(1)) -PAC learner (see Theorem 2 in their paper for more details; we simply set d = Θ( H ) to get our lower bound).

We do not know if coverability is also insufficient for the generative model setting; we conjecture that one may be able to show, using a similar construction, that coverability is insufficient when C (Π) is large, showing that one cannot adapt to benign problem instances.

D Proofs for Section 4

D.1 Proof of Theorem 1

Algorithm 2 TrajectoryTree (Kearns et al., 1999)

We show that, with minor changes, the TrajectoryTree algorithm of Kearns et al. (1999) attains the guarantee in Theorem 1. The pseudocode can be found in Algorithm 2. The key modification is line 6: we simply observe that only ( s, a ) pairs which are reachable by some π ∈ Π in the current

tree ̂ T i need to be sampled (in contrast, in the original algorithm of Kearns et al. (1999), they sample all A H transitions).

Fix any π ∈ Π . For every trajectory tree i ∈ [ n ] , the algorithm has collected enough transitions so that ̂ v π i is well-defined, by line 6 of the algorithm. By the sampling process, it is clear that the values { ̂ v π i } i ∈ [ n ] are i.i.d. generated. We claim that they are unbiased estimates of V π . Observe that one way of defining V π is the expected value of the following process:

For every ( s, a ) ∈ S × A , independently sample a next state s ′ ∼ P ( ·| s, a ) and a reward r ∼ R ( s, a ) to define a deterministic MDP ̂ M

Return the value ̂ v π to be the value of π run on ̂ M .

Define the law of this process as Q . The sampling process of TrajectoryTree (call the law of this process Q ) can be viewed as sampling the subset of ̂ M det which is reachable by some π ∈ Π . Thus, we have

where the second equality is due to the law of total probability, and the third equality is due to the fact that ̂ v π is measurable with respect to the trajectory tree ̂ T . Thus, { ̂ v π i } i ∈ [ n ] are unbiased estimates of V π .

Therefore, by Hoeffding's inequality (Lemma 17) we see that | V π -̂ V π | ≤ √ log(2 /δ ) 2 n . Applying union bound we see that when the number of trajectory trees exceeds n ≳ log( | Π | /δ ) ε 2 , with probability at least 1 -δ , for all π ∈ Π , the estimates satisfy | V π -̂ V π | ≤ ε/ 2 . Thus the TrajectoryTree algorithm returns an ε -optimal policy. Since each trajectory tree uses at most H · C (Π) queries to the generative model, we have the claimed sample complexity bound.

D.2 Proof of Theorem 2

Fix any worst-case deterministic MDP M ⋆ which witnesses C (Π) at layer h ⋆ . Since C (Π) is a property depending on the dynamics of M ⋆ , we can assume that M ⋆ has zero rewards. We can also assume that the algorithm knows M ⋆ and h ⋆ (this only makes the lower bound stronger). We construct a family of instances M ⋆ where all the MDPs in M ⋆ have the same dynamics as M ⋆ but different nonzero rewards at the reachable ( s, a ) pairs at layer h ⋆ .

Observe that we can embed a multi-armed bandit instance with C (Π) arms using the class M ⋆ . The value of any policy π ∈ Π is exactly the reward that it receives at the unique ( s, a ) pair in layer h ⋆ that it reaches. Any ( ε, δ ) -PAC algorithm that works over the family of instances M ⋆ must return a policy ˆ π that reaches an ( s, a ) pair in layer h ⋆ with near-optimal reward. Furthermore, in the generative model setting, the algorithm can only receive information about a single ( s, a ) pair. Thus, such a PAC algorithm must also be able to PAC learn the best arm for multi-armed bandits with C (Π) arms. Therefore, we can directly apply existing PAC lower bounds which show that the sample complexity of ( ε, δ ) -PAC learning the best arm for K -armed multi-armed bandits is at least Ω( K ε 2 · log 1 δ ) (see, e.g., Mannor and Tsitsiklis, 2004).

D.3 Proof of Corollary 1

The upper bound is obtained by a simple modification of the argument in the proof of Theorem 1. In terms of data collection, the trajectory tree collected every time is the same fixed deterministic MDP (with different rewards); furthermore, one can always execute line 6 and line 9 for a deterministic MDP since the algorithm can execute a sequence of actions to get to any new ( s, a ) pair required by line 9. Thus in every episode of online interaction we are guaranteed to add the new ( s, a ) pair to the trajectory tree.

The lower bound trivially extends because the proof of Appendix D.2 uses a family of MDPs with deterministic transitions (that are even known to the algorithm beforehand).

E Proofs for Section 5

In this section, we prove Theorem 3, which shows a superpolynomial lower bound on the sample complexity required to learn bounded spanning capacity classes, ruling out poly( C (Π) , H, log | Π | ) sample complexity for online RL. We restate the theorem below with the precise constants:

Theorem 5 (Lower bound for online RL) . Let h 0 ∈ N and c ∈ (0 , 1) be universal constants. Fix any H ≥ h 0 . Let ε ∈ (1 / 2 cH , 1 / (100 H )) and ℓ ∈ { 2 , . . . , H } such that 1 /ε ℓ ≤ 2 H . There exists a policy class Π ( ℓ ) of size 1 / (6 ε ℓ ) with C (Π ( ℓ ) ) ≤ O ( H 4 ℓ +2 ) and a family of MDPs M with state space S of size H · 2 2 H +1 , binary action space, horizon H such that: for any ( ε/ 16 , 1 / 8) -PAC algorithm, there exists an M ∈ M in which the algorithm has to collect at least

E.1 Construction of State Space, Action Space, and Policy Class

State and Action Spaces. We define the state space S . In every layer h ∈ [ H ] , there will be 2 2 H +1 states. The states will be paired up, and each state will be denoted by either j [ h ] or j ′ [ h ] , so S h = { j [ h ] : j ∈ [2 2 H ] } ∪ { j ′ [ h ] : j ∈ [2 2 H ] } . For any state s ∈ S , we define the index of s , denoted idx( s ) as the unique j ∈ [2 2 H ] such that s ∈ { j [ h ] } h ∈ [ H ] ∪ { j ′ [ h ] } h ∈ [ H ] . In total there are H · 2 2 H +1 states. The action space is A = { 0 , 1 } .

Policy Class. For the given ε and ℓ ∈ { 2 , . . . , H } , we show via a probabilistic argument the existence of a large policy class Π ( ℓ ) which has bounded spanning capacity but is hard to explore. We state several properties in Lemma 2 which will be exploited in the lower bound.

We introduce some additional notation. For any j ∈ [2 2 H ] we denote

that is, Π ( ℓ ) j are the policies which take an action a = 1 on at least one state with index j .

We also define the set of relevant state indices for a given policy π ∈ Π ( ℓ ) as

For any policy π we denote π ( j 1: H ) := ( π ( j [1]) , . . . , π ( j [ H ])) ∈ { 0 , 1 } H to be the vector that represents the actions that π takes on the states in index j . The vector π ( j ′ 1: H ) is defined similarly.

Lemma 2. Let H , ε , and ℓ satisfy the assumptions of Theorem 5. There exists a policy class Π ( ℓ ) of size N = 1 / (6 ε ℓ ) which satisfies the following properties.

For every j ∈ [2 2 H ] we have | Π ( ℓ ) j | ∈ [ εN/ 2 , 2 εN ] .

For every π ∈ Π we have |J π rel | ≥ ε/ 2 · 2 2 H .

For every π ∈ Π ( ℓ ) j , the vector π ( j 1: H ) is unique and always equal to π ( j ′ 1: H ) .

Bounded spanning capacity: C (Π ( ℓ ) ) ≤ c · H 4 ℓ +2 for some universal constant c > 0 .

E.2 Construction of MDP Family

The family M = { M π ⋆ ,ϕ } π ⋆ ∈ Π ( ℓ ) ,ϕ ∈ Φ will be a family of MDPs which are indexed by a policy π ⋆ as well as a decoder function ϕ : S ↦→ { GOOD , BAD } , which assigns each state to be 'good' or 'bad' in a sense that will be described later on. An example construction of an MDP M π ⋆ ,ϕ is illustrated in Figure 1. For brevity, the bracket notation used to denote the layer that each state lies in has been omitted in the figure.

…

1

2

2

2

H

…

50

…

Uniform over

combination locks

2

2

H

…

Relevant Combination Lock for

: all transitions

deterministic

Mπ

⋆ ,

ϕ

Irrelevant Combination Lock for

: all transitions

uniform

to next level

Mπ

⋆ ,

ϕ

…

h

= 1

h

= 2

1

g

1

b

50

g

50

b

h

= 3

h

= 4

h

=

H

1

g

1

b

1

g

1

b

…

…

1

g

1

b

r

H

(

3

4

)

r

H

(

1

2

)

r

H

(

1

2

)

r

H

(

1

2

)

50

g

50

b

50

g

50

b

…

…

50

g

50

b

r

H

(

1

2

)

r

H

(

1

2

)

Decoder Function Class. The decoder function class Φ will be the set of all possible mappings which for every j ∈ [2 2 H ] and h ≥ 2 assign exactly one of j [ h ] or j ′ [ h ] to the label GOOD (where the other is assigned to the label BAD). There are (2 H -1 ) 2 2 H such functions. The label of a state will be used to describe the transition dynamics. Intuitively, a learner who does not know the decoder function ϕ will not be able to tell if a certain state has the label GOOD or BAD when visiting that state for the first time.

Transition Dynamics. The MDP M π ⋆ ,ϕ will be a uniform distribution over 2 2 H combination locks { CL j } j ∈ [2 2 H ] with disjoint states. More formally, s 1 ∼ Uniform( { j [1] } j ∈ [2 2 H ] ) . From each start state j [1] , only the 2 H -2 states corresponding to index j at layers h ≥ 2 will be reachable in the combination lock CL j .

In the following, we will describe each combination lock CL j , which forms the basic building block of the MDP construction.

Good/Bad Set. At every layer h ∈ [ H ] , for each j [ h ] and j ′ [ h ] , the decoder function ϕ assigns one of them to be GOOD and one of them to be BAD. We will henceforth denote j g [ h ] to be the good state and j b [ h ] to be the bad state. Observe that by construction in Eq. (7), for every π ∈ Π ( ℓ ) and h ∈ [ H ] we have π ( j g [ h ]) = π ( j b [ h ]) .

Dynamics of CL j , if j ∈ J π ⋆ rel . Here, the transition dynamics of the combination locks are deterministic. For every h ∈ [ H ] ,

-On good states j g [ h ] we transit to the next good state iff the action is π ⋆ :

✶ -On bad states j b [ h ] we always transit to the next bad state:

̸

Dynamics of CL j , if j / ∈ J π ⋆ rel . If j is not a relevant index for π ⋆ , then the transitions are uniformly random regardless of the current state/action. For every h ∈ [ H ] ,

Reward Structure. The reward function is nonzero only at layer H , and is defined as

That is, we get 3 / 4 whenever we reach the H -th good state for an index j which is relevant for π ⋆ , and 1 / 2 reward otherwise.

Reference MDPs. We define several reference MDPs.

In the reference MDP M 0 , the initial start state is again taken to be the uniform distribution, i.e., s 1 ∼ Uniform( { j [1] } j ∈ [2 2 H ] ) , and all the combination locks behave the same and have uniform transitions to the next state along the chain: for every h ∈ [ H ] and j ∈ [2 2 H ] ,

The rewards for M 0 are Ber(1 / 2) for every ( s, a ) ∈ S H ×A .

For any decoder ϕ ∈ Φ , the reference MDP M 0 ,π ⋆ ,ϕ has the same transitions as M π ⋆ ,ϕ but the rewards are Ber(1 / 2) for every ( s, a ) ∈ S H ×A .

E.3 Proof of Theorem 5

We are now ready to prove the lower bound using the construction of the MDP family M .

Value Calculation. Consider any M π ⋆ ,ϕ ∈ M . For any policy π ∈ A S we use V π ⋆ ,ϕ ( π ) to denote the value of running π in MDP M π ⋆ ,ϕ . By construction we can see that

̸

where in the above, we defined for any s 1 , π (idx( s 1 ) 1: H ) = π ( j 1: H ) = ( π ( j [1]) , . . . , π ( j [ H ])) , where j denotes idx( s 1 ) . Informally speaking, the second term counts the additional reward that π gets for solving a combination lock rooted at a relevant state index idx( s 1 ) ∈ J π ⋆ rel . By Property (2) and (3) of Lemma 2, we additionally have V π ⋆ ,ϕ ( π ⋆ ) ≥ 1 / 2 + ε/ 8 , as well as V π ⋆ ,ϕ ( π ) = 1 / 2 for all other π = π ⋆ ∈ Π ( ℓ ) .

By Eq. (4), if π is an ε/ 16 -optimal policy on M π ⋆ ,ϕ it must satisfy

Averaged Measures. We define the following measures which will be used in the analysis.

Define P π ⋆ [ · ] = 1 | Φ | ∑ ϕ ∈ Φ P π ⋆ ,ϕ [ · ] to be the averaged measure where we first pick ϕ uniformly among all decoders and then consider the distribution induced by M π ⋆ ,ϕ .

Define the averaged measure P 0 ,π ⋆ [ · ] = 1 | Φ | ∑ ϕ ∈ Φ P 0 ,π ⋆ ,ϕ [ · ] where we pick ϕ uniformly and then consider the distribution induced by M 0 ,π ⋆ ,ϕ .

For both averaged measures the expectations E π ⋆ and E 0 ,π ⋆ are defined analogously.

Algorithm and Stopping Time. Recall that an algorithm A is comprised of two phases. In the first phase, it collects some number of trajectories by interacting with the MDP in episodes. We use η to denote the (random) number of episodes after which A terminates. We also use A t to denote the intermediate policy that the algorithm runs in round t for t ∈ [ η ] . In the second phase, A outputs 6 a

policy ̂ π . We use the notation A f : { τ ( t ) } t ∈ [ η ] ↦→A S to denote the second phase of A which outputs ̂ π as a measurable function of collected data.

For any policy π ⋆ , decoder ϕ , and dataset D we define the event

The event E ( π ⋆ , ϕ, A f ( D )) is measurable with respect to the random variable D , which denotes the collected data.

Under this notation, the PAC learning guarantee on A implies that for every π ⋆ ∈ Π ( ℓ ) , ϕ ∈ Φ we have

Moreover via an averaging argument we also have

Lower Bound Argument. We apply a truncation to the stopping time η . Define T max := 2 H/ 3 . Observe that if P π ⋆ [ η > T max ] > 1 / 8 for some π ⋆ ∈ Π ( ℓ ) then the lower bound immediately follows, since

so there must exist an MDP M π ⋆ ,ϕ for which A collects at least T max / 8 = 2 H/ 3 -3 samples in expectation.

Otherwise we have P π ⋆ [ η > T max ] ≤ 1 / 8 for all π ⋆ ∈ Π ( ℓ ) . This further implies that for all π ⋆ ∈ Π ( ℓ ) ,

However, in the following, we will argue that if Eq. (6) holds then A must query a significant number of samples in M 0 .

Lemma 3 (Stopping Time Lemma) . Let δ ∈ (0 , 1 / 8] . Let A be an ( ε/ 16 , δ ) -PAC algorithm. Let T max ∈ N . Suppose that P π ⋆ [ η < T max and E ( π ⋆ , ϕ, A f ( D ))] ≥ 1 -2 δ for all π ⋆ ∈ Π ( ℓ ) . The expected stopping time for A on M 0 is at least

Using Lemma 3 with δ = 1 / 8 and plugging in the value of | Π ( ℓ ) | and T max , we see that

For the second inequality, we used the fact that ℓ ≥ 2 , H ≥ 10 5 , and ε < 1 / 10 7 .

We have shown that either there exists some MDP M π ⋆ ,ϕ for which A collects at least T max / 8 = 2 H/ 3 -3 samples in expectation, or A must query at least | Π ( ℓ ) | / 20 = 1 / (120 ε ℓ ) trajectories in expectation in M 0 . Putting it all together, the lower bound on the sample complexity is at least

This concludes the proof of Theorem 5.

E.4 Proof of Lemma 2

To prove Lemma 2, we first use a probabilistic argument to construct a certain binary matrix B which satisfies several properties, and then construct Π ( ℓ ) using B and verify it satisfies Properties (1)-(4).

Binary Matrix Construction. First, we define a block-free property of binary matrices.

Definition 5 (Block-free Matrices) . Fix parameters k, ℓ, N, d ∈ N where k ≤ N and l ≤ d . We say a binary matrix B ∈ { 0 , 1 } N × d is ( k, ℓ ) -block-free if the following holds: for every I ⊆ [ N ] with | I | = k , and J ⊆ [ d ] with | J | = ℓ there exists some ( i, j ) ∈ I × J with B ij = 0 .

In words, matrices which are ( k, ℓ ) -block-free do not contain a k × ℓ block of all 1s.

Lemma 4. Fix any ε ∈ (0 , 1 / 10) and ℓ ∈ N . For any

there exists a binary matrix B ∈ { 0 , 1 } N × d with N = 1 / (6 · ε ℓ ) such that:

(Row sum): for every row i ∈ [ N ] , we have ∑ j B ij ≥ εd/ 2 .

(Column sum): for every column j ∈ [ d ] , we have ∑ i B ij ∈ [ εN/ 2 , 2 εN ] .

The matrix B is ( ℓ log d, ℓ ) -block-free.

Proof of Lemma 4. The existence of B is proven using the probabilistic method. Let ˜ B ∈ { 0 , 1 } N × d be a random matrix where each entry is i.i.d. chosen to be 1 with probability ε .

By Chernoff bounds (Lemma 18), for every row i ∈ [ N ] , we have P [ ∑ j B ij ≤ εd 2 ] ≤ exp( -εd/ 8) ; likewise for every column j ∈ [ d ] , we have P [ ∑ j B ij / ∈ [ εN 2 , 2 εN ]] ≤ 2 exp( -εN/ 8) . By union bound, the matrix ˜ B satisfies the first two properties with probability at least 0 . 8 as long as

One can check that under the choice of N = 1 / (6 · ε ℓ ) and the assumption on d , both constraints are met.

Now we examine the probability of ˜ B satisfies the block-free property with parameters ( k := ℓ log d, ℓ ) . Let X be the random variable which denotes the number of submatrices which violate the block-free property in ˜ B , i.e.,

By linearity of expectation, we have

We now plug in the choice k = ℓ log d and observe that as long as N ≤ 1 / (2 e · ε ℓ ) we have E [ X ] ≤ 1 / 2 . By Markov's inequality, P [ X = 0] ≥ 1 / 2 .

Therefore with positive probability, ˜ B satisfies all 3 properties (otherwise we would have a contradiction via inclusion-exlusion principle). Thus, there exists a matrix B which satisfies all of the above three properties, proving the result of Lemma 4.

Policy Class Construction. For the given ε and ℓ ∈ { 2 , . . . , H } we will use Lemma 4 to construct a policy class Π ( ℓ ) which has bounded spanning capacity but is hard to explore. We instantiate Lemma 4 with the given ℓ and d = 2 2 H , and use the resulting matrix B to construct Π ( ℓ ) = { π i } i ∈ [ N ] with | Π ( ℓ ) | = N = 1 / (6 ε ℓ ) .

Recall that we assume that

We claim that under these assumptions, the requirement of Lemma 4 is met:

For the lower bound, we can check that:

where we use the bound ℓ ≤ H and ε ≥ 2 -cH . The last inequality holds for sufficiently small universal constant c ∈ (0 , 1) and sufficiently large H ≥ h 0 .

For the upper bound, we can also check that

where we use the bound ℓ ≥ 2 and ε ≤ 1 / (100 H ) . The last inequality holds for sufficiently large H . We define the policies as follows: for every π i ∈ Π ( ℓ ) we set

The function bit h : [2 H -1] ↦→{ 0 , 1 } selects the h -th bit in the binary representation of the input.

Verifying Properties (1) -(4) of Lemma 2. Properties (1) -(3) are straightforward from the construction of B and Π ( ℓ ) , since π i ∈ Π ( ℓ ) j if and only if B ij = 1 . The only detail which requires some care is that we require that 2 εN < 2 H in order for Property (3) to hold, since otherwise we cannot assign the behaviors of the policies according to Eq. (7). However, by assumption, this always holds, since 2 εN = 1 / (3 ε ℓ -1 ) ≤ 2 H .

We now prove Property (4) that Π ( ℓ ) has bounded spanning capacity. To prove this we will use the block-free property of the underlying binary matrix B .

Fix any deterministic MDP M ⋆ which witnesses C (Π ( ℓ ) ) at layer h ⋆ . To bound C (Π ( ℓ ) ) , we need to count the contribution to C reach h ⋆ (Π; M ⋆ ) from trajectories τ which are produced by some π ∈ Π ( ℓ ) on M . We first define a layer decomposition for a trajectory τ = ( s 1 , a 1 , s 2 , a 2 , . . . , s H , a H ) as the unique tuple of indices ( h 1 , h 2 , . . . h m ) , where each h k ∈ [ H ] , that satisfies the following properties:

The layers satisfy h 1 < h 2 < · · · < h m .

The layer h 1 represents the first layer where a h 1 = 1 .

The layer h 2 represents the first layer where a h 2 = 1 on some state s h 2 such that

The layer h 3 represents the first layer where a h 3 = 1 on some state s h 3 such that

More generally the layer h k , k ∈ [ m ] represents the first layer where a h k = 1 on some state s h k such that

In other words, the layer h k represents the k -th layer for where action is a = 1 on a new state index which τ has never played a = 1 on before.

We will count the contribution to C reach h ⋆ (Π; M ⋆ ) by doing casework on the length of the layer decomposition for any τ . That is, for every length m ∈ { 0 , . . . , H } , we will bound C h ⋆ ( m ) , which is defined to be the total number of ( s, a ) at layer h ⋆ which, for some π ∈ Π ( ℓ ) , a trajectory π ⇝ τ that has a m -length layer decomposition visits. Then we apply the bound

Note that this will overcount, since the same ( s, a ) pair can belong to multiple different trajectories with different length layer decompositions.

Lemma 5. The following bounds hold:

For any m ≤ ℓ , C h ⋆ ( m ) ≤ H m · ∏ m k =1 (2 kH ) = O ( H 4 m ) .

We have ∑ m ≥ ℓ +1 C h ⋆ ( m ) ≤ O ( ℓ · H 4 ℓ +1 ) .

Therefore, applying Lemma 5 to Eq. (8), we have the bound that

This concludes the proof of Lemma 2.

Proof of Lemma 5. All of our upper bounds will be monotone in the value of h ⋆ , so we will prove the bounds for C H ( m ) . In the following, fix any deterministic MDP M ⋆ .

First we start with the case where m = 0 . The trajectory τ must play a = 0 at all times; since there is only one such τ , we have C H (0) = 1 .

Now we will bound C H ( m ) , for any m ∈ { 1 , . . . , ℓ } . Observe that there are ( H m ) ≤ H m ways to pick the tuple ( h 1 , . . . , h m ) . Now we will fix ( h 1 , . . . , h m ) and count the contributions to C H ( m ) for trajectories τ which have this fixed layer decomposition, and then sum up over all possible choices of ( h 1 , . . . , h m ) .

In the MDP M ⋆ , there is a unique state s h 1 which τ must visit. In the layers between h 1 and h 2 , all trajectories are only allowed take 1 on states with index idx( s h 1 ) , but they are not required to. Thus we can compute that the contribution to C h 2 ( m ) from trajectories with the fixed layer decomposition to be at most 2 H . The reasoning is as follows. At h 1 , there is exactly one ( s, a ) pair which is reachable by trajectories with this fixed layer decomposition, since any τ must take a = 1 at s h 1 . Subsequently we can add at most two reachable pairs in every layer h ∈ { h 1 +1 , . . . , h 2 -1 } due to encountering a state j [ h ] or j ′ [ h ] where j = idx( s h 1 ) , and at layer h 2 we must play a = 1 , for a total of 1 + 2( h 2 -h 1 -1) ≤ 2 H . Using similar reasoning the contribution to C h 3 ( m ) from trajectories with this fixed layer decomposition is at most (2 H ) · (4 H ) , and so on. Continuing in this way, we have the final bound of ∏ m k =1 (2 kH ) . Since this holds for a fixed choice of ( h 1 , . . . , h m ) in total we have C H ( m ) ≤ H m · ∏ m k =1 (2 kH ) = O ( H 4 m ) .

When m ≥ ℓ +1 , observe that the block-free property on B implies that for any J ⊆ [2 H ] with | J | = ℓ we have |∩ j ∈ J Π j | ≤ ℓ log 2 2 H . So for any trajectory τ with layer decomposition such that m ≥ ℓ we can redo the previous analysis and argue that there is at most ℓ log 2 2 H multiplicative factor contribution to the value C H ( m ) due to all trajectories which have layer decompositions longer than ℓ . Thus we arrive at the bound ∑ m ≥ ℓ +1 C H ( m ) ≤ O ( H 4 ℓ ) · ℓ log 2 2 H ≤ O ( ℓ · H 4 ℓ +1 ) .

E.5 Proof of Lemma 3

The proof of this stopping time lemma follows standard machinery for PAC lower bounds (Garivier et al., 2019; Domingues et al., 2021; Sekhari et al., 2021). In the following we use KL( P ∥ Q ) to denote the Kullback-Leibler divergence between two distributions P and Q and kl( p ∥ q ) to denote the Kullback-Leibler divergence between two Bernoulli distributions with parameters p, q ∈ [0 , 1] .

For any π ⋆ ∈ Π ( ℓ ) we denote the random variable

the number of episodes for which the algorithm's policy at round t ∈ [ η ∧ T max ] matches that of π ⋆ on a certain relevant state of π ⋆ .

In the sequel we will prove upper and lower bounds on the intermediate quantity ∑ π ⋆ ∈ Π E 0 [ N π ⋆ ] and relate these quantities to E 0 [ η ] .

Step 1: Upper Bound. First we prove an upper bound. We can compute that

Here, the first inequality follows because for every index j and every π ⋆ ∈ Π ( ℓ ) j , each π ⋆ admits a unique sequence of actions (by Property (3) of Lemma 2), so any policy A t can completely match with at most one of the π ⋆ .

Step 2: Lower Bound. Now we turn to the lower bound. We use a change of measure argument.

The inequality ( i ) follows from a change of measure argument using Lemma 6, with ∆( T max ) := T max 2 / 2 H +3 . Here, F η ∧ T max denotes the natural filtration generated by the first η ∧ T max episodes. The inequality ( ii ) follows from Lemma 7, using the fact that M 0 ,π ⋆ ,ϕ and M π ⋆ ,ϕ have identical transitions and only differ in rewards at layer H for the trajectories which reach the end of a relevant combination lock. The number of times this occurs is exactly N π ⋆ . The factor 1 / 7 is a lower bound on kl(1 / 2 ∥ 3 / 4) . The inequality ( iii ) follows by the convexity of KL divergence.

Now we apply Lemma 8 to lower bound the expectation for any F η ∧ T max -measurable random variable Z ∈ [0 , 1] as

where the second inequality follows from the bound kl( p ∥ q ) ≥ (1 -p ) log(1 / (1 -q )) -log(2) (see, e.g., Domingues et al., 2021, Lemma 15).

Now we pick Z = Z π ⋆ := ✶ { η < T max and E ( π ⋆ , ϕ, A f ( D )) } and note that E π ⋆ [ Z π ⋆ ] ≥ 1 -2 δ by assumption. This implies that

Another application of Lemma 6 gives

Summing the above over π ⋆ ∈ Π ( ℓ ) , we get

It remains to prove an upper bound on ∑ π ⋆ E 0 [ Z π ⋆ ] . We calculate that

The last inequality is an application of Markov's inequality.

Now we carefully investigate the sum. For any ϕ ∈ Φ , the sum can be rewritten as

The equality ( i ) follows because regardless of which MDP M π ⋆ we are in, the first state is distributed uniformly over S 1 . The equality ( ii ) follows because once we condition on the first state s 1 , the probability is either 0 or 1.

Fix any start state s 1 . We can write

where the second equality uses the fact that on any index j , each π ⋆ ∈ Π ( ℓ ) j behaves differently (Property (3) of Lemma 2), so A f ( D ) can match at most one of these behaviors. Plugging this back into Eq. (12), averaging over ϕ ∈ Φ , and combining with Eq. (11), we arrive at the bound

We now use this in conjunction with Eq. (10) to arrive at the final lower bound

Step 3: Putting it All Together. Combining Eqs. (9) and (13), plugging in our choice of ∆( T max ) , and simplifying we get

The last inequality follows since δ ≤ 1 / 8 implies log(1 / (2 δ )) ≥ 2 log(2) .

This concludes the proof of Lemma 3.

E.6 Change of Measure Lemma

Lemma 6. Let Z ∈ [0 , 1] be a F T max -measurable random variable. Then, for every π ⋆ ∈ Π ( ℓ ) ,

Proof. First, we note that

Here P 0 [ ·|F t ] denotes the conditional distribution of the t -th trajectory given the first t -1 trajectories. Similarly P 0 ,π ⋆ [ ·|F t ] is the averaged over decoders condition distribution of the t -th trajectory given the first t -1 trajectories. The second inequality follows by chain rule of TV distance (see, e.g., Polyanskiy and Wu, 2022, pg. 152).

Now we examine each term TV( P 0 [ ·|F t -1 ] , P 0 ,π ⋆ [ ·|F t -1 ]) . Fix a history F t -1 and sequence s 1: H where all s i have the same index. We want to bound the quantity

where it is understood that the random variable S ( t ) 1: H is drawn according to the MDP dynamics and algorithm's policy A t (which is in turn a measurable function of F t -1 ).

We observe that the second term is exactly

since the state s 1 appears with probability 1 / |S 1 | and the transitions in M 0 are uniform to the next state in the combination lock, so each sequence is equally as likely.

For the first term, again the state s 1 appears with probability 1 / |S 1 | . Suppose that idx( s 1 ) / ∈ J π ⋆ rel . Then the dynamics of P 0 ,π ⋆ ,ϕ for all ϕ ∈ Φ are exactly the same as M 0 , so again the probability in this case is 1 / ( |S 1 | 2 H -1 ) . Now consider when idx( s 1 ) ∈ J π ⋆ rel . At some point ̂ h ∈ [ H +1] , the policy A t will deviate from π ⋆ for the first time (if A t never deviates from π ⋆ we set ̂ h = H +1) . The layer ̂ h is only a function of s 1 and A t and does not depend on the MDP dynamics. The correct decoder must assign ϕ ( s 1: ̂ h -1 ) = GOOD and ϕ ( s ̂ h : H ) = BAD, so therefore we have

If s 1 / ∈ F t -1 , i.e., we are seeing s 1 for the first time, then the conditional distribution over the labels given by ϕ is the same as the unconditioned distribution:

Otherwise, if s 1 ∈ F t -1 then we bound the conditional probability by 1.

Putting this all together we can compute

Therefore we have the bound

∣ ∣ ∣ P 0 ,π ⋆ [ S ( t ) 1: H = s 1: H | F t -1 ] -P 0 [ S ( t ) 1: H = s 1: H | F t -1 ]∣ ∣ ∣ ≤ 1 |S 1 | ✶ { idx( s 1 ) ∈ J π ⋆ rel , s 1 ∈ F t -1 } . Summing over all possible sequences s 1: H we have

since the only sequences s 1: H for which the difference in the two measures are nonzero are the ones for which s 1 ∈ F t -1 , of which there are ( t -1) · 2 H -1 of them.

Lastly, taking expectations and summing over t = 1 to T max and plugging in the value of |S 1 | = 2 2 H we have the final bound.

The next lemma is a straightforward modification of (Domingues et al., 2021, Lemma 5), with varying rewards instead of varying transitions.

Lemma 7. Let M and M ′ be two MDPs that are identical in transition and differ in the reward distributions, denote r h ( s, a ) and r ′ h ( s, a ) . Assume that for all ( s, a ) we have r h ( s, a ) ≪ r ′ h ( s, a ) . Then for any stopping time η with respect to ( F t ) t ≥ 1 that satisfies P M [ η < ∞ ] = 1 ,

where N η s,a,h := ∑ η t =1 ✶ { ( S ( t ) h , A ( t ) h ) = ( s, a ) } and I η : Ω ↦→ ⋃ t ≥ 1 I t : ω ↦→ I η ( ω ) ( ω ) is the random vector representing the history up to episode η .

Lemma 8 (Lemma 1, Garivier et al. (2019)) . Consider a measurable space (Ω , F ) equipped with two distributions P 1 and P 2 . For any F -measurable function Z : Ω ↦→ [0 , 1] we have

F Proofs for Section 6

F.1 Algorithm Sketch

In this section we provide a high level sketch of Algorithm 1. We will describe the key algorithmic ideas, as well as the empirical estimation of policy-specific MRPs, in the two phases of POPLER .

State Identification Phase

The goal of the state identification phase is to discover all such petal states that are reachable with probability Ω( ε/D ) . The algorithm proceeds in a loop and sequentially grows the set T , which contains tuples of the form ( s, π s ) , where s ∈ ⋃ π ∈ Π S π is a sufficiently reachable petal state (for some policy) and π s denotes a policy that reaches s with probability Ω( ε/D ) . We also denote S rch := { s : ( s, π s ) ∈ T } to denote the set of reachable states in T . Initially, T only contains a dummy start state s ⊤ and a null policy. We will collect data using the DataCollector subroutine that: for a given ( s, π s ) ∈ T , first run π s to reach state s , and if we succeed in reaching s , restart exploration by sampling a policy from Uniform(Π exp ) . Note that DataCollector will be sampleefficient for any ( s, π s ) since Ω( ε/D ) fraction of the trajectories obtained via π s are guaranteed to reach s (by definition of π s and construction of set T ). Initially, we run DataCollector using Uniform(Π exp ) from the start, where we slightly abuse the notation and assume that all trajectories in the MDP start at the dummy state s ⊤ at time step h = 0 .

In every loop, the algorithm attempts to find a new petal state ¯ s for some π ∈ Π that is guaranteed to be Ω( ε/D ) -reachable by π . This is accomplished by constructing a (estimated and partial) version of the policy-specific MRP using the datasets collected up until that loop (line 9). In particular given a policy π and a set S rch π = S + π ∩ S rch , we construct ̂ M π S rch = MRP( S + π , ̂ P π , ̂ R π , H, s ⊤ , s ⊥ ) which essentially compresses our empirical knowledge of the original MDP relevant to the policy π . In particular, for any states s ∈ S π ∪ { s ⊤ } and s ′ ∈ S π ∪ { s ⊥ } residing in different layers h < h ′ in the underlying MDP, we define:

Transition ̂ P π ′

Transition ̂ R π ′

Clearly, the above definition implies that ̂ P π s → s ′ = 0 and ̂ R π s → s ′ = 0 for any s / ∈ S rch π since D s would be empty corresponding to these unexplored states. Furthermore, P π s ⊥ → s ⊥ = 1 and R π s ⊥ → s ⊥ = 0 .

Note that since the set S rch is changing in each iteration of the loop as the algorithm collects more data, the policy-specific MRP ̂ M π S rch also changes in every iteration of the loop - in particular, more and more transitions/rewards are assigned nonzero values due to new states being added to S rch . More details on policy-specific MRPs is given in Appendix F.2.

The key advantage of constructing the empirical versions of policy-specific MRPs is that they allow us to explore and find new leaf states in S π which are reachable with probability at least Ω( ε / 6 D ) . In particular, using standard dynamic programming (subroutine EstReachability ), we can check whether a candidate petal ¯ s is reachable with decent probability by π (lines 11-12); If it is, then we add (¯ s, π ) to the set T and collect a fresh dataset using DataCollector (lines 13-14). Crucially, the importance sampling technique enables us to be sample efficient, since the same dataset D s can be used to evaluate transitions/rewards in Eqs. (14) and (15) for multiple π ∈ Π for which s is a petal state. Furthermore, the number of such datasets we collect must be bounded-each ( s, π s ) ∈ T contributes Ω( ε/D ) to cumulative reachability, but since cumulative reachability is bounded from above by C (Π) (Lemma 1), we know that |T | ≤ O ( D · C (Π) /ε ) .

Evaluation Phase

Next, POPLER moves to the evaluation phase. Using the collected data, it executes the Evaluate subroutine for every π ∈ Π to get estimates ̂ V π (line 21) corresponding to V π . For a given π ∈ Π , the Evaluate subroutine also constructs an empirical policy-specific MRP ̂ M π S rch for every π ∈ Π and computes the value of π via dynamic programming on ̂ M π S rch . While the returned estimate ̂ V π is biased, in the complete proof, we will show that the bias is negligible since it is now only due to the states in the petal S π which are not Ω( ε/D ) -reachable. Thus, we can guarantee that ̂ V π closely estimates V π for every π ∈ Π , and therefore POPLER returns a near-optimal policy.

F.2 Algorithmic Details and Preliminaries

In this subsection, we provide the details of the subroutines that do not appear in the main body, in Algorithms 3, 4, and 5. The transitions and reward functions in line 5 in Algorithm 5 are computed using Eqs. (19) and (20), which are specified below, after introducing additional notation.

Algorithm 3 DataCollector

Algorithm 4 EstReachability

We recall the definition of petals and sunflowers given in the main body (in Definitions 3 and 4). In the rest of this section, we assume that Π is a ( K,D ) -sunflower with Π core and S π for any π ∈ Π .

Definition 6 (Petals and Sunflowers (Definitions 3 and 4 in the main body)) . For a policy set ¯ Π , and states ¯ S ⊆ S , a policy π is said to be a ¯ S -petal on ¯ Π if for all h ≤ h ′ ≤ H , and partial trajectories τ = ( s h , a h , · · · , s h ′ , a h ′ ) that are consistent with π : either τ is also consistent with some π ′ ∈ ¯ Π , or there exists i ∈ ( h, h ′ ] s.t. s i ∈ ¯ S .

A policy class Π is said to be a ( K,D ) -sunflower if there exists a set Π core of Markovian policies with | Π core | ≤ K such that for every policy π ∈ Π there exists a set S π ⊆ S , of size at most D , so that π is an S π -petal on Π core .

Additional notation. Recall that we assumed that the state space S = S 1 × . . . S H is layered. Thus, given a state s , we can infer the layer h such that s ∈ S h . By definition s ⊤ belongs to the layer h = 0 and s ⊥ belongs to the layer h = H . In the following, we define additional notation:

̸

Algorithm 5 Evaluate

( a ) Sets T ( s → s ′ ; ¬ ¯ S ) : For any set ¯ S , and states s, s ′ ∈ S , we define T ( s → s ′ ; ¬ ¯ S ) as the set of all the trajectories that go from s to s ′ without passing through any state in ¯ S in between. More formally, let state s be at layer h , and s ′ be at layer h ′ . Then, T ( s → s ′ ; ¬ ¯ S ) denotes the set of all the trajectories τ = ( s 1 , a 1 , · · · , s H , a H ) that satisfy all of the following:

s h = s , where s h is the state at timestep h in τ .

s h ′ = s ′ , where s h ′ is the state at timestep h ′ in τ .

For all h < ˜ h < h ′ , the state s ˜ h , at time step ˜ h in τ , does not lie in the set ¯ S .

Note that when h ′ ≤ h , we define T ( s → s ′ ; ¬ ¯ S ) = ∅ . Additionally, we define T ( s ⊤ → s ; ¬ ¯ S ) as the set of all trajectories that go to s ′ (from a start state) without going through any state in ¯ S in between. Finally, we define T ( s → s ⊥ ; ¬ ¯ S ) as the set of all the trajectories that go from s at time step h to the end of the episode without passing through any state in ¯ S in between.

Furthermore, we use the shorthand T π ( s → s ′ ) := T ( s → s ′ ; ¬S π ) to denote the set of all the trajectories that go from s to s ′ without passing though any leaf state S π .

( b ) Using the above notation, for any s ∈ S and set ¯ S ⊆ S , we define ¯ d π ( s ; ¬ ¯ S ) as the probability of reaching s (from a start state) without passing through any state in ¯ S in between, i.e.

We next recall the notation of Markov Reward Process and formally define both the population versions of policy-specific MRPs.

Markov Reward Process (MRP). A Markov reward process M = MRP( S , P, R, H, s ⊤ , s ⊥ ) is defined over the state space S with start state s ⊤ and end state s ⊥ , for trajectory length H + 2 . Without loss of generality, we assume that { s ⊤ , s ⊥ } ∈ S . The transition kernel is denoted by P : S × S → [0 , 1] , such that for any s ∈ S , ∑ s ′ P s → s ′ = 1 ; the reward kernel is denoted R : S × S → ∆([0 , 1]) . Throughout, we use the notation → to signify that the transitions and rewards are defined along the edges of the MRP.

A trajectory in M is of the form τ = ( s ⊤ , s 1 , · · · , s H , s ⊥ ) , where s h ∈ S for all h ∈ [ H ] . Furthermore, from any state s ∈ S , the MRP transitions 7 to another state s ′ ∈ S with probability P s → s ′ , and

obtains the rewards r s → s ′ ∼ R s → s ′ . Thus,

Furthermore, in all the MRPs that we consider in the paper, we have P s ⊥ → s ⊥ = 1 and r s ⊥ → s ⊥ = 0 .

Policy-Specific Markov Reward Processes. A key technical tool in our analysis will be policyspecific MRPs that depend on the set S rch of the states that we have explored so far. Recall that for any policy π , S + π = S π ∪ { s ⊤ , s ⊥ } , S rch π = S + π ∩ S rch and S rem π = S + π \ ( S rch π ∪ { s ⊥ } ) . We define the expected and the empirical versions of policy-specific MRPs below; see Figure 2 for an illustration.

( a ) Expected Version of Policy-Specific MRP. We define M π S rch = MRP( S + π , P π , r π , H, s ⊤ , s ⊥ ) where

Transition Kernel P π : For any s ∈ S rch π and s ′ ∈ S + π , we have

P π s → s ′ = E π [ ✶ { τ ∈ T π ( s → s ′ ) } ∣ ∣ s h = s ] , (17) where the expectation above is w.r.t. the trajectories drawn using π in the underlying MDP, and h denotes the time step such that s ∈ S h (again, in the underying MDP). Thus, the transition P π s → s ′ denotes the probability of taking policy π from s and directly transiting to s ′ without visiting any other states in S π . Furthermore, P π s → s ′ = { s ′ = s ⊥ } for all s ∈ S rem π ∪ { s ⊥ } .

r π s → s ′ := E π [ R ( τ h : h ′ ) ✶ { τ ∈ T π ( s → s ′ ) } ∣ ∣ s h = s ] , (18) where R ( τ h : h ′ ) denotes the reward for the partial trajectory τ h : h ′ in the underlying MDP. The reward r π s → s ′ denotes the expectation of rewards collected by taking policy π from s and directly transiting to s ′ without visiting any other states in S π . Furthermore, r π s → s ′ = 0 for all s ∈ S rem π ∪ { s ⊥ } .

✶ · Reward Kernel r π : For any s ∈ S rch π and s ′ ∈ S + π , we have

Throughout the analysis, we use P M [ · ] := P M π S rch [ · ] and E M [ · ] := E M π S rch [ · ] as a shorthand, whenever clear from the context.

( b ) Empirical Version of Policy-Specific MRPs. Since the learner only has sampling access to the underlying MDP, it can not directly construct the MRP M π S rch . Instead, in Algorithm 1, the learner constructs an empirical estimate for M π S rch , defined as ̂ M π S rch = MRP( S + π , ̂ P π , ̂ r π , H, s ⊤ , s ⊥ ) where

Transition Kernel ̂ P π : For any s ∈ S rch π and s ′ ∈ S + π , we have

✶ · Reward Kernel ̂ r π : For any s ∈ S rch π and s ′ ∈ S + π , we have

✶ where Π core denotes the core of the sunflower corresponding to Π and D s denotes a dataset of trajectories collected via DataCollector ( s, π s , Π core , n 2 ) . Furthermore, ̂ P π s → s ′ = { s ′ = s ⊥ } for all s ∈ S rem π ∪ { s ⊥ } .

✶ where Π core denotes the core of the sunflower corresponding to Π , D s denotes a dataset of trajectories collected via DataCollector ( s, π s , Π core , n 2 ) , and R ( τ h : h ′ ) = ∑ h ′ -1 i = h r i . Furthermore, ̂ r π s → s ′ = 0 for all s ∈ S rem π .

The above approximates the MRP given by (14) and (15) in the main body.

and the rewards

s

⊤

s

⊥

s

1

s

2

s

3

h

= 1

h

= 2

P π

s

⊤→

s

⊥

= 0.2

R π

s

⊤→

s

⊥

= 0.9

P π

s

⊤→

s

1

= 0.35

R π

s

⊤→

s

1

= 0.1

P π

s

⊤→

s

2

= 0.4

R π

s

⊤→

s

2

= 0

P π

s

⊤→

s

3

= 0.05

R π

s

⊤ →

s

3

= 1

P π

s

3→

s

⊥

= 1

R π

s

3→

s

⊥

= 0

P π

s

2→

s

⊥

= 1

R π

s

2→

s

⊥

= 0.2

P π

s

1→

s

⊥

= 0.5

R π

s

1→

s

⊥

= 0.1

P π

s

1→

s

2

= 0.4

R π

s

1→

s

2

= 0

P π

s

1→

s

3

= 0.1

R π

s

1→

s

3

= 0.1

h

= 3

P π

s

⊥→

s

⊥

= 1

R π

s

⊥→

s

= 0

h

= 0

h

= 4

⊥

Properties of Trajectories in the Policy-Specific MRPs. We state several properties of trajectories in the policy-specific MRPs, which will be used in the proofs. Let τ = ( s ⊤ , s 1 , · · · , s H , s ⊥ ) denote a trajectory from either M π S rch or ̂ M π S rch .

For some k ≤ H we have s 1 , · · · , s k ∈ S π and s k +1 = · · · = s H = s ⊥ (if k = H we say the second condition is trivially met).

Each state in s 1 , · · · , s k is unique.

Letting h ( s ) denote the layer that a state s ∈ S π is in, we have h ( s 1 ) < · · · < h ( s k ) .

Either (a) s 1 , · · · , s k ∈ S rch π , or (b) s 1 , · · · , s k -1 ∈ S rch π and s k ∈ S rem π .

Parameters Used in Algorithm 1. Here, we list all the parameters that are used in Algorithm 1 and its subroutines:

where C 1 , C 2 > 0 are absolute numerical constants, which will be specified later in the proofs.

F.3 Supporting Technical Results

We start by stating the following variant of the classical simulation lemma (Kearns and Singh, 2002; Agarwal et al., 2019; Foster et al., 2021a).

Lemma 9 (Simulation lemma (Foster et al., 2021a, Lemma F.3)) . Let M = ( S , P, r, H, s ⊤ , s ⊥ ) be a markov reward process. Then, the empirical version ̂ M = ( S , ̂ P, ̂ r, H, s ⊤ , s ⊥ ) corresponding to M

satisfies:

where d M ( s ) is the probability of reaching s under M , and V MRP and ̂ V MRP denotes the value of s ⊤ under M and ̂ M respectively.

The following technical lemma shows that for any policy π , the empirical version of policy-specific MRP closely approximates its expected version.

Lemma 10. Let Algorithm 1 be run with the parameters given in Eq. (21) , and consider any iteration of the while loop in line 5 with the instantaneous set S rch . Further, suppose that |D s | ≥ εn 2 24 D for all s ∈ S rch . Then, with probability at least 1 -δ , the following hold:

( a ) For all π ∈ Π , s ∈ S rch and s ′ ∈ S ∪ { s ⊥ }

( b ) For all π ∈ Π and s ′ ∈ S π ∪ { s ⊥ } ,

In the sequel, we define the event that the conclusion of Lemma 10 holds as E est .

Proof. Fix any π ∈ Π . We first prove the bound for s ∈ S rch π . Let s be at layer h . Fix any policy π ∈ Π , and consider any state s ′ ∈ S π ∪{ s ⊥ } , where s ′ is at layer h ′ . Note that since Π is a ( K,D ) -sunflower, with its core Π core and petals {S π } π ∈ Π , we must have that any trajectory τ ∈ T π ( s → s ′ ) is also consistent with at least one π e ∈ Π core . Furthermore, for any such π e , we have

where the second line holds because both π ⇝ τ h : h ′ and π e ⇝ τ h : h ′ . Next, recall from Eq. (17), that

✶ Furthermore, from Eq. (19), recall that the empirical estimate ̂ P π s → s ′ of P π s → s ′ is given by :

✶ where the dataset D s consists of i.i.d. samples, and is collected in lines 10-12 in Algorithm 3 ( DataCollector ), by first running the policy π s for h timesteps and if the trajectory reaches s , then executing π e ∼ Uniform(Π core ) for the remaining time steps (otherwise this trajectory is rejected). Let the law of this process be q . We thus note that,

✶ where ( i ) follows from the sampling strategy in Algorithm 3 after observing s h = s , and ( ii ) simply uses the relation (22) since both π ′ e ⇝ τ h : h ′ and π ⇝ τ h : h ′ hold. Finally, in ( iii ) , we use the relation (23).

We have shown that ̂ P π s → s ′ is an unbiased estimate of P π s → s ′ for any π and s, s ′ ∈ S + π . Thus, using Hoeffding's inequality (Lemma 17), followed by a union bound, we get that with probability at least 1 -δ/ 4 , for all π ∈ Π , s ∈ S rch π , and s ′ ∈ S π ∪ { s ⊥ } ,

where the additional factor of K appears because for any τ ∈ T π ( s → s ′ ) , there must exist some π e ∈ Π core that is also consistent with τ (as we showed above), which implies that each of the terms in Eq. (24) satisfies the bound a.s.:

Since |D s | ≥ εn 2 24 D by assumption, we have

Repeating a similar argument for the empirical reward estimation in Eq. (19), we get that with probability at least 1 -δ/ 4 , for all π ∈ Π , and s ∈ S rch π and s ′ ∈ S π ∪ { s ⊥ } , we have that

Similarly, we can also get for any π ∈ Π and s ′ ∈ S π ∪ { s ⊥ } , with probability at least 1 -δ/ 2 ,

where the last line simply uses the fact that |D s ⊤ | = n 1 . The final statement is due to a union bound on the above results. This concludes the proof of Lemma 10.

Lemma 11. Fix a policy π ∈ Π and a set of reachable states S rch , and consider the policyspecific MRP M π S rch (as defined by Eqs. (17) and (18) ). Then for any s ∈ S rem π , the quantity ¯ d π ( s ; ¬S rem π ) = d M ( s ) , where d M ( s ) is the occupancy of state s in M π S rch .

Proof. We use ¯ τ to denote a trajectory in M π S rch and τ to denote a 'corresponding' (in a sense which will be described shortly) trajectory in the original MDP M . For any s ∈ S rem π , we have

The first equality is simply due to the definition of d M . For the second equality, we sum up over all possible sequences which start at s ⊤ , pass through some (variable length) sequence of states ¯ s 1 , · · · , ¯ s k ∈ S rch π , then reach s and the terminal state s ⊥ . By definition of the policy-specific MRP, we know that once the MRP transits to a state s ∈ S rem π , it must then transit to s ⊥ and repeat s ⊥ until the end of the episode.

Now fix a sequence ¯ s 1 , · · · , ¯ s k ∈ S rch π . We relate the term in the summand to the probability of corresponding trajectories in the original MDP M . To avoid confusion, we let s h 1 , . . . , s h k ∈ S rch π denote the corresponding sequence of states in the original MDP, which are unique and satisfy h 1 < h 2 < · · · < h k . We also denote s h k +1 = s .

Using the definition of M π S rch , we write

Now we sum over all sequences ¯ s 1 , · · · , ¯ s k ∈ S rch π to get

The second equality follows from the definition of S rem π , and the last line is the definition of the ¯ d notation. This concludes the proof of Lemma 11.

Lemma 12. With probability at least 1 -2 δ , any (¯ s, π ) that is added into T (in line 13 of Algorithm 1) satisfies d π (¯ s ) ≥ ε / 12 D .

Proof. For any (¯ s, π ) ∈ T , when we collect D ¯ s in Algorithm 3, the probability that a trajectory will be accepted (i.e. the trajectory satisfies the 'if' statement in line 10) is exactly d π (¯ s ) . Thus, using Hoeffding's inequality (Lemma 17), with probability at least 1 -δ / D | Π | ,

Since |T | ≤ D | Π | , by union bound, the above holds for every (¯ s, π ) ∈ T with probability at least 1 -δ . Let us denote this event as E data . Under E data , for any (¯ s, π ) that satisfies d π (¯ s ) ≥ ε 12 D ,

where the second inequality follows by the bound on d π (¯ s ) and our choice of parameter n 2 in Eq. (21).

In the following, we prove by induction that every (¯ s, π ) that is added into T in the while loop from lines 7-17 in Algorithm 1 satisfies d π (¯ s ) ≥ ε 12 D . This is trivially true at initialization when T = { ( s ⊤ , Null) } , since every trajectory starts at the dummy state s ⊤ , for which we have d Null ( s ⊤ ) = 1 .

We now proceed to the induction hypothesis. Suppose that in some iteration of the while loop, every tuple (¯ s, π ) ∈ T satisfies d π (¯ s ) ≥ ε / 12 D , and that (¯ s ′ , π ′ ) is a new tuple that will be added to T . We will show that (¯ s ′ , π ′ ) will also satisfy d π ′ (¯ s ′ ) ≥ ε / 12 D .

Recall that S + π ′ = S π ′ ∪ { s ⊤ , s ⊥ } , S rch π ′ = S + π ′ ∩ S rch , and S rem π ′ = S + π ′ \ S rch π ′ . Let M π ′ S rch = MRP( S + π ′ , P π ′ , r π ′ , H, s ⊤ , s ⊥ ) be the policy-specific MRP, where P π ′ and r π ′ are defined in Eqs. (17) and (18) respectively for the policy π ′ . Similarly let ̂ M π ′ S rch =

MRP( S + π ′ , ̂ P π ′ , ̂ r π ′ , H, s ⊤ , s ⊥ ) denote the estimated policy-specific MRP, where ̂ P π ′ and r π ′ are defined using (19) and (20) respectively. Note that for any state s ∈ S rch π ′ , the bound in (25) holds.

For the rest of the proof, we assume that the event E est , defined in Lemma 10, holds (this happens with probability at least 1 -δ ). By definition of E est , we have

Furthermore, note that ̂ d π ′ (¯ s ′ ) ← EstReachability ( S + π ′ , ̂ M π ′ S rch , ¯ s ′ ) since in Algorithm 4 we start with V ( s ) = ✶ { s = ¯ s ′ } . Furthermore, using Lemma 9, we have

where the second inequality follows from (26). Additionally, Lemma 11 states that d M (¯ s ′ ) = ¯ d π ′ (¯ s ′ ; ¬S rem π ′ ) . Therefore we obtain

Thus, if the new state-policy pair (¯ s ′ , π ′ ) is added into T , we will have

Furthermore, by definition of ¯ d we have

so we have proved the induction hypothesis d π ′ (¯ s ′ ) ≥ ε / 12 D for the next round. This concludes the proof of Lemma 12.

The next lemma establishes that Algorithm 1 will terminate after finitely many rounds, and that after termination will have explored all sufficiently reachable states.

Lemma 13. With probability at least 1 -2 δ ,

( a ) The while loop in line 5 in Algorithm 1 will terminate after at most 12 HD C (Π) ε rounds.

( b ) After the termination of the while loop, for any π ∈ Π , the remaining states s ∈ S rem π that are not added to S rch satisfy ¯ d π ( s ; ¬S rem π ) ≤ ε / 4 D .

Notice that according to our algorithm, the same state cannot be added multiple times into S rch . Therefore, |S rch | ≤ D | Π | , and the maximum number of rounds of the while loop is D | Π | (i.e., the while loop eventually terminates).

Proof. We prove each part separately.

( a ) First, note that from the definition of coverability and Lemma 1, we have

Furthermore, Lemma 12 states that every ( s, π s ) ∈ T satisfies d π s ( s ) ≥ ε / 12 D . Thus, at any point in Algorithm 1, we have

Since, S rch ⊆ S , the two bounds indicate that

Since every iteration of the while loop adds one new ( s, π s ) to T , the while loop terminates after at most 12 HD C (Π) / ε many rounds.

( b ) We know that once the while loop has terminated, for every π ∈ Π and ¯ s ∈ S rem π , we must have ̂ d π (¯ s ) ≤ ε / 6 D , or else the condition in line 12 in Algorithm 1 is violated. Fix any such (¯ s, π ) pair. Inspecting the proof of Lemma 12, we see that

To conclude, we get

Lemma 14. Suppose that the conclusions of Lemmas 10 and 13 hold. Then for every π ∈ Π , the estimated value ̂ V π computed in Algorithm 1 satisfies

Proof. We will break up the proof into two steps. First, we show that for any π , the value estimate ̂ V π obtained using the empirical policy-specific MRP ̂ M π S rch is close to its value in the policy-specific MRP M π S rch , as defined via (17) and (18). We denote this quantity as V π MRP . Then, we will show that V π MRP is close to V π , the value of the policy π in the original MDP M .

Part 1: ̂ V π is close to V π MRP . Note that the output ̂ V π of Algorithm 5 is exact the value function of MRP ̂ M π S rch defined by Eqs. (19) and (20). When D = 0 , by part (b) of Lemma 10, we obtain

When D ≥ 1 , using Lemma 10, we have

By the simulation lemma (Lemma 9), we get

Part 2 : V π MRP is close to V π . As in the proof of Lemma 11, let us consider different trajectories ¯ τ that are possible in M π S rch . We can represent ¯ τ = ( s ⊤ , ¯ s 1 , · · · , ¯ s k , s ⊥ , · · · ) where the states ¯ s 1 , · · · , ¯ s k are distinct and all except possibly ¯ s k belong to S rch π , and the states after s ⊥ are just repeats of s ⊥ until the end of the episode. Let s h 1 , s h 2 , . . . , s h k be the same sequence (in the original MDP M ) Again, we have

where recall that P M denotes probability under the M π S rch , and P π denotes the probability under trajectories drawn according to π in the underlying MDP; E M and E π are defined similarly.

Furthermore, the expectation of rewards we collected in M π S rch with trajectories ¯ τ is

✶ Next, we sum over all possible trajectories. However, note that the only trajectories that are possible in M whose corresponding trajectories are not accounted for in M π S rch are precisely those that visit states in S π , after visiting some s h k in the remaining states S rem π (since, by construction, the MRP transitions directly to s ⊥ after encountering a state in S rem π ). Thus,

V π MRP = E π [ R [ τ ]( ✶ { τ ∩ S rem π = ∅} + ✶ {∃ k ∈ [ H ] : s h k ∈ S rem π and ∀ h > h k : s h / ∈ S π } )] , where the first term corresponds to trajectories that do not pass through S rem π , and the second term corresponds to trajectories that passes through some state in S rem π but then does not go through any other state in S π . On the other hand,

Clearly, V π MRP ≤ V π . Furthermore, we also have

̸

̸

where the first inequality follows by just ignoring the second indicator term, and the second inequality follows by taking a union bound over all possible values of S rem π as well as the conclusion of Lemma 13.

Putting it all together, we get that

This concludes the proof of Lemma 14.

F.4 Proof of Theorem 4

We assume the events defined in Lemmas 10, 12 and 13 hold (which happens with probability at least 1 -2 δ ). With our choices of n 1 , n 2 in Eq. (21), the total number of samples used in our algorithm is at most

After the termination of the while loop, we know that for any policy π ∈ Π and s ∈ S rem π we have

Therefore, by Lemma 14, we know for every π ∈ Π , | ̂ V π -V π | ≤ ε . Hence the output policy ̂ π ∈ arg max π ̂ V π satisfies

Rescaling ε by 2 ε and δ by 2 δ concludes the proof of Theorem 4.

F.5 Sunflower Property is Insufficient By Itself

We give an example of a policy class Π for which the sunflower property holds for K,D = poly( H ) but C (Π) = 2 H . Therefore, in light of Theorem 2, the sunflower property by itself cannot ensure statistically efficient agnostic PAC RL in the online access model.

The example is as follows: Consider a binary tree MDP with 2 H -1 states and action space A = { 0 , 1 } . The policy class Π will be able to get to every ( s, a ) pair in layer H . To define the policies, we consider each possible trajectory τ = ( s 1 , a 1 , · · · , s H , a H ) and let:

Thus it is clear that C (Π) = 2 H , but the sunflower property holds with K = 1 , D = H by taking Π core = { π 0 } (the policy which always picks a = 0 ).

G Infinite Policy Classes

In this section we discuss the extensions of our results to infinite policy classes.

G.1 Definitions and Preliminary Lemmas

We will state our results in terms of the Natarajan dimension, which is a generalization of the VC dimension used to study multiclass learning. We note that the results in this section could be stated in terms of other complexity measures from multiclass learning such as the graph dimension and DS dimension (see, e.g., Natarajan, 1989; Shalev-Shwartz and Ben-David, 2014; Daniely and ShalevShwartz, 2014; Brukhim et al., 2022a); for simplicity we analyze guarantees in terms of the Natarajan dimension.

̸

Definition 7 (Natarajan Dimension (Natarajan, 1989)) . Let X be an instance space and Y be a finite label space. Given a class H ⊆ Y X , we define its Natarajan dimension , denoted Ndim( H ) , to be the maximum cardinality of a set C ⊆ X that satisfies the following: there exists h 0 , h 1 : C →Y such that (1) for all x ∈ C , h 0 ( x ) = h 1 ( x ) , and (2) for all B ⊆ C , there exists h ∈ H such that for all x ∈ B , h ( x ) = h 0 ( x ) and for all x ∈ C \ B , h ( x ) = h 1 ( x ) .

A notation we will use throughout is the projection operator. For a hypothesis class H ⊆ Y X and a finite set X = ( x 1 , · · · , x n ) ∈ X n , we define the projection of H on to X as

Lemma 15 (Sauer's Lemma for Natarajan Classes (Haussler and Long, 1995)) . Given a hypothesis class H ⊆ Y X with |Y| = K and Ndim( H ) ≤ d , we have for every X = ( x 1 , · · · , x n ) ∈ X n ,

Theorem 6 (Multiclass Fundamental Theorem (Shalev-Shwartz and Ben-David, 2014)) . For any class H ⊆ Y X with Ndim( H ) = d and |Y| = K , the minimax sample complexity of ( ε, δ ) agnostic PAC learning H can be bounded as

Definition 8 (Pseudodimension) . Let X be an instance space. Given a hypothesis class H ⊆ R X , its pseudodimension, denoted Pdim( H ) , is defined as Pdim( H ) := VC( H + ) , where H + := { ( x, θ ) ↦→ { h ( x ) ≤ θ } : h ∈ H} .

✶ Definition 9 (Covering Numbers) . Given a hypothesis class H ⊆ R X , α > 0 , and X = ( x 1 , · · · , x n ) ∈ X n , the covering number N 1 ( H , α, X ) is the minimum cardinality of a set C ⊂ R n such that for any h ∈ H there exists a c ∈ C such that 1 n ∑ n i =1 | h ( x i ) -c i | ≤ α .

Lemma 16 (Jiang et al. (2017), see also Pollard (2012); Luc et al. (1996)) . Let H ⊂ [0 , 1] X be a realvalued hypothesis class, and let X = ( x 1 , · · · , x n ) be i.i.d. samples drawn from some distribution D on X . Then for any α > 0

Furthermore if Pdim( H ) ≤ d then we have the bound

which is at most δ as long as n ≥ 128 α 2 ( d log 16 e α +log(8 e ( d +1)) + log 1 δ ) .

G.2 Generative Model Lower Bound

First we address the lower bound. Observe that it is possible to achieve a lower bound that depends on Ndim(Π) with the following construction. First, identify the layer h ∈ [ H ] such that the witnessing

set C contains the maximal number of states in S h ; by pigeonhole principle there must be at least Ndim(Π) /H such states in layer h . Then, we construct an MDP which 'embeds' a hard multiclass learning problem at layer h over these states. A lower bound of Ω ( Ndim(Π) Hε 2 · log 1 δ ) follows from Theorem 6.

By combining Theorem 2 with the above we get the following corollary.

Corollary 3 (Lower Bound for Generative Model with Infinite Policy Classes) . For any policy class Π , the minimax sample complexity ( ε, δ ) -PAC learning Π is at least

Again, since the generative model setting is easier than online RL, this lower bound also extends to the online RL setting.

Our bound is additive in C (Π) and Ndim(Π) ; we do not know if it is possible to strengthen this to be a product of the two factors, as we will achieve in the upper bound in the next section.

G.3 Generative Model Upper Bound

For the upper bounds, we can replace the dependence on log | Π | with Ndim(Π) (and additional log factors). In particular, we can modify the analysis of the TrajectoryTree to account for infinite policy classes. Recall that our analysis of TrajectoryTree required us to prove a uniform convergence guarantee for the estimate ̂ V π : with probability at least 1 -δ , for all π ∈ Π , we have | ̂ V π -V π | ≲ ε . We previously used a union bound over | Π | , which gave us the log | Π | dependence. Now we sketch an argument to replace it with Ndim(Π) .

Let T be the set of all possible trajectory trees. We introduce the notation v π : T → R to denote the function that takes as input a trajectory tree ̂ T (for example, as sampled by TrajectoryTree ) and returns the value of running π on it. Then we can rewrite the desired uniform convergence guarantee:

In light of Lemma 16, we will compute the pseudodimension for the function class V Π = { v π : π ∈ Π } . Define the subgraph class

✶ By definition, Pdim( V Π ) = VC( V Π , + ) . Fix any X = { ( ̂ T 1 , θ 1 ) , · · · , ( ̂ T d , θ d ) } ∈ ( T × R ) d . In order to show that VC( V Π , + ) ≤ d for some value of d it suffices to prove that |V Π , + ∣ ∣ X | < 2 d .

For any index t ∈ [ d ] , we also denote π ( ⃗ s i ) ∈ A ≤ H C (Π) to be the vector of actions selected by π on all Π -reachable states in ̂ T i (of which there are at most H · C (Π) ). We claim that

This is true because once the d trajectory trees are fixed, for every π ∈ Π , the value of the vector V { π } , + ∣ ∣ X ∈ { 0 , 1 } d is determined by the trajectory that π takes in every trajectory tree. This in turn is determined by the assignment of actions to every reachable state in all the d trajectory trees, of which there are at most C (Π) · H · d of. Therefore, we can upper bound the size of V Π , + ∣ ∣ X by the number of ways any π ∈ Π assign actions to every state in ̂ T 1 , · · · , ̂ T d .

Applying Lemma 15 to Eq. (29), we get that

For the choice of d = ˜ O (Ndim(Π)) , the previous display is at most 2 d , thus proving the bound on Pdim( V Π ) . Lastly, the bound can be plugged back into Lemma 16 to get a bound on the error of TrajectoryTree : the statement in Eq. (28) holds using

This in turn yields a guarantee on ̂ π returned by TrajectoryTree .

G.4 Online RL Upper Bound

The modified analysis for the online RL upper bound (Theorem 4) proceeds similarly; we sketch the ideas below.

There are two places in the proof of Theorem 4 which require a union bound over | Π | : the event E est (defined by Lemma 10) that the estimated transitions and rewards of the MRPs are close to their population versions, and the event E data (defined by Lemma 12) that the datasets collected are large enough. The latter is easy to address, since we can simply modify the algorithm's while loop to break after O ( HD C (Π) ε ) iterations and union bound over the size of the set |T | instead of the worst-case bound on the size D | Π | . For E data , we follow a similar strategy as the analysis for the generative model upper bound.

Fix a state s . Recall that the estimate for the probability transition kernel in the MDP in Eq. (19) takes the form

We set up some notation. Define the function p s → s ′ : ( S × A × R ) → [0 , | Π core | ] as

π H

✶ with the implicit restriction of the domain to trajectories τ for which the denominator is nonzero. We have E [ p π s → s ′ ( τ )] = P π s → s ′ . Also let Π s = { π ∈ Π : s ∈ S π } .

Restated in this notation, our objective is to show the uniform convergence guarantee

Again, in light of Lemma 16, we need to compute the pseudodimension for the function class P Π s = { p π s → s ′ : π ∈ Π s , s ′ ∈ S π } , since these are all possible transitions that we might use the dataset D s to evaluate. Define the subgraph class

P Π s , + := { ( τ, θ ) ↦→ ✶ { p π s → s ′ ( τ ) ≤ θ } : π ∈ Π s , s ′ ∈ S π } . Fix the set X = { ( τ 1 , θ 1 ) , · · · , ( τ d , θ d ) } ∈ (( S × A × R ) H × R ) d , where the trajectories τ 1 , · · · , τ d pass through s . We also denote S X to be the union of all states which appear in τ 1 , · · · , τ d . In order to show a bound that Pdim( P Π s ) ≤ d it suffices to prove that |P Π s , + ∣ ∣ X | < 2 d .

We first observe that

( ✶ { p π s → s ′ ( τ 1 ) ≤ θ 1 } , · · · , ✶ { p π s → s ′ ( τ d ) ≤ θ d } ) = ⃗ 0 , no matter what π is, contributing at most 1 to the count. Furthermore, once we have fixed s ′ and the { τ 1 , · · · , τ d } the quantities 1 | Π core | ∑ π ′ ∈ Π core ✶ { π e ⇝ τ i,h : h ′ } for every i ∈ [ d ] are constant (do not depend on π ), so we can reparameterize θ ′ i := θ i · 1 | Π core | ∑ π ′ ∈ Π core { π e ⇝ τ i,h : h ′ } to get:

where b i ( π ) := ✶ { ✶ { π ⇝ τ i,h : h ′ } ✶ { τ i ∈ T π ( s → s ′ ) } ≤ θ ′ i } . Now we count how many values the vector ( b 1 ( π ) , · · · , b d ( π )) can take for different π ∈ Π s . Without loss of generality, we can (1) assume that the θ ′ i = 0 (since a product of indicators can only take values in { 0 , 1 } , and if θ ′ ≥ 1 then we must have b i ( π ) = 1 for every π ), and (2) s ′ ∈ τ i for each i ∈ [ d ] (otherwise b i ( π ) = 0 for every π ∈ Π ). So we can rewrite b i ( π ) = ✶ { π ⇝ τ i,h : h ′ } ✶ { τ i ∈ T π ( s → s ′ ) } . For every fixed choice of s ′ we upper bound the size of the set as:

The inequality ( i ) follows by upper bounding by the Cartesian product. The inequality ( ii ) follows because (1) for the first term, the vector ( ✶ { π ⇝ τ 1 ,h : h ′ } , · · · , ✶ { π ⇝ τ d,h : h ′ } ) is determined by the number of possible behaviors π has over all dH states in the trajectories, and (2) for the second term, the vector ( ✶ { τ 1 ∈ T π ( s → s ′ ) } , · · · , ✶ { τ d ∈ T π ( s → s ′ ) } ) is determined by which of the dH states lie in the petal set for S π . The inequality ( iii ) follows by applying Lemma 15 to the first term and Sauer's Lemma to the second term, further noting that every petal S π set has cardinality at most D .

Combining Eqs. (32) and (33) we get the final bound that

To conclude the calculation, we observe that this bound is < 2 d whenever d = ˜ O ( D +Ndim(Π)) , which we can again use in conjunction with Lemma 16 to prove the desired uniform convergence statement found in Eq. (31). Ultimately this allows us to replace the log | Π | with ˜ O ( D +Ndim(Π)) in the upper bound of Theorem 4; the precise details are omitted.

H Connections to Other Complexity Measures

We show relationships between the spanning capacity and several other combinatorial measures of complexity.

For every h ∈ [ H ] we denote the state space at layer h as S h := { s ( j,h ) : j ∈ [ K ] } for some K ∈ N . We will restrict ourselves to binary action spaces A = { 0 , 1 } , but the definitions and results can be extended to larger (but finite) action spaces. In addition, we will henceforth assume that all policy classes Π under consideration satisfy the following stationarity assumption.

Assumption 1. The policy class Π satisfies stationarity : for every π ∈ Π we have

For any π ∈ Π and j ∈ [ K ] , we use π ( j ) as a shorthand to denote the value of π ( s ( j,h ) ) for every h .

The stationarity assumption is not required but is useful for simplifying the definitions and results.

H.1 Definitions and Relationships

First, we state several complexity measures in learning theory. For further discussion on these quantities, see (Foster et al., 2021c; Li et al., 2022).

Definition 10 (Combinatorial Eluder Dimension) . Fix any stationary base policy ¯ π . The combinatorial eluder dimension of Π w.r.t. ¯ π , denoted dim E (Π; ¯ π ) , is the length of the longest sequence ( j 1 , π 1 ) , . . . , ( j N , π N ) such that for every ℓ ∈ [ N ] :

̸

We define the combinatorial eluder dimension of Π as dim E (Π) := sup π ∈ Π dim E (Π; ¯ π ) . 8

Definition 11 (Star Number (Hanneke and Yang, 2015)) . Fix any stationary base policy ¯ π . The star number of Π w.r.t. ¯ π , denoted dim S (Π; ¯ π ) , is the length of the longest sequence ( j 1 , π 1 ) , . . . , ( j N , π N ) such that for every ℓ ∈ [ N ] :

̸

̸

We define the star number of Π as dim S (Π) := sup π ∈ Π dim S (Π; ¯ π ) .

Definition 12 (Threshold Dimension (Alon et al., 2019; Li et al., 2022)) . Fix any stationary base policy ¯ π . The threshold dimension of Π w.r.t. ¯ π , denoted dim T (Π; ¯ π ) , is the length of the longest sequence ( j 1 , π 1 ) , . . . , ( j N , π N ) such that for every ℓ ∈ [ N ] :

̸

We define the threshold dimension of Π as dim T (Π) := sup π ∈ Π dim T (Π; ¯ π ) .

Relationships Between Complexity Measures. From (Li et al., 2022, Theorem 8) we have the relationship for every Π :

The lower bound is obvious from the definitions and in general cannot be improved; the upper bound also cannot be improved beyond constant factors in the exponent (Li et al., 2022).

We also remark that it is clear from the definitions that VC dimension is a lower bound on all three (eluder, star, threshold); however, VC(Π) can be arbitrarily smaller.

H.2 Bounds on Spanning Capacity

Now we investigate bounds on the spanning capacity in terms of the aforementioned quantities.

Theorem 7. For any policy class Π satisfying Assumption 1 we have

We give several remarks on Theorem 7. The proof is deferred to the following subsection.

It is interesting to understand to what degree we can improve the bounds in Theorem 7. On the lower bound side, we note that the each of the terms individually cannot be sharpened:

For the singleton class Π sing we have C (Π sing ) = min { K,H +1 } and dim S (Π sing ) = K .

For the threshold class Π thres := { π i ( j ) ↦→ ✶ { j ≥ i } : i ∈ [ K ] } , when K is a power of two, it can be shown that C (Π thres ) = min { K, 2 H } and dim T (Π thres ) = K .

While we also provide an upper bound in terms of dim E (Π) , we note that there can be a huge gap between the lower bound and the upper bound. In fact, our upper bound is likely very loose since we are not aware of any policy class for which the upper bound is non-vacuous, i.e. 2 dim E (Π) ≪ min { 2 H , | Π | , 2 KH } (implying that our bound improves on Proposition 3). It would be interesting to understand how to improve the upper bound (possibly, to scale polynomially with dim E (Π) , or more directly in terms of some function of dim S (Π) and dim T (Π) ); we leave this as a direction for future research.

Lastly, we remark that the lower bound of C (Π) ≥ min { Ω(dim T (Π)) , 2 H } is a generalization of previous bounds which show that linear policies cannot be learned with poly( H ) sample complexity (e.g., Du et al., 2019b), since linear policies (even in 2 dimensions) have infinite threshold dimension.

H.2.1 Proof of Theorem 7

We will prove each bound separately.

Star Number Lower Bound. Let ¯ π ∈ Π and the sequence ( j 1 , π 1 ) , . . . , ( j N , π N ) witness dim S (Π) = N . We construct a deterministic MDP M for which the cumulative reachability at layer h max := min { N,H } (Definition 1) is at least min { N,H +1 } . The transition dynamics of M are as follows; we will only specify the transitions until h max -1 (afterwards, the transitions can be arbitrary).

The starting state of M at layer h = 1 is s ( j 1 , 1) .

(On-Chain Transitions): For every h < h max ,

̸

✶ · (Off-Chain Transitions): For every h < h max , state index ˜ j = j h , and action a ∈ A ,

✶ We now compute the cumulative reachability at layer h max . If N ≤ H , the the number of ( s, a ) pairs that Π can reach in M is N (namely the pairs ( s ( j 1 ,N ) , 1) , · · · , ( s ( j N ,N ) , 1) ). On the other hand, if N > H , then the number of ( s, a ) pairs that Π can reach in M is H +1 (namely the pairs ( s ( j 1 ,H ) , 1) , · · · , ( s ( j H ,H ) , 1) , ( s ( j H ,H ) , 0) ). Thus we have shown that C (Π) ≥ min { N,H +1 } .

Threshold Dimension Lower Bound. Let ¯ π ∈ Π and the sequence ( j 1 , π 1 ) , . . . , ( j N , π N ) witness dim T (Π) = N . We define a deterministic MDP M as follows. Set h max = min {⌊ log 2 N ⌋ , H } . Up until layer h max , the MDP will be a full binary tree of depth h max ; afterward, the transitions will be arbitrary. It remains to assign state labels to the nodes of the binary tree (of which there are 2 h max -1 ≤ N ). We claim that it is possible to do so in a way so that every policy π ℓ for ℓ ∈ [2 h max ] reaches a different state-action pair at layer h max . Therefore the cumulative reachability of Π on M is at least 2 h max = min { 2 ⌊ log 2 N ⌋ , 2 H } as claimed.

It remains to prove the claim. The states of M are labeled j 2 , · · · , j 2 h max according to the order they are traversed using inorder traversal of a full binary tree of depth h max (Cormen et al., 2022). One can view the MDP M as a binary search tree where the action 0 corresponds to going left and the action 1 corresponds to going right. Furthermore, if we imagine that the leaves of the binary search tree at depth h max are labeled from left to right with the values 1 . 5 , 2 . 5 , · · · , 2 h max +0 . 5 , then it is clear that for any ℓ ∈ [2 h max ] , the trajectory generated by running π ℓ on M is exactly the path obtained by searching for the value ℓ +0 . 5 in the binary search tree. Thus we have shown that the cumulative reachability of Π on M is the number of leaves at depth h max , thus proving the claim.

Eluder Dimension Upper Bound. Let dim E (Π) = N . We only need to prove this statement when N ≤ H , as otherwise the statement already follows from Proposition 3. Let ( M ⋆ , h ⋆ ) be the MDP and layer which witness C (Π) . Also denote s 1 to be the starting state of M ⋆ . For any state s , we denote child 0 ( s ) and child 1 ( s ) to be the states in the next layer which are reachable by taking a = 0 and a = 1 respectively.

For any reachable state s at layer h in the MDP M ⋆ we define the function f ( s ) as follows. For any state s at layer h ⋆ , we set f ( s ) := 1 if the state-action pairs ( s, 0) and ( s, 1) are both reachable by Π ; otherwise we set f ( s ) := 0 . For states in layers h < h ⋆ we set

We claim that for any state s , the contribution to C (Π) by policies that pass through s is at most 2 f ( s ) . We prove this by induction. Clearly, the base case of f ( s ) = 0 or f ( s ) = 1 holds. If only one child of s is reachable by Π then the contribution to C (Π) by policies that pass through s equal to the contribution to C (Π) by policies that pass through the child of s . If both children of s are reachable by Π then the contribution towards C (Π) by policies that pass through s is upper bounded by the

̸

sum of the contribution towards C (Π) by policies that pass through the two children, i.e. it is at most 2 f (child 0 ( s )) +2 f (child 1 ( s )) ≤ 2 f ( s ) . This concludes the inductive argument.

̸

Now we bound f ( s 1 ) . Observe that the quantity f ( s 1 ) counts the maximum number of layers h 1 , h 2 , · · · , h L that satisfy the following property: there exists a trajectory τ = ( s 1 , a 1 , · · · , s H , a H ) for which we can find L policies π 1 , π 2 , · · · , π L so that each policy π ℓ when run on M ⋆ (a) reaches s h ℓ , and (b) takes action π ℓ ( s h ℓ ) = a h ℓ . Thus, by definition of the eluder dimension we have f ( s 1 ) ≤ N . Therefore, we have shown that the cumulative reachability of Π in M ⋆ is at most 2 N .

I Extension: Can Expert Feedback Help in Agnostic PAC RL?

For several policy classes, the spanning capacity may be quite large, and our lower bounds (Theorem 2 and 3) demonstrate an unavoidable dependence on C (Π) . In this section, we investigate whether it is possible to achieve bounds which are independent of C (Π) and instead only depend on poly( A,H, log | Π | ) under a stronger feedback model.

Our motivation comes from practice. It is usually uncommon to learn from scratch: often we would like to utilize domain expertise or prior knowledge to learn with fewer samples. For example, during training one might have access to a simulator which can roll out trajectories to estimate the optimal value function Q ⋆ , or one might have access to expert advice / demonstrations. However, this access does not come for free; estimating value functions with a simulator requires some computation, or the 'expert' might be a human who is providing labels or feedback on the performance of the algorithm. Motivated by this, we consider additional feedback in the form of an expert oracle.

Definition 13 (Expert oracle) . An expert oracle O exp : S × A → R is a function which given an ( s, a ) pair as input returns the Q value of some expert policy π ◦ , denoted Q π ◦ ( s, a ) .

Definition 13 is a natural formulation for understanding how expert feedback can be used for agnostic RL in large state spaces. We do not require π ◦ to be the optimal policy (either over the given policy class Π or over all A S ). The objective is to compete with π ◦ , i.e., with probability at least 1 -δ , return a policy ̂ π such that V ̂ π ≥ V π ◦ -ε using few online interactions with the MDP and calls to O exp .

Asample efficient algorithm (one which uses at most poly( A,H, log | Π | , ε -1 , δ -1 ) online trajectories and calls to the oracle) must use both forms of access. Our lower bounds (Theorem 2 and 3) show that an algorithm which only uses online access to the MDP must use Ω( C (Π)) samples. Likewise, an algorithm which only queries the expert oracle must use Ω( SA ) queries because it does not know the dynamics of the MDP, so the best it can do is just learn the optimal action on every state.

Relationship to Prior Works. The oracle O exp is closely related to several previously considered settings (Golowich and Moitra, 2022; Gupta et al., 2022; Amortila et al., 2022). Prior work (Golowich and Moitra, 2022; Gupta et al., 2022) has studied tabular RL with inexact predictions for either the optimal Q ⋆ or V ⋆ . They assume access to the entire table of values; since we study the agnostic RL setting with a large state space, we formalize access to predictions via the expert oracle. Amortila et al. (2022) study a related expert action oracle under the assumption of linear value functions. They show that in a generative model, with poly( d ) resets and queries to an expert action oracle, one can learn an ε -optimal policy, thus circumventing known hardness results for the linear value function setting. Up to a factor of A , one can simulate queries to the expert action oracle by querying O exp ( s, a ) for each a ∈ A .

I.1 Upper Bound under Realizability

Under realizability (namely, π ◦ ∈ Π ), it is known that the dependence on C (Π) can be entirely removed with few queries to the expert oracle.

Theorem 8. For any Π such that π ◦ ∈ Π , with probability at least 1 -δ , the AggreVaTe algorithm (Ross and Bagnell, 2014) computes an ε -optimal policy using

The proof is omitted; it can be found in (Ross and Bagnell, 2014; Agarwal et al., 2019). We also note that actually we require a slightly weaker oracle than O exp : the AggreVaTe algorithm only queries the value of Q π ◦ on ( s, a ) pairs which are encountered in online trajectories.

I.2 Lower Bound in Agnostic Setting

Realizability of the expert policy used for O exp is a rather strong assumption in practice. For example, one might choose to parameterize Π as a class of neural networks, but one would like to use human annotators to give expert feedback on the actions taken by the learner; here, it is unreasonable to assume that realizability of the expert policy holds.

We sketch a lower bound in Theorem 9 that shows that without realizability ( π ◦ / ∈ Π ), we can do no better than Ω( C (Π)) queries to a generative model or queries to O exp .

Theorem 9 (informal) . For any H ∈ N , C ∈ [2 H ] , there exists a policy class Π with C (Π) = | Π | = C , expert policy π ◦ / ∈ Π , and family of MDPs M with state space S of size O (2 H ) , binary action space, and horizon H such that any algorithm that returns a 1 / 4 -optimal policy must either use Ω( C ) queries to a generative model or Ω( C ) queries to the O exp .

Before sketching the proof, several remarks are in order.

By comparing with Theorem 8, Theorem 9 demonstrates that realizability of the expert policy is crucial for circumventing the dependence on spanning capacity via the expert oracle.

̸

In the lower bound construction of Theorem 9, π ◦ is the optimal policy. Furthermore, while π ◦ / ∈ Π , the lower bound still has the property that V π ◦ = V ⋆ = max π ∈ Π V π ; that is, the best-in-class policy ˜ π := arg max π ∈ Π V π attains the same value as the optimal policy. This is possible because there exist multiple states for which ˜ π ( s ) = π ◦ ( s ) , however these states have d ˜ π ( s ) = 0 . Thus, we also rule out guarantees of the form V ̂ π ≥ max π ∈ Π V π -ε .

Since the oracle O exp is stronger than the expert action oracle (Amortila et al., 2022) (up to a factor of A ), the lower bound extends to this weaker feedback model. Investigating further assumptions that enable statistically tractable agnostic learning with expert feedback is an interesting direction for future work.

Proof Sketch of Theorem 9. We present the construction as well as intuition for the lower bound, leaving out a formal information-theoretic proof.

Construction of MDP Family. We describe the family of MDPs M . In every layer, the state space is S h = { s ( j,h ) : j ∈ [2 h ] } , except at S H where we have an additional terminating state, S H = { s ( j,h ) : j ∈ [2 H ] } ∪ { s ⊥ } . The action space is A = { 0 , 1 } .

The MDP family M = { M b,f ⋆ } b ∈A H -1 ,f ⋆ ∈A S H is parameterized by a bit sequence b ∈ A H -1 as well as a labeling function f ⋆ ∈ A S H . The size of M is 2 H -1 · 2 2 H . We now describe the transitions and rewards for any M b,f ⋆ . In the following, let s b ∈ S H -1 be the state that is reached by playing the sequence of actions ( b [1] , b [2] , · · · , b [ H -2]) for the first H -2 layers.

Transitions. For the first H -2 layers, the transitions are the same for M b,f ⋆ ∈ M . At layer H -1 , the transition depends on b .

-For any h ∈ { 1 , 2 , . . . , H -2 } , the transitions are deterministic and given by a tree process: namely

✶ -At layer H -1 , for the state s b , the transition is P ( s ′ | s b , a ) = ✶ { s ′ = s ⊥ } for any a ∈ A . For all other states, the transitions are uniform to S H , i.e., for any s ∈ S H -1 \{ s b } , a ∈ A , the transition is P ( · | s, a ) = Uniform( S H \{ s ⊥ } ) .

Rewards. The rewards depend on the b ∈ A H -1 and f ⋆ ∈ A S H .

-The reward at layer H -1 is R ( s, a ) = ✶ { s = s b , a = b [ H -1] } . -The reward at layer H is

̸

From the description of the transitions and rewards, we can compute the value of Q ⋆ ( · , · ) .

Layers 1 , · · · , H -2 : For any s ∈ S 1 ∪ S 2 ∪ · · · ∪ S H -2 and a ∈ A , the Q -value is Q ⋆ ( s, a ) = 1 .

Layer H -1 : At s b , the Q -value is Q ⋆ ( s b , a ) = ✶ { a = b [ H -1] } . For other states s ∈ S H -1 \{ s b } , the Q -value is Q ⋆ ( s, a ) = 1 for any a ∈ A .

Lastly, the optimal value is V ⋆ = 1 .

Layer H : At s ⊥ , the Q -value is Q ⋆ ( s ⊥ , a ) = 0 for any a ∈ A . For other states s ∈ S H \{ s ⊥ } , the Q -value is Q ⋆ ( s, a ) = ✶ { a = f ⋆ ( s ) } .

Expert Oracle. The oracle O exp returns the value of Q ⋆ ( s, a ) .

Policy Class. The policy class Π is parameterized by bit sequences of length H -1 . Denote the function bin : { 0 , 1 , . . . , 2 H -1 } ↦→ A H -1 that returns the binary representation of the input. Specifically,

where each π b is defined such that π b ( s ) := b [ h ] if s ∈ S h , and π b ( s ) := 0 otherwise. By construction it is clear that C (Π) = | Π | = C .

Lower Bound Argument. Consider any M b,f ⋆ where b ∈ { bin( i ) : i ∈ { 0 , 1 , . . . , C -1 }} and f ⋆ ∈ A S H . There are two ways for the learner to identify a 1 / 4 -optimal policy in M b,f ⋆ :

Find the value of b , and return the policy π b , which has V π b = 1 .

Estimate ̂ f ≈ f ⋆ , and return the policy π ̂ f which picks arbitrary actions for any s ∈ S 1 ∪ S 2 ∪ · · · ∪ S H -1 and picks π ̂ f ( s ) = ̂ f ( s ) on s ∈ S H .

We claim that in any case, the learner must either use many samples from a generative model or many calls to O exp . First, observe that since the transitions and rewards at layers 1 , · · · , H -2 are known and identical for all M b,f ⋆ ∈ M , querying the generative model on these states does not provide the learner with any information. Furthermore, in layers 1 , · · · , H -2 , every ( s, a ) pair has Q ⋆ ( s, a ) = 1 , so querying O exp on these ( s, a ) pairs also does not provide any information to the learner. Thus, we consider learners which query the generative model or the expert oracle at states in layers H -1 and H .

In order to identify b , the learner must identify which ( s, a ) pair at layer H -1 achieves reward of 1. They can do this either by (1) querying the generative model at a particular ( s, a ) pair and observing if r ( s, a ) = 1 (or if the transition goes to s ⊥ ); or (2) querying O exp at a particular ( s, a ) pair and observing if Q ⋆ ( s, a ) = 0 (which informs the learner that s b = s and b [ H -1] = 1 -a ). In either case, the learner must expend Ω( C ) queries in total in order to identify b .

To learn f ⋆ , the learner must solve a supervised learning problem over S H \ s ⊥ . They can learn the identity of f ⋆ ( s ) by querying either the generative model or the expert oracle on S H . Due to classical supervised learning lower bounds, learning f ⋆ requires Ω ( VC( A S H ) ) = Ω(2 H ) queries.

J Technical Tools

Lemma 17 (Hoeffding's Inequality) . Let Z 1 , · · · , Z n be independent bounded random variables with Z i ∈ [ a, b ] for all i ∈ [ n ] . Then

Lemma 18 (Multiplicative Chernoff Bound) . Let Z 1 , · · · , Z n be i.i.d. random variables taking values in { 0 , 1 } with expectation µ . Then for any δ > 0 ,

Furthermore for any δ ∈ (0 , 1) ,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of sample complexity in finite horizon MDPs from an agnostic perspective. More specifically, the term agnostic refers to the following setting: given a (finite) policy class $\\Pi$, they aim at understanding the number of samples that are required to output a policy that is $\\epsilon$-optimal w.r.t. the best policy within $\\Pi$ is. \n\nTo study the problem, they introduce the concept of *spanning capacity*, a novel complexity metric that depends solely on $\\Pi$ and is independent of the MDP dynamics. \n\nThen, the authors study two interaction settings (i.e., generative model and online setting) and derive the following results:\n- Generative model setting. The authors show that the spanning capacity describes the learning complexity up to an $H \\log(|\\Pi|)$ factor. This is proven by deriving lower and upper bounds that are explicitly dependent on the spanning capacity.\n- Online setting. In this case, the results can be summarized as follows:\n    - The spanning capacity fails at describing the complexity of online RL. Indeed, the authors present a lower bound that shows that a superpolynomial-in-the-horizon number of samples are still needed for agnostic learning (i.e., it cannot be lower-bounded by any polynomial function of the spanning capacity). \n    - By restricting the set $\\Pi$ to \"sunflower\" policies, provably efficient learning is possible. In this sense, the authors propose POPLER, an algorithm whose complexity scales with the spanning capacity and characteristic that depends on the structure of $\\Pi$.\n\nSTRENGTHS:\nProvably efficient learning is a significant and open problem for RL agents, with a longstanding history of different methods and approaches. The authors thoroughly review existing works (Appendix A) and properly contextualize their results within the field. \nIn this sense, novel ideas, such as the ones proposed by the authors, are of interest to the NeurIPS community that is interested in RL theory. \n\nMore specifically, the notion of spanning capacity is, to the best of my knowledge, original and of potential interest. \nIndeed:\n1) As the authors show in Proposition 3, this metric recovers, in worst-case scenarios, existing upper and lower bounds available in the literature. Furthermore, in more favorable settings, it is significantly smaller. \n2) Furthermore, as the author shows in Theorem 1 and 2, it describes the min-max sample complexity (up to H*log(\\Pi) factors) in the generative model setting. Although, due to this mismatch, the problem is still open, the progress remains significant. \n\nIn the online interaction setting, this metric turns out to be not descriptive enough (i.e., Theorem 5, a superpolynomial-in-the-horizon number of samples are still needed for agnostic learning). Although, in some sense, this is a drawback (see weaknesses below), this negative result can potentially provide insights to guide future research in metrics that properly describe provably efficient agnostic RL in online settings. \n\nFinally, (minor) by carefully restricting the set of policies, the authors are able to show that the spanning capacity can be used to upper-bound the complexity in the online setting as well. This adds an additional (but minor, see below why) point in favor of the spanning capacity. \n\nOn the clarity. Overall, the paper is a nice read, and it is relatively easy to follow (except a minor suggestion, see below).\n\nWEAKNESSES:\n**Online RL**\nAs commented above, the authors show that the spanning capacity turns out to be not descriptive enough for providing polynomial bounds on the sample complexity of agnostic RL in an online interaction setting. Nevertheless, the metric is in itself something that the authors propose. For this reason, one might object that this is a potential drawback of the metric that is *designed* by the authors. \nA main question, in this sense, remains open: Is there any metric (e.g., a generalization of the spanning capacity) that characterizes both scenarios? \n\n\n**On sunflower policies: online RL under restrictive assumptions on $\\Pi$**\nCurrently, this point seems the major weakness of the presented work. Indeed, I find their definition quite involved, and it is unclear if it is the result of an artifact of the analysis rather than something necessary to reach polynomial sample complexity in the online setting. The fact that a lower bound is missing reinforces this doubt. I invite the authors to comment on this point. \n\n\n**On the clarity**. I would suggest the authors to improve the clarity on line 206-213, maybe by introducing some figures. Indeed, I still miss this natural interpretation of the spanning capacity. \n\n\n**Minors**:\n- The term \"surprising\" to refer to the mismatch between online RL and the generative model setting might be mitigated. Indeed, the generative model setting is intuitively easier.\n- It is remarked multiple times that other works make stronger assumptions that are rarely satisfied in practice (e.g., realizability). Nevertheless, stronger theoretical results (\\epsilon-optimality) are usually achieved under these assumptions. This should be stated more clearly in my opinion. \n- line 176 typo: \"loogmk\"\n- The definition of the set of MDPs seems improper. To the best of my understanding, state, and action spaces are fixed over this class. In this sense, e.g., M^sto is not the set of all MDPs with horizon H, but the set of all MDPs with horizon H and state-action spaces given by (S,A)\n\nQUESTIONS:\n1) I invite the authors to discuss the relationship between their work and problem-dependent learning complexity analysis (e.g., \"Optimistic PAC RL: the instance-dependent view\", Tirinzoni et al., ALT 2023). \n\n2) See the point on sunflower policies in the weakness section.\n\nLIMITATIONS:\nThe authors discuss the limitations of their work as open questions in the conclusion section.\n\nPotential negative societal impact: the paper deals with foundational research on the sample complexity of RL. I don't see a direct path to negative applications.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the minimax sample complexity of learning the best policy within a given policy class $\\Pi$, i.e., the best sample complexity of learning $\\argmax_{\\pi\\in \\Pi}V^\\pi$ in the worst-case MDPs. As a motivation, Proposition 1 shows that without future assumptions on the structure of $\\Pi$, the best policy is not PAC-learnable in the worst case MDPs. Hence, this paper proposes a new complexity measure, spanning capacity, that counts maximum number of reachable states by policies in $\\Pi$ for all deterministic MDP. With generative models, this paper shows that the spanning capacity characterizes the minimax sample complexity of learning $\\Pi$. For online RL, this paper requires an additional structural assumption called sunflower property and proves corresponding upper bounds.\n\nSTRENGTHS:\nThis paper studies a novel aspect of PAC learning in MDPs (called agnostic reinforcement learning) --- learning the best policy within a policy class with no modeling assumptions for the dynamics or value function class. The agnostic reinforcement learning problem could be a new approach toward RL with function approximations since the realizability assumptions on the complicated dynamics/value function class are no longer needed, and it is indeed unclear whether those assumptions hold for practical environments.\n\nThis paper proves novel upper bounds agnostic reinforcement learning setting by introducing a new complexity measure called the spanning capacity. With generative models, this paper proves that the spanning capacity is necessary and sufficient for agonistic reinforcement learning. This is morally a strong result albeit the simplicity of its proof. This paper also proves results for the classic online RL setting with additional structural assumptions on the underlying MDP. \n\nThis paper is well-written --- exposition of the results is clear with the help of concrete examples. The results and assumptions in this paper are well motivated by corresponding lower bounds (Proposition 1 and Theorem 2). \n\nThis paper mainly focuses on the minimax complexity of agonistic RL. While this perspective eliminates the dependence on the realizability assumptions on the underlying MDP, it is possible that the resulting complexity is over-pessimistic. It would be interesting to have an instance-dependent complexity measure that depends on some property of the underlying MDP.\n\nWEAKNESSES:\nThe sunflower structure in Section 6 lacks necessary motivations and seems a bit out of nowhere. I understand that this assumption helps extend the IS technique to the online RL setting. However, it’s unclear to me whether this assumption is necessary/nature for agonistic RL. Intuitively, why/when should we expect the sunflower structure to hold?\n\n[Minor] The computation complexity of the algorithm is linear in the size of the policy class. Is it possible to have some efficient implementation with the help of some standard computation oracles?\n\nQUESTIONS:\n-\tAlg. 1 takes as input the set $S_\\pi$ and $\\Pi_{core}$ defined by Def. 3. They depend on the underlying MDP (because the partial trajectory is generated by the MDP, if I understand correctly). Is the algorithm assumed to have the knowledge of $S_\\pi$ and $\\Pi_{core}$? If not, how should the algorithm compute $S_\\pi$ and $\\Pi_{core}$?\n\n-\tIn Theorem 3, the spanning capacity is $H^{O(\\ell)}$, while the sample complexity lower bound is $\\epsilon^{-\\ell}$ with $\\epsilon=H^{O(1)}$, which means that the sample complexity lower bound is polynomial in the spanning capacity. Then why is the spanning capacity not sufficient to characterize the minimax sample complexity?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies conditions on which agnostic reinforcement learning is statistically tractable. The paper introduces a new concept of complexity measure called spanning capacity, which sorely depends on the policy class. The authors studies in what cases the sample complexity of agnostic RL can be polynomial to spanning capacity.\nThe contributions include 1) for generative model, the authors show spanning capacity is a necessary and sufficient complexity measure for agnostic RL, 2) for online model, the authors show spanning capacity is insufficient: they prove a lower bound superpolynomial to the spanning capacity of policy class. 3) the authors propose a strong assumption of the policy class called sunflower structure, and propose an algorithm that is statistically efficient to spanning capacity under the sunflower structure assumption.\n\nSTRENGTHS:\n- Clarify: the paper is very well-written. I appreciate the intuitions and examples given in the main text. In addition, the proofs in the appendix are well organized and easy to follow.\n- Novelty and Significance: While prior work has studied the sample complexity for RL problems, what assumptions are sufficient or necessary for statistically efficient is still not clear. The concept spanning capability is new and reasonable to me.\n- Technical correctness: I didn’t find any technical errors in the proofs included in the appendix, but I admit I was not able to go through all the details in the appendix.\n\nWEAKNESSES:\n- The (K,D)-sunflower structure is an overly strong assumption and not reasonable enough. It seems to only work when there is a small subset of policies (so K is small) that covers most states (so D is small). \n- Although Algorithm 1 is sample efficient and only needs to collect polynomial trajectotires, the algorithm itself is not efficient: the algorithm needs to traverse all policies in $\\Pi$ to find the policy that can reach states which cannot be reached by $\\Pi_{\\text{core}}$. The complexity of Algorithm 1 is at least $O(|\\Pi|)$.\n- For Theorem 4, it seems the sample complexity is about $O({1}/{\\epsilon^4})$, which is not optimal on the order of $\\epsilon$. This also implies the sample complexity will be larger than $\\widetilde{\\mathcal{O}} (\\min\\{A^H. |\\Pi|, HSA\\}/\\epsilon^2)$ when $\\epsilon$ is small enough.\n\nQUESTIONS:\n- in Definition 2, how is the second equation established? It is not clear to me how to make the inf on $\\mu$ to the sum on $(s,a)$ pairs.\n- It seems coverability coefficient is always no larger than the span capability. Is coverability a sufficient complexity measure for generative model? Moreover, I note that the authors claim coverability is insufficient for online RL. However, considering that spanning capability is also insufficient for online RL, where is the key difference.\n- in appendix Lemma 2(1), what does $N$ means. The authors have not introduced this notation in Section E.\n\n\nOne minor error: in appendix Line 865 and 867, it should be $j\\in [2^{2H}]$ instead of $j\\in [2^{H}]$.\n\nLIMITATIONS:\nTheoretical work. No limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies agnostic learning in RL, and seeks to characterize when, given some policy class $\\Pi$, it is possible to learn an $\\epsilon$-optimal policy in $\\Pi$ regardless of the MDP. They propose a novel complexity measure—the spanning capacity—which depends only on the policy class $\\Pi$ (i.e. is independent of the underlying MDP) which they show is both a necessary and sufficient measure of learnability when the learner has generative access to the MDP. In the fully online setting, however, it is shown that the spanning capacity is no longer a sufficient measure of capacity. To characterize efficient learning in the online setting, they introduce the notion of a ``sunflower’’ policy class, and show that if a policy class has bounded spanning capacity and is a sunflower class, then efficient learning is possible in the online setting as well.\n\nSTRENGTHS:\n1. I believe this work makes an interesting and novel contribution to the RL literature. There has been much interest over the last several years in developing general complexity measures that characterize when efficient learning in RL is possible, but existing work has typically made stronger assumptions—for example, access to a model or value function class where realizability holds (i.e. the true model/value function is in the class)—than this work, which does not assume realizability, and simply considers the question of finding the best policy in some set of policies.\n2. The results in this work show that there is a formal separation between the generative model setting and online access setting in policy learning framework considered here. To my knowledge, such a result has not been previously known, and is an interesting observation. \n3. In the generative setting, the characterization given by the spanning capacity is very clean: for any policy class $\\Pi$, efficient learning is possible if and only if the spanning capacity is small. While this is minimax over MDPs (see below), it is not minimax over policy classes—it applies to any policy class. \n4. The spanning capacity is a very intuitive measure of complexity and admits a clean interpretation.\n5. The paper is very well-written and reads very nicely.\n\nWEAKNESSES:\n1. While the spanning capacity does provide a clean characterization of the learnability of a given policy class, it is a very worst-case measure over problem instances. It could be the case that the spanning capacity of a given policy class is very large, implying that efficient learning is not possible in general with the given policy class, but where on many MDPs efficient learning with the given policy class is still possible. In particular, for the spanning capacity to be large, there only needs to exist a single deterministic MDP for which it is difficult to find the optimal policy in class on—there could be many MDPs where it is very easy to find the optimal policy in class on, but this is not captured by the spanning capacity. In contrast, in practice we are typically interested not just in the policy class but the interaction between the policy class and the underlying dynamics of the MDP, as this characterizes how feasible it is to actually learn a good policy for a given problem. Due to this worst-case nature, for settings with large state or action spaces, the examples given of policy classes with bounded spanning capacity are very simple and not representative of the sorts of policy classes that would actually be used (in theory or practice). \n2. While it is shown that the sunflower condition is a sufficient condition for learning in the online setting, it is not clear it is a necessary condition (and furthermore, the scaling on $(K,D)$ given in Theorem 4 is likely not tight). Some discussion of this would be helpful. \n3. In principle, ignoring computation cost, I believe the optimal values of $K$ and $D$ for Theorem 4 could be computed a priori (before interacting with the MDP). This would be helpful to clarify around Theorem 4 (in particular on lines 296-298). Since POPLER takes as input $K$ and $D$, the bounded stated in Theorem 4 cannot be optimized for $K$ and $D$ after the fact, but can be optimized if the optimal $K$ and $D$ are computed before running POPLER (I found this somewhat unclear from the discussion on 296-298).\n4. I understand that the main contributions of this paper are statistical, but it would still be helpful to comment on the computational efficiency of the proposed algorithms.\n5. Several additional papers that should be cited: [1] also considers agnostic RL (albeit with stronger assumptions on the realizability of the model class). [2,3] are additional works on RL with function approximation that are generally relevant to this work.\n6. Minor typo: line 103 should read “access” not “saccess”. \n\n[1] Wagenmaker, Andrew, and Kevin G. Jamieson. \"Instance-dependent near-optimal policy identification in linear mdps via online experiment design.\" Advances in Neural Information Processing Systems 35 (2022): 5968-5981.\n\n[2] Foster, Dylan J., Noah Golowich, and Yanjun Han. \"Tight guarantees for interactive decision making with the decision-estimation coefficient.\" arXiv preprint arXiv:2301.08215 (2023).\n\n[3] Zhong, Han, et al. \"GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond.\" CoRR (2022).\n\nQUESTIONS:\n1. The DEC [4] is a recently proposed measure of complexity that applies in the realizable setting. While the setting considered here does not assume realizability, and so is not directly comparable to the DEC, some comparison with the DEC would be helpful. In particular, one could apply the DEC to the model class $\\mathcal{M}$ the set of all MDPs with no more than $S$ states, $A$ actions, and horizon $H$ (and with the decision space set to the given policy class $\\Pi$). One would get realizability for free in this setting, so the characterization given by the DEC could be compared to the spanning capacity. How would the DEC scale with this choice of $\\mathcal{M}$ for the policy classes presented on lines 200-204? \n\n2. Are there examples where $S$ and $H$ are both large, the policy set is reasonably expressive (e.g. it contains an $\\epsilon$-optimal policy for a reasonable class of MDPs), and the spanning capacity is bounded? For example, say we have some featurization $\\phi(s,a) \\in \\mathbb{R}^d$, and our class $\\Pi$ is defined as $\\Pi = ( \\pi^w \\ : \\ w \\in \\mathcal{W} )$ for $\\pi^w(s) = argmax_{a} \\phi(s,a)^\\top w$ and $\\mathcal{W}$, for example, a cover of $\\mathbb{R}^d$. Can the spanning capacity be shown to scale polynomially with $d$ in this case, or is the best that can be shown still the bounds given in Proposition 3? It is known that smoothed versions of such policy classes are sufficient for settings such as linear MDPs when $\\mathcal{W}$ is an $\\epsilon$-cover of $\\mathbb{R}^d$ (that is, such a policy class can be shown to contain an $\\epsilon$-optimal policy class for any linear MDP), so if the spanning capacity is bounded by $poly(d)$ in this case (even though $|\\Pi|$ scales exponentially in $d$) would provide a compelling large state-space example where spanning capacity scales reasonably. \n\n[4] Foster, Dylan J., et al. \"The statistical complexity of interactive decision making.\" arXiv preprint arXiv:2112.13487 (2021).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-24"}