SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we investigated when agnostic RL is statistically tractable in large state and action spaces, and introduced spanning capacity as a natural measure of complexity that only depends on the policy class, and is independent of the MDP rewards and transitions. We first showed that bounded spanning capacity is both necessary and sufficient for agnostic PAC RL under the generative access model. However, we also provided a negative result showing that bounded spanning capacity does not suffice for online RL, thus showing a surprising separation between agnostic RL with a generative model and online interaction. We then provided an additional structural assumption, called the sunflower property, that allows for statistically efficient learning in online RL. Our sample complexity bound for online RL is obtained using a novel exploration algorithm called POPLER that relies on certain policy-specific Markov Reward Processes to guide exploration, and takes inspiration from the classical importance sampling method and reward-free exploration algorithms for tabular MDPs.
Our results pave the way for several future lines of inquiry. The most important direction is exploring complexity measures to tightly characterize the minimax sample complexity for online RL. On the upper bound side, Theorem 4 shows that bounded spanning capacity along with an additional sunflower property is sufficient for online RL. On the lower bound side, while bounded spanning capacity is necessary (by Theorem 2), we do not know if the sunflower property is also necessary. Another direction is understanding if one can improve the statistical complexity of policy-based learning using stronger feedback models; in Appendix I we explore whether receiving expert feedback in the form of the optimal value function  $\{Q^{\star}(s,a)\}_{a\in \mathcal{A}}$  on the visited states can help. Surprisingly, the answer depends on realizability of the optimal policy  $\pi^{\star}\in \Pi$ : when  $\pi^{\star}\in \Pi$ ,  $Q^{\star}$  feedback can be utilized to achieve an  $O(\mathrm{poly}(\log |\Pi|,H,1 / \varepsilon))$  sample complexity bound, with no dependence on  $\mathfrak{C}(\Pi)$ ; if  $\pi^{\star}\notin \Pi$ , then one can not hope to learn with less than  $\Omega (\mathfrak{C}(\Pi))$  samples in the worst case. Understanding the role of  $\pi^{\star}$ -realizability and exploring the benefits of other feedback models in agnostic RL are interesting future research directions. Other research directions include sharpening the sample complexity bound in Theorem 4, extending POPLER for regret minimization, and developing computationally efficient algorithms.
We thank Dylan Foster, Pritish Kamath, Jason D. Lee, Wen Sun, and Cong Ma for helpful discussions. GL and NS are partially supported by the NSF and the Simons Foundation. Part of this work was completed while GL was visiting Princeton University. AR acknowledges support from the ONR through award N00014-20-1-2336, ARO through award W911NF-21-1-0328, and from the DOE through award DE-SC0022199.
Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellert Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning, pages 3692-3702. PMLR, 2019.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. PC-PG: Policy cover directed exploration for provable policy gradient learning. Advances in Neural Information Processing Systems, 2020.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4506, 2021.
Naman Agarwal, Brian Bullins, and Karan Singh. Variance-reduced conservative policy iteration. In International Conference on Algorithmic Learning Theory, pages 3-33. PMLR, 2023.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.
Aymen Al-Marjani, Andrea Tirinzoni, and Emilie Kaufmann. Active coverage for pac reinforcement learning. arXiv preprint arXiv:2306.13601, 2023.
Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private pac learning implies finite littlestone dimension. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pages 852-860, 2019.
Philip Amortila, Nan Jiang, Dhruv Madeka, and Dean P Foster. A few expert queries suffices for sample-efficient rl with resets and linear value approximation. arXiv preprint arXiv:2207.08342, 2022.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 2008.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
James Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic programming. Advances in neural information processing systems, 16, 2003.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786, 2019.
Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM), 36(4):929-965, 1989.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730-2775. PMLR, 2022.
Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Reward-free model-based reinforcement learning with linear function approximation. Advances in Neural Information Processing Systems, 34: 1582-1593, 2021a.
Zihan Zhang, Xiangyang Ji, and Simon Du. Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon. In Conference on Learning Theory, pages 4528-4531. PMLR, 2021b.
Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. CoRR, 2022.
A Detailed Comparison to Related Works 20
B Examples of Policy Classes 22
C Proofs for Section 3 24
C.1 Proof of Lemma 1 24
C.2 Coverability is Not Sufficient for Online RL 26
D Proofs for Section 4 26
D.1 Proof of Theorem 1 26
D.2 Proof of Theorem 2 27
D.3 Proof of Corollary 1 27
E Proofs for Section 5 28
E.1 Construction of State Space, Action Space, and Policy Class 28
E.2 Construction of MDP Family 28
E.3 Proof of Theorem 5 30
E.4 Proof of Lemma 2 31
E.5 Proof of Lemma 3 34
E.6 Change of Measure Lemma 37
F Proofs for Section 6 39
F.1 Algorithm Sketch 39
F.2 Algorithmic Details and Preliminaries 40
F.3 Supporting Technical Results 43
F.4 Proof of Theorem 4 49
F.5 Sunflower Property is Insufficient By Itself 49
G Infinite Policy Classes 51
G.1 Definitions and Preliminary Lemmas 51
G.2 Generative Model Lower Bound 51
G.3 Generative Model Upper Bound 52
G.4 Online RL Upper Bound 53
H Connections to Other Complexity Measures 54
H.1 Definitions and Relationships 54
H.2 Bounds on Spanning Capacity 55
H.2.1 Proof of Theorem 7 56
I Extension: Can Expert Feedback Help in Agnostic PAC RL? 57
I.1 Upper Bound under Realizability 57
I.2 Lower Bound in Agnostic Setting 58
J Technical Tools 60
A Detailed Comparison to Related Works
Reinforcement Learning (RL) has seen substantial progress over the past few years, with several different directions of work being pursued for efficiently solving RL problems that occur in practice. The classical approach to solving an RL problem is to model it as a tabular MDP. A long line of work (Sutton and Barto, 2018; Agarwal et al., 2019; Kearns and Singh, 2002; Brafman and Tennenholtz, 2002; Auer et al., 2008; Azar et al., 2017; Gheshlaghi Azar et al., 2013; Jin et al., 2018) has studied provably sample-efficient learning algorithms that can find the optimal policy in tabular RL. Unfortunately, the sample complexity of such algorithms unavoidably scales with the size of the state/action spaces, so they fail to be efficient in practical RL problems with large state/action spaces. In order to develop algorithms for the more practical large state/action RL settings, various assumptions have been considered in the prior works. In the following, we provide a detailed comparison of our setup and assumptions with the existing literature.
RL with Function Approximation. A popular paradigm for developing algorithms for MDPs with large state/action spaces is to use function approximation to either model the MDP dynamics or optimal value functions. Over the last decade, there has been a long line of work (Jiang et al., 2017; Dann et al., 2018; Sun et al., 2019; Du et al., 2019a; Wang et al., 2020c; Du et al., 2021; Foster et al., 2021a; Jin et al., 2021a; Zhong et al., 2022; Foster et al., 2023) in understanding structural conditions on the function class and the underlying MDP that enable statistically efficient RL. However, all of these works rely crucially on the realizability assumption, namely that the true model / value function belong to the chosen class. Unfortunately, such an assumption is too strong to hold in practice. Furthermore, the prior works using function approximation make additional assumptions like Bellman Completeness that are difficult to verify for the underlying task.
In our work, we study the problem of agnostic RL to sidestep these challenges. In particular, instead of modeling the value/dynamics, the learner now models "good policies" for the underlying task, and the learning objective is to find a policy that can perform as well as the best in the chosen policy class. We note that while a realizable value class/dynamics class  $\mathcal{F}$  can be converted into a realizable policy class  $\Pi_{\mathcal{F}}$  by choosing the greedy policies for each value function/dynamics, the converse is not true. Thus, our agnostic RL objective relies on a strictly weaker modeling assumption.
Connections to Decision-Estimation Coefficient (DEC). The seminal work of Foster et al. (2021a) provides a unified complexity measure called Decision-Estimation Coefficient (DEC) that characterizes the complexity of model-based RL. Given the generality of the E2D algorithm of Foster et al. (2021a), one may be wondering if our results can be recovered using their framework via a model-based approach. In particular, can we recover the sample complexity bound in Theorems 1 or 4 by considering the model class  $\mathcal{M}^{\mathrm{det}}$  or  $\mathcal{M}^{\mathrm{sto}}$  along with the decision set II. To the best of our knowledge, the framework of Foster et al. (2021a) do not directly recover our results, however, the complexity measures are closely related. Note that one can upper-bound the DEC by the coverability coefficient Xie et al. (2022); furthermore we show that  $\mathfrak{C}(\Pi)$  is worst-case coverability (Lemma 1) so it follows that DEC is upper-bounded by  $\mathfrak{C}(\Pi)$ . However, the algorithm in Foster et al. (2021a) achieves regret bounds which scale with  $\mathrm{DEC}\cdot \log |\mathcal{M}^{\mathrm{sto}}|$ , which can be vacuous in the large state-space setting since  $\log |\mathcal{M}^{\mathrm{sto}}|\propto |S|$ ; In contrast, our upper bounds have no explicit dependence on  $|S|$ .
RL with Rich Observations. Various settings have been studied where the dynamics are determined by a simple latent state space, but instead of observing the latent states directly, the learner receives rich observations corresponding to the underlying latent states. These include the Block MDP (Krishnamurthy et al., 2016; Du et al., 2019a; Misra et al., 2020; Mhammedi et al., 2023), Low-Rank MDPs (Uehara et al., 2021; Huang et al., 2023), Exogenous MDPs (Efroni et al., 2021; Xie et al., 2022; Efroni et al., 2022), etc. However, all of these prior works assume that the learner is given a realizable decoder class (consisting of functions that map observations to latent states) that contains the true decoder for the underlying MDP. Additionally, they require strong assumptions on the underlying latent state space dynamics, e.g. it is tabular or low-rank, in order to make learning tractable. Thus, their guarantees are not agnostic. In fact, given a realizable decoder class and additional structure on the latent state dynamics, one can construct a policy class that contains the optimal policy for the MDP, but the converse is not true. Thus, our agnostic RL setting is strictly more general.
Relation to Exponential Lower Bounds for RL with Function Approximation. Recently, many statistical lower bounds have been developed in RL with function approximation under only realizability. A line of work including (Wang et al., 2020b; Zanette, 2021; Weisz et al., 2021; Foster et al., 2021b) showed that the sample complexity scales exponentially in the horizon  $H$  for learning the optimal policy for RL problems where only the optimal value function  $Q^{\star}$  is linear w.r.t. the given features. Similarly, Du et al. (2019b) showed that one may need exponentially in  $H$  even if the optimal policy is linear w.r.t. the true features. These lower bounds can be extended to our agnostic RL setting, giving similar exponential in  $H$  lower bounds for agnestic RL, thus supplementing the well-known lower bounds (Krishnamurthy et al., 2016) which show that agnestic RL is intractable without additional structural assumptions on the policy class. However, to recall, the focus of this paper is to propose assumptions, like Definition 1 or 4, that circumvent these lower bounds and allow for sample efficient agnestic RL.
Importance Sampling for RL. Various importance sampling based estimators (Xie et al., 2019; Jiang and Li, 2016; Gottesman et al., 2019; Yin and Wang, 2020; Thomas and Brunskill, 2016; Nachum et al., 2019) have been developed in RL literature to provide reliable off-policy evaluation in offline RL. However, these methods also require realizable value function approximation and rely on additional assumptions on the off-policy/offline data, in particular, that the offline data covers the state/action space that is explored by the comparator policy. We note that this line of work does not directly overlap with our current approach but provides a valuable tool for dealing with off-policy data.
Agnostic RL in Low-Rank MDPs. Sekhari et al. (2021) explored agnostic PAC RL in low-rank MDPs, and showed that one can perform agnostic learning w.r.t. any policy class for MDPs that have a small rank. While their guarantees are similar to ours, i.e., they compete with the best policy in the given class and do not assume access to a realizable dynamics / value-function class, the key objectives of the two works are complementary. Sekhari et al. (2021) explore assumptions on the underlying MDP dynamics which suffice for agnostic learning for any given policy class, whereas we ask what assumptions on the given policy class suffice for agnostic learning for any underlying dynamics. Exploring the benefits of structure in both the policy class and the underlying MDP in agnostic RL is an interesting direction for future research.
Policy Gradient Methods. A significant body of work in RL, in both theory (Agarwal et al., 2021; Abbasi-Yadkori et al., 2019; Bhandari and Russo, 2019; Liu et al., 2020; Agarwal et al., 2020; Zhan et al., 2021; Xiao, 2022) and practice (Kakade, 2001; Kakade and Langford, 2002; Levine and Koltun, 2013; Schulman et al., 2015, 2017), studies policy-gradient based methods that directly search for the best policy in a given policy class. These approaches often leverage mirror-descent style analysis, and can deliver guarantees that are similar to ours, i.e. the returned policy can compete with any policy in the given class, which is an agnostic guarantee in some sense. However, these works primarily study smooth and parametric policy classes, e.g. tabular and linear policy classes, which limits their applicability for a broader range of problem instances. Furthermore, they require strong additional assumptions to work: for instance, that the learner is given a good reset distribution that can cover the occupancy measure of the policy that we wish to compare to, and that the policy class satisfies a certain "policy completeness assumption"; both of which are difficult to verify in practice. In contrast, our work makes no such assumptions but instead studies what kind of policy classes are learnable for any MDP.
CPI, PSDP, and Other Reductions to Supervised Learning. Various RL methods have been developed that return a policy that performs as well as the best policy in the given policy class, by reducing the RL problem from supervised learning. The key difference from policy-gradient based methods (discussed previously) is that these approaches do not require a smoothly parameterized policy class, but instead rely on access to a supervised learning oracle w.r.t. the given policy class. Popular approaches include Conservative Policy Iteration (CPI) (Kakade and Langford, 2002; Kakade, 2003; Brukhim et al., 2022b; Agarwal et al., 2023), PSDP (Bagnell et al., 2003), Behavior Cloning (Ross and Bagnell, 2010; Torabi et al., 2018), etc. We note that these algorithms rely on additional assumptions, including "policy completeness assumption" and a good sampling / reset distribution that covers the policies that we wish to compare to; in comparison, we do not make any such assumptions in our work.
Efficient RL via reductions to online regression oracles w.r.t. the given policy class has also been studied, see, e.g., DAgger (Ross et al., 2011), AggreVaTe (Ross and Bagnell, 2014), etc. However, these algorithms rely on much stronger feedback. In particular the learner, on the states which it visits, can query an expert policy (that we wish to complete with) for its actions or the value function. On the other hand, in this paper, we restrict ourselves to the standard RL setting where the learner only gets instantaneous reward signal. In Appendix I we investigate whether such stronger feedback can be used for agnostic RL.
Reward-Free RL. From a technical viewpoint, our algorithm (Algorithm 1) share similarities to algorithms developed in the reward-free RL literature (Jin et al., 2020). In reward-free RL, the goal of the learner is to output a dataset or a set of policies, after interacting with the underlying MDP, that can be later used for planning (with no further interaction with the MDP) for downstream reward functions. The key ideas in our Algorithm 1, in particular, that the learner first finds states  $\mathcal{I}$  that are  $\Omega(\varepsilon)$ -reachable and corresponding policies that can reach them, and then outputs datasets  $\{\mathcal{D}_s\}_{s \in \mathcal{I}}$  that can be later used for evaluating any policy  $\pi \in \Pi$ , share similarities to algorithmic ideas used in reward-free RL. However, our algorithm strictly generalizes prior works in reward-free RL, and in particular can work with large state-action spaces where the notion of reachability as well as the offline RL objective, is defined w.r.t. the given policy class. In comparison, prior reward-free RL works compete with the best policy for the underlying MDP, and make structure assumptions on the dynamics, e.g. tabular structure (Jin et al., 2020; Menard et al., 2021; Li et al., 2023) or linear dynamics (Wang et al., 2020a; Zanette et al., 2020; Zhang et al., 2021a; Wagenmaker et al., 2022), to make the problem tractable.
Instance Optimal Measures. Several recent works including Wagenmaker and Jamieson (2022); Tirinzoni et al. (2023); Bottou et al. (2013); Al-Marjani et al. (2023) have explored instance-dependent complexity measures for PAC RL. At a high level, these instance-dependent bounds are obtained via similar algorithmic ideas to ours that combine reward-free exploration with policy elimination. However, there are major differences. Firstly, these prior works in instance-dependent PAC RL operate under additional modeling assumptions on the MDP dynamics, e.g., that it is a tabular or linear MDP. Secondly, they require additional reachability assumptions on the state space, which is restrictive for MDPs with large states/actions; in fact, their sample complexity bounds typically have a dependence on the number of states/actions in the lower order terms. Finally, they implicitly assume that the optimal policy  $\pi^{\star} \in \Pi$ , and thus the provided algorithms do not transfer cleanly to the agnostic PAC RL setting considered in our paper.
Other Complexity Measures for RL. A recent work by Mou et al. (2020) proposed a new notion of eluder dimension for the policy class, and provide upper bounds for policy-based RL when the class  $\Pi$  has bounded eluder dimension. However, they require various additional assumptions: that the policy class contains the optimal policy, the learner has access to a generative model, and that the optimal value function has a gap. On the other hand, we do not make any such assumptions and characterize learnability in terms of spanning capacity or the size of the minimal sunflower in  $\Pi$ . We discuss connections to the eluder dimension, as well as other classical complexity measures in learning theory in Appendix H.
B Examples of Policy Classes
In this section, we will prove that examples in Section 3 have both bounded spanning capacity and the sunflower property with small  $K$  and  $D$ . To facilitate our discussion, we define the following notation: for any policy class  $\Pi$  we let
$$
\mathfrak{C}_{h}(\Pi):= \max_{M\in \mathcal{M}^{\mathrm{det}}}C_{h}^{\mathrm{reach}}(\Pi ;M),
$$
where  $C_h^{\mathrm{reach}}(\Pi; M)$  is defined in Definition 1. That is,  $\mathfrak{C}_h(\Pi)$  is the per-layer spanning capacity of  $\Pi$ . Then as defined in Definition 1, we have
$$
\mathfrak{C}(\Pi) = \max_{h\in [H]}\mathfrak{C}_{h}(\Pi).
$$
Tabular MDP. Since there are at most  $|S_h|$  states in layer  $h$ , it is obvious that  $\mathfrak{C}_h(\Pi) \leq |S_h|A$ , so therefore  $\mathfrak{C}(\Pi) \leq SA$ . Additionally, if we choose  $\Pi_{\mathrm{core}} = \{\pi_a : \pi_a(s) = a, a \in \mathcal{A}\}$  to be the set
of policies which play the constant  $a$  for each  $a \in \mathcal{A}$  and  $S_{\pi} = S$  for every  $\pi \in \Pi$ , then any partial trajectory which satisfies the condition in Definition 4 is of the form  $(s_h, a_h)$ , which is consistent with  $\pi_{a_h} \in \Pi_{\mathrm{core}}$ . Hence  $\Pi$  is a  $(A, S)$ -sunflower.
Contextual Bandit. Since there is only one layer, any deterministic MDP has a single state with at most  $A$  actions possible, so  $\mathfrak{C}(\Pi) \leq A$ . Additionally, if we choose  $\Pi_{\mathrm{core}} = \{\pi_a : \pi_a(s) \equiv a, a \in \mathcal{A}\}$ , and  $S_{\pi} = \emptyset$  for every  $\pi \in \Pi$ , then any partial trajectory which satisfies the condition in Definition 4 is in the form  $(s, a)$ , which is consistent with  $\pi_a \in \Pi_{\mathrm{core}}$ . Hence  $\Pi$  is a  $(A, 0)$ -sunflower.
$H$ -Layer Contextual Bandit. By induction, it is easy to see that any deterministic MDP spans at most  $A^{h - 1}$  states in layer  $h$ , each of which has at most  $A$  actions. Hence  $\mathfrak{C}(\Pi)\leq A^H$ . Additionally, if we choose
$$
\Pi_ {\text {c o r e}} = \left\{\pi_ {a _ {1}, \dots , a _ {H}}: \pi_ {a _ {1}, \dots , a _ {H}} \left(s _ {h}\right) \equiv a _ {h}, a _ {1}, \dots , a _ {H} \in \mathcal {A} \right\}
$$
and  $S_{\pi} = \emptyset$  for every  $\pi \in \Pi$ , then any partial trajectory which satisfies the condition in Definition 4 is in the form  $(s_1, a_1, \dots, s_H, a_H)$ , which is consistent with  $\pi_{a_1, a_2, \dots, a_H} \in \Pi_{\mathrm{core}}$ . Hence  $\Pi$  is a  $(A^H, 0)$ -sunflower.