Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?

Modern deep learning methods provide effective means to learn good representations . However , is a good representation itself sufficient for sample efficient reinforcement learning ? This question has largely been studied only with respect to ( worst-case ) approximation error , in the more classical approximate dynamic programming literature . With regards to the statistical viewpoint , this question is largely unexplored , and the extant body of literature mainly focuses on conditions which permit sample efficient reinforcement learning with little understanding of what are necessary conditions for efficient reinforcement learning . This work shows that , from the statistical viewpoint , the situation is far subtler than suggested by the more traditional approximation viewpoint , where the requirements on the representation that suffice for sample efficient RL are even more stringent . Our main results provide sharp thresholds for reinforcement learning methods , showing that there are hard limitations on what constitutes good function approximation ( in terms of the dimensionality of the representation ) , where we focus on natural representational conditions relevant to value-based , model-based , and policy-based learning . These lower bounds highlight that having a good ( valuebased , model-based , or policy-based ) representation in and of itself is insufficient for efficient reinforcement learning , unless the quality of this approximation passes certain hard thresholds . Furthermore , our lower bounds also imply exponential separations on the sample complexity between 1 ) value-based learning with perfect representation and value-based learning with a good-but-not-perfect representation , 2 ) value-based learning and policy-based learning , 3 ) policy-based learning and supervised learning and 4 ) reinforcement learning and imitation learning . 1 INTRODUCTION . Modern reinforcement learning ( RL ) problems are often challenging due to the huge state space . To tackle this challenge , function approximation schemes are often employed to provide a compact representation , so that reinforcement learning can generalize across states . A common paradigm is to first use a feature extractor to transform the raw input to features ( a succinct representation ) and then apply a linear predictor on top of the features . Traditionally , the feature extractor is often handcrafted ( Sutton & Barto , 2018 ) , while more modern methods often train a deep neural network to extract features . The hope of this paradigm is that , if there exists a good low dimensional ( linear ) representation , then efficient reinforcement learning is possible . Empirically , combining various RL function approximation algorithms with neural networks for feature extraction has lead to tremendous successes on various tasks ( Mnih et al. , 2015 ; Schulman et al. , 2015 ; 2017 ) . A major problem , however , is that these methods often require a large amount of samples to learn a good policy . For example , deep Q-network requires millions of samples to solve certain Atari games ( Mnih et al. , 2015 ) . Here , one may wonder if there are fundamental statistical limitations on such methods , and , if so , under what conditions it would be possible to efficiently learn a good policy ? In the supervised learning context , it is well-known that empirical risk minimization is a statistically efficient method when using a low-complexity hypothesis space ( Shalev-Shwartz & Ben-David , 2014 ) , e.g . a hypothesis space with bounded VC dimension . For example , polynomial number of samples suffice for learning a near-optimal d-dimensional linear classifier , even in the agnostic setting1 . In contrast , in the more challenging RL setting , we seek to understand if efficient learning is possible ( say from a sample complexity perspective ) when we have access to an accurate ( and compact ) parametric representation — e.g . our policy class contains a near-optimal policy or our hypothesis class accurately approximates the optimal value function . In particular , this work focuses on the following question : Is a good representation sufficient for sample-efficient reinforcement learning ? . This question has largely been studied only with respect to approximation error in the more classical approximate dynamic programming literature , where it is known that algorithms are stable to certain worst-case approximation errors . With regards to sample efficiency , this question is largely unexplored , where the extant body of literature mainly focuses on conditions which are sufficient for efficient reinforcement learning though there is little understanding of what are necessary conditions for efficient reinforcement learning . In reinforcement learning , there is no direct analogue of empirical risk minimization as in the supervised learning context , and it is not evident what are the statistical limits of learning based on properties of our underlying hypothesis class ( which may be value-based , policy-based , or model-based ) . Many recent works have provided polynomial upper bounds under various sufficient conditions , and in what follows we list a few examples . For value-based learning , the work of Wen & Van Roy ( 2013 ) showed that for deterministic systems2 , if the optimal Q-function can be perfectly predicted by linear functions of the given features , then the agent can learn the optimal policy exactly with polynomial number of samples . Recent work ( Jiang et al. , 2017 ) further showed that if certain complexity measure called Bellman rank is bounded , then the agent can learn a near-optimal policy efficiently . For policy-based learning , Agarwal et al . ( 2019 ) gave polynomial upper bounds which depend on a parameter that measures the difference between the initial distribution and the distribution induced by the optimal policy . Our Contributions . This paper gives , perhaps surprisingly , strong negative results to this question . The main results are exponential lower bounds in terms of planning horizonH for value-based , modelbased , and policy-based algorithms with given good representations3 . Notably , the requirements on the representation that suffice for sample efficient RL are even more stringent than the more traditional approximation viewpoint . A comprehensive summary of previous upper bounds and our lower bounds is given in Table 1 , and here we briefly summarize our hardness results . 1 . For value-based learning , we show even if Q-functions of all policies can be approximated by linear functions of the given representation with approximation error δ = Ω ( √ H d ) where d is the dimension of the representation and H is the planning horizon , then the agent still needs to sample exponential number of trajectories to find a near-optimal policy . 2 . For model-based learning , we show even if the transition matrix and the reward function can be approximated by linear functions of the given representation with approximation error δ = Ω ( √ H d ) ( in ` ∞ sense ) , the agent still needs to sample exponential number of trajectories to find a near-optimal policy . 3 . We show even if optimal policy can be perfectly predicted by a linear function of the given representation with a strictly positive margin , the agent still requires exponential number of trajectories to find a near-optimal policy . 1Here we only study the sample complexity and ignore the computational complexity . 2MDPs where both reward and transition are deterministic . 3Our results can be easily extend to infinite horizon MDPs with discount factors by replacing the planning horizon H with 1 1−γ , where γ is the discount factor . We omit the discussion on discount MDPs for simplicity . These lower bounds hold even in deterministic systems and even if the agent knows the transition model . Note these negative results apply to the case where the Q-function , the model , or the optimal policy can be predicted well by a linear function of the given representation . Since the class of linear functions is a strict subset of many more complicated function classes , including neural networks in particular , our negative results imply lower bounds for these more complex function classes as well . Our results highlight the following conceptual insights : • The requirements on the representation that suffice for sample efficient RL are significantly more stringent than the more traditional approximation viewpoint ; our statistical lower bounds show that there are hard thresholds on the worst-case approximation quality of the representation which are not necessary from the approximation viewpoint . • Since our lower bounds apply even when the agent knows the transition model , the hardness is not due to the difficulty of exploration in the standard sense . The unknown reward function is sufficient to make the problem exponentially difficult . • Our lower bounds are not due to the agent ’ s inability to perform efficient supervised learning , since our assumptions do admit polynomial sample complexity upper bounds if the data distribution is fixed . • Our lower bounds are not pathological in nature and suggest that these concerns may arise in practice . In a precise sense , almost all feature extractors induce a hard MDP instance in our construction ( see Section 4.4 ) . Instead , one interpretation is that the hardness is due to a distribution mismatch in the following sense : the agent does not know which distribution to use for minimizing a ( supervised ) learning error ( see Kakade ( 2003 ) for discussion ) , and even a known transition model is not information-theoretically sufficient to reduce the sample complexity . Furthermore , our work implies several interesting exponential separations on the sample complexity between : 1 ) value-based learning with perfect representation and value-based learning with a goodbut-not-perfect representation , 2 ) value-based learning and policy-based learning , 3 ) policy-based learning and supervised learning and 4 ) reinforcement learning and imitation learning . We provide more details in Section 5 . 2 RELATED WORK . A summary of previous upper bounds , together with lower bounds proved in this paper , is provided in Table 1 . Some key assumptions are formally stated in Section 3 and Section 4 . Our lower bounds highlight that classical complexity measures in supervised learning including small approximation error and margin , and standard assumptions in reinforcement learning including optimality gap and deterministic systems , are not enough for efficient RL with function approximation . We need additional assumptions , e.g. , ones used in previous upper bounds , for efficient RL . 2.1 PREVIOUS LOWER BOUNDS . Existing exponential lower bounds , to our knowledge , construct unstructured MDPs with an exponentially large state space and reduce a bandit problem with exponentially many arms to an MDP ( Krishnamurthy et al. , 2016 ; Sun et al. , 2017 ) . However , these lower bounds can not apply to MDPs whose transition models , value functions , or policies can be approximated with some natural function classes , e.g. , linear functions , neural networks , etc . The current paper gives the first set of lower bounds for RL with linear function approximation ( and thus also hold for super classes of linear functions such as neural networks ) . 2.2 PREVIOUS UPPER BOUNDS . We divide previous algorithms ( with provable guarantees ) into three classes : those that utilize uncertainty-based bonuses ( e.g . UCB variants or Thompson sampling variants ) ; approximate dynamic programming variants ( which often make assumptions with respect to concentrability coefficients ) ; and direct policy search-based methods ( such as conserve policy iteration ( CPI , see Kakade ( 2003 ) ) or policy gradient methods , which make assumptions with respect to distribution mismatch coefficients ) . with δ = Ω ( √ H d ) . Exact linear π∗ : Assumption 4.4 . Margin : Assumption 4.5 . Exact Linear Qπ for all π : Assumption 4.2 with δ = 0 . Approximate Linear Qπ for all π : Assumption 4.2 with δ = Ω ( √ H d ) . DetMDP : deterministic system defined in Section 3.1 . Bellman-rank : Definition 5 in Jiang et al . ( 2017 ) . Low Var : Assumption 1 in Du et al . ( 2019b ) . Gap : Assumption 3.1 . Bounded Distribution Mismatch Coefficient : Definition 3.3 in Agarwal et al . ( 2019 ) . ` ∞ Approx Linear MDP : Assumption 4.3 with δ = Ω ( √ H d ) . X : there exists an algorithm with polynomial sample complexity to find a near-optimal policy . X× : requires certain condition on the initial distribution . × : exponential number of samples is required. ? : open problem . The first class of methods include those based on witness rank , Belman rank , and the Eluder dimension , while the latter two classes of algorithms make assumptions either on concentrability coefficients or on distribution mismatch coefficients ( see Agarwal et al . ( 2019 ) ; Scherrer ( 2014 ) for discussions ) . Uncertainty bonus-based algorithms . Now we discuss existing theoretical results on value-based learning with function approximation . The most relevant work is Wen & Van Roy ( 2013 ) which showed in deterministic systems , if the optimal Q-function is within a pre-specified function class which has bounded Eluder dimension , for which the class of linear functions is a special case , then the agent can learn the optimal policy using polynomial number of samples . This result has recently been generalized by Du et al . ( 2019a ) which can deal with stochastic reward and low variance transition but requires strictly positive optimality gap . As we listed in Table 1 , it is an open problem whether the condition that the optimal Q-function is linear itself is sufficient for efficient RL . Li et al . ( 2011 ) proposed a Q-learning algorithm which requires the Know-What-It-Knows oracle . However , it is in general unknown how to implement such oracle in practice . Jiang et al . ( 2017 ) proposed the concept of Bellman Rank to characterize the sample complexity of value-based learning methods and gave an algorithm that has polynomial sample complexity in terms of the Bellman Rank , though the proposed algorithm is not computationally efficient . Bellman rank is bounded for a wide range of problems , including MDP with small number of hidden states , linear MDP , LQR , etc . Later work gave computationally efficient algorithms for certain special cases ( Dann et al. , 2018 ; Du et al. , 2019a ; Yang & Wang , 2019b ; Jin et al. , 2019 ) . Recently , Witness rank , a generalization of Bellman rank to model-based methods , is studied in Sun et al . ( 2019 ) . Approximate dynamic programming-based algorithms . We now discuss approximate dynamic programming-based results characterized in terms of the concentrability coefficient . While classical approximate dynamic programming results typically require ` ∞-bounded errors , the notion of concentrability ( originally due to ( Munos , 2005 ) ) permits sharper bounds in terms of average-case function approximation error , provided that the concentrability coefficient is bounded ( e.g . see Munos ( 2005 ) ; Szepesvári & Munos ( 2005 ) ; Antos et al . ( 2008 ) ; Geist et al . ( 2019 ) ) . Under the assumption that this problem-dependent parameter is bounded , Munos ( 2005 ) ; Szepesvári & Munos ( 2005 ) and Antos et al . ( 2008 ) proved sample complexity and error bounds for approximate dynamic programming methods when there is a data collection policy ( under which value-function fitting occurs ) that induces a finite concentrability coefficient . The assumption that the concentrability coefficient is finite is in fact quite limiting . See Chen & Jiang ( 2019 ) which provides a more detailed discussion on this quantity . Direct policy search-based algorithms . Stronger guarantees over approximate dynamic programming-based algrithm can be obtained with direct policy search-based methods , where instead of having a bounded concentrability coefficient , one only needs to have a bounded distribution mismatch coefficient . The latter assumption requires the agent to have access to a “ good ” initial state distribution ( e.g . a measure which has coverage over where an optimal policy tends to visit ) ; note that this assumption does not make restrictions over the class of MDPs . There are two classes of algorithms that fall into this category . First , there is Conservative Policy Iteration ( Kakade & Langford , 2002 ) , along with Policy Search by Dynamic Programming ( PSDP ) ( Bagnell et al. , 2004 ) , and other boosting-style of policy search-based methods Scherrer & Geist ( 2014 ) ; Scherrer ( 2014 ) , which have guarantees in terms of bounded distribution mismatch ratio . Second , more recently , Agarwal et al . ( 2019 ) showed that policy gradient styles of algorithms also have comparable guarantees . Recent extensions . Subsequent to this work , the work by Van Roy & Dong ( 2019 ) and Lattimore & Szepesvari ( 2019 ) made notable contributions to the misspecified linear bandit problem . In particular , both papers found that Theorem 4.1 in our paper can be extended to the misspecified linear bandit problem and gave upper bounds for this problem showing that our lower bound has tight dependency on δ and d. Lattimore & Szepesvari ( 2019 ) further gave an upper bound for the setting where the Q-functions of all policies can be approximated by linear functions with small approximation errors and the agent can interact with the environment using a generative model . This upper bound also demonstrates that our lower bound has tight dependency on δ and d .

Summary: This paper presents theoretical lower bounds on sample complexities to learn good policies in reinforcement learning. The derived theorems show that there exists MDPs which require an exponential number of samples to learn a near-optimal policy even if a good-but-not-perfect representation is given to the agent for both value-based and policy-based learning. These results constitute the first lower bounds for RL with linear function approximation.