An Exponential Lower Bound for Linearly Realizable MDP with Constant Suboptimality Gap

1 Introduction . There has been substantial recent theoretical interest in understanding the means by which we can avoid the curse of dimensionality and obtain sample-efficient reinforcement learning ( RL ) methods [ Wen and Van Roy , 2017 , Du et al. , 2019b , a , Wang et al. , 2019 , Yang and Wang , 2019 , Lattimore et al. , 2020 , Yang and Wang , 2020 , Jin et al. , 2020 , Cai et al. , 2020 , Zanette et al. , 2020 , Weisz et al. , 2020 , Du et al. , 2020 , Zhou et al. , 2020b , a , Modi et al. , 2020 , Jia et al. , 2020 , Ayoub et al. , 2020 ] . Here , the extant body of literature largely focuses on sufficient conditions for efficient reinforcement learning . Our understanding of what are the necessary conditions for efficient reinforcement learning is far more limited . With regards to the latter , arguably , the most natural assumption is linear realizability : we assume that the optimal Q-function lies in the linear span of a given feature map . The goal is to the obtain polynomial sample complexity under this linear realizability assumption alone . This “ linear Q∗ problem ” was a major open problem ( see Du et al . [ 2019a ] for discussion ) , and a recent hardness result by Weisz et al . [ 2020 ] provides a negative answer . In particular , the result shows that even with access to a generative model , any algorithm requires an exponential number 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of samples ( in the dimension d of the feature mapping ) to find a near-optimal policy , provided the action space has exponential size . With this question resolved , one may naturally ask what is the source of hardness for the construction in Weisz et al . [ 2020 ] and if there are additional assumptions that can serve to bypass the underlying source of this hardness . Here , arguably , it is most natural to further examine the suboptimality gap in the problem , which is the gap between the optimal Q-value of the best action and that of the second-best action ; the construction in Weisz et al . [ 2020 ] does in fact fundamentally rely on having an exponentially small gap . Instead , if we assume the gap is lower bounded by a constant for all states , we may hope that the problem becomes substantially easier since with a finite number of samples ( appropriately obtained ) , we can identify the optimal policy itself ( i.e. , the gap assumption allows us to translate value-based accuracy to the identification of the optimal policy itself ) . In fact , this intuition is correct in the following sense : with a generative model , it is not difficult to see that polynomial sample complexity is possible under the linear realizability assumption plus the suboptimality gap assumption , since the suboptimality gap assumption allows us to easily identify an optimal action for all states , thus making the problem tractable ( see Section C in Du et al . [ 2019a ] for a formal argument ) . More generally , the suboptimality gap assumption is widely discussed in the bandit literature [ Dani et al. , 2008 , Audibert and Bubeck , 2010 , Abbasi-Yadkori et al. , 2011 ] and the reinforcement learning literature [ Simchowitz and Jamieson , 2019 , Yang et al. , 2020 ] to obtain fine-grained sample complexity upper bounds . More specifically , under the realizability assumption and the suboptimality gap assumption , it has been shown that polynomial sample complexity is possible if the transition is nearly deterministic [ Du et al. , 2019b , 2020 ] ( also see Wen and Van Roy [ 2017 ] ) . However , it remains unclear whether the suboptimality gap assumption is sufficient to bypass the hardness result in Weisz et al . [ 2020 ] , or the same exponential lower bound still holds even under the suboptimality gap assumption , when the transition could be stochastic and the generative model is unavailable . For the construction in Weisz et al . [ 2020 ] , at the final stage , the gap between the value of the optimal action and its non-optimal counterparts will be exponentially small , and therefore the same construction does not imply an exponential sample complexity lower bound under the suboptimality gap assumption . Our contributions . In this work , we significantly strengthen the hardness result in Weisz et al . [ 2020 ] . In particular , we show that in the online RL setting ( where a generative model is unavailable ) with exponential-sized action space , the exponential sample complexity lower bound still holds even under the suboptimality gap assumption . Complementing our hardness result , we show that under the realizability assumption and the suboptimality gap assumption , our hardness result can be bypassed if one further assumes the low variance assumption in Du et al . [ 2019b ] 1 , or a hypercontractivity assumption . Hypercontractive distributions include Gaussian distributions ( with arbitrary covariance matrices ) , uniform distributions over hypercubes and strongly log-concave distributions [ Kothari and Steinhardt , 2017 ] . This condition has been shown powerful for outlier-robust linear regression [ Kothari and Steurer , 2017 ] , but has not yet been introduced for reinforcement learning with linear function approximation . Our results have several interesting implications , which we discuss in detail in Section 6 . Most notably , our results imply an exponential separation between the standard reinforcement learning setting and the generative model setting . Moreover , our construction enjoys greater simplicity , making it more suitable to be generalized for other RL problems or to be presented for pedagogical purposes . 1We note that the sample complexity of the algorithm in Du et al . [ 2019b ] has at least linear dependency on the number of actions , which is not sufficient for bypassing our hardness results which assumes an exponential-sized action space . 2 Related work . Previous hardness results . Existing exponential lower bounds in RL [ Krishnamurthy et al. , 2016 , Chen and Jiang , 2019 ] usually construct unstructured MDPs with an exponentially large state space . Du et al . [ 2019a ] prove that under the approximate version of the realizability assumption , i.e. , the optimal Q-function lies in the linear span of a given feature mapping approximately , any algorithm requires an exponential number of samples to find a near-optimal policy . The main idea in Du et al . [ 2019a ] is to use the Johnson-Lindenstrauss lemma [ Johnson and Lindenstrauss , 1984 ] to construct a large set of near-orthogonal feature vectors . Such idea is later generalized to other settings , including those in Wang et al . [ 2020a ] , Kumar et al . [ 2020 ] , Van Roy and Dong [ 2019 ] , Lattimore et al . [ 2020 ] . Whether the exponential lower bound still holds under the exact version of the realizability assumption is left as an open problem in Du et al . [ 2019a ] . The above open problem is recently solved by Weisz et al . [ 2020 ] . They show that under the exact version of the realizability assumption , any algorithm requires an exponential number of samples to find a near-optimal policy assuming an exponential-sized action space . The construction in Weisz et al . [ 2020 ] also uses the Johnson-Lindenstrauss lemma to construct a large set of near-orthogonal feature vectors , with additional subtleties to ensure exact realizability . Very recently , under the exact realizability assumption , strong lower bounds are proved in the offline setting [ Wang et al. , 2020b , Zanette , 2020 , Amortila et al. , 2020 ] . These work focus on the offline RL setting , where a fixed data distribution with sufficient coverage is given and the agent can not interact with the environment in an online manner . Instead , we focus on the online RL setting in this paper . Existing upper bounds . For RL with linear function approximation , most existing upper bounds require representation conditions stronger than realizability . For example , the algorithms in Yang and Wang [ 2019 , 2020 ] , Jin et al . [ 2020 ] , Cai et al . [ 2020 ] , Zhou et al . [ 2020b , a ] , Modi et al . [ 2020 ] , Jia et al . [ 2020 ] , Ayoub et al . [ 2020 ] assume that the transition model lies in the linear span of a given feature mapping , and the algorithms in Wang et al . [ 2019 ] , Lattimore et al . [ 2020 ] , Zanette et al . [ 2020 ] assume completeness properties of the given feature mapping . In the remaining part of this section , we mostly focus on previous upper bounds that require only realizability as the representation condition . For deterministic systems , under the realizability assumption , Wen and Van Roy [ 2017 ] provide an algorithm that achieves polynomial sample complexity . Later , under the realizability assumption and the suboptimality gap assumption , polynomial sample complexity upper bounds are shown if the transition is deterministic [ Du et al. , 2020 ] , a generative model is available [ Du et al. , 2019a ] , or a low-variance condition holds [ Du et al. , 2019b ] . Compared to the original algorithm in Du et al . [ 2019b ] , our modified algorithm in Section 5 works under a similar low-variance condition . However , the sample complexity in Du et al . [ 2019b ] has at least linear dependency on the number of actions , whereas our sample complexity in Section 5 has no dependency on the size of the action space . Finally , Shariff and Szepesvári [ 2020 ] obtain a polynomial upper bound under the realizability assumption when the features for all state-action pairs are inside the convex hull of a polynomial-sized coreset and the generative model is available to the agent . 3 Preliminaries . 3.1 Markov decision process ( MDP ) and reinforcement learning . An MDP is specified by ( S , A , H , P , { Rh } h∈ [ H ] ) , where S is the state space , A is the action space with |A| = A , H ∈ Z+ is the planning horizon , P : S × A → ∆S is the transition function and Rh : S ×A → ∆R is the reward distribution . Throughout the paper , we occasionally abuse notation and use a scalar a to denote the single-point distribution at a . A ( stochastic ) policy takes the form π = { πh } h∈ [ H ] , where each πh : S → ∆A assigns a distribution over actions for each state . We assume that the initial state is drawn from a fixed distribution , i.e . s1 ∼ µ . Starting from the initial state , a policy π induces a random trajectory s1 , a1 , r1 , · · · , sH , aH , rH via the process ah ∼ πh ( · ) , rh ∼ R ( ·|sh , ah ) and sh+1 ∼ P ( ·|sh , ah ) . For a policy π , denote the distribution of sh in its induced trajectory by Dπh . Given a policy π , the Q-function ( action-value function ) is defined as Qπh ( s , a ) : = E [ H∑ h′=h rh′ |sh = s , ah = a , π ] , while V πh ( s ) : = Ea∼πh ( s ) [ Qπh ( s , a ) ] . We denote the optimal policy by π∗ , and the associated optimal Q-function and value function by Q∗ and V ∗ respectively . Note that Q∗ and V ∗ can also be defined via the Bellman optimality equation2 : V ∗h ( s ) = max a∈A Q∗h ( s , a ) , Q∗h ( s , a ) = E [ Rh ( s , a ) + V ∗ h+1 ( sh+1 ) |sh = s , ah = a ] . The online RL setting . In this paper , we aim to prove lower bound and upper bound in the online RL setting . In this setting , in each episode , the agent interacts with the unknown environment using a policy and observes rewards and the next states . We remark that the hardness result by Weisz et al . [ 2020 ] operates in the setting where a generative model is available to the agent so that the agent can transit to any state . Also , it is known that with a generative model , under the linear realizability assumption plus the suboptimality gap assumption , one can find a near-optimal policy with polynomial number of samples ( see Section C in Du et al . [ 2019a ] for a formal argument ) .

Summary: This paper considers online RL with linear realizability of Q* and a suboptimality gap assumption. It shows an exponential sample complexity lower bound holds for this setting, but under an additional assumption of either (a) or (b), there is an algorithm (modified DMQ) that achieves a polynomial upper bound, showing sample-efficient RL is possible in this setting. The additional assumption required for this is either (a) low variance of value differences between some policy pi and the optimal policy, or (b) hypercontractivity of the distribution of features encountered along any policy.  The negative result is a modification of Weisz et al 2020’s construction of hard MDPs into a “leaky” graph that allows for a large suboptimality gap. The positive result is similar in algorithm and analysis to Du et al 2019b, with the novelties that (1) a number-of-actions factor in the sample complexity is avoided by sampling actions from an optimal design, and (2) the method is shown to work for assumption (b).   (1) is important to make the separation between the lower and upper bounds clear as the lower bound uses exponentially many actions. As the authors point out, these results also imply that there is an exponential separation between online RL and RL with a generative model, as in the latter, sample-efficient learning is possible in the same setting as this paper’s lower bound.