An Exponential Lower Bound for Linearly Realizable MDP with Constant Suboptimality Gap

1 Introduction . There has been substantial recent theoretical interest in understanding the means by which we can avoid the curse of dimensionality and obtain sample-efficient reinforcement learning ( RL ) methods [ Wen and Van Roy , 2017 , Du et al. , 2019b , a , Wang et al. , 2019 , Yang and Wang , 2019 , Lattimore et al. , 2020 , Yang and Wang , 2020 , Jin et al. , 2020 , Cai et al. , 2020 , Zanette et al. , 2020 , Weisz et al. , 2020 , Du et al. , 2020 , Zhou et al. , 2020b , a , Modi et al. , 2020 , Jia et al. , 2020 , Ayoub et al. , 2020 ] . Here , the extant body of literature largely focuses on sufficient conditions for efficient reinforcement learning . Our understanding of what are the necessary conditions for efficient reinforcement learning is far more limited . With regards to the latter , arguably , the most natural assumption is linear realizability : we assume that the optimal Q-function lies in the linear span of a given feature map . The goal is to the obtain polynomial sample complexity under this linear realizability assumption alone . This “ linear Q∗ problem ” was a major open problem ( see Du et al . [ 2019a ] for discussion ) , and a recent hardness result by Weisz et al . [ 2020 ] provides a negative answer . In particular , the result shows that even with access to a generative model , any algorithm requires an exponential number 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of samples ( in the dimension d of the feature mapping ) to find a near-optimal policy , provided the action space has exponential size . With this question resolved , one may naturally ask what is the source of hardness for the construction in Weisz et al . [ 2020 ] and if there are additional assumptions that can serve to bypass the underlying source of this hardness . Here , arguably , it is most natural to further examine the suboptimality gap in the problem , which is the gap between the optimal Q-value of the best action and that of the second-best action ; the construction in Weisz et al . [ 2020 ] does in fact fundamentally rely on having an exponentially small gap . Instead , if we assume the gap is lower bounded by a constant for all states , we may hope that the problem becomes substantially easier since with a finite number of samples ( appropriately obtained ) , we can identify the optimal policy itself ( i.e. , the gap assumption allows us to translate value-based accuracy to the identification of the optimal policy itself ) . In fact , this intuition is correct in the following sense : with a generative model , it is not difficult to see that polynomial sample complexity is possible under the linear realizability assumption plus the suboptimality gap assumption , since the suboptimality gap assumption allows us to easily identify an optimal action for all states , thus making the problem tractable ( see Section C in Du et al . [ 2019a ] for a formal argument ) . More generally , the suboptimality gap assumption is widely discussed in the bandit literature [ Dani et al. , 2008 , Audibert and Bubeck , 2010 , Abbasi-Yadkori et al. , 2011 ] and the reinforcement learning literature [ Simchowitz and Jamieson , 2019 , Yang et al. , 2020 ] to obtain fine-grained sample complexity upper bounds . More specifically , under the realizability assumption and the suboptimality gap assumption , it has been shown that polynomial sample complexity is possible if the transition is nearly deterministic [ Du et al. , 2019b , 2020 ] ( also see Wen and Van Roy [ 2017 ] ) . However , it remains unclear whether the suboptimality gap assumption is sufficient to bypass the hardness result in Weisz et al . [ 2020 ] , or the same exponential lower bound still holds even under the suboptimality gap assumption , when the transition could be stochastic and the generative model is unavailable . For the construction in Weisz et al . [ 2020 ] , at the final stage , the gap between the value of the optimal action and its non-optimal counterparts will be exponentially small , and therefore the same construction does not imply an exponential sample complexity lower bound under the suboptimality gap assumption . Our contributions . In this work , we significantly strengthen the hardness result in Weisz et al . [ 2020 ] . In particular , we show that in the online RL setting ( where a generative model is unavailable ) with exponential-sized action space , the exponential sample complexity lower bound still holds even under the suboptimality gap assumption . Complementing our hardness result , we show that under the realizability assumption and the suboptimality gap assumption , our hardness result can be bypassed if one further assumes the low variance assumption in Du et al . [ 2019b ] 1 , or a hypercontractivity assumption . Hypercontractive distributions include Gaussian distributions ( with arbitrary covariance matrices ) , uniform distributions over hypercubes and strongly log-concave distributions [ Kothari and Steinhardt , 2017 ] . This condition has been shown powerful for outlier-robust linear regression [ Kothari and Steurer , 2017 ] , but has not yet been introduced for reinforcement learning with linear function approximation . Our results have several interesting implications , which we discuss in detail in Section 6 . Most notably , our results imply an exponential separation between the standard reinforcement learning setting and the generative model setting . Moreover , our construction enjoys greater simplicity , making it more suitable to be generalized for other RL problems or to be presented for pedagogical purposes . 1We note that the sample complexity of the algorithm in Du et al . [ 2019b ] has at least linear dependency on the number of actions , which is not sufficient for bypassing our hardness results which assumes an exponential-sized action space . 2 Related work . Previous hardness results . Existing exponential lower bounds in RL [ Krishnamurthy et al. , 2016 , Chen and Jiang , 2019 ] usually construct unstructured MDPs with an exponentially large state space . Du et al . [ 2019a ] prove that under the approximate version of the realizability assumption , i.e. , the optimal Q-function lies in the linear span of a given feature mapping approximately , any algorithm requires an exponential number of samples to find a near-optimal policy . The main idea in Du et al . [ 2019a ] is to use the Johnson-Lindenstrauss lemma [ Johnson and Lindenstrauss , 1984 ] to construct a large set of near-orthogonal feature vectors . Such idea is later generalized to other settings , including those in Wang et al . [ 2020a ] , Kumar et al . [ 2020 ] , Van Roy and Dong [ 2019 ] , Lattimore et al . [ 2020 ] . Whether the exponential lower bound still holds under the exact version of the realizability assumption is left as an open problem in Du et al . [ 2019a ] . The above open problem is recently solved by Weisz et al . [ 2020 ] . They show that under the exact version of the realizability assumption , any algorithm requires an exponential number of samples to find a near-optimal policy assuming an exponential-sized action space . The construction in Weisz et al . [ 2020 ] also uses the Johnson-Lindenstrauss lemma to construct a large set of near-orthogonal feature vectors , with additional subtleties to ensure exact realizability . Very recently , under the exact realizability assumption , strong lower bounds are proved in the offline setting [ Wang et al. , 2020b , Zanette , 2020 , Amortila et al. , 2020 ] . These work focus on the offline RL setting , where a fixed data distribution with sufficient coverage is given and the agent can not interact with the environment in an online manner . Instead , we focus on the online RL setting in this paper . Existing upper bounds . For RL with linear function approximation , most existing upper bounds require representation conditions stronger than realizability . For example , the algorithms in Yang and Wang [ 2019 , 2020 ] , Jin et al . [ 2020 ] , Cai et al . [ 2020 ] , Zhou et al . [ 2020b , a ] , Modi et al . [ 2020 ] , Jia et al . [ 2020 ] , Ayoub et al . [ 2020 ] assume that the transition model lies in the linear span of a given feature mapping , and the algorithms in Wang et al . [ 2019 ] , Lattimore et al . [ 2020 ] , Zanette et al . [ 2020 ] assume completeness properties of the given feature mapping . In the remaining part of this section , we mostly focus on previous upper bounds that require only realizability as the representation condition . For deterministic systems , under the realizability assumption , Wen and Van Roy [ 2017 ] provide an algorithm that achieves polynomial sample complexity . Later , under the realizability assumption and the suboptimality gap assumption , polynomial sample complexity upper bounds are shown if the transition is deterministic [ Du et al. , 2020 ] , a generative model is available [ Du et al. , 2019a ] , or a low-variance condition holds [ Du et al. , 2019b ] . Compared to the original algorithm in Du et al . [ 2019b ] , our modified algorithm in Section 5 works under a similar low-variance condition . However , the sample complexity in Du et al . [ 2019b ] has at least linear dependency on the number of actions , whereas our sample complexity in Section 5 has no dependency on the size of the action space . Finally , Shariff and Szepesvári [ 2020 ] obtain a polynomial upper bound under the realizability assumption when the features for all state-action pairs are inside the convex hull of a polynomial-sized coreset and the generative model is available to the agent . 3 Preliminaries . 3.1 Markov decision process ( MDP ) and reinforcement learning . An MDP is specified by ( S , A , H , P , { Rh } h∈ [ H ] ) , where S is the state space , A is the action space with |A| = A , H ∈ Z+ is the planning horizon , P : S × A → ∆S is the transition function and Rh : S ×A → ∆R is the reward distribution . Throughout the paper , we occasionally abuse notation and use a scalar a to denote the single-point distribution at a . A ( stochastic ) policy takes the form π = { πh } h∈ [ H ] , where each πh : S → ∆A assigns a distribution over actions for each state . We assume that the initial state is drawn from a fixed distribution , i.e . s1 ∼ µ . Starting from the initial state , a policy π induces a random trajectory s1 , a1 , r1 , · · · , sH , aH , rH via the process ah ∼ πh ( · ) , rh ∼ R ( ·|sh , ah ) and sh+1 ∼ P ( ·|sh , ah ) . For a policy π , denote the distribution of sh in its induced trajectory by Dπh . Given a policy π , the Q-function ( action-value function ) is defined as Qπh ( s , a ) : = E [ H∑ h′=h rh′ |sh = s , ah = a , π ] , while V πh ( s ) : = Ea∼πh ( s ) [ Qπh ( s , a ) ] . We denote the optimal policy by π∗ , and the associated optimal Q-function and value function by Q∗ and V ∗ respectively . Note that Q∗ and V ∗ can also be defined via the Bellman optimality equation2 : V ∗h ( s ) = max a∈A Q∗h ( s , a ) , Q∗h ( s , a ) = E [ Rh ( s , a ) + V ∗ h+1 ( sh+1 ) |sh = s , ah = a ] . The online RL setting . In this paper , we aim to prove lower bound and upper bound in the online RL setting . In this setting , in each episode , the agent interacts with the unknown environment using a policy and observes rewards and the next states . We remark that the hardness result by Weisz et al . [ 2020 ] operates in the setting where a generative model is available to the agent so that the agent can transit to any state . Also , it is known that with a generative model , under the linear realizability assumption plus the suboptimality gap assumption , one can find a near-optimal policy with polynomial number of samples ( see Section C in Du et al . [ 2019a ] for a formal argument ) .

Summary: This is a technical paper that considers online RL with linear function approximation under the assumptions: 1) Q^* is realizable, 2) gap = V^*(s) - Q^*(s,a) > 0 for all (s,a). The first result is an unfortunate impossibility result that gives an exponential sample lower bound without further assumptions. Then under additional assumptions on system dynamics, a variation of existing algorithm (Difference Maximization Q-Learning) returns a near-optimal policy in polynomial time.