Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms

1 Introduction . Modern Reinforcement Learning ( RL ) commonly engages practical problems with an enormous number of states , where function approximation must be deployed to approximate the true value function using functions from a prespecified function class . Function approximation , especially based on deep neural networks , lies at the heart of the recent practical successes of RL in domains such as Atari [ 2 ] , Go [ 3 ] , robotics [ 4 ] , and dialogue systems [ 5 ] . Despite its empirical success , RL with function approximation raises a new series of theoretical challenges when comparing to the classic tabular RL : ( 1 ) generalization , to generalize knowledge from the visited states to the unvisited states due to the enormous state space . ( 2 ) limited expressiveness , to handle the complicated issues where true value functions or intermediate steps computed in the algorithm can be functions outside the prespecified function class . ( 3 ) exploration , to address the tradeoff between exploration and exploitation when above challenges are present . Consequently , most existing theoretical results on efficient RL with function approximation rely on relatively strong structural assumptions . For instance , many require that the MDP admits a linear approximation [ 6–8 ] , or that the model is precisely Linear Quadratic Regulator ( LQR ) [ 9–11 ] . Most of these structural assumptions rarely hold in practical applications . This naturally leads to one of the most fundamental questions in RL . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . What are the minimal structural assumptions that empower sample-efficient RL ? We advance our understanding of this grand question via the following two steps : ( 1 ) identify a rich class of RL problems ( with weak structural assumptions ) that cover many practical applications of interests ; ( 2 ) design sample-efficient algorithms that provably learn any RL problem in this class . The attempts to find weak or minimal structural assumptions that allow statistical learning can be traced in supervised learning where VC dimension [ 12 ] or Rademacher complexity [ 13 ] is proposed , or in online learning where Littlestone dimension [ 14 ] or sequential Rademacher complexity [ 15 ] is developed . In the area of reinforcement learning , there are two intriguing lines of recent works that have made significant progress in this direction . To begin with , Jiang et al . [ 1 ] introduces a generic complexity notion—Bellman rank , which can be proved small for many RL problems including linear MDPs [ 7 ] , reactive POMDPs [ 16 ] , etc . [ 1 ] further propose an hypothesis elimination-based algorithm— OLIVE for sample-efficient learning of problems with low Bellman rank . On the other hand , recent work by Wang et al . [ 17 ] considers general function approximation with low Eluder dimension [ 18 ] , and designs a UCB-style algorithm with regret guarantee . Noticeably , generalized linear MDPs [ 6 ] and kernel MDPs ( see Appendix C ) are subclasses of low Eluder dimension problems , but not low Bellman rank . In this paper , we make the following three contributions . • We introduce a new complexity measure for RL—Bellman Eluder ( BE ) dimension . We prove that the family of RL problems of low BE dimension is remarkably rich , which subsumes both low Bellman rank problems and low Eluder dimension problems—two arguably most generic tractable function classes so far in the literature ( see Figure 1 ) . The family of low BE dimension further includes new problems such as kernel reactive POMDPs ( see Appendix C ) which were not known to be sample-efficiently learnable . • We design a new optimization-based algorithm—GOLF , which provably learns nearoptimal policies of low BE dimension problems in a number of samples that is polynomial in all relevant parameters , but independent of the size of state-action space . Our regret or sample complexity guarantees match [ 8 ] which is minimax optimal when specified to the linear setting . Our rates further improve upon [ 1 , 17 ] in low Bellman rank and low Eluder dimension settings , respectively . • We reanalyze the hypothesis elimination based algorithm—OLIVE proposed in [ 1 ] . We show it can also learn RL problems with low BE dimension sample-efficiently , under slightly weaker assumptions but with worse sample complexity comparing to GOLF . 1.1 Related works . This section reviews prior theoretical works on RL , under Markov Decision Process ( MDP ) models . 1The family of low Bellman rank problems and low Bellman Eluder dimension problems include both Q-type and V-type variants . Please refer to Section 3.1 and Appendix B for more details . We remark that there has been a long line of research on function approximation in the batch RL setting [ see , e.g. , 19–22 ] . In this setting , agents are provided with exploratory data or simulator , so that they do not need to explicitly address the challenge of exploration . In this paper , we do not make such assumption , and attack the exploration problem directly . In the following we focus exclusively on the RL results in the general setting where exploration is required . Tabular RL . Tabular RL concerns MDPs with a small number of states and actions , which has been thoroughly studied in recent years [ see , e.g. , 23–30 ] . In the episodic setting with non-stationary dynamics , the best regret bound Õ ( √ H2|S||A|T ) is achieved by both model-based [ 27 ] and modelfree [ 30 ] algorithms . Moreover , the bound is proved to be minimax-optimal [ 29 , 31 ] . This minimax bound suggests that when the state-action space is enormous , RL is information-theoretically hard without further structural assumptions . RL with linear function approximation . A recent line of work studies RL with linear function approximation [ see , e.g. , 7 , 6 , 32 , 8 , 33–36 ] These papers assume certain completeness conditions , as well as the optimal value function can be well approximated by linear functions . Under one formulation of linear approximation , the minimax regret bound Õ ( d √ T ) is achieved by algorithm ELEANOR [ 8 ] , where d is the ambient dimension of the feature space . RL with general function approximation . Beyond the linear setting , there is a flurry line of research studying RL with general function approximation [ see , e.g. , 37 , 1 , 36 , 38 , 17 , 39 , 40 ] . Among them , [ 1 ] and [ 17 ] are the closest to our work . Jiang et al . [ 1 ] propose a complexity measure named Bellman rank and design an algorithm OLIVE with PAC guarantees for problems with low Bellman rank . We note that low Bellman rank is a special case of low BE dimension . When specialized to the low Bellman rank setting , our result for OLIVE exactly matches the guarantee in [ 1 ] . Our result for GOLF requires an additional completeness assumption , but provides sharper sample complexity guarantee . Wang et al . [ 17 ] propose a UCB-type algorithm with a regret guarantee under the assumption that the function class has a low eluder dimension . Again , we will show that low Eluder dimension is a special case of low BE dimension . Comparing to [ 17 ] , our algorithm GOLF works under a weaker completeness assumption , with a better regret guarantee . Finally , we remark that the algorithms proposed in [ 1 , 33 , 41 ] and this paper are all computationally inefficient in general . We notice several existing works [ e.g. , 7 , 17 ] can be computationally efficient given suitable regression oracles but they require stronger representation conditions and also achieve worse regret guarantees . Relation to bilinear classes Concurrent to this work , Du et al . [ 41 ] propose a new general tractable class of RL problems—bilinear class with low effective dimension ( also known as low critical information gain in Du et al . [ 41 ] ) . We comment on the similarities and differences between two works as follows . In terms of algorithms , both Algorithm 2 in this paper and the algorithm proposed in Du et al . [ 41 ] are based on OLIVE originally proposed in Jiang et al . [ 1 ] . The two algorithms share similar guarantees in terms of assumptions and complexity results . More importantly , our work further develops a new type of algorithm for general function approximation—GOLF , a natural and clean algorithm which can be viewed as an optimistic version of classical algorithm—Fitted Q-Iteration [ 42 ] . GOLF gives much sharper sample complexity guarantees compared to [ 41 ] for various settings , and is minimaxoptimal when applied to the linear setting [ 8 ] . In terms of richness of new classes identified , it depends on ( a ) what structure of MDP the complexity measures are applied to , and ( b ) what complexity measures are used . For ( a ) , BE dimension applies to the Bellman error , while the bilinear class allows general surrogate losses of the Bellman error . For ( b ) , this paper uses Eluder dimension while Du et al . [ 41 ] uses effective dimension . It can be shown that low effective dimension always implies low Eluder dimension ( see Appendix C.2 ) . In short , Du et al . [ 41 ] is more general in ( a ) , while our work is more general in ( b ) . As a result , neither work fully captures the other . In particular , our BE framework covers a majority of the examples identified in Du et al . [ 41 ] including low occupancy complexity , linear Q ? /V ? , Q ? state aggregation , feature selection/FLAMBE . Nevertheless , our work can not address examples with model-based function approximation ( e.g. , low witness rank [ 36 ] ) while [ 41 ] can . On the other hand , Du et al . [ 41 ] can not address the class of RL problems with low Eluder dimension [ 17 ] while our work can . Moreover , for several classes of RL problems that both works cover , our complexity measure is sharper . For example , in the setting of function approximation with generalized linear functions , the BE dimension is Õ ( d ) where d is the ambient dimension of the feature vectors , while the effective dimension under the generalized bilinear framework of Du et al . [ 41 ] is at least Ω̃ ( d2 ) . 1.2 Paper organization . We present preliminaries in Section 2 , the definition of Bellman-Eluder dimension as well as its relations to existing complexity notions in Section 3 . We present the results for algorithm GOLF in Section 4 , and conclude in Section 5 . Due to space limit , we postpone the results for algorithm OLIVE to Appendix A . Further discussions on Q-type versus V-type variants of BE dimension , as well as the practical examples will be provided in Appendix B and C. All the proofs are postponed to the appendix . 2 Preliminaries . We consider episodic Markov Decision Process ( MDP ) , denoted byM = ( S , A , H , P , r ) , where S is the state space , A is the action space , H is the number of steps in each episode , P = { Ph } h∈ [ H ] is the collection of transition measures with Ph ( s′ | s , a ) equal to the probability of transiting to s′ after taking action a at state s at the hth step , and r = { rh } h∈ [ H ] is the collection of reward functions with rh ( s , a ) equal to the deterministic reward received after taking action a at state s at the hth step . 2 Throughout this paper , we assume reward is non-negative , and ∑H h=1 rh ( sh , ah ) ≤ 1 for all possible sequence ( s1 , a1 , . . . , sH , aH ) . In each episode , the agent starts at a fixed initial state s1 . Then , at each step h ∈ [ H ] , the agent observes its current state sh , takes action ah , receives reward rh ( sh , ah ) , and causes the environment to transit to sh+1 ∼ Ph ( · | sh , ah ) . Without loss of generality , we assume there is a terminating state send which the environment will always transit to at step H + 1 , and the episode terminates when send is reached . Policy and value functions A ( deterministic ) policy π is a collection of H functions { πh : S → A } Hh=1 . We denote V πh : S → R as the value function at step h for policy π , so that V πh ( s ) gives the expected sum of the remaining rewards received under policy π , starting from sh = s , till the end of the episode . In symbol , V πh ( s ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s ] . Similarly , we denote Qπh : S ×A → R as the Q-value function at step h for policy π , where Qπh ( s , a ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s , ah = a ] . There exists an optimal policy π ? , which gives the optimal value function for all states [ 43 ] , in the sense , V π ? h ( s ) = supπ V π h ( s ) for all h ∈ [ H ] and s ∈ S . For notational simplicity , we abbreviate V π ? as V ? . We similarly define the optimal Q-value function as Q ? . Recall that Q ? satisfies the Bellman optimality equation : Q ? h ( s , a ) = ( ThQ ? h+1 ) ( s , a ) : = rh ( s , a ) + Es′∼Ph ( ·|s , a ) max a′∈A Q ? h+1 ( s ′ , a′ ) . ( 1 ) for all ( s , a , h ) ∈ S ×A× [ H ] . We also call Th the Bellman operator at step h. 2We study deterministic reward for notational simplicity . Our results readily generalize to random rewards . -optimality and regret We say a policy π is -optimal if V π1 ( s1 ) ≥ V ? 1 ( s1 ) − . Suppose an agent interacts with the environment for K episodes . Denote by πk the policy the agent follows in episode k ∈ [ K ] . The ( accumulative ) regret is defined as Reg ( K ) : = K∑ k=1 [ V ? 1 ( s1 ) − V π k 1 ( s1 ) ] . The objective of reinforcement learning is to find an -optimal policy within a small number of interactions or to achieve sublinear regret .

Summary: This paper introduces a new notion of dimension (i.e., the BE dimension) to measure the complexity of a function class when applied to RL problems. The BE dimension summarizes two previously prevailing notions: the Bellman rank and the Eluder dimension, and is also able to cover more tractable cases than before. An algorithm is proposed to be provably efficient with a low BE dimension which improves previous results based on low Bellman rank and low Eluder dimension.