Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms

1 Introduction . Modern Reinforcement Learning ( RL ) commonly engages practical problems with an enormous number of states , where function approximation must be deployed to approximate the true value function using functions from a prespecified function class . Function approximation , especially based on deep neural networks , lies at the heart of the recent practical successes of RL in domains such as Atari [ 2 ] , Go [ 3 ] , robotics [ 4 ] , and dialogue systems [ 5 ] . Despite its empirical success , RL with function approximation raises a new series of theoretical challenges when comparing to the classic tabular RL : ( 1 ) generalization , to generalize knowledge from the visited states to the unvisited states due to the enormous state space . ( 2 ) limited expressiveness , to handle the complicated issues where true value functions or intermediate steps computed in the algorithm can be functions outside the prespecified function class . ( 3 ) exploration , to address the tradeoff between exploration and exploitation when above challenges are present . Consequently , most existing theoretical results on efficient RL with function approximation rely on relatively strong structural assumptions . For instance , many require that the MDP admits a linear approximation [ 6–8 ] , or that the model is precisely Linear Quadratic Regulator ( LQR ) [ 9–11 ] . Most of these structural assumptions rarely hold in practical applications . This naturally leads to one of the most fundamental questions in RL . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . What are the minimal structural assumptions that empower sample-efficient RL ? We advance our understanding of this grand question via the following two steps : ( 1 ) identify a rich class of RL problems ( with weak structural assumptions ) that cover many practical applications of interests ; ( 2 ) design sample-efficient algorithms that provably learn any RL problem in this class . The attempts to find weak or minimal structural assumptions that allow statistical learning can be traced in supervised learning where VC dimension [ 12 ] or Rademacher complexity [ 13 ] is proposed , or in online learning where Littlestone dimension [ 14 ] or sequential Rademacher complexity [ 15 ] is developed . In the area of reinforcement learning , there are two intriguing lines of recent works that have made significant progress in this direction . To begin with , Jiang et al . [ 1 ] introduces a generic complexity notion—Bellman rank , which can be proved small for many RL problems including linear MDPs [ 7 ] , reactive POMDPs [ 16 ] , etc . [ 1 ] further propose an hypothesis elimination-based algorithm— OLIVE for sample-efficient learning of problems with low Bellman rank . On the other hand , recent work by Wang et al . [ 17 ] considers general function approximation with low Eluder dimension [ 18 ] , and designs a UCB-style algorithm with regret guarantee . Noticeably , generalized linear MDPs [ 6 ] and kernel MDPs ( see Appendix C ) are subclasses of low Eluder dimension problems , but not low Bellman rank . In this paper , we make the following three contributions . • We introduce a new complexity measure for RL—Bellman Eluder ( BE ) dimension . We prove that the family of RL problems of low BE dimension is remarkably rich , which subsumes both low Bellman rank problems and low Eluder dimension problems—two arguably most generic tractable function classes so far in the literature ( see Figure 1 ) . The family of low BE dimension further includes new problems such as kernel reactive POMDPs ( see Appendix C ) which were not known to be sample-efficiently learnable . • We design a new optimization-based algorithm—GOLF , which provably learns nearoptimal policies of low BE dimension problems in a number of samples that is polynomial in all relevant parameters , but independent of the size of state-action space . Our regret or sample complexity guarantees match [ 8 ] which is minimax optimal when specified to the linear setting . Our rates further improve upon [ 1 , 17 ] in low Bellman rank and low Eluder dimension settings , respectively . • We reanalyze the hypothesis elimination based algorithm—OLIVE proposed in [ 1 ] . We show it can also learn RL problems with low BE dimension sample-efficiently , under slightly weaker assumptions but with worse sample complexity comparing to GOLF . 1.1 Related works . This section reviews prior theoretical works on RL , under Markov Decision Process ( MDP ) models . 1The family of low Bellman rank problems and low Bellman Eluder dimension problems include both Q-type and V-type variants . Please refer to Section 3.1 and Appendix B for more details . We remark that there has been a long line of research on function approximation in the batch RL setting [ see , e.g. , 19–22 ] . In this setting , agents are provided with exploratory data or simulator , so that they do not need to explicitly address the challenge of exploration . In this paper , we do not make such assumption , and attack the exploration problem directly . In the following we focus exclusively on the RL results in the general setting where exploration is required . Tabular RL . Tabular RL concerns MDPs with a small number of states and actions , which has been thoroughly studied in recent years [ see , e.g. , 23–30 ] . In the episodic setting with non-stationary dynamics , the best regret bound Õ ( √ H2|S||A|T ) is achieved by both model-based [ 27 ] and modelfree [ 30 ] algorithms . Moreover , the bound is proved to be minimax-optimal [ 29 , 31 ] . This minimax bound suggests that when the state-action space is enormous , RL is information-theoretically hard without further structural assumptions . RL with linear function approximation . A recent line of work studies RL with linear function approximation [ see , e.g. , 7 , 6 , 32 , 8 , 33–36 ] These papers assume certain completeness conditions , as well as the optimal value function can be well approximated by linear functions . Under one formulation of linear approximation , the minimax regret bound Õ ( d √ T ) is achieved by algorithm ELEANOR [ 8 ] , where d is the ambient dimension of the feature space . RL with general function approximation . Beyond the linear setting , there is a flurry line of research studying RL with general function approximation [ see , e.g. , 37 , 1 , 36 , 38 , 17 , 39 , 40 ] . Among them , [ 1 ] and [ 17 ] are the closest to our work . Jiang et al . [ 1 ] propose a complexity measure named Bellman rank and design an algorithm OLIVE with PAC guarantees for problems with low Bellman rank . We note that low Bellman rank is a special case of low BE dimension . When specialized to the low Bellman rank setting , our result for OLIVE exactly matches the guarantee in [ 1 ] . Our result for GOLF requires an additional completeness assumption , but provides sharper sample complexity guarantee . Wang et al . [ 17 ] propose a UCB-type algorithm with a regret guarantee under the assumption that the function class has a low eluder dimension . Again , we will show that low Eluder dimension is a special case of low BE dimension . Comparing to [ 17 ] , our algorithm GOLF works under a weaker completeness assumption , with a better regret guarantee . Finally , we remark that the algorithms proposed in [ 1 , 33 , 41 ] and this paper are all computationally inefficient in general . We notice several existing works [ e.g. , 7 , 17 ] can be computationally efficient given suitable regression oracles but they require stronger representation conditions and also achieve worse regret guarantees . Relation to bilinear classes Concurrent to this work , Du et al . [ 41 ] propose a new general tractable class of RL problems—bilinear class with low effective dimension ( also known as low critical information gain in Du et al . [ 41 ] ) . We comment on the similarities and differences between two works as follows . In terms of algorithms , both Algorithm 2 in this paper and the algorithm proposed in Du et al . [ 41 ] are based on OLIVE originally proposed in Jiang et al . [ 1 ] . The two algorithms share similar guarantees in terms of assumptions and complexity results . More importantly , our work further develops a new type of algorithm for general function approximation—GOLF , a natural and clean algorithm which can be viewed as an optimistic version of classical algorithm—Fitted Q-Iteration [ 42 ] . GOLF gives much sharper sample complexity guarantees compared to [ 41 ] for various settings , and is minimaxoptimal when applied to the linear setting [ 8 ] . In terms of richness of new classes identified , it depends on ( a ) what structure of MDP the complexity measures are applied to , and ( b ) what complexity measures are used . For ( a ) , BE dimension applies to the Bellman error , while the bilinear class allows general surrogate losses of the Bellman error . For ( b ) , this paper uses Eluder dimension while Du et al . [ 41 ] uses effective dimension . It can be shown that low effective dimension always implies low Eluder dimension ( see Appendix C.2 ) . In short , Du et al . [ 41 ] is more general in ( a ) , while our work is more general in ( b ) . As a result , neither work fully captures the other . In particular , our BE framework covers a majority of the examples identified in Du et al . [ 41 ] including low occupancy complexity , linear Q ? /V ? , Q ? state aggregation , feature selection/FLAMBE . Nevertheless , our work can not address examples with model-based function approximation ( e.g. , low witness rank [ 36 ] ) while [ 41 ] can . On the other hand , Du et al . [ 41 ] can not address the class of RL problems with low Eluder dimension [ 17 ] while our work can . Moreover , for several classes of RL problems that both works cover , our complexity measure is sharper . For example , in the setting of function approximation with generalized linear functions , the BE dimension is Õ ( d ) where d is the ambient dimension of the feature vectors , while the effective dimension under the generalized bilinear framework of Du et al . [ 41 ] is at least Ω̃ ( d2 ) . 1.2 Paper organization . We present preliminaries in Section 2 , the definition of Bellman-Eluder dimension as well as its relations to existing complexity notions in Section 3 . We present the results for algorithm GOLF in Section 4 , and conclude in Section 5 . Due to space limit , we postpone the results for algorithm OLIVE to Appendix A . Further discussions on Q-type versus V-type variants of BE dimension , as well as the practical examples will be provided in Appendix B and C. All the proofs are postponed to the appendix . 2 Preliminaries . We consider episodic Markov Decision Process ( MDP ) , denoted byM = ( S , A , H , P , r ) , where S is the state space , A is the action space , H is the number of steps in each episode , P = { Ph } h∈ [ H ] is the collection of transition measures with Ph ( s′ | s , a ) equal to the probability of transiting to s′ after taking action a at state s at the hth step , and r = { rh } h∈ [ H ] is the collection of reward functions with rh ( s , a ) equal to the deterministic reward received after taking action a at state s at the hth step . 2 Throughout this paper , we assume reward is non-negative , and ∑H h=1 rh ( sh , ah ) ≤ 1 for all possible sequence ( s1 , a1 , . . . , sH , aH ) . In each episode , the agent starts at a fixed initial state s1 . Then , at each step h ∈ [ H ] , the agent observes its current state sh , takes action ah , receives reward rh ( sh , ah ) , and causes the environment to transit to sh+1 ∼ Ph ( · | sh , ah ) . Without loss of generality , we assume there is a terminating state send which the environment will always transit to at step H + 1 , and the episode terminates when send is reached . Policy and value functions A ( deterministic ) policy π is a collection of H functions { πh : S → A } Hh=1 . We denote V πh : S → R as the value function at step h for policy π , so that V πh ( s ) gives the expected sum of the remaining rewards received under policy π , starting from sh = s , till the end of the episode . In symbol , V πh ( s ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s ] . Similarly , we denote Qπh : S ×A → R as the Q-value function at step h for policy π , where Qπh ( s , a ) : = Eπ [ H∑ h′=h rh′ ( sh′ , ah′ ) | sh = s , ah = a ] . There exists an optimal policy π ? , which gives the optimal value function for all states [ 43 ] , in the sense , V π ? h ( s ) = supπ V π h ( s ) for all h ∈ [ H ] and s ∈ S . For notational simplicity , we abbreviate V π ? as V ? . We similarly define the optimal Q-value function as Q ? . Recall that Q ? satisfies the Bellman optimality equation : Q ? h ( s , a ) = ( ThQ ? h+1 ) ( s , a ) : = rh ( s , a ) + Es′∼Ph ( ·|s , a ) max a′∈A Q ? h+1 ( s ′ , a′ ) . ( 1 ) for all ( s , a , h ) ∈ S ×A× [ H ] . We also call Th the Bellman operator at step h. 2We study deterministic reward for notational simplicity . Our results readily generalize to random rewards . -optimality and regret We say a policy π is -optimal if V π1 ( s1 ) ≥ V ? 1 ( s1 ) − . Suppose an agent interacts with the environment for K episodes . Denote by πk the policy the agent follows in episode k ∈ [ K ] . The ( accumulative ) regret is defined as Reg ( K ) : = K∑ k=1 [ V ? 1 ( s1 ) − V π k 1 ( s1 ) ] . The objective of reinforcement learning is to find an -optimal policy within a small number of interactions or to achieve sublinear regret .

Summary: The paper provides a new set of assumptions under which,  given a realizable Q-function / V-function class, one can efficiently find an almost optimal policy. They define the notion of bellman eluder (BE) dimension and provide regret bounds for RL problems for which the BE dimension is bounded. In particular, the regret bounds do not depend on the size of the state space |S|, and only scale with BE dimension, H and log(|F|), where F denotes the given realizable value function class. Their algorithm is based on optimism with local confidence intervals.   They also show that almost all the RL settings where we know how to get a sample efficient algorithm have small BE dimensions. In addition to this, they also provide an example setting (kernel reactive POMDPs) where BE dimension is small and hence their algorithm is sample efficient, but we did not know any algorithm for this setting before the paper.