Agnostic Reinforcement Learning with Low-Rank MDPs and Rich Observations

( H4dK3d log |⇧| ) / '' 2 where H is the length of episodes , K is the number of actions and `` > 0 is the desired sub-optimality . We also provide a nearly matching lower bound for this agnostic setting that shows that the exponential dependence on rank is unavoidable , without further assumptions . 1 Introduction . Reinforcement Learning ( RL ) has achieved several remarkable empirical successes in the last decade , which include playing Atari 2600 video games at superhuman levels [ Mnih et al. , 2015 ] , AlphaGo or AlphaGo Zero surpassing champions in Go [ Silver et al. , 2018 ] , AlphaStar ’ s victory over top-ranked professional players in StarCraft [ Vinyals et al. , 2019 ] , or practical self-driving cars . These applications all correspond to the setting of rich observations , where the state space is very large and where observations may be images , text or audio data . In contrast , most provably efficient RL algorithms are still limited to the classical tabular setting where the state space is small [ Kearns and Singh , 2002 , Brafman and Tennenholtz , 2002 , Azar et al. , 2017 , Dann et al. , 2019 ] and do not scale to the rich observation setting . To derive guarantees for large state spaces , much of the existing work in RL theory relies on a realizability and a low-rank assumption [ Krishnamurthy et al. , 2016 , Jiang et al. , 2017 , Dann et al. , 2018 , Du et al. , 2019a , Misra et al. , 2020 , Agarwal et al. , 2020a ] . Different notions of rank have been adopted in the literature , including that of a low-rank transition matrix [ Jin and Luo , 2019 ] , a low Bellman rank [ Jiang et al. , 2017 ] , Wittness rank [ Sun et al. , 2019 ] , Eluder dimension [ Osband and Van Roy , 2014 ] , Bellman-Eluder dimension [ Jin et al. , 2021 ] , or bilinear classes [ Du et al. , 2021 ] . These studies also show that learning without any such structural assumptions requires a sample size that grows exponentially in the time horizon of the MDP [ Dann and Brunskill , 2015 , Krishnamurthy et al. , 2016 , Du et al. , 2019b ] . The choice of the most suitable and most general notion of rank is the topic of much active research in RL theory . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In comparison , the realizability assumption has received much less attention . This is the strong premise that the optimal value function belongs to the class of functions considered , which typically does not hold in practice . In many applications , the optimal value function Q ? is highly complex and we can not hope to accurately approximate it in the absence of some strong domain knowledge . Can we relax the realizability assumption in RL ? Value-function realizability can be viewed as the analogue of the PAC-realizability assumption in classical statistical learning theory . That assumption rarely holds , which has motivated the development and analysis of numerous algorithms for the agnostic PAC learnability model . Those algorithms provably learn to predict as well as the best predictor in the given function class , independently of whether the Bayes predictor belongs to the class . The counterparts of such results in reinforcement learning are mostly unavailable , which prompts the following question : Can we derive a theory of agnostic reinforcement learning ? Here , we precisely initiate that study . In this agnostic setting , we adopt common structural assumptions , e.g . small rank of the transition matrix , but seek to learn to perform as well as the best policy in the given policy class , independently of how close this class represents the Bellman-optimal policy . Specifically , we study agnostic Reinforcement Learning ( RL ) with a fixed policy class ⇧ in the episodic MDPs with rich observations . Provably sample-efficient algorithms for agnostic RL would be highly desirable but it is still unknown to what degree learning is possible in this setting . Our work provides new insights about learnability with structural assumptions in the absence of ( approximate ) realizability in RL . Agnostic RL without any additional structural assumptions has been considered in the past . By evaluating each policy in the class individually , one can easily obtain a sample complexity upper bound of O ( |⇧|/ '' 2 ) . Kearns et al . [ 2000 ] also showed that an upper bound of ( KH log |⇧| ) / '' 2 is possible , where K is the number of actions and H is the time horizon . However , as discussed in prior work such as [ Krishnamurthy et al. , 2016 ] , bounds of this form are rather unsatisfactory as one of them admits a linear dependence on the size of the function class , which is prohibitively large , and the other one admits an exponential dependence on the length of the episodes H , which is typically long . Using existing constructions , one can derive a lower bound on the sample complexity of the form min { |⇧| , KH } / '' 2 in the rich observation setting . This further justifies our adoption of rank as a natural structural assumption . Our Contributions : The following highlights our main technical contributions , where d is the rank of the state transition matrix induced by any policy in the class ⇧ , and is assumed to be small . • We provide a uniform exploration-based algorithm that can find an `` -sub-optimal policy w.r.t . the policy class ⇧ after collecting O ( ( HK/d ) 4d log ( d|⇧| ) / '' 2 ) samples in the MDP . This bound shows that one can achieve a sample complexity that is polynomial in both H and log |⇧| , while being exponential in rank d only ( which we assume is small ) . In addition to the sample complexity bound obtained here , the algorithmic techniques itself might be of independent interest and useful beyond this work . The algorithm is based on showing that for every policy , the expected rewards follows an autoregressive model of degree d. Thus obtaining samples of O ( d ) -length paths for a policy we show that one can extrapolate expected rewards for the entire episode . • We complement this upper bound with a sample complexity lower bound of ⌦ ( H/d ) d/2 / '' 2 ( when K = 2 ) , thereby showing that the HO ( d ) term in the upper bound is unavoidable . The lower bound also highlights which structures in the policy class induce the HO ( d ) terms thus shedding light on what structural assumptions could help alleviate the exponential dependence on the rank . • Finally , we seek to improve upon the Hd term and provide an adaptive algorithm that admits a sample complexity that depends on the eigenspectrum of the transition matrix of the MDP ; while in the worst case that bound matches the above one , it provides a significantly better guarantee when the eigenspectrum is more favorable . However , we view the main benefit of our work to be the initiation of the study of agnostic reinforcement learning and the presentation of an in-depth analysis of a natural structural assumption within that setting . This can form the basis for future research in this domain with alternative and perhaps more favorable rank-type assumptions . 2 Problem Setup . We consider an episodic Markov decision process with episode length H 2 N , observation space X and action space A : = { 1 , . . . , K } . For ease of exposition , we assume that the observation space X is finite ( albeit extremely large ) , but our results can be readily extended to countably infinite and possibly uncountably infinite observation spaces . Each episode is a sequence ( ( x1 , a1 , r1 ) , ( x2 , a2 , r2 ) , . . . , ( xH , aH , rH ) ) 2 ( X ⇥A⇥ R ) H , where the initial observation x0 is drawn from the initial distribution µ0 2 ( X ) , the actions are generated by the learning agent and all the following observations are sampled from the transition kernel xh+1 ⇠ T ( ·|xh , ah ) 2 ( X ) that depends on the previous observation and action . Finally , the rewards rh are drawn from a subGaussian distribution with mean r ( xh , ah ) where r : X ⇥ A 7 ! [ 0 , 1 ] . The learning agent does not know the transition kernel T , the initial distribution µ0 , or the reward function r. In our setting , the agent is given a policy class ⇧ ✓ { X 7 ! ( A ) } consisting of policies that map observations to distributions over the actions A . For any policy ⇡ 2 ⇧ , we denote by T⇡ 2 RX⇥X , the transition matrix induced by ⇡ , i.e. , for any x , x0 2 X , [ T⇡ ] ( x0 , x ) = Ea⇠⇡ ( x ) T ( x0 | x , a ) . Assumption 1 ( Low-rank transition ) . There exists d 2 N such that rank ( T⇡ ) d for all ⇡ 2 ⇧ . For the main part of the paper , we assume that the learner knows the value of d , but later extend our results to the case where d is unknown . We define ⇡ = ( ⇡1 , . . . , ⇡d ) > 2 Cd to denote the eigenvalues of the transition matrix T⇡ with rank at most d. Without any loss of generality , assume that | ⇡1 | | ⇡2 | . . . | ⇡d | . We denote by P⇡ the distribution over episodes when following policy ⇡ and by E⇡ its expectation . We call the expected rewards obtained at time h by policy ⇡ expected policy rewards : R⇡h : = E⇡ [ r ( xh , ah ) ] . ( 1 ) The value function of ⇡ at time h is given by V ⇡h ( x ) = E ⇡ hPH h0=h r ( xh0 , ah0 ) | xh0 = x i . Further , when using V ⇡ without a time index and arguments , we mean the value or expected H-step return : V ⇡ : = E [ V ⇡0 ( x0 ) ] = E⇡ h HX h=1 r ( xh , ah ) i = HX h=1 R⇡h , ( 2 ) the value function averaged over initial observations . Learning objective . The goal of the learner is to return a policy e⇡ , after interacting with the MDP for n episodes of length H , such that the value of the returned policy is as close as possible to the value of the best policy in ⇧ , that is , V e⇡ max ⇡2⇧ V ⇡ `` , where the error `` is a small as possible and may depend on n , the policy class ⇧ and the MDP . 3 Related work . We give a brief overview of the most closely related works here , and defer a more detailed discussion to Appendix A . Recently there has been great interest in designing RL algorithms with general function approximation [ Jiang et al. , 2017 , Dann et al. , 2018 , Sun et al. , 2019 , Du et al. , 2019a , Wang et al. , 2020 , Du et al. , 2021 ] . In particular , Jiang et al . [ 2017 ] introduced the notion of Bellman rank , a measure of complexity that depends on the underlying environment and the value function class F , and provide statistically efficient algorithms for learning problems for which Bellman rank is bounded . This was later extended to model-based algorithms by Sun et al . [ 2019 ] . While these algorithms work across a variety of problem settings , their sample complexity scales with log ( |F| ) . Furthermore , these algorithms also require the optimal value function f⇤ to be realized in F . In our work , we do not assume that the learner has access to a value function class F . In fact , given a value function class F , we can construct the policy class ⇧F that corresponds to greedy policies induced by the class F . However , given just a policy class ⇧ , one can not construct a value function class , without additional knowledge of the underlying dynamics . Our Assumption 1 implies that for any policy ⇡ 2 ⇧ , the transition dynamics exhibits a low-rank decomposition with dimension d , that is T⇡ ( x0|x ) = h ⇡ ( x ) , ⇡ ( x0 ) i , for some d-dimensional feature maps ⇡ , ⇡ : X 7 ! Rd . Low rank MDPs and linear transition models have recently gained a lot of attention in the RL literature [ Yang and Wang , 2020 , Jin et al. , 2020 , Modi et al. , 2020 , Wang et al. , 2021a ] . The works most closely related to our setup are those of Jin et al . [ 2020 ] and Yang and Wang [ 2020 ] , who give algorithms to find an optimal policy in low rank MDPs with known feature maps . Similarly , the other algorithms also assume that the learner either observes the feature ( x ) , or the feature ( x ) . Agarwal et al . [ 2020a ] and Modi et al . [ 2021 ] learn under weaker assumptions and only assume that the learner has access to a function class that realizes . However , in our setup , the learner neither observes the features ⇡ , ⇡ nor has access to a realizable function class for them , and thus these methods are not applicable . Several of the works mentioned above recognize the issue of a strict realizability assumption and provide results only when the function class contains a good approximation to the optimal value function of model . However , the goal in our agnostic setting is more ambitious . We would like to find a policy that can compete with the best policy in the given class ⇧ , independent of how close the best return in the class max⇡2⇧ V ⇡ is to the return of the optimal policy V ⇡ ? for that MDP . There have also been several approaches for provably efficient RL with non-parametric function classes [ Yang et al. , 2020 , Long et al. , 2021 , Shah et al. , 2020 ] . However , these approaches still aim to learn the optimal value function and their regret necessarily scales with the complexity of the optimal value function in the RKHS which can be very high . Instead , in our agnostic setting we would like to be able to quickly identify the best policy from the given policy class with low complexity containing a good but not necessarily optimal policy . Finally , there are a few prior works in agnostic RL that directly compete against a policy class [ Abbasi-Yadkori et al. , 2013 , Azar et al. , 2013 ] . However , they either make strong assumptions on the feedback model , e.g . AbbasiYadkori et al . [ 2013 ] assumes that the the agent fully observed the current transition kernel and reward instead of just a sample from it , or the provided bound [ Azar et al. , 2013 ] scales linearly with the size of the policy class , instead of logarithmically .

Summary: This paper studies RL with linear function approximation when transition is low rank, but the features are unknown. The goal is to compete against a policy class that may not contain any near-optimal policy. The paper proposes an algorithm based on autoregression and shows a sample complexity bound that is exponential in the rank; a corresponding lower bound suggests that this dependence is unavoidable.