Online learning in MDPs with linear function approximation and bandit feedback. 

( √ dHT ) , where H is the number of steps in each episode and d is the dimensionality of the feature map . 1 Introduction . We study the problem of online learning in episodic Markov Decision Processes ( MDP ) , modelling a sequential decision making problem where the interaction between a learner and its environment is divided into T episodes of fixed length H . At each time step of the episode , the learner observes the current state of the environment , chooses one of the K available actions , and earns a reward . Consequently , the state of the environment changes according to the transition function of the underlying MDP , as a function of the previous state and the action taken by the learner . A key distinguishing feature of our setting is that we assume that the reward function can change arbitrarily between episodes , and the learner only has access to bandit feedback : instead of being able to observe the reward function at the end of the episode , the learner only gets to observe the rewards that it actually received . As traditional in this line of work , we aim to design algorithms for the learner with theoretical guarantees on her regret , which is the difference between the total reward accumulated by the learner and the total reward of the best stationary policy fixed in hindsight . Unlike most previous work on this problem , we allow the state space to be very large and aim to prove performance guarantees that do not depend on the size of the state space , bringing theory one step closer to practical scenarios where assuming finite state spaces is unrealistic . To address the challenge of learning in large state spaces , we adopt the classic RL technique of using linear function approximation and suppose that we have access to a relatively low-dimensional feature map that can be used to represent policies and value functions . We will assume that the feature map is expressive enough so that all action-value functions can be expressed as linear functions of the features , and that the learner has full knowledge of the transition function of the MDP . Our main contribution is designing a computationally efficient algorithm called ONLINE Q-REPS , and prove that in the setting described above , its regret is at most O ( √ dHTD ( µ∗‖µ0 ) ) , where d is the dimensionality of the feature map and D ( µ∗‖µ0 ) is the relative entropy between the state-action distribution µ∗ induced by the optimal policy and an initial distribution µ0 given as input to the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithm . Notably , our results do not require the likelihood ratio between these distributions to be uniformly bounded , and the bound shows no dependence on the eigenvalues of the feature covariance matrices . Our algorithm itself requires solving a d2-dimensional convex optimization problem at the beginning of each episode , which can be solved to arbitrary precision ε in time polynomial in d and 1/ε , independently of the size of the state-action space . Our work fits into a long line of research considering online learning in Markov decision processes . The problem of regret minimization in stationary MDPs with a fixed reward function has been studied extensively since the work of Burnetas and Katehakis [ 6 ] , Auer and Ortner [ 2 ] , Tewari and Bartlett [ 31 ] , Jaksch et al . [ 14 ] , with several important advances made in the past decade [ 9 , 10 , 4 , 13 , 15 ] . While most of these works considered small finite state spaces , the same techniques have been very recently extended to accommodate infinite state spaces under the assumption of realizable function approximation by Jin et al . [ 17 ] and Yang and Wang [ 33 ] . In particular , the notion of linear MDPs introduced by Jin et al . [ 17 ] has become a standard model for linear function approximation and has been used in several recent works ( e.g. , 22 , 32 , 1 ) . Even more relevant is the line of work considering adversarial rewards , initiated by Even-Dar et al . [ 12 ] , who consider online learning in continuing MDPs with full feedback about the rewards . They proposed a MDP-E algorithm , that achieves O ( τ2 √ T logK ) regret , where τ is an upper bound on the mixing time of the MDP . Later , Neu et al . [ 25 ] proposed an algorithm which guarantees Õ ( √ τ3KT/α ) regret with bandit feedback , essentially assuming that all states are reachable with probability α > 0 under all policies . In our work , we focus on episodic MDPs with a fixed episode length H . The setting was first considered in the bandit setting by Neu et al . [ 23 ] , who proposed an algorithm with a regret bound of O ( H2 √ TK/α ) . Although the number of states does not appear explicitly in the bound , the regret scales at least linearly with the size of the state space X , since |X | ≤ H/α . Later work by Zimin and Neu [ 35 ] , Dick et al . [ 11 ] eliminated the dependence on α and proposed an algorithm achieving Õ ( √ TH|X |K ) regret . Regret bounds for the full-information case without prior knowledge of the MDP were achieved by Neu et al . [ 24 ] and Rosenberg and Mansour [ 30 ] , of order Õ ( H|X |K √ T ) and Õ ( H|X | √ KT ) , respectively . These results were recently extended to handle bandit feedback about the rewards by Jin et al . [ 16 ] , ultimately resulting in a regret bound of Õ ( H|X | √ KT ) . As apparent from the above discussion , all work on online learning in MDPs with adversarial rewards considers finite state spaces . The only exception we are aware of is the recent work of Cai et al . [ 7 ] , whose algorithm OPPO is guaranteed to achieve Õ ( √ d3H3T ) , assuming that the learner has access to d-dimensional features that can perfectly represent all action-value functions . While Cai , Yang , Jin , and Wang [ 7 ] remarkably assumed no prior knowledge of the MDP parameters , their guarantees are only achieved in the full-information case . This is to be contrasted with our results that are achieved for the much more restrictive bandit setting , albeit with the stronger assumption of having full knowledge of the underlying MDP , as required by virtually all prior work in the bandit setting , with the exception of Jin et al . [ 16 ] . Our results are made possible by a careful combination of recently proposed techniques for contextual bandit problems and optimal control in Markov decision processes . In particular , a core component of our algorithm is a regularized linear programming formulation of optimal control in MDPs due to Bas-Serrano et al . [ 5 ] , which allows us to reduce the task of computing near-optimal policies in linear MDPs to a low-dimensional convex optimization problem . A similar algorithm design has been previously used for tabular MDPs by Zimin and Neu [ 35 ] , Dick et al . [ 11 ] , with the purpose of removing factors of 1/α from the previous state-of-the-art bounds of Neu et al . [ 23 ] . Analogously to this improvement , our methodology enables us to make strong assumptions on problem-dependent constants like likelihood ratios between µ∗ and µ0 or eigenvalues of the feature covariance matrices . Another important building block of our method is a version of the recently proposed Matrix Geometric Resampling procedure of Neu and Olkhovskaya [ 21 ] that enables us to efficiently estimate the reward functions . Incorporating these estimators in the algorithmic template of Bas-Serrano et al . [ 5 ] is far from straightforward and requires several subtle adjustments . Notation . We use 〈· , ·〉 to denote inner products in Euclidean space and by ‖·‖ we denote the Euclidean norm for vectors and the operator norm for matrices . For a symmetric positive definite matrix A , we use λmin ( A ) to denote its smallest eigenvalue . We write tr ( A ) for the trace of a matrix A and use A < 0 to denote that an operator A is positive semi-definite , and we use A < B to denote A−B < 0 . For a d-dimensional vector v , we denote the corresponding d× d diagonal matrix by diag ( v ) . For a positive integer N , we use [ N ] to denote the set of positive integers { 1 , 2 , . . . , N } . Finally , we will denote the set of all probability distributions over any set X by ∆X . 2 Preliminaries . An episodic Markovian Decision Process ( MDP ) , denoted by M = ( X , A , H , P , r ) is defined by a state space X , action space A , episode length H ∈ Z+ , transition function P : X ×A → ∆X and a reward function r : X ×A → [ 0 , 1 ] . For convenience , we will assume that both X and A are finite sets , although we allow the state space X to be arbitrarily large . Without significant loss of generality , we will assume that the set of available actions is the same A in each state , with cardinality |A| = K. Furthermore , without any loss of generality , we will assume that the MDP has a layered structure , satisfying the following conditions : • The state set X can be decomposed into H disjoint sets : X = ∪Hh=1Xh , • X1 = { x1 } and XH = { xH } are singletons , • transitions are only possible between consecutive layers , that is , for any xh ∈ Xh , the distribution P ( ·|x , a ) is supported on Xh+1 for all a and h ∈ [ H − 1 ] . These assumptions are common in the related literature ( e.g. , 23 , 35 , 30 ) and are not essential to our analysis ; their primary role is simplifying our notation . In the present paper , we consider an online learning problem where the learner interacts with its environment in a sequence of episodes t = 1 , 2 , . . . , T , facing a different reward functions rt,1 , . . . rt , H+1 selected by a ( possibly adaptive ) adversary at the beginning of each episode t. Oblivious to the reward function chosen by the adversary , the learner starts interacting with the MDP in each episode from the initial state Xt,1 = x1 . At each consecutive step h ∈ [ H − 1 ] within the episode , the learner observes the state Xt , h , picks an action At , h and observes the reward rt , h ( Xt , h , At , h ) . Then , unless h = H , the learner moves to the next state Xt , h+1 , which is generated from the distribution P ( ·|Xt , h , At , h ) . At the end of step H , the episode terminates and a new one begins . The aim of the learner is to select its actions so that the cumulative sum of rewards is as large as possible . Our algorithm and analysis will make use of the concept of ( stationary stochastic ) policies π : X → ∆A . A policy π prescribes a behaviour rule to the learner by assigning probability π ( a|x ) to taking action a at state x . Let τπ = ( ( X1 , A1 ) , ( X2 , A2 ) , . . . , ( XH , AH ) ) be a trajectory generated by following the policy π through the MDP . Then , for any xh ∈ Xh , ah ∈ A we define the occupancy measure µπh ( x , a ) = Pπ [ ( x , a ) ∈ τπ ] . We will refer to the collection of these distributions across all layers h as the occupancy measure induced by π and denote it as µπ = ( µπ1 , µ π 2 , . . . , µ π H ) . We will denote the set of all valid occupancy measures by U and note that this is a convex set , such that for every element µ ∈ U the following set of linear constraints is satisfied : ∑ a∈A µh+1 ( x , a ) = ∑ x′ , a′∈Xh×A P ( x|x′ , a′ ) µh ( x′ , a′ ) , ∀x ∈ Xh+1 , h ∈ [ H − 1 ] , ( 1 ) as well as ∑ a µ1 ( x1 , a ) = 1 . From every valid occupancy measure µ , a stationary stochastic policy π = π1 , . . . , πH−1 can be derived as πµ , h ( a|x ) = µh ( x , a ) / ∑ a′ µh ( x , a ′ ) . For each h , introducing the linear operators E and P through their action on a set state-action distribution uh as ( ETuh ) ( x ) = ∑ a∈A uh ( x , a ) and ( P T huh ) ( x ) = ∑ x′ , a′∈Xh , A P ( x|x ′ , a′ ) uh ( x ′ , a′ ) , the constraints can be simply written as ETµh+1 = P Thµh for each h. We will use the inner product notation for the sum over the set of states and actions : 〈µh , rh〉 = ∑ ( x , a ) ∈ ( Xh×A ) µh ( x , a ) rt , h ( x , a ) . Using this notation , we formulate our objective as selecting a sequence of policies πt for each episode t in a way that it minimizes the total expected regret defined as RT = sup π∗ T∑ t=1 H∑ h=1 ( Eπ∗ [ rt , h ( X∗h , A∗h ) ] − Eπt [ rt ( Xt , h , At , h ) ] ) = sup µ∗∈U T∑ t=1 H∑ h=1 〈µ∗h − µ πt h , rt , h〉 , where the notations Eπ∗ [ · ] and Eπt [ · ] emphasize that the state-action trajectories are generated by following policies π∗ and πt , respectively . As the above expression suggests , we can reformulate our online learning problem as an instance of online linear optimization where in each episode t , the learner selects an occupancy measure µt ∈ U ( with µt = µπt ) and gains reward ∑H h=1〈µt , h , rt , h〉 . Intuitively , the regret measures the gap between the total reward gained by the learner and that of the best stationary policy fixed in hindsight , with full knowledge of the sequence of rewards chosen by the adversary . This performance measure is standard in the related literature on online learning in MDPs , see , for example Neu et al . [ 23 ] , Zimin and Neu [ 35 ] , Neu et al . [ 24 ] , Rosenberg and Mansour [ 30 ] , Cai et al . [ 7 ] . In this paper , we focus on MDPs with potentially enormous state spaces , which makes it difficult to design computationally tractable algorithms with nontrivial guarantees , unless we make some assumptions . We particularly focus on the classic technique of relying on linear function approximation and assuming that the reward functions occurring during the learning process can be written as a linear function of a low-dimensional feature map . We specify the form of function approximation and the conditions our analysis requires as follows : Assumption 1 ( Linear MDP with adversarial rewards ) . There exists a feature map ϕ : X ×A → Rd and a collection of d signed measures m = ( m1 , . . . , md ) on X , such that for any ( x , a ) ∈ X ×A the transition function can be written as P ( ·|x , a ) = 〈m ( · ) , ϕ ( x , a ) 〉 . Furthermore , the reward function chosen by the adversary in each episode t can be written as rt , h ( x , a ) = 〈θt , h , ϕ ( x , a ) 〉 for some θt , h ∈ Rd . We assume that the features and the parameter vectors satisfy ‖ϕ ( x , a ) ‖ ≤ σ and that the first coordinate ϕ1 ( x , a ) = 1 for all ( x , a ) ∈ X ×A . Also we assume that ‖θt , h‖ ≤ R. Online learning under this assumption , but with a fixed reward function , has received substantial attention in the recent literature , particularly since the work of Jin et al . [ 17 ] who popularized the term “ Linear MDP ” to refer to this class of MDPs . This has quickly become a common assumption for studying reinforcement learning algorithms ( Cai et al . [ 7 ] , Jin et al . [ 17 ] , Neu and Pike-Burke [ 22 ] , Agarwal et al . [ 1 ] ) . This is also a special case of factored linear models ( Yao et al . [ 34 ] , Pires and Szepesvári [ 29 ] ) . Linear MDPs come with several attractive properties that allow efficient optimization and learning . In this work , we will exploit the useful property shown by Neu and Pike-Burke [ 22 ] and Bas-Serrano et al . [ 5 ] that all occupancy measures in a linear MDP can be seen to satisfy a relaxed version of the constraints in Equation ( 1 ) . Specifically , for all h , defining the feature matrix Φh ∈ R ( Xh×A ) ×d with its action on the distribution u as ΦThu = ∑ x , a∈Xh , A uh ( x , a ) ϕ ( x , a ) , we define UΦ as the set of state-action distributions ( µ , u ) = ( ( µ1 , . . . , µH ) , ( u1 , . . . , uH ) ) satisfying the following constraints : ETuh+1 = P T hµh ( ∀h ) , ΦThuh = ΦThµh ( ∀h ) , ETu1 = 1 . ( 2 ) It is easy to see that for all feasible ( µ , u ) pairs , u satisfies the original constraints ( 1 ) if the MDP satisfies Assumption 1 : since the transition operator can be written as Ph = ΦhMh for some matrix Mh . In this case , we clearly have ETuh+1 = P T hµh = M T hΦ T hµh = M T hΦ T huh = P T huh , ( 3 ) showing that any feasible u is indeed a valid occupancy measure . Furthermore , due to linearity of the rewards in Φ , we also have 〈uh , rt , h〉 = 〈µh , rt , h〉 for all feasible ( µ , u ) ∈ UΦ . While the number of variables and constraints in Equation ( 2 ) is still very large , it has been recently shown that approximate linear optimization over this set can be performed tractably [ 22 , 5 ] . Our own algorithm design described in the next section will heavily build on these recent results .

Summary: This paper studies the problem of online learning in a finite horizon MDP with known transition function and adversarially changing rewards at each episode and only bandit feedback from those rewards. In particular, the authors consider the case when the state space is large warranting linear function approximation with the assumption that the MDP is a linear MDP with a known feature map. The proposed method is essentially a follow-the-leader algorithm using estimated rewards, which are found by estimating the parameter of the reward function.