Online learning in MDPs with linear function approximation and bandit feedback. 

( √ dHT ) , where H is the number of steps in each episode and d is the dimensionality of the feature map . 1 Introduction . We study the problem of online learning in episodic Markov Decision Processes ( MDP ) , modelling a sequential decision making problem where the interaction between a learner and its environment is divided into T episodes of fixed length H . At each time step of the episode , the learner observes the current state of the environment , chooses one of the K available actions , and earns a reward . Consequently , the state of the environment changes according to the transition function of the underlying MDP , as a function of the previous state and the action taken by the learner . A key distinguishing feature of our setting is that we assume that the reward function can change arbitrarily between episodes , and the learner only has access to bandit feedback : instead of being able to observe the reward function at the end of the episode , the learner only gets to observe the rewards that it actually received . As traditional in this line of work , we aim to design algorithms for the learner with theoretical guarantees on her regret , which is the difference between the total reward accumulated by the learner and the total reward of the best stationary policy fixed in hindsight . Unlike most previous work on this problem , we allow the state space to be very large and aim to prove performance guarantees that do not depend on the size of the state space , bringing theory one step closer to practical scenarios where assuming finite state spaces is unrealistic . To address the challenge of learning in large state spaces , we adopt the classic RL technique of using linear function approximation and suppose that we have access to a relatively low-dimensional feature map that can be used to represent policies and value functions . We will assume that the feature map is expressive enough so that all action-value functions can be expressed as linear functions of the features , and that the learner has full knowledge of the transition function of the MDP . Our main contribution is designing a computationally efficient algorithm called ONLINE Q-REPS , and prove that in the setting described above , its regret is at most O ( √ dHTD ( µ∗‖µ0 ) ) , where d is the dimensionality of the feature map and D ( µ∗‖µ0 ) is the relative entropy between the state-action distribution µ∗ induced by the optimal policy and an initial distribution µ0 given as input to the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithm . Notably , our results do not require the likelihood ratio between these distributions to be uniformly bounded , and the bound shows no dependence on the eigenvalues of the feature covariance matrices . Our algorithm itself requires solving a d2-dimensional convex optimization problem at the beginning of each episode , which can be solved to arbitrary precision ε in time polynomial in d and 1/ε , independently of the size of the state-action space . Our work fits into a long line of research considering online learning in Markov decision processes . The problem of regret minimization in stationary MDPs with a fixed reward function has been studied extensively since the work of Burnetas and Katehakis [ 6 ] , Auer and Ortner [ 2 ] , Tewari and Bartlett [ 31 ] , Jaksch et al . [ 14 ] , with several important advances made in the past decade [ 9 , 10 , 4 , 13 , 15 ] . While most of these works considered small finite state spaces , the same techniques have been very recently extended to accommodate infinite state spaces under the assumption of realizable function approximation by Jin et al . [ 17 ] and Yang and Wang [ 33 ] . In particular , the notion of linear MDPs introduced by Jin et al . [ 17 ] has become a standard model for linear function approximation and has been used in several recent works ( e.g. , 22 , 32 , 1 ) . Even more relevant is the line of work considering adversarial rewards , initiated by Even-Dar et al . [ 12 ] , who consider online learning in continuing MDPs with full feedback about the rewards . They proposed a MDP-E algorithm , that achieves O ( τ2 √ T logK ) regret , where τ is an upper bound on the mixing time of the MDP . Later , Neu et al . [ 25 ] proposed an algorithm which guarantees Õ ( √ τ3KT/α ) regret with bandit feedback , essentially assuming that all states are reachable with probability α > 0 under all policies . In our work , we focus on episodic MDPs with a fixed episode length H . The setting was first considered in the bandit setting by Neu et al . [ 23 ] , who proposed an algorithm with a regret bound of O ( H2 √ TK/α ) . Although the number of states does not appear explicitly in the bound , the regret scales at least linearly with the size of the state space X , since |X | ≤ H/α . Later work by Zimin and Neu [ 35 ] , Dick et al . [ 11 ] eliminated the dependence on α and proposed an algorithm achieving Õ ( √ TH|X |K ) regret . Regret bounds for the full-information case without prior knowledge of the MDP were achieved by Neu et al . [ 24 ] and Rosenberg and Mansour [ 30 ] , of order Õ ( H|X |K √ T ) and Õ ( H|X | √ KT ) , respectively . These results were recently extended to handle bandit feedback about the rewards by Jin et al . [ 16 ] , ultimately resulting in a regret bound of Õ ( H|X | √ KT ) . As apparent from the above discussion , all work on online learning in MDPs with adversarial rewards considers finite state spaces . The only exception we are aware of is the recent work of Cai et al . [ 7 ] , whose algorithm OPPO is guaranteed to achieve Õ ( √ d3H3T ) , assuming that the learner has access to d-dimensional features that can perfectly represent all action-value functions . While Cai , Yang , Jin , and Wang [ 7 ] remarkably assumed no prior knowledge of the MDP parameters , their guarantees are only achieved in the full-information case . This is to be contrasted with our results that are achieved for the much more restrictive bandit setting , albeit with the stronger assumption of having full knowledge of the underlying MDP , as required by virtually all prior work in the bandit setting , with the exception of Jin et al . [ 16 ] . Our results are made possible by a careful combination of recently proposed techniques for contextual bandit problems and optimal control in Markov decision processes . In particular , a core component of our algorithm is a regularized linear programming formulation of optimal control in MDPs due to Bas-Serrano et al . [ 5 ] , which allows us to reduce the task of computing near-optimal policies in linear MDPs to a low-dimensional convex optimization problem . A similar algorithm design has been previously used for tabular MDPs by Zimin and Neu [ 35 ] , Dick et al . [ 11 ] , with the purpose of removing factors of 1/α from the previous state-of-the-art bounds of Neu et al . [ 23 ] . Analogously to this improvement , our methodology enables us to make strong assumptions on problem-dependent constants like likelihood ratios between µ∗ and µ0 or eigenvalues of the feature covariance matrices . Another important building block of our method is a version of the recently proposed Matrix Geometric Resampling procedure of Neu and Olkhovskaya [ 21 ] that enables us to efficiently estimate the reward functions . Incorporating these estimators in the algorithmic template of Bas-Serrano et al . [ 5 ] is far from straightforward and requires several subtle adjustments . Notation . We use 〈· , ·〉 to denote inner products in Euclidean space and by ‖·‖ we denote the Euclidean norm for vectors and the operator norm for matrices . For a symmetric positive definite matrix A , we use λmin ( A ) to denote its smallest eigenvalue . We write tr ( A ) for the trace of a matrix A and use A < 0 to denote that an operator A is positive semi-definite , and we use A < B to denote A−B < 0 . For a d-dimensional vector v , we denote the corresponding d× d diagonal matrix by diag ( v ) . For a positive integer N , we use [ N ] to denote the set of positive integers { 1 , 2 , . . . , N } . Finally , we will denote the set of all probability distributions over any set X by ∆X . 2 Preliminaries . An episodic Markovian Decision Process ( MDP ) , denoted by M = ( X , A , H , P , r ) is defined by a state space X , action space A , episode length H ∈ Z+ , transition function P : X ×A → ∆X and a reward function r : X ×A → [ 0 , 1 ] . For convenience , we will assume that both X and A are finite sets , although we allow the state space X to be arbitrarily large . Without significant loss of generality , we will assume that the set of available actions is the same A in each state , with cardinality |A| = K. Furthermore , without any loss of generality , we will assume that the MDP has a layered structure , satisfying the following conditions : • The state set X can be decomposed into H disjoint sets : X = ∪Hh=1Xh , • X1 = { x1 } and XH = { xH } are singletons , • transitions are only possible between consecutive layers , that is , for any xh ∈ Xh , the distribution P ( ·|x , a ) is supported on Xh+1 for all a and h ∈ [ H − 1 ] . These assumptions are common in the related literature ( e.g. , 23 , 35 , 30 ) and are not essential to our analysis ; their primary role is simplifying our notation . In the present paper , we consider an online learning problem where the learner interacts with its environment in a sequence of episodes t = 1 , 2 , . . . , T , facing a different reward functions rt,1 , . . . rt , H+1 selected by a ( possibly adaptive ) adversary at the beginning of each episode t. Oblivious to the reward function chosen by the adversary , the learner starts interacting with the MDP in each episode from the initial state Xt,1 = x1 . At each consecutive step h ∈ [ H − 1 ] within the episode , the learner observes the state Xt , h , picks an action At , h and observes the reward rt , h ( Xt , h , At , h ) . Then , unless h = H , the learner moves to the next state Xt , h+1 , which is generated from the distribution P ( ·|Xt , h , At , h ) . At the end of step H , the episode terminates and a new one begins . The aim of the learner is to select its actions so that the cumulative sum of rewards is as large as possible . Our algorithm and analysis will make use of the concept of ( stationary stochastic ) policies π : X → ∆A . A policy π prescribes a behaviour rule to the learner by assigning probability π ( a|x ) to taking action a at state x . Let τπ = ( ( X1 , A1 ) , ( X2 , A2 ) , . . . , ( XH , AH ) ) be a trajectory generated by following the policy π through the MDP . Then , for any xh ∈ Xh , ah ∈ A we define the occupancy measure µπh ( x , a ) = Pπ [ ( x , a ) ∈ τπ ] . We will refer to the collection of these distributions across all layers h as the occupancy measure induced by π and denote it as µπ = ( µπ1 , µ π 2 , . . . , µ π H ) . We will denote the set of all valid occupancy measures by U and note that this is a convex set , such that for every element µ ∈ U the following set of linear constraints is satisfied : ∑ a∈A µh+1 ( x , a ) = ∑ x′ , a′∈Xh×A P ( x|x′ , a′ ) µh ( x′ , a′ ) , ∀x ∈ Xh+1 , h ∈ [ H − 1 ] , ( 1 ) as well as ∑ a µ1 ( x1 , a ) = 1 . From every valid occupancy measure µ , a stationary stochastic policy π = π1 , . . . , πH−1 can be derived as πµ , h ( a|x ) = µh ( x , a ) / ∑ a′ µh ( x , a ′ ) . For each h , introducing the linear operators E and P through their action on a set state-action distribution uh as ( ETuh ) ( x ) = ∑ a∈A uh ( x , a ) and ( P T huh ) ( x ) = ∑ x′ , a′∈Xh , A P ( x|x ′ , a′ ) uh ( x ′ , a′ ) , the constraints can be simply written as ETµh+1 = P Thµh for each h. We will use the inner product notation for the sum over the set of states and actions : 〈µh , rh〉 = ∑ ( x , a ) ∈ ( Xh×A ) µh ( x , a ) rt , h ( x , a ) . Using this notation , we formulate our objective as selecting a sequence of policies πt for each episode t in a way that it minimizes the total expected regret defined as RT = sup π∗ T∑ t=1 H∑ h=1 ( Eπ∗ [ rt , h ( X∗h , A∗h ) ] − Eπt [ rt ( Xt , h , At , h ) ] ) = sup µ∗∈U T∑ t=1 H∑ h=1 〈µ∗h − µ πt h , rt , h〉 , where the notations Eπ∗ [ · ] and Eπt [ · ] emphasize that the state-action trajectories are generated by following policies π∗ and πt , respectively . As the above expression suggests , we can reformulate our online learning problem as an instance of online linear optimization where in each episode t , the learner selects an occupancy measure µt ∈ U ( with µt = µπt ) and gains reward ∑H h=1〈µt , h , rt , h〉 . Intuitively , the regret measures the gap between the total reward gained by the learner and that of the best stationary policy fixed in hindsight , with full knowledge of the sequence of rewards chosen by the adversary . This performance measure is standard in the related literature on online learning in MDPs , see , for example Neu et al . [ 23 ] , Zimin and Neu [ 35 ] , Neu et al . [ 24 ] , Rosenberg and Mansour [ 30 ] , Cai et al . [ 7 ] . In this paper , we focus on MDPs with potentially enormous state spaces , which makes it difficult to design computationally tractable algorithms with nontrivial guarantees , unless we make some assumptions . We particularly focus on the classic technique of relying on linear function approximation and assuming that the reward functions occurring during the learning process can be written as a linear function of a low-dimensional feature map . We specify the form of function approximation and the conditions our analysis requires as follows : Assumption 1 ( Linear MDP with adversarial rewards ) . There exists a feature map ϕ : X ×A → Rd and a collection of d signed measures m = ( m1 , . . . , md ) on X , such that for any ( x , a ) ∈ X ×A the transition function can be written as P ( ·|x , a ) = 〈m ( · ) , ϕ ( x , a ) 〉 . Furthermore , the reward function chosen by the adversary in each episode t can be written as rt , h ( x , a ) = 〈θt , h , ϕ ( x , a ) 〉 for some θt , h ∈ Rd . We assume that the features and the parameter vectors satisfy ‖ϕ ( x , a ) ‖ ≤ σ and that the first coordinate ϕ1 ( x , a ) = 1 for all ( x , a ) ∈ X ×A . Also we assume that ‖θt , h‖ ≤ R. Online learning under this assumption , but with a fixed reward function , has received substantial attention in the recent literature , particularly since the work of Jin et al . [ 17 ] who popularized the term “ Linear MDP ” to refer to this class of MDPs . This has quickly become a common assumption for studying reinforcement learning algorithms ( Cai et al . [ 7 ] , Jin et al . [ 17 ] , Neu and Pike-Burke [ 22 ] , Agarwal et al . [ 1 ] ) . This is also a special case of factored linear models ( Yao et al . [ 34 ] , Pires and Szepesvári [ 29 ] ) . Linear MDPs come with several attractive properties that allow efficient optimization and learning . In this work , we will exploit the useful property shown by Neu and Pike-Burke [ 22 ] and Bas-Serrano et al . [ 5 ] that all occupancy measures in a linear MDP can be seen to satisfy a relaxed version of the constraints in Equation ( 1 ) . Specifically , for all h , defining the feature matrix Φh ∈ R ( Xh×A ) ×d with its action on the distribution u as ΦThu = ∑ x , a∈Xh , A uh ( x , a ) ϕ ( x , a ) , we define UΦ as the set of state-action distributions ( µ , u ) = ( ( µ1 , . . . , µH ) , ( u1 , . . . , uH ) ) satisfying the following constraints : ETuh+1 = P T hµh ( ∀h ) , ΦThuh = ΦThµh ( ∀h ) , ETu1 = 1 . ( 2 ) It is easy to see that for all feasible ( µ , u ) pairs , u satisfies the original constraints ( 1 ) if the MDP satisfies Assumption 1 : since the transition operator can be written as Ph = ΦhMh for some matrix Mh . In this case , we clearly have ETuh+1 = P T hµh = M T hΦ T hµh = M T hΦ T huh = P T huh , ( 3 ) showing that any feasible u is indeed a valid occupancy measure . Furthermore , due to linearity of the rewards in Φ , we also have 〈uh , rt , h〉 = 〈µh , rt , h〉 for all feasible ( µ , u ) ∈ UΦ . While the number of variables and constraints in Equation ( 2 ) is still very large , it has been recently shown that approximate linear optimization over this set can be performed tractably [ 22 , 5 ] . Our own algorithm design described in the next section will heavily build on these recent results .

Summary: This paper presents a theoretical approach for online learning in Markov decision processes (MDPs) with time-varying adversarial rewards and infinite, and possibly continuous, state spaces. The proposed approach is based on linear assumptions on the structure of the MDP and approximate solutions of a linear programming problem. Theoretical regret bounds and discussions on the implementation of a tractable algorithmic solution are provided.