Efficient Reinforcement Learning in Factored MDPs with Application to Constrained RL

√ nH|Si| , where |Si| is the cardinality of the factored state sub- space , H is the planning horizon and n is the number of factored transitions . We also provide a lower bound , which shows near-optimality of our algorithm w.r.t . timestep T , horizon H and factored state-action subspace cardinality . Finally , as an application , we study a new formulation of constrained RL , RL with knapsack constraints ( RLwK ) , and provide the first sample-efficient algorithm based on FMDP-BF . 1 INTRODUCTION . Reinforcement learning ( RL ) is concerned with sequential decision making problems where an agent interacts with a stochastic environment and aims to maximize its cumulative rewards . The environment is usually modeled as a Markov Decision Process ( MDP ) whose transition kernel and reward function are unknown to the agent . A main challenge of the agent is efficient exploration in the MDP , so as to minimize its regret , or the related sample complexity of exploration . Extensive study has been done on the tabular case , in which almost no prior knowledge is assumed on the MDP dynamics . The regret or sample complexity bounds typically depend polynomially on the cardinality of state and action spaces ( e.g. , Strehl et al. , 2009 ; Jaksch et al. , 2010 ; Azar et al. , 2017 ; Dann et al. , 2017 ; Jin et al. , 2018 ; Dong et al. , 2019 ; Zanette & Brunskill , 2019 ) . Moreover , matching lower bounds ( e.g. , Jaksch et al. , 2010 ) imply that these results can not be improved without additional assumptions . On the other hand , many RL tasks involve large state and action spaces , for which these regret bounds are still excessively large . In many practical scenarios , one can often take advantage of specific structures of the MDP to develop more efficient algorithms . For example , in robotics , the state may be high-dimensional , but the subspaces of the state may evolve independently of others , and only depend on a lowdimensional subspace of the previous state . Formally , these problems can be described as factored MDPs ( Boutilier et al. , 2000 ; Kearns & Koller , 1999 ; Guestrin et al. , 2003 ) . Most relevant to the present work is Osband & Van Roy ( 2014b ) , who proposed a posterior sampling algorithm and a UCRL-like algorithm that both enjoy √ T regret , where T is the maximum timestep . Their regret bounds have a linear dependence on the time horizon and each factored state subspace . It is unclear whether this bound is tight or not . In this work , we tackle this problem by proposing algorithms with improved regret bounds , and developing corresponding lower bounds for episodic FMDPs . We propose a sample- and computationefficient algorithm called FMDP-BF based on the principle of optimism in the face of uncertainty , and prove its regret bounds . We also provide a lower bound , which implies that our algorithm is near-optimal with respect to the timestep T , the planning horizon H and factored state-action subspace cardinality |X [ Zi ] | . As an application , we study a novel formulation of constrained RL , known as RL with knapsack constraints ( RLwK ) , which we believe is natural to capture many scenarios in real-life applications . We apply FMDP-BF to this setting , to obtain a statistically efficient algorithm with a regret bound that is near-optimal in terms of T , S , A , and H . Our contributions are summarized as follows : 1 . We propose an algorithm for FMDP , and prove its regret bound that improves on the previous result of Osband & Van Roy ( 2014b ) by a factor of √ nH|Si| . 2 . We prove a regret lower bound for FMDP , which implies that our regret bound is nearoptimal in terms of timestep T , horizon H and factored state-action subspace cardinality . 3 . We apply FMDP-BF in RLwK , a novel constrained RL setting with knapsack constraints , and prove a regret bound that is near-optimal in terms of T , S , A and H . 2 PRELIMINARIES . We consider the setting of a tabular episodic Markov decision process ( MDP ) , ( S , A , H , P , R ) , where S is the set of states , A is the action set , H is the number of steps in each episode . P is the transition probability matrix so that P ( ·|s , a ) gives the distribution over states if action a is taken on state s , and R ( s , a ) is the reward distribution of taking action a on state s with support [ 0 , 1 ] . We use R̄ ( s , a ) to denote the expectation E [ R ( s , a ) ] . In each episode , the agent starts from an initial state s1 that may be arbitrarily selected . At each step h ∈ [ H ] , the agent observes the current state sh ∈ S , takes action ah ∈ A , receives a reward rh sampled from R ( sh , ah ) , and transits to state sh+1 with probability P ( sh+1|sh , ah ) . The episode ends when sH+1 is reached . A policy π is a collection of H policy functions { πh : S → A } h∈ [ H ] . We use V πh : S → R to denote the value function at step h under policy π , which gives the expected sum of remaining rewards received under policy π starting from sh = s , i.e . V πh ( s ) = E [ ∑H h′=hR ( sh′ , πh′ ( sh′ ) ) | sh = s ] . Accordingly , we define Qπh ( s , a ) as the expected Q-value function at step h : Q π h ( s , a ) = E [ R ( sh , ah ) + ∑H h′=h+1R ( sh′ , πh′ ( sh′ ) ) | sh = s , ah = a ] . We use V ∗h and Q ∗ h to denote the optimal value and Q-functions under optimal policy π∗ at step h. The agent interacts with the environment for K episodes with policy πk = { πk , h : S → A } h∈ [ H ] determined before the k-th episode begins . The agent ’ s goal is to maximize its cumulative rewards∑K k=1 ∑H h=1 rk , h over T = KH steps , or equivalently , to minimize the following expected regret : Reg ( K ) def = K∑ k=1 [ V ∗1 ( sk,1 ) − V πk 1 ( sk,1 ) ] , where sk,1 is the initial state of episode k . 2.1 FACTORED MDPS . A factored MDP is an MDP whose rewards and transitions exhibit certain conditional independence structures . We start with the formal definition of factored MDP ( Boutilier et al. , 2000 ; Osband & Van Roy , 2014b ; Xu & Tewari , 2020 ; Lu & Van Roy , 2019 ) . LetP ( X , Y ) denote the set of functions that map x ∈ X to the probability distribution on Y . Definition 1 . ( Factored set ) Let X = X1 × · · · × Xd be a factored set . For any subset of indices Z ⊆ { 1 , 2 , . . . , d } , we define the scope set X [ Z ] : = ⊗i∈ZXi . Further , for any x ∈ X , define the scope variable x [ Z ] ∈ X [ Z ] to be the value of the variables xi ∈ Xi with indices i ∈ Z . If Z is a singleton , we will write x [ i ] for x [ { i } ] . Definition 2 . ( Factored reward ) The reward function classR ⊂ P ( X , R ) is factored over S ×A = X = X1 × · · · × Xd with scopes Z1 , · · · , Zm if for all R ∈ R , x ∈ X , there exist functions { Ri ∈ P ( X [ Zi ] , [ 0 , 1 ] ) } mi=1 such that r ∼ R ( x ) is equal to 1 m ∑m i=1 ri with each ri ∼ Ri ( x [ Zi ] ) individually observed . We use R̄i to denote the expectation E [ Ri ] . Definition 3 . ( Factored transition ) The transition function class P ⊂ P ( X , S ) is factored over S × A = X = X1 × · · · × Xd and S = S1 × · · · × Sn with scopes Z1 , · · · , Zn if and only if , for all P ∈ P , x ∈ X , s ∈ S , there exist functions { Pj ∈ P ( X [ Zj ] , Sj ) } nj=1 such that P ( s | x ) =∏n j=1 Pj ( s [ j ] | x [ Zj ] ) . A factored MDP is an MDP with factored rewards and transitions . A factored MDP is fully characterized by M = ( { Xi } di=1 ; { ZRi } m i=1 ; { Ri } mi=1 ; { Sj } n j=1 ; { ZPj } n j=1 ; { Pj } nj=1 ; H ) , where X = S × A , { ZRi } mi=1 and { ZPj } nj=1 are the scopes for the reward and transition functions , which we assume to be known to the agent . An excellent example of factored MDP is given by Osband & Van Roy ( 2014 ) , about a large production line with d machines in sequence with Si possible states for machine i . Over a single time-step each machine can only be influenced by its direct neighbors . For this problem , the scopes ZRi and ZPi of machine i ∈ { 2 , ... , d − 1 } can be defined as { i − 1 , i , i + 1 } , and the scopes of machine 1 and machine d are { 1 , 2 } and { d− 1 , d } respectively . Another possible example to explain factored MDP is about robotics . For a robot , the transition dynamics of its different parts ( e.g . its legs and arms ) may be relatively independent . In that case , the factored transition can be defined for each part separately . For notation simplicity , we use X [ i : j ] and S [ i : j ] to denote X [ ∪k=i , ... , jZk ] and ⊗jk=iSk respectively . Similarly , We use P [ i : j ] ( s′ [ i : j ] | s , a ) to denote ∏j k=i P ( s′ [ k ] | ( s , a ) [ ZPk ] ) . For every V : S → R and the right-linear operators P , we define PV ( s , a ) def= ∑ s′∈S P ( s′ | s , a ) V ( s′ ) . A state-action pair can be represented as ( s , a ) or x . We also use ( s , a ) [ Z ] to denote the corresponding x [ Z ] for notation convenience . We mainly focus on the case where the total time step T = KH is the dominant factor , and assume that T ≥ |Xi| ≥ H during the analysis .

Summary: The authors study the factor MDP problem in an online and episodic setting. They provide two main contributions on this question. First, they propose an OFU type algorithm which enjoys a better regret bound than Osband & Van Roy (2014) by a factor of  $\sqrt{nH \Gamma}$. The improvement is brought about by a refined consideration on the confidence radius' dependence on the variances of the rewards, which carries a similar idea to the design of UCBVI-CH by Azar et al. 2017 for the tabular case.  The second contribution is on the generalization to an episodic FMDP with knapsacks problem, where the authors generalize the approach in the first contribution to provide a regret bound.