Efficient Reinforcement Learning in Factored MDPs with Application to Constrained RL

√ nH|Si| , where |Si| is the cardinality of the factored state sub- space , H is the planning horizon and n is the number of factored transitions . We also provide a lower bound , which shows near-optimality of our algorithm w.r.t . timestep T , horizon H and factored state-action subspace cardinality . Finally , as an application , we study a new formulation of constrained RL , RL with knapsack constraints ( RLwK ) , and provide the first sample-efficient algorithm based on FMDP-BF . 1 INTRODUCTION . Reinforcement learning ( RL ) is concerned with sequential decision making problems where an agent interacts with a stochastic environment and aims to maximize its cumulative rewards . The environment is usually modeled as a Markov Decision Process ( MDP ) whose transition kernel and reward function are unknown to the agent . A main challenge of the agent is efficient exploration in the MDP , so as to minimize its regret , or the related sample complexity of exploration . Extensive study has been done on the tabular case , in which almost no prior knowledge is assumed on the MDP dynamics . The regret or sample complexity bounds typically depend polynomially on the cardinality of state and action spaces ( e.g. , Strehl et al. , 2009 ; Jaksch et al. , 2010 ; Azar et al. , 2017 ; Dann et al. , 2017 ; Jin et al. , 2018 ; Dong et al. , 2019 ; Zanette & Brunskill , 2019 ) . Moreover , matching lower bounds ( e.g. , Jaksch et al. , 2010 ) imply that these results can not be improved without additional assumptions . On the other hand , many RL tasks involve large state and action spaces , for which these regret bounds are still excessively large . In many practical scenarios , one can often take advantage of specific structures of the MDP to develop more efficient algorithms . For example , in robotics , the state may be high-dimensional , but the subspaces of the state may evolve independently of others , and only depend on a lowdimensional subspace of the previous state . Formally , these problems can be described as factored MDPs ( Boutilier et al. , 2000 ; Kearns & Koller , 1999 ; Guestrin et al. , 2003 ) . Most relevant to the present work is Osband & Van Roy ( 2014b ) , who proposed a posterior sampling algorithm and a UCRL-like algorithm that both enjoy √ T regret , where T is the maximum timestep . Their regret bounds have a linear dependence on the time horizon and each factored state subspace . It is unclear whether this bound is tight or not . In this work , we tackle this problem by proposing algorithms with improved regret bounds , and developing corresponding lower bounds for episodic FMDPs . We propose a sample- and computationefficient algorithm called FMDP-BF based on the principle of optimism in the face of uncertainty , and prove its regret bounds . We also provide a lower bound , which implies that our algorithm is near-optimal with respect to the timestep T , the planning horizon H and factored state-action subspace cardinality |X [ Zi ] | . As an application , we study a novel formulation of constrained RL , known as RL with knapsack constraints ( RLwK ) , which we believe is natural to capture many scenarios in real-life applications . We apply FMDP-BF to this setting , to obtain a statistically efficient algorithm with a regret bound that is near-optimal in terms of T , S , A , and H . Our contributions are summarized as follows : 1 . We propose an algorithm for FMDP , and prove its regret bound that improves on the previous result of Osband & Van Roy ( 2014b ) by a factor of √ nH|Si| . 2 . We prove a regret lower bound for FMDP , which implies that our regret bound is nearoptimal in terms of timestep T , horizon H and factored state-action subspace cardinality . 3 . We apply FMDP-BF in RLwK , a novel constrained RL setting with knapsack constraints , and prove a regret bound that is near-optimal in terms of T , S , A and H . 2 PRELIMINARIES . We consider the setting of a tabular episodic Markov decision process ( MDP ) , ( S , A , H , P , R ) , where S is the set of states , A is the action set , H is the number of steps in each episode . P is the transition probability matrix so that P ( ·|s , a ) gives the distribution over states if action a is taken on state s , and R ( s , a ) is the reward distribution of taking action a on state s with support [ 0 , 1 ] . We use R̄ ( s , a ) to denote the expectation E [ R ( s , a ) ] . In each episode , the agent starts from an initial state s1 that may be arbitrarily selected . At each step h ∈ [ H ] , the agent observes the current state sh ∈ S , takes action ah ∈ A , receives a reward rh sampled from R ( sh , ah ) , and transits to state sh+1 with probability P ( sh+1|sh , ah ) . The episode ends when sH+1 is reached . A policy π is a collection of H policy functions { πh : S → A } h∈ [ H ] . We use V πh : S → R to denote the value function at step h under policy π , which gives the expected sum of remaining rewards received under policy π starting from sh = s , i.e . V πh ( s ) = E [ ∑H h′=hR ( sh′ , πh′ ( sh′ ) ) | sh = s ] . Accordingly , we define Qπh ( s , a ) as the expected Q-value function at step h : Q π h ( s , a ) = E [ R ( sh , ah ) + ∑H h′=h+1R ( sh′ , πh′ ( sh′ ) ) | sh = s , ah = a ] . We use V ∗h and Q ∗ h to denote the optimal value and Q-functions under optimal policy π∗ at step h. The agent interacts with the environment for K episodes with policy πk = { πk , h : S → A } h∈ [ H ] determined before the k-th episode begins . The agent ’ s goal is to maximize its cumulative rewards∑K k=1 ∑H h=1 rk , h over T = KH steps , or equivalently , to minimize the following expected regret : Reg ( K ) def = K∑ k=1 [ V ∗1 ( sk,1 ) − V πk 1 ( sk,1 ) ] , where sk,1 is the initial state of episode k . 2.1 FACTORED MDPS . A factored MDP is an MDP whose rewards and transitions exhibit certain conditional independence structures . We start with the formal definition of factored MDP ( Boutilier et al. , 2000 ; Osband & Van Roy , 2014b ; Xu & Tewari , 2020 ; Lu & Van Roy , 2019 ) . LetP ( X , Y ) denote the set of functions that map x ∈ X to the probability distribution on Y . Definition 1 . ( Factored set ) Let X = X1 × · · · × Xd be a factored set . For any subset of indices Z ⊆ { 1 , 2 , . . . , d } , we define the scope set X [ Z ] : = ⊗i∈ZXi . Further , for any x ∈ X , define the scope variable x [ Z ] ∈ X [ Z ] to be the value of the variables xi ∈ Xi with indices i ∈ Z . If Z is a singleton , we will write x [ i ] for x [ { i } ] . Definition 2 . ( Factored reward ) The reward function classR ⊂ P ( X , R ) is factored over S ×A = X = X1 × · · · × Xd with scopes Z1 , · · · , Zm if for all R ∈ R , x ∈ X , there exist functions { Ri ∈ P ( X [ Zi ] , [ 0 , 1 ] ) } mi=1 such that r ∼ R ( x ) is equal to 1 m ∑m i=1 ri with each ri ∼ Ri ( x [ Zi ] ) individually observed . We use R̄i to denote the expectation E [ Ri ] . Definition 3 . ( Factored transition ) The transition function class P ⊂ P ( X , S ) is factored over S × A = X = X1 × · · · × Xd and S = S1 × · · · × Sn with scopes Z1 , · · · , Zn if and only if , for all P ∈ P , x ∈ X , s ∈ S , there exist functions { Pj ∈ P ( X [ Zj ] , Sj ) } nj=1 such that P ( s | x ) =∏n j=1 Pj ( s [ j ] | x [ Zj ] ) . A factored MDP is an MDP with factored rewards and transitions . A factored MDP is fully characterized by M = ( { Xi } di=1 ; { ZRi } m i=1 ; { Ri } mi=1 ; { Sj } n j=1 ; { ZPj } n j=1 ; { Pj } nj=1 ; H ) , where X = S × A , { ZRi } mi=1 and { ZPj } nj=1 are the scopes for the reward and transition functions , which we assume to be known to the agent . An excellent example of factored MDP is given by Osband & Van Roy ( 2014 ) , about a large production line with d machines in sequence with Si possible states for machine i . Over a single time-step each machine can only be influenced by its direct neighbors . For this problem , the scopes ZRi and ZPi of machine i ∈ { 2 , ... , d − 1 } can be defined as { i − 1 , i , i + 1 } , and the scopes of machine 1 and machine d are { 1 , 2 } and { d− 1 , d } respectively . Another possible example to explain factored MDP is about robotics . For a robot , the transition dynamics of its different parts ( e.g . its legs and arms ) may be relatively independent . In that case , the factored transition can be defined for each part separately . For notation simplicity , we use X [ i : j ] and S [ i : j ] to denote X [ ∪k=i , ... , jZk ] and ⊗jk=iSk respectively . Similarly , We use P [ i : j ] ( s′ [ i : j ] | s , a ) to denote ∏j k=i P ( s′ [ k ] | ( s , a ) [ ZPk ] ) . For every V : S → R and the right-linear operators P , we define PV ( s , a ) def= ∑ s′∈S P ( s′ | s , a ) V ( s′ ) . A state-action pair can be represented as ( s , a ) or x . We also use ( s , a ) [ Z ] to denote the corresponding x [ Z ] for notation convenience . We mainly focus on the case where the total time step T = KH is the dominant factor , and assume that T ≥ |Xi| ≥ H during the analysis .

Summary: This paper proposes a reinforcement learning algorithm FMDP-BF for episodic Factored MDPs. Similar to previous works, FMDP-BF follows the principle of "optimism in the face of uncertainty" to efficiently explore to achieve low regret. Compared to algorithms for general MDPs, FMDP-BF leverages the factorization structure of FMDP and results in exponential regret reduction in the size of the sate space. Compared to previous results for FMDP, the proposed algorithm utilizes Bernstein-type confidence bounds and new bounds on the factored transitions to reduce the regret order. The paper also connects a class of constrained RL to FMDP and  provide a sample efficient for the constrained RL problem.