Oracle-Efficient Regret Minimization in Factored MDPs with Unknown Structure

1 Introduction . Reinforcement learning ( RL ) considers an agent interacting with an unknown stochastic environment with the aim of maximizing its expected cumulative reward . This is usually modeled by a Markov decision process ( MDP ) with a finite number of states . The vast majority of provably-efficient RL has focused on the tabular case , where the state space is assumed to be small . Starting with the UCRL2 algorithm [ Jaksch et al. , 2010 ] , near-optimal regret bounds were proved [ Azar et al. , 2017 , Fruit et al. , 2018 , Jin et al. , 2018 , Zanette and Brunskill , 2019 , Efroni et al. , 2019 ] . Unfortunately , many real-world RL applications involve problems with a huge state space , yielding the tabular MDP model impractical as it requires the regret to unavoidably scale polynomially with the number of states . In many practical scenarios , prior knowledge about the environment can be leveraged in order to develop more efficient algorithms . A popular way to model additional knowledge about the structure of the environment is by factored MDPs ( FMDPs ; Boutilier et al . [ 1995 , 1999 ] ) . The state of an FMDP is composed of d components , called factors , and each component is determined by only m other factors , called its scope . FMDPs arise naturally in many applications like games , robotics , image-based applications and production lines ( where only neighbouring machines affect one another ) . The common property of all these examples is the huge state space exponential in d , but the very small scope size m ( e.g. , in images each pixel is a factor and it depends only on neighboring pixels ) . The key benefit of FMDPs is the combinatorial state space that allows compact representation . That is , although the number of states is exponential in d , the FMDP representation is only exponential in m ( which is much smaller ) and polynomial in d. Early works [ Kearns and Koller , 1999 , Guestrin et al. , 2002 , Strehl , 2007 , Szita and Lőrincz , 2009 ] show that FMDPs also reduce the sample complexity exponentially , thus avoiding polynomial dependence on the number of states . Recently , this was further extended to algorithms with near-optimal regret bounds [ Osband and Van Roy , 2014 , Xu and Tewari , 2020 , Tian et al. , 2020 , Chen et al. , 2021 , Talebi et al. , 2021 ] . However , all these works make the strong assumption that the underlying FMDP structure is fully known to the learner in advance . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper we provide the first regret minimization algorithm for FMDPs with unknown structure , thus solving an open problem from Osband and Van Roy [ 2014 ] . Our algorithm is built on a novel concept of consistent scopes and guarantees near-optimal √ T regret that scales polynomially with the FMDP encoding and is therefore exponentially smaller than the number of states ( and the regret in tabular MDPs ) . Moreover , our algorithm features an innovative construction that can incorporate elimination of inconsistent scopes into the optimistic regret minimization framework , while maintaining computational efficiency given oracle-access to an FMDP planner . Keeping computational oracle-efficiency is a difficult challenge in factored MDPs and especially hard when structure is unknown , since the number of possible structure configurations is highly exponential . Furthermore , our algorithm easily accommodates any level of structure knowledge , and is therefore extremely useful when additional prior domain knowledge is available . We note that while structure learning in FMDPs was previously studied by Strehl et al . [ 2007 ] , Diuk et al . [ 2009 ] , Chakraborty and Stone [ 2011 ] , Hallak et al . [ 2015 ] , Guo and Brunskill [ 2017 ] , none of them provide regret guarantees . To make our algorithms compatible with existing approximate FMDP planners , we also study FMDPs with non-factored actions . To the best of our knowledge , existing planners require small non-factored action space which is not compatible with the FMDP regret minimization literature . To mitigate this gap , we show that even when the oracle is limited to non-factored actions , a variant of our algorithm can still be implemented efficiently and achieve similar near-optimal regret bounds . Finally , we leverage the techniques presented in this paper to prove a novel lower bound for regret minimization in FMDPs with known structure . This is the first lower bound to show that the regret must scale exponentially with the scope size m , and the first to utilize connections between different factors in a non-trivial way ( i.e. , with scope size larger than 1 ) . Furthermore , it improves previous lower bounds by a factor of √ d and closes the gap to the state-of-the-art regret bound of Chen et al . [ 2021 ] , thus establishing the minimax optimal regret in this setting . Our algorithms make oracle use of FMDP planners . However , even where an FMDP can be represented concisely , solving for the optimal policy may take exponentially long in the most general case [ Goldsmith et al. , 1997 , Littman , 1997 ] . Our focus in this paper is upon the statistical aspect of the learning problem , and we therefore assume oracle-access to an FMDP planner . We emphasize that the oracle assumption appears in all previous regret minimization algorithms . Furthermore , except for the DORL algorithm of Xu and Tewari [ 2020 ] , all previous algorithms run in time exponential in d even with access to a planning oracle ( and known structure ) . We stress that in many cases of interest , effective approximate planners do exist [ Boutilier et al. , 2000 , Koller and Parr , 2000 , Schuurmans and Patrascu , 2001 , Guestrin et al. , 2001 , 2003 , Sanner and Boutilier , 2005 , Delgado et al. , 2011 ] . 2 Preliminaries . An infinite-horizon average-reward MDP is described by a tuple M = ( S , A , P , R ) , where S and A are finite state and action spaces , respectively , P : S ×A→ ∆S is the transition function1 , and R : S ×A→ ∆ [ 0,1 ] is the reward function with expectation r ( s , a ) = E [ R ( s , a ) ] . The interaction between the MDP and the learner proceeds as follows . The learner starts in an arbitrary initial state s1 ∈ S. For t = 1 , 2 , . . . , the learner observes the current state st ∈ S , picks an action at ∈ A and earns a reward rt sampled from R ( st , at ) . Then , the environment draws the next state st+1 ∼ P ( · | st , at ) and the process continues . A policy π : S → A is a mapping from states to actions , and its gain is defined by the average-reward criterion : λ ( M , π , s ) def= limT→∞ 1T E [ ∑T t=1 r ( s t , π ( st ) ) | s1 = s ] , where st+1 ∼ P ( · | st , π ( st ) ) . In order to derive non-trivial regret bounds , one must constrain the connectivity of the MDP [ Bartlett and Tewari , 2009 ] . We focus on communicating MDPs , i.e. , MDPs with finite diameter D < ∞ . Definition 1 . Let T ( s′ | M , π , s ) be the random variable for the first time step in which state s′ is reached when playing a stationary policy π in an MDP M with initial state s. The diameter of M is defined as D ( M ) def= maxs6=s′∈S minπ : S→A E [ T ( s′ |M , π , s ) ] . For communicating MDPs , neither the optimal policy nor its gain depend on the initial state s1 . We denote them by π ? ( M ) = arg maxπ : S→A λ ( M , π , s 1 ) and λ ? ( M ) = λ ( M , π ? , s1 ) , respectively . 1∆X denotes the set of distributions over a set X . We measure the performance of the learner by the regret . That is , the difference between the expected gain of the optimal policy in T steps and the cumulative reward obtained by the learner up to time T , i.e. , RegT ( M ) def = ∑T t=1 ( λ ? ( M ) − rt ) , where rt ∼ R ( st , at ) and at is chosen by the learner . 2.1 Factored MDPs . Factored MDPs inherit the above definitions , but also possess some conditional independence structure that allows compact representation . We follow the factored MDP definition of Osband and Van Roy [ 2014 ] , which generalizes the original definition of Boutilier et al . [ 2000 ] , Kearns and Koller [ 1999 ] to allow a factored action space as well . We start with a definition of a factored set and scope operation . Definition 2 . A set X is called factored if it can be written as a product of n sets X1 , . . . , Xn , i.e. , X = X1 × · · · ×Xn . For any subset of indices Z = { i1 , . . . , i|Z| } ⊆ { 1 , . . . , n } , define the scope set X [ Z ] = Xi1 × · · · ×Xi|Z| . Further , for any x ∈ X define the scope variable x [ Z ] ∈ X [ Z ] to be the value of the variables xi ∈ Xi with indices i ∈ Z . For singleton sets we write x [ i ] for x [ { i } ] . Next , we define the factored reward and transition functions . We use the notations X = S ×A for the state-action space , d for the number of state factors and n for the number of state-action factors . Definition 3 . A reward functionR is called factored overX = X1×· · ·×Xn with scopesZr1 , . . . , Zr ` if there exist functions { Rj : X [ Zrj ] → ∆ [ 0,1 ] } ` j=1 with expectations rj ( x [ Zrj ] ) = E [ Rj ( x [ Zrj ] ) ] such that for all x ∈ X : R ( x ) = 1 ` ∑ ` j=1Rj ( x [ Z r j ] ) . Note that when a reward r = 1 ` ∑ ` j=1 rj is sampled from R ( x ) , the learner observes every rj individually . Definition 4 . A transition functionP is called factored overX = X1×· · ·×Xn and S = S1×· · ·×Sd with scopes ZP1 , . . . , Z P d if there exist functions { Pi : X [ ZPi ] → ∆Si } di=1 such that for all x ∈ X and s′ ∈ S : P ( s′ | x ) = ∏d i=1 Pi ( s ′ [ i ] | x [ ZPi ] ) . That is , given a state-action pair x , factor i of s′ is independent of its other factors , and is determined only by x [ ZPi ] . Thus , a factored MDP ( FMDP ) is defined by an MDP whose reward and transition functions are both factored , and is fully characterized by the tuple M = ( { Xi } ni=1 , { Si , ZPi , Pi } di=1 , { Zrj , Rj } ` j=1 ) . As opposed to previous works [ Osband and Van Roy , 2014 , Xu and Tewari , 2020 , Tian et al. , 2020 , Chen et al. , 2021 ] that assume known factorization , in this paper the learner does not have any prior knowledge of the scopes ZP1 , . . . , Z P d or Z r 1 , . . . , Z r ` , and they need to be learned from experience . However , the learner has a bound m on the size of the scopes , i.e. , |ZPi | ≤ m and |Zrj | ≤ m ∀i , j . See remarks on unknown scope size and variable scope sizes in Appendix B . Remark ( FMDP encoding size ) . Let the action factorization A = Ad+1 × · · · × An , factor size W = max { max1≤i≤d |Si| , maxd+1≤i≤n |Ai| } and L = maxZ : |Z|=m |X [ Z ] | . The encoding size is O ( dWL+ ` L+ ( d+ ` ) m log n ) . Importantly , the encoding is only polynomial in d while the number of states W d is exponential . It is however exponential in the ( much smaller ) scope size as L ≈Wm .

Summary: This paper studies regret minimization in factored MDPs with unknown structure. The algorithm combines optimism in the face of uncertainty principle and a structure learning approach. With planning oracles, both the sample and computation complexity depend polynomially on the problem size. This paper also provides a regret lower bound and empirical evaluations.