Oracle-Efficient Regret Minimization in Factored MDPs with Unknown Structure

1 Introduction . Reinforcement learning ( RL ) considers an agent interacting with an unknown stochastic environment with the aim of maximizing its expected cumulative reward . This is usually modeled by a Markov decision process ( MDP ) with a finite number of states . The vast majority of provably-efficient RL has focused on the tabular case , where the state space is assumed to be small . Starting with the UCRL2 algorithm [ Jaksch et al. , 2010 ] , near-optimal regret bounds were proved [ Azar et al. , 2017 , Fruit et al. , 2018 , Jin et al. , 2018 , Zanette and Brunskill , 2019 , Efroni et al. , 2019 ] . Unfortunately , many real-world RL applications involve problems with a huge state space , yielding the tabular MDP model impractical as it requires the regret to unavoidably scale polynomially with the number of states . In many practical scenarios , prior knowledge about the environment can be leveraged in order to develop more efficient algorithms . A popular way to model additional knowledge about the structure of the environment is by factored MDPs ( FMDPs ; Boutilier et al . [ 1995 , 1999 ] ) . The state of an FMDP is composed of d components , called factors , and each component is determined by only m other factors , called its scope . FMDPs arise naturally in many applications like games , robotics , image-based applications and production lines ( where only neighbouring machines affect one another ) . The common property of all these examples is the huge state space exponential in d , but the very small scope size m ( e.g. , in images each pixel is a factor and it depends only on neighboring pixels ) . The key benefit of FMDPs is the combinatorial state space that allows compact representation . That is , although the number of states is exponential in d , the FMDP representation is only exponential in m ( which is much smaller ) and polynomial in d. Early works [ Kearns and Koller , 1999 , Guestrin et al. , 2002 , Strehl , 2007 , Szita and Lőrincz , 2009 ] show that FMDPs also reduce the sample complexity exponentially , thus avoiding polynomial dependence on the number of states . Recently , this was further extended to algorithms with near-optimal regret bounds [ Osband and Van Roy , 2014 , Xu and Tewari , 2020 , Tian et al. , 2020 , Chen et al. , 2021 , Talebi et al. , 2021 ] . However , all these works make the strong assumption that the underlying FMDP structure is fully known to the learner in advance . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper we provide the first regret minimization algorithm for FMDPs with unknown structure , thus solving an open problem from Osband and Van Roy [ 2014 ] . Our algorithm is built on a novel concept of consistent scopes and guarantees near-optimal √ T regret that scales polynomially with the FMDP encoding and is therefore exponentially smaller than the number of states ( and the regret in tabular MDPs ) . Moreover , our algorithm features an innovative construction that can incorporate elimination of inconsistent scopes into the optimistic regret minimization framework , while maintaining computational efficiency given oracle-access to an FMDP planner . Keeping computational oracle-efficiency is a difficult challenge in factored MDPs and especially hard when structure is unknown , since the number of possible structure configurations is highly exponential . Furthermore , our algorithm easily accommodates any level of structure knowledge , and is therefore extremely useful when additional prior domain knowledge is available . We note that while structure learning in FMDPs was previously studied by Strehl et al . [ 2007 ] , Diuk et al . [ 2009 ] , Chakraborty and Stone [ 2011 ] , Hallak et al . [ 2015 ] , Guo and Brunskill [ 2017 ] , none of them provide regret guarantees . To make our algorithms compatible with existing approximate FMDP planners , we also study FMDPs with non-factored actions . To the best of our knowledge , existing planners require small non-factored action space which is not compatible with the FMDP regret minimization literature . To mitigate this gap , we show that even when the oracle is limited to non-factored actions , a variant of our algorithm can still be implemented efficiently and achieve similar near-optimal regret bounds . Finally , we leverage the techniques presented in this paper to prove a novel lower bound for regret minimization in FMDPs with known structure . This is the first lower bound to show that the regret must scale exponentially with the scope size m , and the first to utilize connections between different factors in a non-trivial way ( i.e. , with scope size larger than 1 ) . Furthermore , it improves previous lower bounds by a factor of √ d and closes the gap to the state-of-the-art regret bound of Chen et al . [ 2021 ] , thus establishing the minimax optimal regret in this setting . Our algorithms make oracle use of FMDP planners . However , even where an FMDP can be represented concisely , solving for the optimal policy may take exponentially long in the most general case [ Goldsmith et al. , 1997 , Littman , 1997 ] . Our focus in this paper is upon the statistical aspect of the learning problem , and we therefore assume oracle-access to an FMDP planner . We emphasize that the oracle assumption appears in all previous regret minimization algorithms . Furthermore , except for the DORL algorithm of Xu and Tewari [ 2020 ] , all previous algorithms run in time exponential in d even with access to a planning oracle ( and known structure ) . We stress that in many cases of interest , effective approximate planners do exist [ Boutilier et al. , 2000 , Koller and Parr , 2000 , Schuurmans and Patrascu , 2001 , Guestrin et al. , 2001 , 2003 , Sanner and Boutilier , 2005 , Delgado et al. , 2011 ] . 2 Preliminaries . An infinite-horizon average-reward MDP is described by a tuple M = ( S , A , P , R ) , where S and A are finite state and action spaces , respectively , P : S ×A→ ∆S is the transition function1 , and R : S ×A→ ∆ [ 0,1 ] is the reward function with expectation r ( s , a ) = E [ R ( s , a ) ] . The interaction between the MDP and the learner proceeds as follows . The learner starts in an arbitrary initial state s1 ∈ S. For t = 1 , 2 , . . . , the learner observes the current state st ∈ S , picks an action at ∈ A and earns a reward rt sampled from R ( st , at ) . Then , the environment draws the next state st+1 ∼ P ( · | st , at ) and the process continues . A policy π : S → A is a mapping from states to actions , and its gain is defined by the average-reward criterion : λ ( M , π , s ) def= limT→∞ 1T E [ ∑T t=1 r ( s t , π ( st ) ) | s1 = s ] , where st+1 ∼ P ( · | st , π ( st ) ) . In order to derive non-trivial regret bounds , one must constrain the connectivity of the MDP [ Bartlett and Tewari , 2009 ] . We focus on communicating MDPs , i.e. , MDPs with finite diameter D < ∞ . Definition 1 . Let T ( s′ | M , π , s ) be the random variable for the first time step in which state s′ is reached when playing a stationary policy π in an MDP M with initial state s. The diameter of M is defined as D ( M ) def= maxs6=s′∈S minπ : S→A E [ T ( s′ |M , π , s ) ] . For communicating MDPs , neither the optimal policy nor its gain depend on the initial state s1 . We denote them by π ? ( M ) = arg maxπ : S→A λ ( M , π , s 1 ) and λ ? ( M ) = λ ( M , π ? , s1 ) , respectively . 1∆X denotes the set of distributions over a set X . We measure the performance of the learner by the regret . That is , the difference between the expected gain of the optimal policy in T steps and the cumulative reward obtained by the learner up to time T , i.e. , RegT ( M ) def = ∑T t=1 ( λ ? ( M ) − rt ) , where rt ∼ R ( st , at ) and at is chosen by the learner . 2.1 Factored MDPs . Factored MDPs inherit the above definitions , but also possess some conditional independence structure that allows compact representation . We follow the factored MDP definition of Osband and Van Roy [ 2014 ] , which generalizes the original definition of Boutilier et al . [ 2000 ] , Kearns and Koller [ 1999 ] to allow a factored action space as well . We start with a definition of a factored set and scope operation . Definition 2 . A set X is called factored if it can be written as a product of n sets X1 , . . . , Xn , i.e. , X = X1 × · · · ×Xn . For any subset of indices Z = { i1 , . . . , i|Z| } ⊆ { 1 , . . . , n } , define the scope set X [ Z ] = Xi1 × · · · ×Xi|Z| . Further , for any x ∈ X define the scope variable x [ Z ] ∈ X [ Z ] to be the value of the variables xi ∈ Xi with indices i ∈ Z . For singleton sets we write x [ i ] for x [ { i } ] . Next , we define the factored reward and transition functions . We use the notations X = S ×A for the state-action space , d for the number of state factors and n for the number of state-action factors . Definition 3 . A reward functionR is called factored overX = X1×· · ·×Xn with scopesZr1 , . . . , Zr ` if there exist functions { Rj : X [ Zrj ] → ∆ [ 0,1 ] } ` j=1 with expectations rj ( x [ Zrj ] ) = E [ Rj ( x [ Zrj ] ) ] such that for all x ∈ X : R ( x ) = 1 ` ∑ ` j=1Rj ( x [ Z r j ] ) . Note that when a reward r = 1 ` ∑ ` j=1 rj is sampled from R ( x ) , the learner observes every rj individually . Definition 4 . A transition functionP is called factored overX = X1×· · ·×Xn and S = S1×· · ·×Sd with scopes ZP1 , . . . , Z P d if there exist functions { Pi : X [ ZPi ] → ∆Si } di=1 such that for all x ∈ X and s′ ∈ S : P ( s′ | x ) = ∏d i=1 Pi ( s ′ [ i ] | x [ ZPi ] ) . That is , given a state-action pair x , factor i of s′ is independent of its other factors , and is determined only by x [ ZPi ] . Thus , a factored MDP ( FMDP ) is defined by an MDP whose reward and transition functions are both factored , and is fully characterized by the tuple M = ( { Xi } ni=1 , { Si , ZPi , Pi } di=1 , { Zrj , Rj } ` j=1 ) . As opposed to previous works [ Osband and Van Roy , 2014 , Xu and Tewari , 2020 , Tian et al. , 2020 , Chen et al. , 2021 ] that assume known factorization , in this paper the learner does not have any prior knowledge of the scopes ZP1 , . . . , Z P d or Z r 1 , . . . , Z r ` , and they need to be learned from experience . However , the learner has a bound m on the size of the scopes , i.e. , |ZPi | ≤ m and |Zrj | ≤ m ∀i , j . See remarks on unknown scope size and variable scope sizes in Appendix B . Remark ( FMDP encoding size ) . Let the action factorization A = Ad+1 × · · · × An , factor size W = max { max1≤i≤d |Si| , maxd+1≤i≤n |Ai| } and L = maxZ : |Z|=m |X [ Z ] | . The encoding size is O ( dWL+ ` L+ ( d+ ` ) m log n ) . Importantly , the encoding is only polynomial in d while the number of states W d is exponential . It is however exponential in the ( much smaller ) scope size as L ≈Wm .

Summary: This work concerns reinforcement learning in factored MDPs. The first part of the work combines previous work on regret minimization in FMDPs using a planning oracle (following prior work by Osband and Van Roy and Xu and Tewari) with work on structure learning in FMDPs. In this context, structure learning means the following: in order for FMDPs to possess compact transition and reward functions, one typically assumes they are represented by graphical models in which each factor depends on a limited number of factors from the previous step; the structure learning problem is essentially to identify these dependencies. Here, Xu and Tewari's algorithm is modified so that it identifies the structure during learning, while maintaining its regret bound. The previous works in this area provided sample complexity bounds rather than regret bounds. In addition to the regret bounds, experiments are provided, demonstrating that the algorithm is somewhat competitive with the algorithm of Osband and Van Roy that is provided with the dependency structure.  In more detail, as in the algorithm of Xu and Tewari, this algorithm uses the idea of augmenting the actions that originally appeared in Jaksch et al.; essentially, in addition to allowing the planner to select an action in the actual MDP, Xu and Tewari allow the planner to select one of the extremal transition functions. Here, the possible dependency structures are added to the options presented to the planner.  An additional transformation is proposed that converts the resulting large action space to a sequence of moves in an augmented state space. The same transformation could be applied to the algorithm of Xu and Tewari as well as the unknown structure algorithm, so this could be considered a second, separate part of the work.  The final part of the work is an improved lower bound that in particular shows that regret must depend exponentially on the scope sizes, and asymptotically matches the regret obtained by some recent works (e.g., Tian et al. 2020)