PROVABLY BENEFITS OF DEEP HIERARCHICAL RL

1 INTRODUCTION . Reinforcement learning ( RL ) is a powerful tool to solve sequential decision making problems in various domains , including computer games ( Mnih et al. , 2013 ) , Go ( Silver et al. , 2016 ) , robotics ( Schulman et al. , 2015 ) . A particular feature in these successful applications of RL is that these tasks are concrete enough to be solved by primitive actions and do not require high-level planning . Indeed , when the problem is complex and requires high-level planning , directly applying RL algorithms can not solve the problem . An example is the Atari game Montezuma ’ s Revenge , in which the agent needs to find keys , kills monsters , move to correct rooms , etc . This is notoriously difficulty problem that requires more sophisticated high-level planning . Hierarchical reinforcement learning ( HRL ) is powerful framework that explicitly incorporate highlevel planning . Roughly speaking , HRL divides the decision problem into multiple layers and each layer has its own state space . States in higher layers represent more abstraction and thus higher layer states are some time named meta-states . When number of layers of abstraction is large , we call this framework , deep hierarchical reinforcement learning ( deep HRL ) . In deep HRL , the agent makes decisions by looking at states from all layers . The dependency on higher layer states represents high-level planning . HRL has been successfully applied to many domains that require high-level planning , including autonomous driving ( Chen et al. , 2018 ) , recommendation system ( Zhao et al. , 2019 ) , robotics ( Morimoto & Doya , 2001 ) . Recently , extension to imitation learning has also been studied ( Le et al. , 2018 ) . While being a practically powerful framework , theoretical understanding on HRL is still limited . Can we provably show the benefit of using HRL instead of naı̈ve RL ? In particular , what can we gain from multi-layer abstraction of deep HRL ? Existing theories mostly focus on option RL setting , which transits from the upper layer to lower layer when some stopping criterion is met ( Fruit et al. , 2017 ) . This is different from our setting requiring horizons in each layer to be the same , which is common in computer games and autonomous driving Moreover , the number of samples needed in Fruit et al . ( 2017 ) is proportional to the total number of states and total number of actions . In our setting , both the total number of states and number of actions can be exponentially large and hence their algorithm becomes impractical . We initiate the study of rigorously characterizing the complexity of deep HRL and explaining its benefits compared to classical RL . We study the most basic form , tabular deep HRL in which there are total L-layers and each layer has its own state space S ` for ` ∈ [ L ] . One can simply apply classical RL algorithm on the enlarged state space S = S1 × S2 × · × SL . The sample complexity will however depend on the size of the enlarged states space |S| = ΠL ` =1 |S ` | . In this paper , we show because of the hierarchical structure , we can reduce the sample complexity exponentially , from ∝ poly ( ΠL ` |S ` | ) to ∝ ∑L ` =1 poly ( |S ` | ) . We achieve this via a model-based algorithm which carefully constructs confidence of the model in a hierarchical manner . We fully exploit the structure that lower-level MDPs of different high-level states share the same transition probability , which allows us to combine the information collected at different high-level states and use it to give an accurate estimator of the model for all low-level MDPs . Due to this information aggregation , we are able to improve the sample complexity bound . To our knowledge , this is the first theoretical result quantifying the complexity of deep HRL and explain its benefit comparing to classical RL . Organization . This paper is organized as follows . In Section 2 we discuss related work . In Section 3 , we review basic RL concepts and formalize deep HRL . In Section 4 , we present our main algorithm and present its theoretical guarantees . In Section 5 , we give a proof sketch of our main theorem . We conclude in Section 6 and defer some technical lemmas to appendix . 2 RELATED WORK . We are going to provide several related literatures on tabular MDP and hierarchical reinforcement learning in this section . As for tabular MDP , many works focus on solving MDP with a simulator which can provide samples of the next state and reward given the current state-action pair . These work includes Lattimore & Hutter ( 2012 ) ; Azar et al . ( 2013 ) ; Sidford et al . ( 2018b ; a ) ; Agarwal et al . ( 2019 ) . Since we do not need to consider the balance between exploration and exploitation , this setting is easier than the setting of minimizing the regret . There are also a line of work on analysis of the regret bound in RL setting . Jaksch et al . ( 2010 ) and Agrawal & Jia ( 2017 ) propose a model-based reinforcement learning algorithm , which estimates the transition model using past samples and add a bonus to the estimation . Their algorithms achieve regret bound Õ ( √ H4S2AT ) and Õ ( √ H3S2AT ) respectively . Later , the UCBVI algorithm in Azar et al . ( 2017 ) adds bonus term to the Q function directly , and achieves the regret bound Õ ( √ H2SAT ) , which matches the lower bound when the number of episode is sufficiently large . Adopting the technique of variance reduction , the vUCQ algorithm in Kakade et al . ( 2018 ) improves the lower order term in the regret bound . Jin et al . ( 2018 ) proposed a model-free Q-learning algorithm is proved to achieve regret bound Õ ( √ H3SAT ) . Hierarchical reinforcement learning Barto & Mahadevan ( 2003 ) are also broadly studied in Dayan & Hinton ( 1993 ) ; Parr & Russell ( 1998 ) ; Sutton et al . ( 1999 ) ; Dietterich ( 2000 ) ; Stolle & Precup ( 2002 ) ; Bacon et al . ( 2017 ) ; Florensa et al . ( 2017 ) ; Frans et al . ( 2017 ) . The option framework , which is studied in Sutton et al . ( 1999 ) ; Precup ( 2001 ) , is another popular formulation used in hierarchical RL . In Fruit et al . ( 2017 ) , the regret analysis is carried on option reinforcement learning , but their analysis only applies to the setting of option RL . To our current knowledge , there is no such work analyzing the regret bound of multi-level hierarchical RL . 3 PRELIMINARIES . 3.1 EPISODIC MARKOV DECISION PROCESS . In this paper , we consider finite horizon Markov decision process ( MDP ) . An MDP is specified by a tuple ( S , A , H , P , r ) , where S is the ( possibly uncountable ) state space , A is a finite action space , H ∈ Z+ is a planning horizon , P : S×A → 4 ( S ) is the transition function , and r : S×A → [ 0 , 1 ] is the reward function . At each state s ∈ S , an agent is able to interact with the MDP by playing an action a ∈ A . Once an action a is played on state s , the agent receives an immediate reward r ( s , a ) ∈ [ 0 , 1 ] 1 , and the state transitions to next state s′ with probability P ( s′|s , a ) . Starting 1Here we only consider cases where rewards are in [ 0 , 1 ] , but it is easy to know that this result can generalize to rewards in a different range using standard reduction Sidford et al . ( 2018a ) . We can also generalize our from some initial state s1 ∈ S ( draw from some distribution ) , the agent is able to play H steps ( an episode ) and then the system resets to another initial state s1 sampled from the initial distribution . For an MDP , our goal is to obtain an optimal ( will be precise shortly ) policy , π : S → A , which is a function that maps each state to an action . If an agent always follows the action given by a policy π , then it induces a random trajectory for an episode : s1 , a1 , r1 , s2 , a2 , r2 , . . . , sH , aH , rH where r1 = r ( s1 , a1 ) , s2 ∼ P ( ·|s1 , a1 ) , a2 ∼ π ( s2 ) , etc . The value function and Q-function at step h of a given policy is defined as V πh ( s ) = Eπ [ ∑H h′=h rh′ ∣∣sh = s ] and Qπh ( s , a ) = Eπ [ ∑H h′=h ∣∣sh = s , ah = a ] , where the expectation is over all sample trajectories . Then the optimal policy , π∗ , is defined to be the policy with largest V π1 ( s1 ) for all s1 ∈ S. For any optimal policy , its value and Q-function satisfy the following Bellman equation ∀s ∈ S , a ∈ A , h ∈ [ H ] : Q∗h ( s , a ) = r ( s , a ) + Es′∼P ( ·|s , a ) V ∗h+1 ( s′ ) , V ∗h ( s ) = max a∈A Q∗h ( s , a ) and V ∗H+1 ( s ) = 0 . ( 1 ) We consider the MDP problem in the online learning setting , where the probability transition is unknown . However , our goal is still to collect the maximum amount reward , i.e. , play a policy that is comparable to the optimal one . Therefore , the agent needs to learn to play through trial and error , i.e. , improving the policy by learning from experiences . Suppose we allow the agent to play in total K ≥ 1 episodes . For each episode , the agent is following a policy πk , which is computed based on her experiences collected from episodes 1 , 2 , . . . , k − 1 . To measure the performance of the agent , we use the following standard regret formulation , which compares the reward collected by the agent to the performance of an optimal policy . R ( K ) = K∑ k=1 ( V ∗1 ( s1 ) − V πk 1 ( s1 ) ) . ( 2 ) Note that , if the agent learns nothing , we then expect R ( K ) ∝ K. But if the agent is able to learn , then the average regret , R ( K ) /K , which measures the average error per step , goes to 0 when K becomes large . In the online MDP literature , model based algorithms ( e.g . Jaksch et al . ( 2010 ) ) achieves regret R ( K ) ≤ Õ ( √ H2|S|2|A|HK ) .

Summary: This paper proposes a new kind of episodic finite MDPs called "deep hierarchical MDP" (hMDP). An L-layer hMDP can be *roughly* thought of as L episodic finite MDPs stacked together. A variant of UCRL2 [JOA10] is proposed to solve these hMDPs and some results from its regret analysis are provided.