Provable Hierarchy-Based Meta-Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has demonstrated tremendous successes in many domains ( Schulman et al. , 2015 ; Vinyals et al. , 2019 ; Schrittwieser et al. , 2020 ) , learning near-optimal policies despite limited supervision . Nevertheless , RL remains difficult to apply to problems requiring temporally extended planning and/or exploration ( Ecoffet et al. , 2021 ) . A promising approach to this problem is hierarchical reinforcement learning ( HRL ) , which has seen continued interest due to its appealing biological basis . In its most basic form , HRL seeks to decompose tasks into a sequence of skills , each of which is easier to learn individually than the full task . By restricting the agent to using learned skills , the search space over policies can be greatly reduced . Furthermore , learned skills can induce simpler state and/or action spaces , simplifying the learning problem . Finally , learned skills with useful semantic behavior can be reused across tasks , enabling transfer learning . Naturally , a hierarchy-based learner is limited by the quality of skills that are made available and/or learned . Accordingly , many empirical works have proposed algorithms for online skill learning in the context of a single RL task ( Nachum et al. , 2019a ; 2018 ) . These approaches have been experimentally demonstrated to be effective in finding useful and interpretable skills . Other approaches consider the skill learning problem in the context of meta-RL ( Frans et al. , 2018 ) , or in the rewardfree setting ( Eysenbach et al. , 2018 ) . Nevertheless , the heuristics and algorithms proposed in these empirical works do not provide any provable guarantees on the quality of learned skills . On the other hand , theoretical analyses have mostly focused on how learners benefits from having access to skills . For example , Fruit & Lazaric ( 2017 ) provide a regret bound on learning with skills in the infinite-horizon average reward case . Meanwhile , in the meta-RL setting , Brunskill & Li ( 2014 ) considers the problem of finding and using skills in a continual learning setting and provides a sample complexity analysis . However , these analyses either sidestep the question of how the skills are obtained , or do not address the problem in a computationally tractable manner . In this work , we aim to take a step towards providing provable guarantees for hierarchy learning through tractable algorithms . We focus on the meta-RL setting , in which a learner extracts skills from a set of provided tasks which are then used in a downstream task . We work in the tabular case , assuming the transition dynamics of the given tasks share latent hierarchical structure induced by predetermined clustering and bottlenecks . Our contributions are as follows : 1 . “ Diversity conditions ” ensuring hierarchy recovery . We develop natural optimismbased coverage conditions which ensure that bottlenecks embedded in the transition dynamics are detectable by solving provided meta-training tasks . 2 . A tractable hierarchy-learning algorithm . We provide an algorithm that provably learns the latent hierarchy from interactions , assuming the coverage conditions above . Our method has sample complexity scaling as O ( TKS ) in the leading term compared to O ( TS2A ) for a brute-force method , where T are the number of tasks , S is the number of states , A is the number of actions , and K SA is the number of skills to learn . 3 . Regret bounds on downstream tasks . We provide regret bounds for learners that apply the extracted hierarchy from meta-training on downstream tasks . Furthermore , we show an exponential regret separation between hierarchy-based and hierarchy-oblivious learners for a family of task distributions , corroborating prevailing intuitions regarding when/why HRL helps . In particular , hierarchy-based learners incur regret bounded by O ( √ H2N ) while hierarchy-oblivious learners incur worst-case regret of at least O ( 2H/2 √ H2N ) . 2 NOTATION . We now introduce notation which we will use throughout the paper . We write [ K ] : = { 1 , . . . , K } . Furthermore , we use the standard notations O , Θ , Ω to denote orders of growth , and Õ , Θ̃ , Ω̃ to indicate suppressed logarithmic factors . We use δ ( x ) to denote the Dirac delta measure on x . We work with finite-horizon Markov decision processes ( MDPs ) , defined as a tuple M = ( S , A , P , r , H ) , where S is the set of states , A is the set of actions , P : S × A × S → [ 0 , 1 ] are the transition dynamics , r : S × A → [ 0 , 1 ] is the reward function , and H is the horizon . We assume stationary dynamics unless otherwise noted , in which case P ( h ) is the dynamics at time step h. For constants relating to horizons , we will define [ H ] : = { 0 , . . . , H − 1 } . Given a policy π : [ H ] × S → A , we define the value functions V πh ( s ) : = E [ H−1∑ k=h r ( sk , ak ) ∣∣∣∣∣ sh = s ] and Qπh ( s , a ) : = E [ H−1∑ k=h r ( sk , ak ) ∣∣∣∣∣ ( sh , ah ) = ( s , a ) ] where sk+1 ∼ P ( · | sk , ak ) and ak = πk ( sk ) . Furthermore , we write V ∗ and Q∗ to denote optimal value functions obtained by maximizing over π ( and are attained by the optimal policy π∗ ) . When a learner plays π1 , . . . , πN inM , we define its regret as RegretN ( M ) : = N∑ t=1 V ∗0 ( s0 ) − V πt 0 ( s0 ) . We use to denote a terminal state . We let τπ denote the ( random ) trajectory generated by π . For a state s and length-H trajectory τ , we write s ∈ τπ if sh = s for some h ∈ [ H ] . We define ( s , a ) ∈ τπ similarly . Finally , given an MDPM , M ( 2H ) denotes a copy ofM with a doubled horizon . 3 SETTING . We work in the tabular meta-RL setting . The learner can access T meta-training MDPs { ( S , A , Pt , rt , H ) } t∈ [ T ] = { Mt } t∈ [ T ] . Note that Pt and rt both vary across tasks . We set S : = |S| and A : = |A| . After interacting with these tasks , the learner is presented with a meta-test MDP ( S , A , PTg , rTg , H ) , where the learner seeks to minimize its regret . We assume , without loss of generality , that the MDPs have a shared starting state s0 . For meta-learning to succeed , the MDPs must have shared structure . We focus on the following notion of shared hierarchical structure : Definition 3.1 ( Latent Hierarchy ) . Let { Zc } be a partition of the state space S into clusters . We associate with each cluster Z a set of entrances Ent ( Z ) ⊆ Z and a set of exits Ext ( Z ) ⊆ Z × A . We say that the tasks have a latent hierarchy with respect to ( { Zc } , Ent ( · ) , Ext ( · ) ) if for any Zc : ( a ) For any ( s , a ) ∈ ( Zc×A ) \Ext ( Zc ) , Pt ( · | s , a ) is constant over t and supported on Zc . ( b ) For any ( s , a ) ∈ Ext ( Zc ) , there exists t , t′ with t 6= t′ such that Pt ( · | s , a ) 6= Pt′ ( · | s , a ) . Furthermore , Pt ( · | s , a ) is supported on ⋃ c Ent ( Zc ) for any t ∈ [ T ] . Definition 3.1 partitions the shared state space of the MDPs into clusters such that ( 1 ) non-exit ( s , a ) dynamics do not change between the MDPs and ( 2 ) exits are bottlenecks between clusters . Example 3.1 ( Gated Four-Room ) . Figure 1 illustrates the gated four-room environment along with a sample task . The environment has a latent hierarchy with respect to the rooms outlined by colored gates ( which can be open or closed ) . Entrances are colored aqua , while exits are marked by arrows . Although we assume a single fixed starting state , we can model task-dependent initial states by appending a dummy state s0 . A dummy action a0 then takes the agent to the starting state for the task . Observe that ( s0 , a0 ) is an exit , and therefore must transition to an entrance . y To see how Definition 3.1 captures intuitive notions of hierarchy in practical settings , we provide an example of a continuous setting roughly fitting into our framework : Example 3.2 ( The Alchemy benchmark ) . Alchemy ( Wang et al. , 2021 ) is a recently proposed empirical benchmark for meta-RL , where the agent needs to place a stone in a series of potions to obtain some desired appearance , as illustrated in Figure 2 . Dipping a stone into a potion traverses an edge ( determined by the potion ) in a graph where nodes are possible stone appearances . We focus on task distributions that randomize the edges of this graph ( i.e. , potion positions and feasible stone appearances are fixed ) . Then , the set of obtainable MDPs has a latent hierarchy where dipping the stone into any of the potions is an exit . Indeed , other than dipping the stone into a potion , all other actions ( e.g. , moving the stone around the room ) have the same dynamics in all tasks . y For convenience , we will define several relevant notions . First , for any clusterZ , we define its interior Z◦ : = ( Z×A ) \Ext ( Z ) . Furthermore , we let Ent ( S ) : = ⋃ c Ent ( Zc ) denote the set of all entrances and Ext ( S ) : = ⋃ c Ext ( Zc ) the set of all exits . Finally , we define K : = |Ext ( S ) | L : = |Ent ( S ) | M : = sup c |Ext ( Zc ) | so that K and L are the total number of exits and entrances , respectively , while M is the maximal number of exits from any cluster . Query Model . We work in the online setting , where the agent interacts with the tasks by playing policies from the initial state s0 . During meta-training , we allow the agent to interact with the environments using an unbounded number of timesteps for each trajectory before resetting . We then compute query complexity in terms of the total number of timesteps spent in all tasks in total .

Summary: This paper proposes a novel formulation  to analyze the provable benefits of hierarchical RL algorithms. Based on this new formulation of hierarchical structures, they propose new algorithms to learn the latent hierarchy and apply the extracted hierarchy on the downstream tasks. Under several assumptions, they prove that their algorithms enjoy better regret or sample complexity bounds than brute-force methods.