Provable Hierarchy-Based Meta-Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has demonstrated tremendous successes in many domains ( Schulman et al. , 2015 ; Vinyals et al. , 2019 ; Schrittwieser et al. , 2020 ) , learning near-optimal policies despite limited supervision . Nevertheless , RL remains difficult to apply to problems requiring temporally extended planning and/or exploration ( Ecoffet et al. , 2021 ) . A promising approach to this problem is hierarchical reinforcement learning ( HRL ) , which has seen continued interest due to its appealing biological basis . In its most basic form , HRL seeks to decompose tasks into a sequence of skills , each of which is easier to learn individually than the full task . By restricting the agent to using learned skills , the search space over policies can be greatly reduced . Furthermore , learned skills can induce simpler state and/or action spaces , simplifying the learning problem . Finally , learned skills with useful semantic behavior can be reused across tasks , enabling transfer learning . Naturally , a hierarchy-based learner is limited by the quality of skills that are made available and/or learned . Accordingly , many empirical works have proposed algorithms for online skill learning in the context of a single RL task ( Nachum et al. , 2019a ; 2018 ) . These approaches have been experimentally demonstrated to be effective in finding useful and interpretable skills . Other approaches consider the skill learning problem in the context of meta-RL ( Frans et al. , 2018 ) , or in the rewardfree setting ( Eysenbach et al. , 2018 ) . Nevertheless , the heuristics and algorithms proposed in these empirical works do not provide any provable guarantees on the quality of learned skills . On the other hand , theoretical analyses have mostly focused on how learners benefits from having access to skills . For example , Fruit & Lazaric ( 2017 ) provide a regret bound on learning with skills in the infinite-horizon average reward case . Meanwhile , in the meta-RL setting , Brunskill & Li ( 2014 ) considers the problem of finding and using skills in a continual learning setting and provides a sample complexity analysis . However , these analyses either sidestep the question of how the skills are obtained , or do not address the problem in a computationally tractable manner . In this work , we aim to take a step towards providing provable guarantees for hierarchy learning through tractable algorithms . We focus on the meta-RL setting , in which a learner extracts skills from a set of provided tasks which are then used in a downstream task . We work in the tabular case , assuming the transition dynamics of the given tasks share latent hierarchical structure induced by predetermined clustering and bottlenecks . Our contributions are as follows : 1 . “ Diversity conditions ” ensuring hierarchy recovery . We develop natural optimismbased coverage conditions which ensure that bottlenecks embedded in the transition dynamics are detectable by solving provided meta-training tasks . 2 . A tractable hierarchy-learning algorithm . We provide an algorithm that provably learns the latent hierarchy from interactions , assuming the coverage conditions above . Our method has sample complexity scaling as O ( TKS ) in the leading term compared to O ( TS2A ) for a brute-force method , where T are the number of tasks , S is the number of states , A is the number of actions , and K SA is the number of skills to learn . 3 . Regret bounds on downstream tasks . We provide regret bounds for learners that apply the extracted hierarchy from meta-training on downstream tasks . Furthermore , we show an exponential regret separation between hierarchy-based and hierarchy-oblivious learners for a family of task distributions , corroborating prevailing intuitions regarding when/why HRL helps . In particular , hierarchy-based learners incur regret bounded by O ( √ H2N ) while hierarchy-oblivious learners incur worst-case regret of at least O ( 2H/2 √ H2N ) . 2 NOTATION . We now introduce notation which we will use throughout the paper . We write [ K ] : = { 1 , . . . , K } . Furthermore , we use the standard notations O , Θ , Ω to denote orders of growth , and Õ , Θ̃ , Ω̃ to indicate suppressed logarithmic factors . We use δ ( x ) to denote the Dirac delta measure on x . We work with finite-horizon Markov decision processes ( MDPs ) , defined as a tuple M = ( S , A , P , r , H ) , where S is the set of states , A is the set of actions , P : S × A × S → [ 0 , 1 ] are the transition dynamics , r : S × A → [ 0 , 1 ] is the reward function , and H is the horizon . We assume stationary dynamics unless otherwise noted , in which case P ( h ) is the dynamics at time step h. For constants relating to horizons , we will define [ H ] : = { 0 , . . . , H − 1 } . Given a policy π : [ H ] × S → A , we define the value functions V πh ( s ) : = E [ H−1∑ k=h r ( sk , ak ) ∣∣∣∣∣ sh = s ] and Qπh ( s , a ) : = E [ H−1∑ k=h r ( sk , ak ) ∣∣∣∣∣ ( sh , ah ) = ( s , a ) ] where sk+1 ∼ P ( · | sk , ak ) and ak = πk ( sk ) . Furthermore , we write V ∗ and Q∗ to denote optimal value functions obtained by maximizing over π ( and are attained by the optimal policy π∗ ) . When a learner plays π1 , . . . , πN inM , we define its regret as RegretN ( M ) : = N∑ t=1 V ∗0 ( s0 ) − V πt 0 ( s0 ) . We use to denote a terminal state . We let τπ denote the ( random ) trajectory generated by π . For a state s and length-H trajectory τ , we write s ∈ τπ if sh = s for some h ∈ [ H ] . We define ( s , a ) ∈ τπ similarly . Finally , given an MDPM , M ( 2H ) denotes a copy ofM with a doubled horizon . 3 SETTING . We work in the tabular meta-RL setting . The learner can access T meta-training MDPs { ( S , A , Pt , rt , H ) } t∈ [ T ] = { Mt } t∈ [ T ] . Note that Pt and rt both vary across tasks . We set S : = |S| and A : = |A| . After interacting with these tasks , the learner is presented with a meta-test MDP ( S , A , PTg , rTg , H ) , where the learner seeks to minimize its regret . We assume , without loss of generality , that the MDPs have a shared starting state s0 . For meta-learning to succeed , the MDPs must have shared structure . We focus on the following notion of shared hierarchical structure : Definition 3.1 ( Latent Hierarchy ) . Let { Zc } be a partition of the state space S into clusters . We associate with each cluster Z a set of entrances Ent ( Z ) ⊆ Z and a set of exits Ext ( Z ) ⊆ Z × A . We say that the tasks have a latent hierarchy with respect to ( { Zc } , Ent ( · ) , Ext ( · ) ) if for any Zc : ( a ) For any ( s , a ) ∈ ( Zc×A ) \Ext ( Zc ) , Pt ( · | s , a ) is constant over t and supported on Zc . ( b ) For any ( s , a ) ∈ Ext ( Zc ) , there exists t , t′ with t 6= t′ such that Pt ( · | s , a ) 6= Pt′ ( · | s , a ) . Furthermore , Pt ( · | s , a ) is supported on ⋃ c Ent ( Zc ) for any t ∈ [ T ] . Definition 3.1 partitions the shared state space of the MDPs into clusters such that ( 1 ) non-exit ( s , a ) dynamics do not change between the MDPs and ( 2 ) exits are bottlenecks between clusters . Example 3.1 ( Gated Four-Room ) . Figure 1 illustrates the gated four-room environment along with a sample task . The environment has a latent hierarchy with respect to the rooms outlined by colored gates ( which can be open or closed ) . Entrances are colored aqua , while exits are marked by arrows . Although we assume a single fixed starting state , we can model task-dependent initial states by appending a dummy state s0 . A dummy action a0 then takes the agent to the starting state for the task . Observe that ( s0 , a0 ) is an exit , and therefore must transition to an entrance . y To see how Definition 3.1 captures intuitive notions of hierarchy in practical settings , we provide an example of a continuous setting roughly fitting into our framework : Example 3.2 ( The Alchemy benchmark ) . Alchemy ( Wang et al. , 2021 ) is a recently proposed empirical benchmark for meta-RL , where the agent needs to place a stone in a series of potions to obtain some desired appearance , as illustrated in Figure 2 . Dipping a stone into a potion traverses an edge ( determined by the potion ) in a graph where nodes are possible stone appearances . We focus on task distributions that randomize the edges of this graph ( i.e. , potion positions and feasible stone appearances are fixed ) . Then , the set of obtainable MDPs has a latent hierarchy where dipping the stone into any of the potions is an exit . Indeed , other than dipping the stone into a potion , all other actions ( e.g. , moving the stone around the room ) have the same dynamics in all tasks . y For convenience , we will define several relevant notions . First , for any clusterZ , we define its interior Z◦ : = ( Z×A ) \Ext ( Z ) . Furthermore , we let Ent ( S ) : = ⋃ c Ent ( Zc ) denote the set of all entrances and Ext ( S ) : = ⋃ c Ext ( Zc ) the set of all exits . Finally , we define K : = |Ext ( S ) | L : = |Ent ( S ) | M : = sup c |Ext ( Zc ) | so that K and L are the total number of exits and entrances , respectively , while M is the maximal number of exits from any cluster . Query Model . We work in the online setting , where the agent interacts with the tasks by playing policies from the initial state s0 . During meta-training , we allow the agent to interact with the environments using an unbounded number of timesteps for each trajectory before resetting . We then compute query complexity in terms of the total number of timesteps spent in all tasks in total .

Summary: The authors develop a theoretical framework to discover the latent hierarchical structures shared across meta-training RL tasks, and propose a tractable hierarchy-learning algorithm with provable guarantees. The paper seems to be theoretically solid, and is well organized. I appreciate the proposed notions, such as latent hierarchy, $\beta$-dynamics separation, $\alpha$-importance, and so on. I believe they can provide researchers with a novel perspective for studying hierarchical RL.