Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning

1 Introduction . Deep reinforcement learning ( RL ) has demonstrated wide success in a variety of sequential decisionmaking problems , i.e. , board games [ 38 , 42 ] , video games [ 1 , 26 , 38 ] , and robotic control tasks [ 15 , 30 , 53 ] . However , solving complex and long-horizon tasks has still remained a major challenge in RL , where hierarchical reinforcement learning ( HRL ) provides a promising direction by enabling control at multiple time scales via a hierarchical structure . Among HRL frameworks , goal-conditioned HRL has long been recognized as an effective paradigm [ 5 , 20 , 29 , 37 ] , showing significant success in a variety of long and complex tasks , e.g. , navigation with locomotion [ 21 , 54 ] . The framework comprises a high-level policy and a low-level policy ; the former breaks the original task into a series of subgoals , and the latter aims to reach those subgoals . The effectiveness of goal-conditioned HRL depends on the acquisition of effective and semantically meaningful subgoals . To this end , several strategies have been proposed , e.g. , learning the subgoal representation space [ 6 , 11 , 21 , 29 , 31 , 32 , 35 , 44 , 50 ] or utilizing domain-specific knowledge for pre-defining subgoal space [ 28 , 54 ] . However , the learned or pre-defined subgoal space is often too large , which results in poor high-level exploration , leading to inefficient training . To address this issue , Zhang et al . [ 54 ] recently proposed a reduction of the high-level action space into the k-step adjacency region around the current state . This approach , however , is limited in that it considers all the states within the k-step adjacency region equally as the candidate actions for the high-level policy , without considering the novelty of states , which is crucial for exploration . 1Code is available https : //github.com/junsu-kim97/HIGL 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contribution . In this paper , we present HIerarchical reinforcement learning Guided by Landmarks ( HIGL ) , a novel framework for training a high-level policy that generates a subgoal toward landmarks , i.e. , promising states to explore . HIGL consists of the following key ingredients ( see Figure 1 ) : • Landmark sampling : To effectively sample landmarks that represent promising states to explore , it is important to sample landmarks that cover a wide area of state space and contain novel states . To this end , we propose two sampling schemes : ( a ) coverage-based sampling scheme that samples located as far away from each other as possible and ( b ) novelty-based sampling scheme that stores novel states encountered during training and utilizes them as landmarks . We find that our method successfully samples diverse and novel landmarks . • Landmark-guided subgoal generation : Among the sampled landmarks , we select the most urgent landmark by our landmark selection scheme with the shortest path planning algorithm . Then we propose to shift the action space of a high-level policy toward a selected landmark . Because HIGL constructs a high-level action space that is both ( a ) reachable from the current state and ( b ) shifted towards a promising landmark state , we find that the proposed method can effectively guide the subgoal generation of a high-level policy . We demonstrate the effectiveness of HIGL on various long-horizon continuous control tasks based on MuJoCo simulator [ 48 ] , which is widely used in the HRL literature [ 9 , 16 , 28 , 29 ] . In our experiments , HIGL significantly outperforms the prior state-of-the-art method , i.e. , HRAC [ 54 ] , especially in complex environments with sparse reward signals . For example , HIGL achieves the success rate of 65.1 % in Ant Maze ( sparse ) environment , where HRAC only achieves 17.6 % . 2 Related work . Goal-conditioned HRL . By introducing a hierarchy consisting of a high-level policy and a lowlevel policy , goal-conditioned HRL has been successful in a wide range of tasks . Notably , Nachum et al . [ 28 ] proposed an off-policy correction method for goal-conditioned HRL , and Levy et al . [ 19 ] successfully trained multiple levels of policies in parallel with hindsight transitions . However , the acquisition of effective and semantically meaningful subgoals still remains a challenge . Several works have been proposed to address this problem , including learning-based approaches [ 6 , 11 , 21 , 29 , 31 , 32 , 35 , 44 , 50 ] and domain knowledge-based approaches [ 28 , 54 ] . The work closest to ours is HRAC [ 54 ] that reduces the high-level action space to the k-adjacent region of the current state . Our work differs in that we explicitly consider the novelty of each state within the region instead of treating all states in an equal manner . Subgoal discovery . Identifying useful subgoals to guide training ( low-level ) policy has long been recognized as an effective way to solve complex RL problems [ 23–25 , 43 ] . Recent works provide a subgoal by ( 1 ) constructing an environmental graph and ( 2 ) planning in the graph [ 8 , 14 , 17 , 22 , 36 , 41 , 51 , 52 ] . For ( 1 ) , one important point is to build a graph that represents the entire map enough with a limited number of nodes . To this end , several approaches such as farthest point sampling [ 14 ] and sparsification [ 17 ] were proposed . For ( 2 ) , traditional planning algorithms , e.g. , the Bellman-Ford algorithm , are usually used for offering the most emergent node to reach the final goal . These works have shown promising results in complex RL tasks but have a scalability issue due to increasing planning time with a larger map . Meanwhile , our framework differs from this line of works since ours “ train ” high-level policy rather than hard-coded high-level planning to generate a subgoal . 3 Preliminaries . We formulate a control task with a finite-horizon , goal-conditioned Markov decision process ( MDP ) [ 45 ] defined as a tuple ( S , G , A , p , r , γ , H ) , where S is the state space , G is the goal space , A is the action space , p ( s′|s , a ) is the transition dynamics , r ( s , a ) is the reward function , γ ∈ [ 0 , 1 ) is the discount factor , and H is the horizon . Goal-conditioned HRL . We consider a framework that consists of two hierarchies : a high-level policy π ( g|s ; θhigh ) and a low-level policy π ( a|s , g ; θlow ) , where each policy parameterized by neural networks whose parameters are θhigh and θlow , respectively . At each timestep t , The highlevel policy generates a high-level action , i.e. , subgoal gt ∈ G , by either sampling from its policy gt ∼ π ( g|st ; θhigh ) when t ≡ 0 ( mod k ) , or otherwise using a pre-defined goal transition process gt = h ( gt−1 , st−1 , st ) , i.e. , h ( gt−1 , st−1 , st ) = gt−1+st−1−st for relative subgoal scheme [ 28 , 54 ] , h ( gt−1 , st−1 , st ) = gt−1 for absolute subgoal scheme.2 The low-level policy observes the state st and goal gt , and performs a low-level atomic action at ∼ π ( a|st , gt ; θlow ) . Then , the reward function for the high-level policy is given as the sum of m external rewards from the environment as follows : rhigh ( τt , m ) = m−1∑ i=0 r ( si , ai ) , ( 1 ) where τt , m = { ( st , gt , at ) , · · · , ( st+m−1 , gt+m−1 , at+m−1 ) } denotes a trajectory segment of sizem . The goal of a high-level policy is to maximize the expected sum of rhigh by providing the low-level policy with an intrinsic reward proportional to the distance in the goal space G. Specifically , in a relative subgoal scheme [ 28 , 54 ] , the reward function for a low-level policy is defined as : rlow ( st , gt , at , st+1 ) = −‖gt , ϕ ( st+1 − st ) ‖2 , ( 2 ) where ϕ : S → G is a goal mapping function that maps a state to a goal . Instead , if one replaces with an absolute subgoal scheme , the reward function is substituted as follows : rlow ( st , gt , at , st+1 ) = −‖gt , ϕ ( st+1 ) ‖2 , ( 3 ) Random network distillation . One line of exploration algorithms introduce novelty of a state , that is calculated by prediction errors [ 3 , 13 , 34 , 39 ] , visit-counts [ 2 , 33 , 47 ] , or state entropy estimate [ 12 , 18 , 27 , 40 ] . One well-known method is Random Network Distillation ( RND ) [ 3 ] , which utilizes the prediction error of a neural network as a novelty score . Specifically , let f be a neural network with fixed parameters θ̄ and f̂ be a neural network parameterized by θ. RND updates θ by minimizing the expected mean squared prediction error of f , Es∼B‖f̂ ( s ; θ ) − f ( s ; θ̄ ) ‖2 . Then the novelty score of a state s is defined as : n ( s ) = ‖f̂ ( s ; θ ) − f ( s ; θ̄ ) ‖2 . ( 4 ) The novelty score n ( s ) is likely to be higher for novel states dissimilar to the ones the predictor network θ has been trained on . 2In a relative subgoal scheme , the high-level policy gives a subgoal representing how far the low-level policy should move from its current state . Whereas , in an absolute subgoal scheme , the high-level policy provides an absolute position where the low-level policy should reach . Adjacency network . Let dst ( s , s′ ) be the shortest transition distance from state s to state s′ , i.e. , dst ( s , s ′ ) is the expected number of steps an optimal agent should take to reach the state s′ from s. To estimate dst , Zhang et al . [ 54 ] introduce an adjacency network ψ parameterized by φ , that discriminates whether two states are k-step adjacent or not . The network learns a mapping from a goal space to an adjacency space by minimizing the following contrastive-like loss : Ladj ( φ ) = Esi , sj∈S [ l ·max ( ‖ψφ ( gi ) − ψφ ( gj ) ‖2 − εk , 0 ) + ( 1− l ) ·max ( εk + δ − ||ψφ ( gi ) − ψφ ( gj ) ||2 , 0 ) ] , ( 5 ) where δ > 0 is a margin between embeddings , εk is a scaling factor , and l ∈ { 0 , 1 } represents the label indicating k-step adjacency derived from the k-step adjacency matrixM that stores the adjacency information of the explored states . The equation ( 5 ) penalizes adjacent state embeddings ( l = 1 ) with large Euclidean distances , while non-adjacent state embeddings ( l = 0 ) with small Euclidean distances . Then the shortest transition distance can be estimated as follows : d̂st ( s , s ′ ; φ ) = k εk ||ψφ ( g1 ) , ψφ ( g2 ) ||2 ≈ dst ( s , s′ ) . ( 6 )

Summary: Paper introduces HIGL, a hierarchical RL framework based on landmarks. The main idea is to limit the higher level action space (which is often taken to be the full state space), by restricting it to specific landmarks. These landmarks are defined bases on two criteria: coverage of the visited state space, and novelty of the visited states. These landmarks are then integrated in an HRL framework. Experiments on three U-maze like environments show that their method performs well.