Active Hierarchical Exploration with Stable Subgoal Representation Learning

1 INTRODUCTION . Goal-conditioned hierarchical reinforcement learning ( GCHRL ) has long demonstrated great potential to solve temporally extended tasks ( Dayan & Hinton , 1993 ; Schmidhuber & Wahnsiedler , 1993 ; Vezhnevets et al. , 2017 ; Péré et al. , 2018 ; Nair & Finn , 2019 ) , where a high-level policy periodically sets subgoals to a low-level policy , and the low-level policy is intrinsically rewarded for reaching those subgoals . Early GCHRL studies used a hand-designed subgoal space , such as positions of robots ( Nachum et al. , 2018 ; Levy et al. , 2019 ) or objects in images ( Kulkarni et al. , 2016a ) . To alleviate the dependency on domain-specific knowledge , recent works investigate learning subgoal representations along with hierarchical policies ( Nachum et al. , 2019a ; Dilokthanakul et al. , 2019 ; Li et al. , 2021 ) , which have shown promise in a more general setting . Although the exploration ability of HRL is boosted via decomposing tasks and reusing low-level policies , existing GCHRL methods struggle in long-horizon tasks with sparse external rewards . The high-level policy learning in GCHRL relies on external rewards , and those tasks are hard to solve with naive high-level exploration strategies . Furthermore , as the high-level policy makes decisions in a latent subgoal space learned simultaneously with the hierarchical policy , the instability of the online-learned subgoal space results in severe non-stationarity of the high-level state transition function and hinders effective high-level exploration . To address these challenges , we develop an active Hierarchical Exploration approach with Stable Subgoal representation learning ( HESS ) . Our approach adopts a contrastive objective for subgoal representation learning inspired by Li et al . ( 2021 ) . To learn a subgoal space that could effectively guide hierarchical exploration , we propose a novel state-specific regularization that contributes to both stable and efficient subgoal representation learning . The proposed regularization constrains the changes for those embeddings that have already well satisfied the representation learning objective . As the hierarchical agent gradually expands its exploration to new state regions , the regularization allows for updates of the embeddings that underfit the learning objective . Benefiting from the proposed regularization , we could improve the representation learning efficiency without hurting its stability via the prioritized sampling technique ( Hinton , 2007 ) , which prioritizes training samples with larger losses . Building upon our stable subgoal representation learning , we design an active exploration strategy for high-level policy learning . As shown in previous work ( Strehl & Littman , 2008 ; Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ) , the visit count provides a simple and effective novelty measure for exploration . However , with online subgoal representation learning , the visit count for subgoals is defined in a changing space . Therefore , such a novelty measure alone is not sufficient for efficient high-level exploration , although the stability regularization could improve its accuracy . Our insight is that desirable novel subgoals should be reachable and effectively guide the agent to unexplored areas . Thus we design a novel potential measure to regularize the novelty measure , which indicates the reachability to the neighbor regions of the visited state embeddings . With the regularized novelty measure , our proposed active exploration strategy directly selects state embeddings with high potential and novelty as subgoals without introducing intrinsic rewards . It is more efficient than the reactive exploration methods that need to learn how to maximize the intrinsic rewards before performing exploratory behaviors ( Tang et al. , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2018 ) , since in the active exploration approach , the subgoal measures have direct effects on the behavior policy . Furthermore , the active exploration strategy naturally avoid introducing additional non-stationarity ( i.e. , induced by dynamically changing intrinsic rewards ) into HRL . We compare the proposed method HESS with state-of-the-art baselines in a number of difficult control tasks with sparse rewards . Note that the environment setting in this paper is much more challenging to exploration than the multi-task and deceptive dense-reward ones in the baselines . Experimental results demonstrate that HESS significantly outperforms existing baselines . In addition , we perform multiple ablations illustrating the importance of the various components of HESS . 2 BACKGROUND . We consider a Markov Decision Process ( MDP ) defined as a tuple ( S , A , P , r , γ ) , where S is a state space , A is an action space , P ( s′|s , a ) is an unknown transition function , r : S×A→ R is a reward function , and γ ∈ [ 0 , 1 ) is a discount factor . Let π ( a|s ) denote a stochastic policy over actions given states . The objective of reinforcement learning ( RL ) is to learn a policy that maximizes the expected cumulative discounted rewards : maxπ EP , π [ ∑T t=0 γ tr ( st , at ) ] , where T denotes the horizon length . In GCHRL , a two-level hierarchical policy πhier is composed of a high-level policy πh ( gt|st ) and a low-level policy πl ( at|slt , gt ) , as illustrated in Figure 1 . A latent subgoal space G is abstracted by a representation function φ ( s ) : S → Rd . The high-level policy πh samples a subgoal gt from a neighborhood ∆ ( φ ( st ) ) of the current latent state φ ( st ) every c steps , i.e. , when t ≡ 0 ( mod c ) : ∆ ( φ ( st ) ) = { gt ∈ G ∣∣D ( gt , φ ( st ) ) ≤ rg } , where c is a fixed constant , D is a distance function , and rg is a radius of the neighborhood . The sampled subgoal is repeated , gt = gt−1 , when t 6≡ 0 ( mod c ) . πh is trained to optimize the expected extrinsic rewards . The low-level controller takes a primitive action at ∈ A every step , and is intrinsically rewarded to reach subgoals set by the high level , rlt = −D ( gt , φ ( st ) ) . To provide dense non-zero rewards for low-level policy learning , we employ L2 distance as D. Furthermore , to keep the low-level reward function rl stationary while learning φ , we concatenate states and state embeddings together as the low-level states , sl = s||φ ( s ) . Previous work ( Li et al. , 2021 ) minimizes a contrastive triplet loss Ltri to learn φ ( s ) , where positive pairs are adjacent states in trajectories , and negative pairs are states c steps apart . Ltri ( st , st+1 , st+c ) = ||φ ( st ) − φ ( st+1 ) ||2 +max ( 0 , δ − ||φ ( st ) − φ ( st+c ) ||2 ) , ( 1 ) where δ is a margin parameter . In this work , we also adopt the contrastive loss in Equation 1 . 3 METHOD . In GCHRL , the low-level policy is optimized with intrinsic rewards generated by subgoals , but external rewards for the high level are still sparse and hard to explore . Furthermore , the changing subgoal space makes it even more challenging for high-level exploration . In this section , we first present a novel regularization that contributes to stable and efficient subgoal representation learning , and then introduce two measures for subgoals , novelty and potential . Finally , we design an active hierarchical exploration strategy based on these two measures . 3.1 STABLE SUBGOAL REPRESENTATION LEARNING . Both stability and efficiency are crucial to subgoal representation learning . The stability of subgoal representation contributes to the stationarity of both the high-level transition function and the low-level reward function . Meanwhile , a fast learned subgoal representation could provide effective guidance to exploration . However , stability seems at odds with efficiency , e.g. , training neural networks with smaller learning rates leads to better stability but is slower ( Goodfellow et al. , 2016 ) . In this subsection , we develop a novel state-specific regularization that resolves the stability-efficiency dilemma in subgoal representation learning . To achieve stable representation learning , we propose a regularization Ls which restricts the representation change during each update . Ls works by anchoring the current representation φ ( s ) to the old subgoal representation φold ( s ) before the update as follows : Ls = Es∼B [ λ ( s ) ||φ ( s ) − φold ( s ) ||2 ] , ( 2 ) where B is a replay buffer , and λ ( s ) is a function that controls the weight of regularization for different states . States with smaller representation losses fit the learning objective well , so λ ( s ) should be larger for those states . In practice , before each representation update , we rank the triplets in the buffer with Ltri ( st , st+1 , st+c ) . For the top k % of the triplets with the minimum representation losses , we set λ ( s ) = λ0 > 0 for the anchor states ( st ) in these triplets , and for the other states , λ ( s ) = 0 . The stability regularization enables us to use prioritized sampling ( Hinton , 2007 ) to improve representation learning efficiency without hurting its stability . The overall loss for subgoal representation learning is : Lφ = E ( st , st+1 , st+c ) ∼Bp [ Ltri ( st , st+1 , st+c ) ] + Ls , ( 3 ) where Bp is a prioritized replay buffer , and states with larger Ltri have higher probabilities of being sampled for training . 3.2 MEASURES FOR SUBGOALS . Novelty Measure : Inspired by count-based exploration methods ( Strehl & Littman , 2008 ; Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Tang et al. , 2017 ) , we formulate the novelty of subgoals with visit counts . As desirable subgoals should incentivize the agent to explore faraway novel states , we consider both immediate counts n ( φ ( si ) ) and expected cumulative counts of future states , and the novelty measure N ( φ ( si ) ) is defined as follows : N ( φ ( si ) ) = Eπhier [ b ( T−i ) /cc∑ j=0 γjn ( φ ( si+jc ) ) ] . ( 4 ) In practice , we partition the low-dimensional continuous latent space into discrete cells , i.e. , the state embeddings are mapped into cells containing them . By maintaining how many times each cell is visited , we could estimate the visit count n ( φ ( s ) ) . In practice , the novelty measure is approximated with the data from the replay buffer , and the implementation detail is described in Appendix B . Potential Measure : With online representation learning , the novelty measure is a mixture of counts in the past and current representation spaces , so it might mislead the exploration , as demonstrated in Figure 3 . Our insight is that desirable novel subgoals should be reachable and effectively guide the agent to unexplored areas . Therefore , we design a potential measure for subgoals to regularize the novelty measure . In the following , we first introduce a subgoal generation mechanism , which is involved in the definition of the potential measure . To guide the low-level controller to reach unexplored states , the subgoals pursued by the low level had better be in unexplored areas as well . Therefore , we propose to add some perturbations to the subgoal gt selected from the replay buffer and obtain an imagined subgoal ge , and then pass ge to the low-level policy . To enable ge in an unexplored or less explored area , the perturbation is conducted as extending gt in the direction of gt − φ ( st ) , as illustrated in Figure 2 . The imagined subgoal is ge = gt + de ( gt − φ ( st ) ) /||gt − φ ( st ) ||2 , where de denotes an extended distance . As ge is imagined , and may have not been visited before , it could be inherently unreachable due to the transition function of the MDP or online representation learning , e.g. , there may be obstacles in navigation tasks . To encourage the agent to explore in promising and reachable directions , we define a measure of potential U ( gt ) for the selected subgoal gt as the expected negative distance between the ending state φ ( st+c ) and ge : U ( gt ) = Est , at , ... , st+c [ −D ( φ ( st+c ) , ge ) ] , where st ∼ ρh , at ∼ πl ( at|slt , ge ) , st+1 ∼ P ( st+1|st , at ) . ( 5 ) ρh is the high-level state distribution under the hierarchical policy πhier . Building on ( Li et al. , 2021 ) , the representation learned with the contrastive objective preserves temporal coherence , and the distances between nearby features approximately represent the number of transitions between them ( Oord et al. , 2018 ) . Hence , with higher potential U ( gt ) , ge is more reachable , and thus exploring the direction of gt is more promising to expand the explored areas . The potential U ( gt ) in Equation 5 is estimated from the data in buffer as well . To calculate the novelty , we partition the continuous representation space into discrete cells . Similarly , we maintain the potential of a cell by averaging the potential of features in that cell , and use the potential of the cell to represent that of the states inside it . Illustrative example : In Figure 3 ( a ) and 3 ( b ) , we visualize the visitation density at 0.2M steps in the Ant Maze task in Figure 5 . The visitation density is the counts normalized by the total number of transitions in the buffer . Comparing Figure 3 ( a ) to 3 ( b ) , we found a mismatch between the density in the x , y coordinate space and the representation space , especially in the red box , since the updated representation at 0.2M steps has projected the state embeddings to somewhere new , and the feature changes are noted by the black arrows in Figure 3 ( c ) . Furthermore , the counts in frontiers of the latent explored areas would not increase with more exploration , as the online learned representation keeps changing . With the potential measure , we could distinguish the promising novel areas from the unpromising ones , illustrated in Figure 3 ( d ) .

Summary: 1. This paper investigates learning stable subgoals within a deep hierarchical reinforcement learning setup.   2. Two controllers are learned from the same experience replay buffer. The high level controller serves as a meta controller and the low level controller serves as a goal-achieving agent. The high level controller communicates abstract goals to the low level controllers.   3. The high level controller is optimized using an extrinsically specified reward function. The low level controller optimizes the intrinsic goal communicated by the high level controller. The subgoals are changed after a deterministic time length (known option termination).  4. The subgoals are designed with the key insight that "desirable novel subgoals should be reachable and effectively guide the agent to unexplored areas.". Typically count-based, predicted or successor feature based rewards have been used as novelty measures. However, these fall short in terms of reasoning about reachability of states. To handle this, a potential measure for subgoals is proposed which regularizes the novelty measure.  5. To go to unexplored states, a directional goal is synthesized/imagined using the current state and a directional vector. The potential function makes sure that this is approximately reachable by formulating reward as the expected negative distance between the ending state and imagined goal. This is similar to Feudal networks (Vezhnevets et al.) but goes beyond it to handle diversity and reachability.   6. The approach is validated on a set of hard to explore continuous environments with reasonably strong and relevant baselines.