Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks

1 INTRODUCTION . Reinforcement learning ( RL ) has been succesfully applied to sequential-decision making tasks , such as learning how to play video games in Atari ( Mnih et al. , 2013 ) , mastering the game of Go ( Silver et al. , 2017 ) or continuous control in robotics ( Lillicrap et al. , 2015 ; Levine et al. , 2015 ; Schulman et al. , 2017 ) . However , despite the success of RL agents in learning control policies for myopic tasks , such as reaching a nearby target , they lack the ability to effectively reason over extended horizons . In this paper , we consider continuous control tasks that require planning over long horizons in navigation environments with sparse rewards . The task becomes particularly challenging with sparse and delayed rewards since an agent needs to infer which actions caused the reward in a domain where most samples give no signal at all . Common techniques to mitigate the issue of sparse rewards include learning from demonstrations ( Schaal , 1999 ; Peng et al. , 2018 ) or using enhanced exploration strategies ( Bellemare et al. , 2016 ; Pathak et al. , 2017 ; Andrychowicz et al. , 2017 ) . Hierarchical Reinforcement Learning ( HRL ) has been proposed in part to solve such tasks . Typically , a sequential decision making task is split into several simpler subtasks of different temporal and functional abstraction levels ( Sutton et al. , 1999 ; Andre & Russell , 2002 ) . Although the hierarchies would ideally be learned in parallel , most methods resort to curriculum learning ( Frans et al. , 2017 ; Florensa et al. , 2017 ; Bacon et al. , 2016 ; Vezhnevets et al. , 2017 ) . Recent goal-conditioned hierarchical architectures have successfully trained policies jointly via off-policy learning ( Levy et al. , 2019 ; Nachum et al. , 2018 ; 2019 ) . However , these methods often do not generalize to unseen environments as we show in Section 5.1 . We argue that this is due to a lack of true separation of planning and low-level control across the hierarchy . In this paper , we consider two main problems , namely functional decomposition of HRL architectures in navigation-based domains and generalization of RL agents to unseen environments ( figure 1 ) . To address these issues , we propose a novel multi-level HRL architecture that enables both functional decomposition and temporal abstraction . We introduce a 3-level hierarchy that decouples the major roles in a complex navigation task , namely planning and low-level control . The benefit of a modular design is twofold . First , layers have access to only task-relevant information for a predefined task , which significantly improves the generalization ability of the overall policy . Hence , this enables policies learned on a single task to solve randomly configured environments . Second , the planning and control layers are modular and thus allow for composition of cross-agent architectures . We empirically show that the planning layer of the hierarchy can be transferred successfully to new agents . During training we provide global environment information only to the planning layer , whereas the full internal state of the agent is only accessible by the control layer . The actions of the top and middle layers are in the form of displacement in space . Similarly , the goals of the middle and lowest layers are relative to the current position . This prevents the policies from overfitting to the global position in an environment and hence encourages generalization to new environments . In our framework ( see figure 2 ) , the planner ( i.e. , the highest level policy π2 ) learns to find a trajectory leading the agent to the goal . Specifically , we learn a value map of the environment by means of a value propagation network ( Nardelli et al. , 2019 ) . To prevent the policy from issuing too ambitious subgoals , an attention network estimates the range of the lower level policy π0 ( i.e. , the agent ) . This attention mask also ensures that the planning considers the agent performance . The action of π2 is the position which maximizes the masked value map , which serves as goal input to the policy π1 . The middle layer implements an interface between the upper planner and lower control layer , which refines the coarser subgoals into shorter and reachable targets for the agent . The middle layer is crucial in functionally decoupling the abstract task of planning ( π2 ) from agent specific continuous control . The lowest layer learns a control policy π0 to steer the agent to intermediate goals . While the policies are functionally decoupled , they are trained together and must learn to cooperate . In this work , we focus on solving long-horizon tasks with sparse rewards in complex continuous navigation domains . We first show in a maze environment that generalization causes challenges for state-of-the-art approaches . We then demonstrate that training with the same environment configuration ( i.e. , fixed start and goal positions ) can generalize to randomly configured environments . Lastly , we show the benefits of functional decomposition via transfer of individual layers between different agents . In particular , we train our method with a simple 2DoF ball agent in a maze environment to learn the planning layer which is later used to steer a more complex agent . The results indicate that the proposed decomposition of policy layers is effective and can generalize to unseen environments . In summary our main contributions include : • A novel multi-layer HRL architecture that allows functional decomposition and temporal abstraction for navigation tasks . • This architecture enables generalization beyond training conditions and environments . • Functional decomposition that allows transfer of individual layers across different agents . 2 RELATED WORK . Learning hierarchical policies has seen lasting interest ( Sutton et al. , 1999 ; Schmidhuber , 1991 ; Dietterich , 1999 ; Parr & Russell , 1998 ; McGovern & Barto , 2001 ; Dayan & Hinton , 2000 ) , but many approaches are limited to discrete domains or induce priors . More recent works ( Bacon et al. , 2016 ; Vezhnevets et al. , 2017 ; Tirumala et al. , 2019 ; Nachum et al. , 2018 ; Levy et al. , 2019 ) have demonstrated HRL architectures in continuous domains . Vezhnevets et al . ( 2017 ) introduce FeUdal Networks ( FUN ) , which was inspired by feudal reinforcement learn- ing ( Dayan & Hinton , 2000 ) . In FUN , a hierarchic decomposition is achieved via a learned state representation in latent space . While being able to operate in continuous state space , the approach is limited to discrete action spaces . Tirumala et al . ( 2019 ) introduce hierarchical structure into KLdivergence regularized RL using latent variables and induces semantically meaningful representations . The separation of concerns between high-level and low-level policy is guided by information asymmetry theory . Transfer of resulting structure can solve or speed up training of new tasks or different agents . Nachum et al . ( 2018 ) present HIRO , an off-policy HRL method with two levels of hierarchy . The non-stationary signal of the upper policy is mitigated via off-policy corrections , while the lower control policy benefits from densely shaped rewards . Nachum et al . ( 2019 ) propose an extension of HIRO , which we call HIRO-LR , by learning a representation space from environment images , replacing the state and subgoal space with neural representations . While HIRO-LR can generalize to a flipped environment , it needs to be retrained , as only the learned space representation generalizes . Contrarily , HiDe generalizes without retraining . Levy et al . ( 2019 ) introduce Hierarchical Actor-Critic ( HAC ) , an approach that can jointly learn multiple policies in parallel . The policies are trained in sparse reward environments via different hindsight techniques . HAC , HIRO and HIRO-LR consist of a set of nested policies where the goal of a policy is provided by the top layer . In this setting the goal and state space of the lower policy is identical to the action space of the upper policy . This necessitates sharing of the state space across layers . Overcoming this limitation , we introduce a modular design to decouple the functionality of individual layers . This allows us to define different state , action and goal spaces for each layer . Global information about the environment is only available to the planning layer , while lower levels only receive information that is specific to the respective layer . Furthermore , HAC and HIRO have a state space that includes the agent ’ s position and the goal position , while ( Nachum et al. , 2019 ) and our method both have access to global information in the form of a top-down view image . In model-based reinforcement learning much attention has been given to learning of a dynamics model of the environment and subsequent planning ( Sutton , 1990 ; Sutton et al. , 2012 ; Wang et al. , 2019 ) . Eysenbach et al . ( 2019 ) propose a planning method that performs a graph search over the replay buffer . However , they require to spawn the agent at different locations in the environment and let it learn a distance function in order to build the search graph . Unlike model-based RL , we do not learn state transitions explicitly . Instead , we learn a spatial value map from collected rewards . Recently , differentiable planning modules that can be trained via model-free reinforcement learning have been proposed ( Tamar et al. , 2016 ; Oh et al. , 2017 ; Nardelli et al. , 2019 ; Srinivas et al. , 2018 ) . Tamar et al . ( 2016 ) establish a connection between convolutional neural networks and Value Iteration ( Bertsekas , 2000 ) . They propose Value Iteration Networks ( VIN ) , an approach where modelfree RL policies are additionally conditioned on a fully differrentiable planning module . MVProp ( Nardelli et al. , 2019 ) extends this work by making it more parameter-efficient and generalizable . The planning layer in our approach is based on MVProp , however contrary to prior work we do not rely on a fixed neighborhood mask to sequentially provide actions in its vicinity in order to reach a goal . Instead we propose to learn an attention mask which is used to generate intermediate goals for the underlying layers . Gupta et al . ( 2017 ) learn a map of indoor spaces and planning on it using a multi-scale VIN . In their setting , the policy is learned from expert actions using supervised learning . Moreover , the robot operates on discrete set of high level macro actions . Srinivas et al . ( 2018 ) propose Universal Planning Networks ( UPN ) , which learn how to plan an optimal action trajectory via a latent space representation . In contrast to our approach , the method relies on expert demonstrations and transfer to harder tasks can only be achieved after retraining . 3 BACKGROUND . 3.1 GOAL-CONDITIONED REINFORCEMENT LEARNING . We model a Markov Decision Process ( MDP ) augmented with a set of goals G. We define the MDP as a tupleM = { S , A , G , R , T , ρ0 , γ } , where S and A are set of states and actions , respectively , Rt = r ( st , at , gt ) a reward function , γ a discount factor ∈ [ 0 , 1 ] , T = p ( st+1|st , at ) the transition dynamics of the environment and ρ0 = p ( s1 ) the initial state distribution , with st ∈ S and at ∈ A . Each episode is initialized with a goal g ∈ G and an initial state is sampled from ρ0 . We aim to find a policy π : S × G → A , which maximizes the expected return . We train our policies by using an actor-critic framework where the goal augmented action-value function is defined as : Q ( s , g , a ) = Eat∼π , st+1∼T [ T∑ i=t γi−tr ( st , gt , at ) |st = s , gt = g , at = a ] ( 1 ) The Q-function ( critic ) and the policy π ( actor ) are approximated by using neural networks with parameters θQ and θπ . The objective for θQ minimizes the loss : L ( θQ ) = EM [ ( Q ( st , gt , at ; θ Q ) − yt ) 2 ] , where yt = r ( st , gt , at ) + γQ ( st+1 , gt+1 , at+1 ; θ Q ) . ( 2 ) The policy parameters θπ are trained to maximize the Q-value : L ( θπ ) = Eπ [ Q ( st , gt , at ; θ Q ) |st , gt , at = π ( st , gt ; θπ ) ] ( 3 ) To address the issue of sparse rewards , we utilize Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) , a technique to improve sample-efficiency in training goal-conditioned environments . The insight is that the desired goals of transitions stored in the replay buffer can be relabeled as states that were achieved in hindsight . Such data augmentation allows learning from failed episodes , which may generalize enough to solve the intended goal .

Summary: This paper addresses hierarchical deep reinforcement learning (RL), an important problem in control learning and RL. Based on my understanding of this paper and recent prior work, the most important difference between the proposed approach (HiDe) and other recent approaches, such as HIRO and HAC, is that the top-level goal proposal policy uses a learned planner based on VIN and a learned attention mask to decide on a subgoal. There are also other differences, e.g., this policy outputs a goal position that is relative to the agent's position, rather than an absolute position. HiDe seems to demonstrate impressive transferability to both unseen mazes and new agent embodiments, which are important problems to address for hierarchical RL.