Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

1 INTRODUCTION . Many real-world tasks may be decomposed into natural hierarchical structures . To navigate a large building , one first needs to learn how to walk and turn before combining these behaviors to achieve robust navigation ; to wash dishes , one first needs to learn basic object grasping and handling before composing a sequence of these primitives to successfully clean a collection of plates . Accordingly , hierarchy is an important topic in the context of reinforcement learning ( RL ) , in which an agent learns to solve tasks from trial-and-error experience , and the use of hierarchical reinforcement learning ( HRL ) has long held the promise to elevate the capabilities of RL agents to more complex tasks ( Dayan & Hinton , 1993 ; Schmidhuber , 1993 ; Parr & Russell , 1998 ; Barto & Mahadevan , 2003 ) . Recent work has made much progress towards delivering on this promise ( Levy et al. , 2017 ; Frans et al. , 2018 ; Vezhnevets et al. , 2017 ; Nachum et al. , 2019 ) . For example , Nachum et al . ( 2018a ; b ; 2019 ) use HRL to solve both simulated and real-world quadrupedal manipulation tasks , whereas state-of-the-art non-hierarchical methods are shown to make negligible progress on the same tasks . Levy et al . ( 2017 ) demonstrate similar results on complex navigation tasks , showing that HRL can find good policies with 3-5x fewer environment interactions than non-hierarchical methods . While the empirical success of HRL is clear , the underlying reasons for this success are more difficult to explain . Prior works have motivated the use of HRL with a number of intuitive arguments : high-level actions are proposed at a lower temporal frequency than the atomic actions of the environment , effectively shortening the length of episodes ; high-level actions often correspond to more semantically meaningful behaviors than the atomic actions of the environment , so both exploration and learning in this high-level action space is easier ; and so on . These claims are easy to understand intuitively , and some may even be theoretically motivated ( e.g. , shorter episodes are indeed easier to learn ; see Strehl et al . ( 2009 ) ; Azar et al . ( 2017 ) ) . On the other hand , the gap between any theoretical setting and the empirical settings in which these hierarchical systems excel is wide . Furthermore , in Markovian systems , there is no theoretical representational benefit to imposing temporally extended , hierarchical structures , since non-hierarchical policies that make a decision at every step can be optimal ( Puterman , 2014 ) . Nevertheless , the empirical advantages of hierarchy are self-evident in a number of recent works , which raises the question , why is hierarchy beneficial in these settings ? Which of the claimed benefits of hierarchy contribute to its empirical successes ? In this work , we answer these questions via empirical analysis on a suite of tasks encompassing locomotion , navigation , and manipulation . We devise a series of experiments to isolate and evaluate the claimed benefits of HRL . Surprisingly , we find that most of the empirical benefit of hierarchy in our considered settings can be attributed to improved exploration . Given this observation , we propose a number of exploration methods that are inspired by hierarchy but are much simpler to use and implement . These proposed exploration methods enable non-hierarchical RL agents to achieve performance competitive with state-of-the-art HRL . Although our analysis is empirical and thus our conclusions are limited to the tasks we consider , we believe that our findings are important to the field of HRL . Our findings reveal that only a subset of the claimed benefits of hierarchy are achievable by current state-of-the-art methods , even on tasks that were previously believed to be approachable only by HRL methods . Thus , more work must be done to devise hierarchical systems that achieve all of the claimed benefits . We also hope that our findings can provide useful insights for future research on exploration in RL . Our findings show that exploration research can be informed by successful techniques in HRL to realize more temporally extended and semantically meaningful exploration strategies . 2 RELATED WORK . Due to its intuitive and biological appeal ( Badre & Frank , 2011 ; Botvinick , 2012 ) , the field of HRL has been an active research topic in the machine learning community for many years . A number of different architectures for HRL have been proposed in the literature ( Dayan & Hinton , 1993 ; Kaelbling , 1993 ; Parr & Russell , 1998 ; Sutton et al. , 1999 ; Dietterich , 2000 ; Florensa et al. , 2017 ; Heess et al. , 2017 ) . We consider two paradigms specifically – the options framework ( Precup , 2000 ) and goal-conditioned hierarchies ( Nachum et al. , 2018b ) , due to their impressive success in recent work ( Frans et al. , 2018 ; Levy et al. , 2017 ; Nachum et al. , 2018a ; 2019 ) , though an examination of other architectures is an important direction for future research . One traditional approach to better understanding and justifying the use of an algorithm is through theoretical analysis . In tabular environments , there exist bounds on the sample complexity of learning a near-optimal policy dependent on the number of actions and effective episode horizon ( Brunskill & Li , 2014 ) . This bound can be used to motivate HRL when the high-level action space is smaller than the atomic action space ( smaller number of actions ) or the higher-level policy operates at a temporal abstraction greater than one ( shorter effective horizon ) . Previous work has also analyzed HRL ( specifically , the options framework ) in the more general setting of continuous states ( Mann & Mannor , 2014 ) . However , these theoretical statements rely on having access to near-optimal options , which are typically not available in practice . Moreover , while simple synthetic tasks can be constructed to demonstrate these theoretical benefits , it is unclear if any of these benefits actually play a role in empirical successes demonstrated in more complex environments . In contrast , our empirical analysis is specifically devised to isolate and evaluate the observed practical benefits of HRL . Our approach to isolating and evaluating the benefits of hierarchy via empirical analysis is partly inspired by previous empirical analysis on the benefits of options ( Jong et al. , 2008 ) . Following a previous flurry of research , empirical demonstrations , and claimed intuitive benefits of options in the early 2000 ’ s , Jong et al . ( 2008 ) set out to systematically evaluate these techniques . Similar to our findings , exploration was identified as a key benefit , although realizing this benefit relied on the use of specially designed options and excessive prior knowledge of the task . Most of the remaining observed empirical benefits were found to be due to the use of experience replay ( Lin , 1992 ) , and the same performance could be achieved with experience replay alone on a non-hierarchical agent . Nowadays , experience replay is an ubiquitous component of RL algorithms . Moreover , the hierarchical paradigms of today are largely model-free and achieve more impressive practical results than the gridworld tasks evaluated by Jong et al . ( 2008 ) . Therefore , we present our work as a recalibration of the field ’ s understanding with regards to current state-of-the-art hierarchical methods . 3 HIERARCHICAL REINFORCEMENT LEARNING . We briefly summarize the HRL methods and environments we evaluate on . We consider the typical two-layer hierarchical design , in which a higher-level policy solves a task by directing one or more lower-level policies . In the simplest case , the higher-level policy chooses a new high-level action every c timesteps.1 In the options framework , the high-level action is a discrete choice , indicating which of m lower-level policies ( called options ) to activate for the next c steps . In goalconditioned hierarchies , there is a single goal-conditioned lower-level policy , and the high-level action is a continuous-valued goal state which the lower-level is directed to reach . Lower-level policy training operates differently in each of the HRL paradigms . For the options framework , we follow Bacon et al . ( 2017 ) ; Frans et al . ( 2018 ) , training each lower-level policy to maximize environment reward . We train m separate Q-value functions to minimize errors , E ( st , at , Rt , st+1 ) = ( Qlo , m ( st , at ) −Rt − γQlo , m ( st+1 , πlo , m ( st+1 ) ) 2 , ( 1 ) over single-step transitions , and the m option policies are learned to maximize this Q-value Qlo , m ( st , πlo , m ( st ) ) . In contrast , for HIRO ( Nachum et al. , 2018a ) and HAC ( Levy et al. , 2017 ) , the lower-level policy and Q-function are goal-conditioned . That is , a Q-function is learned to minimize errors , E ( st , gt , at , rt , st+1 , gt+1 ) = ( Qlo ( st , gt , at ) − rt − γQlo ( st+1 , gt+1 , πlo ( st+1 , gt+1 ) ) 2 , ( 2 ) over single-step transitions , where gt is the current goal ( high-level action updated every c steps ) and rt is an intrinsic reward measuring negative L2 distance to the goal . The lower-level policy is then trained to maximize the Q-value Qlo ( st , gt , πlo ( st , gt ) ) . For higher-level training we follow Nachum et al . ( 2018a ) ; Frans et al . ( 2018 ) and train based on temporally-extended c-step transitions ( st , gt , Rt : t+c−1 , st+c ) , where gt is a high-level action ( discrete identifier for options , goal for goal-conditioned hierarchies ) and Rt : t+c−1 = ∑c−1 k=0Rt+k is the c-step sum of environment rewards . That is , a Q-value function is learned to minimize errors , E ( st , gt , Rt : t+c−1 , st+c ) = ( Qhi ( st , gt ) −Rt : t+c−1 − γQhi ( st+c , πhi ( st+c ) ) ) 2 . ( 3 ) In the options framework where high-level actions are discrete , the higher-level policy is simply πhi ( s ) : = argmaxg Qhi ( s , g ) . In goal-conditioned HRL where high-level actions are continuous , the higher-level policy is learned to maximize the Q-value Qhi ( s , πhi ( s ) ) . Note that higher-level training in HRL is distinct from the use of multi-step rewards or n-step returns ( Hessel et al. , 2018 ) , which proposes to train a non-hierarchical agent with respect to transi- 1We restrict our analysis to hierarchies using fixed c , although evaluating variable-length temporal abstractions are an important avenue for future work . tions ( st , at , Rt : t+crew−1 , st+crew ) ; i.e. , the Q-value of a non-HRL policy is learned to minimize , E ( st , at , Rt : t+c−1 , st+c ) = ( Q ( st , at ) −Rt : t+crew−1 − γQ ( st+crew , π ( st+crew ) ) ) 2 , ( 4 ) while the policy is learned to choose atomic actions to maximize Q ( s , π ( s ) ) . In contrast , in HRL both the rewards and the actions gt used in the Q-value regression loss are temporally extended . However , as we will see in Section 5.2 , the use of multi-step rewards alone can achieve almost all of the benefits associated with hierarchical training ( controlling for exploration benefits ) . For our empirical analysis , we consider four difficult tasks involving simulated robot locomotion , navigation , and object manipulation ( see Figure 1 ) . To alleviate issues of goal representation learning in goal-conditioned HRL , we fix the goals to be relative x , y coordinates of the agent , which are a naturally good representation for our considered tasks . We note that this is only done to better control our empirical analysis , and that goal-conditioned HRL can achieve good performance on our considered tasks without this prior knowledge ( Nachum et al. , 2018b ) . We present the results of two goal-conditioned HRL methods : HIRO ( Nachum et al. , 2018a ) and HIRO with goal relabelling ( inspired by HAC ; Levy et al . ( 2017 ) ) and an options implementation based on Frans et al . ( 2018 ) in Figure 1 . HRL methods can achieve strong performance on these tasks , while non-hierarchical methods struggle to make any progress at all . In this work , we strive to isolate and evaluate the key properties of HRL which lead to this stark difference .

Summary: This paper evaluates the benefits of using hierarchical RL (HRL) methods compared to regular shallow RL methods for fully observed MDPs. The goal of the work is to isolate and evaluate the benefits of using HRL on different control tasks (AntMaze, AntPush, AntBlock, AntBlockMaze). They find that the major benefit of HRL comes in the form of better exploration, compared to the ease of learning policies. They claim that the use of multi-step rewards alone is sufficient to provide the benefits associated with HRL. They also provide two exploration methods that are not hierarchical in nature but achieve similar performance:  a) Explore and Exploit and b) Switching Ensemble.