Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
Authors: Ofir Nachum
Haoran Tang
Xingyu Lu
Shixiang Gu
Honglak Lee
Sergey Levine
Venue:  (2020)
Keywords: rl
hierarchy
reinforcement learning

Abstract: Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation.
Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement. 

Under review as a conference paper at ICLR 2020
WHY DOES HIERARCHY (SOMETIMES) WORK So WELL IN REINFORCEMENT LEARNING?
Anonymous authors Paper under double-blind review
ABSTRACT
Hierarchical reinforcement learning has demonstrated significant success at solv- ing difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architec- tures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL suite of tasks encompassing locomotion, navigation, and manipulation. Sur- prisingly, we find that most of the observed benefits hierarchy can be attributed improved exploration, as opposed to easier policy learning or imposed hierar- chical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while the same time being much simpler to use and implement.
INTRODUCTION
Many real-world tasks may be decomposed into natural hierarchical structures. To navigate large building, one first needs to learn how to walk and turn before combining these behaviors to achieve robust navigation; wash dishes, one first needs learn basic object grasping and handling before composing sequence of these primitives to successfully clean a collection of plates. Accord- ingly, hierarchy is an important topic in the context of reinforcement learning (RL), in which an agent learns to solve tasks from trial-and-error experience, and the use of hierarchical reinforcement learning (HRL) has long held the promise to elevate the capabilities of RI. agents to more com- plex tasks (Dayan & Hinton, 1993. Schmidhuber 1993 Parr & Russell 1998 Barto & Mahadevan 2003).
Recent work has made much progress towards delivering on this promise (Levy et al.) 2017 Frans al. 2018 Vezhnevets et al., 2017 Nachum et al. 2019). For example, Nachum et al. (201 8a b 2019 use HRL to solve both simulated and real-world quadrupedal manipulation tasks, whereas state-of-the -fate-of-the-art
-inahierallliaal methods are shown to make negligible progress on the same tasks. Levy et al. (2017) demonstrate similar results on complex navigation tasks, showing that HRL can find good policies with 3-5x fewer environment interactions than non-hierarchical methods.
While the empirical success of HRL is clear, the underlying reasons for this success more diffi- cult to explain. Prior works have motivated the use of HRL with number of intuitive arguments: high-level actions are proposed a lower temporal frequency than the atomic actions of the envi- ronment, effectively shortening the length of episodes; high-level actions often correspond to more semantically meaningful behaviors than the atomic actions of the environment, so both exploration and learning in this high-level action space is easier; and so on. These claims are easy to understand intuitively, and some may even theoretically motivated (e.g., shorter episodes are indeed easier to lear;; see (Strehl et al. (2009); : (Azar et al. (2017)) On the other hand, the gap between any theoretical setting and the empirical settings in which these hierarchical systems excel is wide. Furthermore, in Markovian systems, there is no theoretical representational benefit to imposing temporally ex- tended, hierarchical structures, since -inrrrieaall policies that make a decision at every step can be optimal (Puterman 2014). Nevertheless, the empirical advantages of hierarchy in a number of recent works, which raises the question, why is hierarchy beneficial in these settings? Which of the claimed benefits of hierarchy contribute its empirical successes?
self-evident

---

Under review as a conference paper at ICLR 2020
In this work, answer these questions via empirical analysis suite of tasks encompassing locomotion, navigation. and manipulation. We devise a series of experiments to isolate and evaluate the claimed benefits of HRL. Surprisingly, we find that most of the empirical benefit of hierarchy in our considered settings can be attributed to improved exploration. Given this observation, we propose a number of exploration methods that inspired by hierarchy but are much simpler to use and implement. These proposed exploration methods enable non-hierarchical RL agents to achieve performance competitive with state-of-the-art HRL. Although our analysis empirical and thus our conclusions are limited to the tasks we consider, we believe that our findings are important to the field of HRL. Our findings reveal that only a subset of the claimed benefits of hierarchy are achievable by current state-of-the-art methods, even on tasks that were previously believed to be approachable only by HRL methods. Thus, more work must done to devise hierarchical systems that achieve all of the claimed benefits. We also hope that our findings can provide useful insights for future research on exploration in RL. Our findings show that exploration research can be informed by successful techniques in HRL to realize more temporally extended and semantically meaningful exploration strategies.
2 RELATED WORK
Due to its intuitive and biological appeal (Badre & Frank 2011 [Botvinck 2012). the field of HRI
has been an active research topic in the machine learning community for many years. A number
of different architectures for HRL have been proposed in the literature (Dayan & Hinton 1993
Kaelbling. 1993. Parr & Russell, 1998 Sutton et al. 1999 Dietterich 2000; Florensa et al. 2017
Heess et al. 2217) We consider two paradigms specifically the options framework (Precup 2000)
and goal-conditioned hierarchies (Nachum et al. 2018b). due to their impressive success in recent
work (Frans et al.) 2018 Levy et al. 2017| Nachum al. 20188 2019), though an examination of other architectures is an important direction for future research.
One traditional approach to better understanding and justifying the use of an algorithm is through theoretical analysis. In tabular environments, there exist bounds on the sample complexity of learn- ing a near-optimal policy dependent on the number of actions and effective episode horizon (Brun- skill & Li, 2014). This bound can be used to motivate HRL when the high-level action space is smaller than the atomic action space (smaller number of actions) or the higher-level policy op- erates at temporal abstraction greater than one (shorter effective horizon). Previous work has also analyzed HRL (specifically, the options framework) in the more general setting of continuous states (Mann & Mannor 2014). However, these theoretical statements rely having access to near-optimal options. which are typically not available in practice. Moreover, while simple syn- thetic tasks can be constructed to demonstrate these theoretical benefits, it is unclear if any of these benefits actually play a role in empirical successes demonstrated in more complex environments. In contrast, our empirical analysis is specifically devised to isolate and evaluate the observed practical benefits of HRL.
Our approach isolating and evaluating the benefits of hierarchy via empirical analysis is partly inspired by previous empirical analysis on the benefits of options (Jong et al.) 2008). Following a previous flurry of research, empirical demonstrations, and claimed intuitive benefits of options in the early 2000's, Jong et al. (2008) set out to systematically evaluate these techniques. Similar to our findings, exploration was identified as a key benefit, although realizing this benefit relied on the use of specially designed options and excessive prior knowledge of the task. Most of the remaining observed empirical benefits were found to be due to the use of experience replay (Lin) 1992), and the same performance could be achieved with experience replay alone on a non-hierarchical agent. Nowadays, experience replay is an ubiquitous component of RL algorithms. Moreover, the hier- archical paradigms of today are largely model-free and achieve more impressive practical results than the gridworld tasks evaluated by Dong et al. (2008). Therefore, we present our work as a re- calibration of the field's understanding with regards to current ofate-of-the-art hierarchical methods.
3 HIERARCHICAL REINFORCEMENT LEARNING
We briefly summarize the HRL methods and environments we evaluate on. We consider the typ- ical two-layer hierarchical design, in which a higher-level policy solves a task by directing one or more lower-level policies. In the simplest case, the higher-level policy chooses new high-level
2

---

Under review as a conference paper at ICLR 2020
AntMaze
AntPush
AntHllokk
AntHlookMaaze
10-
HIRO
BIRO GR
Options
Shallow
Miuitsettp cwsnede (Few S]
Mults-etap roomadd (Caen
Figure 1: We consider four difficult tasks, where the agent (magenta) simulated quadrupedal robot. In AntMaze, the agent must navigate to the end of U-shaped corridor (target given by green arrow); in AntPush, the agent must navigate the target by first pushing block obstacle to the right; in AntBlock and Maze the agent must push a small red block to the target location;
Nachum et al. (2018b) for more details. Task success rates are plotted for three HRL algorithms
HIRO (Nachum et al. 2018a). HIRO with goal relabelling (inspired by Levy et al. (2017), 7017), and Options (Frans et al.J 2018)) and shallow nonnhierarchhiaa) agents with and without the use of multi-step rewards (n-step returns) over 10M training steps, averaged over 5 seeds. In this work, we isolate and evaluate the key properties of hierarchy which yield the stark difference in empirical performance between HRL and non-HRL methods.
action every timesteps In the options framework, the high-level action is a discrete choice, indicating which of m lower-level policies (called options) to activate for the next e steps. goal- conditioned hierarchies, there is a single goal-conditioned lower-level policy, and the high-level action is a continuous-valle goal state which the lower-level directed to reach.
Lower-level policy training operates differently in each of the HRL paradigms. For the options framework, we follow [Bacon et al. (2017); Frans et al. (2018), training each lower-level policy to maximize environment reward. We train separate Q-value functions minimize errors,
E(st, az, Re, 81+1) (Qta,
R. -QQi,m (st+1, To (8(+1))²
over single-step transitions, and the m option policies are learned to maximize this Q-value Q1a.m ml (st)) In contrast, for HIRO (Nachum al. 2018a) and HAC (Levy et al. 2017). the lower-level policy and Q-function are goal-conditioned. That is, a Q-function is learned to minimize errors.
(st, St. Tt, St+1, It+1) = (Qio (St, Gt, at) - - 710 Ir+1,
(2)
over single-step transitions, where Jt is the current goal (high-level action updated every e steps) and r, is an intrinsic reward measuring negative L2 distance to the goal. The lower-level policy is then trained to maximize the Q-value (1, St: (8t, (t))))))
For higher-level training we follow Nachum et al. (0188): Frans et al. (2018) and train based on temporally-extended c-step transitions (st, It, Resto- Sttc), where gr high-level action (dis- crete identifier for options, goal for coul-conditioned hierarchies) and R = Ei- R is the c-step sum of environment rewards. That is, Q-value function learned to minimize errors,
E(st: It, R St+e) (Qui(se, gt) Ret+e-1 Qui Thi(s
In the options framework where high-level actions are discrete, the higher-level policy is simply (s) arg max, Qui (s. In goal-conditioned HRL where high-level actions are continuous, the higher-level policy is learned to maximize the Q-value Qni(s. mni(s)).
Note that higher-level training in HRL is distinct from the use multi-step rewards or n-step re- turns (Hessel et al. 2018), which proposes to train non- non-hierarchical agent with respect to transi-
We restrict our analysis to hierarchies using fixed although evaluating variable-length temporal abstrac- tions important avenue for future work.
3

---

Under review as a conference paper at ICLR 2020
tions (st, oc, Re:t+rr -1: St-czew ); i.e., the Q-value of HiR is learned to minimize,
(4) while the policy learned choose atomic actions maximize Q(s,, ग (s)). In contrast, in HRL both the rewards and the actions 9r used in the Q-value regression loss are temporally extended. However, as we will see in Section the use of multi-step rewards alone can achieve almost all of the benefits associated with hierarchical training (controlling for exploration benefits).
E(s,, at 11 (Q(st, at) Res++ +Q( T ($t+ 1))
For our empirical analysis, we consider four difficult tasks involving simulated robot locomotion, navigation, and object manipulation (see ". To alleviate issues of goal representation learn- ing in goal-conditioned HRL, we fix the goals to be relative æ, y coordinates of the agent, which are a naturally good representation for our considered tasks. We note that this is only done to better control our empirical analysis, and that goal-conditioned HRL can achieve good performance on considered tasks without this prior knowledge (Nachum et al., 2018b). We present the results of two goal-come conditioned HRL methods: HIRO (Nachum et al. 2018a) and HIRO with goal relabelling (inspired by HAC; Levy et al. (2117) and an options implementation based on |rans al. (2018) in Figure T HRL methods can achieve strong performance on these tasks, while non-hierarchical methods struggle make any progress at all. In this work, we strive to isolate and evaluate the key properties of HRL which lead to this stark difference.
HYPOTHESES OF THE BENEFITS OF HIBRARCHY
We begin by listing out the claimed benefits of hierarchical learning. These hypotheses can be organized into several overlapping categories. The first set hypotheses (HI and H2 below) rely on the fact that HRL uses temporally extended actions; i.e., the high-level policy operates at lower temporal frequency than the atomic actions the environment. The second set (H3 and H4 below) rely on the fact that HRL uses semantically meaningfid actions- - high-level actions often correspond to more semantic behaviors than the natural low-level atomic actions exposed by the MDP. For example, in robotic navigation, the atomic actions may correspond to torques applied the robot's joints, while the high-level actions in goal-conditioned HRL correspond to locations to which the robot might navigate. In options, there are many paradigms which are explicitly designed to achieve better exploration (McGovern & Barto 2001 Kulkarni et al. 2016, Machado et al., 2017a|b) b). In the more undirected form of options that we use, it argued that semantic behaviors naturally arise from unsupervised specialization of behaviors (Frans et al.) (2018). The four hypotheses may also be categorized as hierarchical truining (HI and H3) and hierarchical exploration (H2 and H4).
(H1) Temporally extended training. High-level actions correspond to multiple environment steps. To the high-level agent, episodes are effectively shorter. Thus, rewards are propagated faster and learning should improve.
(H2) Temporally extended exploration. Since high-level actions correspond multiple envi- ronment steps, exploration in the high-level is mapped to environment exploration which is temporally correlated across steps. This way, an HRL agent explores the environment more efficiently. As a motivating example. the distribution associated with a random (Gaussian) walk is wider when the random noise is temporally correlated.
(H3) Semantic training. High-level actor and critic networks are trained with respect to semanti- cally meaningful actions. These semantic actions are more correlated with future values, and thus casier to Icarn, compared to training with respect to the atomic actions of the environ- ment. For example, in robot navigation task it is easier to learn future values with respect to deltas in x-y coordinates rather than robot joint torques.
(H4) Semantic exploration. Exploration strategies (in the simplest case, random action noise) are applied to semantically meaningful actions, and are thus more meaningful than the same strategies would be if applied to the atomic actions of the environment. For example, in robot navigation task it intuitively makes more sense to explore at the level of x-y coordinates rather than robot joint torques.
Due to space constraints, see the Appendix for an additional hypothesis based on modularity.
5 EXPERIMENTS
Our experiments are aimed at studying the hypotheses outlined in the previous section, analyzing which of the intuitive benefits of HRL are actually present in practice. We begin by evaluating
4

---

Under review as a conference paper at ICLR 2020
the performance of HRL when varying the length of temporal abstraction used for training and exploration (Section HI and H2), finding that although this has some impact on results, it is not enough to account for the stark difference between HRL and non non-hierarchical methods observed in Figure T We then look at the training hypotheses more closely (Section 5.2] H1 and H3). We find that, controlling for exploration, hierarchical training is only useful far it utilizes multi-step rewards, and furthermore the use of multi-step rewards is possible with non-hierarchical agent. Given this surprising finding, we focus on the exploration question itself (Section 5.3 H2 and H4). We propose two exploration strategies, inspired by HRL, which enable non-hierarchical agents to achieve performance competitive with HRL.
5.1 EVALUATING THE BENEFTS OF TEMPORAL ABSTRACTION (HI AND H2)
We begin by evaluating the merits of Hypotheses HI and H2, both of which appeal to the tempo- rally extended nature of high-level actions. In our considered hierarchies, temporal abstraction is hyperparameter. Each high-level action operates for environment time steps. Accordingly, the choice of impacts two main components of learning:
During training, the higher-level policy is trained with respect to temporally extended tran- sitions of the form (st: 9t. Res+c-1, St+c) (see Section 3 for details).
During experience collection, high-level action is sampled and updated every steps.
The first of these implementation details corresponds to HI (temporally extended training) while the second corresponds to H2 (temporally extended exploration), and we can vary these two param- eters independently to study the two hypotheses separately. Accordingly, we take HIRO, the best performing HRL method from Figure [ and implement it so that these two instances of temporal abstraction are decoupled into separate choices Grain for training horizon and Cexpl for experience collection horizon. We evaluate performance across different choices of these two hyperparameters.
Varying Train Horizon
AntMuree
Amtuush
AntBlock
AnBlockMazee
1.-
0.0
10
Varying Exploration Horizon
AntMaaee
AmtPush
AntBoock
locMaee
0.6
68
60
Peepl Geopl 2
Feepl
Goopl
10
15
Figure We present the results for different HRL methods while changing the temporal abstraction used for training (Cusin- top) or the temporal abstraction used for experience collection (Cexple bot- tom). Average success rates and standard errors are calculated for randomly seeded runs, trained for 10M steps with carly stopping. Recall that our HRL baselines varying Ctrain we find that the choice of horizon matters only so far as Ctrain > 1. For Cexpl= while there exists correlation between performance and temporal abstraction, using temporal abstrac- tion (Gexpl 1) can still make non-negligible progress compared to the shallow policies in Figure
Ctrain Guxpl 10. When
The results are presented in Figure 2 showing performance for different values of Ctrain (top) and Cexpl (bottom); recall that our baseline method uses Ctrain Cexpl 10. The strongest effect of Curain is observed in AntMaze and AntPush, where the difference between Ctrain = and Ctrain > crucial to adequately solving these tasks. Otherwise. while there is some noticeable difference between specific choices of Grrain (as long as Ctrain > 1), there is no clear pattern suggesting that a larger value of Curann is better.
5

---

Under review as a conference paper at ICLR 2020
AntMuaee
AnsPush
AntBlock
AnttlookMMaze
66-
0.5
66--
62
5.0
2.0
Shadow
Shadlot
3
Sharlow
Shadow, 10
We evaluate and compare the performance of training a non-hierarchical shadow agent trained on experience collected by hierarchical agent, thus disentangling the potential benefits of HRL for exploration from the potential benefits of HRL for training. In all environments except AntMaze, the shadow agent can achieve performance competitive with HRI., given appropriate multi-step reward horizon (Crew 3 performs best). Overall, this suggests that the effect of hier- archy on ease of training (as opposed exploration) is modest, and can mostly be replicated by non-hierarchical agent given good experience and the use of multi-step rewards.
Figure 3:
For Gexpl, the effect seems slightly stronger. In AntMaze, there is observed effect, while in AntPush, AntBlock, and AntBlockMaze there exists some correlation suggesting higher values of Gexpl do yield better performance. Even so, Cexpl is often able to make non-negligible progress towards adequately solving the tasks, as compared to a non-hierarchial shallow policy (Figure [
Overall, these results provide intriguing insights into the impact of temporally abstracted training and exploration. While temporally extended training appears to help on these tasks, it is enough to have Ctrain > 1. Temporally extended exploration appears have stronger effect, although it alone does not adequately explain the difference between an HRL agent that can solve the task and a non-hierarchical one that cannot make any progress. Where then does the benefit come from? In the next sections, we will delve deeper into the impact of hierarchy on training and exploration.
5.2 EVALUATING THE BENEFITS OF HIERARCILICAL TRAINING (HI AND H3)
The previous section suggested that temporally extended training (H1) has at most a modest impact on the performance of HRL. In this section, we take closer look at the benefits of hierarchy on training and study Hypothesis H3, which suggests that high- actions used by HRL are easier for learning as compared to the atomic actions of the MDP. In goal-conditioned hierarchies for example, H3 claims that it is easier for RI. to learn policy and value functions based on delta x-y commands (goals), than it is to learn policy and value functions based on atomic joint-torque actions exposed by the environment. In this section we aim to isolate this supposed benefit from other confounding factors, such potential exploration benefits. Therefore, we devise an experiment to disentangle exploration from action representation, by training a standard non-hierarchical agent (a shadow agent) on experience collected by hierarchical agent. If the benefits of HRL stem primarily from exploration, we would expect the shadow agent to do well; if representation of high-level actions matters for training, we would expected HRL to do better.
Accordingly, we augment our HRL implementation (specifically, HIRO) with an additional parallel shadow agent, represented by a standard single-level policy and value function. Each agent the HRL agent and the non-hierarchical shadow agent - has its own replay buffer and collects its own experience from the environment. During training, we train the HRL agent usual, while the shadow agent is trained on batches of experience gathered from both replay buffers (70% from the shadow agent's experience and 30% from the HRL agent's 's experience, chosen based on appropriate tuning). This way, any need for exploration is fulfilled by the experience gathered by the agent. Will the hinrareaalllaall agent's 's policy still be able to learn? Or does training with higher-level that uses semantically meaningful high-level actions make learning casier?
We present the results our experiments in Figure 3 While the potential impacts of Hypotheses H2 and H4 (exploration) are neutralized by our setup, the impact of Hypothesis HI (which Section 5.1 showed has a modest impact) still remains. As an attempt to control for this factor, we also consider setup where the non-hie oo-hieearcchcal shadow agent receives multi-step rewards (see Section 3 for an overview of multi-step rewards). Different temporal extents for the multi-step rewards are indicated by Crew in the figure legend.

---

Under review as a conference paper at ICLR 2020
AntMuate
AnsPush
AntBlock
Antid locMaaze
66-
66--
62
00
20
Optizoaa
FALE (rowtic
EAB Cowndd
BAB Iruuiss
EdcE (Sasiicc
10)
68--
56-
62
6.0
2.0
HERO
Oy4icars
SB (npwitch
SB Couettck -3
SE (Sawitch 5)
SE (trectack 10)
Figure 4: We compare the performance of HRL to Explore & Exploit (E&E) and Switching En- semble (SE) - two non -on-hierarc exploration methods that make use of HRL-inspired temporally extended modulation of behaviors (length of modulation given by Cswitch). We find that the non- hierarchical methods are able to match the performance of HRL on these tasks (with the only excep- tions being Explore & Exploit AntBlockMaze and Switching Ensemble AntPush), suggesting that exploration is the key to success on these tasks. These results also make clear the importance of temporally extended exploration; using Cswitch is almost always better than Cswitch =1.
The results Figure show that learning from atomic actions, without higher-level action represen- tations, is feasible, and can achieve similar performance as HRL. On AntMaze, we observe slight drop performance, but otherwise performance competitive with HRI. The results across differ- ent multi-step reward horizons Crew also provide further insight into the conclusions of Section 5 As suggested by the results of Section 5.1 temporally abstracted training does affect performance, especially for AntMaze and AntPush. Still. while temporally abstracted training is important, these results show that the same benefit can be achieved by simply using multi-step rewards (which are much simpler to implement than using temporally extended actions). To confirm that multi-step rewards are not the only component necessary for success, see Figure in which a non- non-hierarchical shallow agent with multi-step rewards is unable to make non-negligible progress these tasks.
Overall, we conclude that the high-level action representations used by HRL methods in our do- mains are not factor for the success of these methods, outside of their potential benefits for exploration. The only observed benefit of high-level action representations in training is due to the use of multi-step rewards, and this can be easily incorporated into con-hierarchical agent training.
5.3 EVALUATING THE BENEFITS OF HIERARCHICAL EXPLORATION
The findings the previous section show that training a non-hierarchical agent on 'good' experience (from hierarchical agent) performs about as well the hierarchical agent itself. If representing the policy and value function in terms of temporally extended, abstract actions is not crucial to achieving good performance, the next most-likely explanation is that the 'good' experience itself is the key. That is, good exploration is the key component to the success of HRL. This is the claim proposed by Hypotheses H2 (temporally extended exploration) and H4 (semantic exploration). In this section, we attempt to extend the experiments presented in Section 5. to better understand the impact of good exploration on the performance of non- non-hieran non-hierarchical agent. We will show that it is possible to enable non-hierarchica agents to achieve results competitive with HRL by using two exploration methods inspired by HRL: Explore & Exploit and Switching Ensemble.
Explore & Exploit is inspired by the hypothesis that goal-reaching is good exploration strategy in- dependent of hierarchy (Barames & Oudeyer 2010). Thus, we propose training two non-hierarchical agents trained to maximize environment rewards (similar the higher-level policy in HRL), and the other trained to reach goals (similar to the lower-level policy in goal-conditioned HRL). Unlike in HRI., each policy operates the atomic actions of the environments, and the goal for the

---

Under review as a conference paper at ICLR 2020
explore agent is sampled randomly according to an tion 5 and damping 0.8) as opposed to a learned policy. During experience collection, we randomly switch between the explore and exploit agents every Switch timesteps. Specifically, every Cywitch steps we randomly sample one of the two agents (with probability 0.2, 0.8 for the explore and ex- ploit agents, respectively), and this chosen agent is used for sampling the subsequent Cswitch atomic actions. Both agents share the same replay buffer for training. In this way, we preserve the benefits of goal-directed exploration temporally extended and based on goal-reaching in semantically meaningful space- without explicit hierarchies of policies.
proces: (standard devia-
Our other proposed exploration method, Switching Ensemble, is inspired by the options framework, in which multiple lower-level policies interact to solve task based on their shared experience. We propose a simple variant of this approach that removes the higher-level policy. We train multiple (specifically, five) non-hierarchical agents to maximize environment rewards. During experience collection, we choose one of these agents uniformly at random every Cswitch timesteps. This way, we again maintain the spirit of exploration used in HRL - temporally extended and based on multiple interacting agents - while avoiding the use of explicit hierarchies of policies. This approach is related to the use of randomized value functions for exploration (Osband et al. 2014 2016 Plappert et al. 2017; Fortunato et al.] 2017) and may have a Bayesian interpretation (Gal & Ghahramani, 2016 although our proposal is unique for having a mechanism (Cswitch) to control the temporally extended nature of the exploration. For both of these methods, we utilize multi-step environment rewards (Grew = 3), which we found work well in Section 5.2 (Figure
Our findings are presented in Figure 4 We find that the proposed alternatives are able to achieve performance similar to HRL, with the only exceptions being Explore & Exploit on MatelockMaze and Switching Ensemble on AntPush. Overall, these methods are able to bridge the gap in empirical performance between HRL and non- non-hierarchical methods from Figure confirming the importance of good exploration on these tasks. Notably, these results show the benefit of temporally extended exploration even for non- non-hierarchical agents - using Cswitch > is often significantly better than using Cswitch = (switching the agent every step). Furthermore, the good performance of Explore & Exploit suggests that semantic exploration ggoal-reaching is beneficial, and likely plays an im- portant role in the success of goal-conditioned HRL methods. The success of Switching Ensemble further shows that an explicit higher-level policy used to direct multiple agents is not necessary in these environments.
Overall, these results suggest that the success of HRL on these tasks is largely due to better explo- ration. That is, goal-conditioned and obtions-based hierarchies are better exploring these environ- ments as opposed to discovering high-level representations which make policy and value function training easier. Furthermore, these benefits can be achieved without explicit hierarchies of policies. Indeed, the results of Figure Jshww that non- non-hicrarchical agents can achieve similar performance as state-of-the-ar HRL, as long as they (I) use multi-step rewards in training and (2) use temporally- extended exploration (based on either goal-reaching or randomized value functions).
Beyond the core analysis in our experiments, we also studied the effects of modularity using separate networks to represent higher and lower-level policciss. Due to space constraints, these results are presented the Appendix. These experiments confirm that the use of separate networks is beneficial for HRI. We further confirm that using separate networks for the Explore & Exploit and Switching Ensemble methods is crucial for their effectiveness.
6 DISCUSSION AND CONCLUSION
Looking back at the initial set of hypotheses from Section 4 we can draw number of conclusions based on our empirical analysis. In terms of the benefits of training. it is clear that training with respect to semantically meaningful abstract actions (H3) has a negligible effect on the success of HRL (as seen from our shadow experiments; Figure 3). Moreover, temporally extended training (H1) is only important insofar as it enables the use multi-step rewards, as opposed to training with respect to temporally extended actions (Figure 3. The main, and arguably most surprising, benefit of hierarchy is due to exploration. This is evidenced by the fact that temporally extended
'The use of OU noise for temporally-corrllted correlated exploration was first used by Lillicrap et al. 2015). where OU noise is added to the actions of a deterministic policy. In contrast, in our application. OU noise is used to determinee temporally correlated goal; i.c., the OU noise is used as input opposed to added to the output of a policy.
8

---

Under review as a conference paper at ICLR 2020
Hypothesis
Experiments
Important?
Yes, but only for the use of multi-step rewards (n-step returns).
(H1) Temporal training
Figures 2
Yes, and this is important even for non-hierarchical exploration.
(H2) Temporal exploration
Figures 2
(H3) Semantic training
Figure
No.
Yes, and this is important even for non-hierarchical exploration.
(H4) Semantic exploration
Figure
Figure 5: A summary of our conclusions the benefits of hierarchy.
goal-reaching and agent-switching can enable non-hierarchical agents to solve tasks that otherwise can only be solved, in our experiments, by hierarchical agents (Figure(4) These results suggest that the empirical effectiveness of hierarchical agents simply reflects the improved exploration that these agents can attain.
These conclusions suggest several future directions. First, our results show that current state-of-the- art HRL methods only achieve subset of their claimed benefits. More research needs to be done to fully realize all of the benefits, especially with respect to semantic and temporally extended training. Second, our results suggest that hierarchy can be used as inspiration for better exploration meth- ods, and we encourage future work to investigate more variants of the non-hierarchical exploration strategies proposed.
Still, our empirical analysis has limitations. Our results and conclusions are restricted to a limited set tasks and hierarchical designs. The use of other hierarchical designs may lead different con- clusions. Additionally, conclusions may be different for different task settings. For example, the use of hierarchy in multi-task settings may beneficial for better transfer, a benefit that we did not evaluate. In addition, tasks with more complex environments and/or sparser rewards may benefit from other mechanisms for encouraging exploration (e.g., count-based exploration), which would be a complementary investigation to this study. An examination of different hierarchical structures and more varied settings is an important direction for future research.
REFERENCES
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In Proceedings the 34th International Conference on Machine Learning- Volume 70, pp. 263 262.272.J0 JMLR. org. 2017.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pp. 1726 1734, 2017.
David Badre and Michael Frank. Mechanisms of hierarchical reinforcement learning cortico- striatal circuits 2: Evidence from fmri. Cerebral cortex, 22(3):527-536 2011.
Adrien Baranes and Pierre-Yvel Oudeyer. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In 2010 IEEE/RS.J International Conference on Intelligent Robots and Systems, pp. 1773. IEEE, 2010.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(4):341-37, 2003.
Matthew Michael Botvinick. Hierarchical reinforcement learning and decision making. Current opinion in neurobiology, 22(6))956-92) 962, 2012.
Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In International conference on machine learning, pp. 316-324, 2014.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural infor- mation processing systems, pp. 271-278, 1993.

---

Under review as a conference paper at ICLR 2020
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. Journal of Artificial Intelligence Research, 13:227-303, 2000.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint
2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv: 7706. 10295, 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hicrarchies. International Conference Learning Representations (ICLR), 2018.
Scott Fujimoto, Herke Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1802.0000 2018.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. Ali Eslami, Martin Riedmiller, al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:/707.02240 2017.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems- systems-Volume Volume pp. 299-306. International Foundation for Autonomous Agents and Multiagent Systems, 2008.
Leslie Pack Kaelbling. Hierarchical learning in stochastic domains: Preliminary results. In Pro- ceedings of the tenth international conference on machine learning, volume 951, pp. 167-173, 1993.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hicrarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016.
Andrew Levy. Robert Platt, and Kate Saenko.
Hierarchical actor-critic.
arXiv preprint
arXiv:1712.00948, 2017.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (.X9v0/50900277/ 2015.
Long-Ji Ji Lin. Self-improving reactive agents based reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-33) -221311,11111 1992.
Marios C Machado, Marc Bellemare, and Michael Bowling. A laplacian framework for option discovery reinforcement learning. In Proceedings of the 34th International Conference on Machine Coarning-Volume 70, pp. 2295-2304. JMLR. org, 2017a
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint 1700 1089, 2017b.
Timothy Mann and Shie Mannor. Scaling up approximate value iteration with options: Better poli- cies with fewer iterations. In International conference on machine learning, pp. 127-135, 2014.
Amy McGovern and Andrew Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. 2001.
10

---

Under review as a conference paper at ICLR 2020
Ofir Nachum, Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. NeurlPS, 2018a.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:1810.01257, 2018b.
Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-agent manipula-
tion via locomotion using hierarchical sim2real. arXiv preprint arXiv:/908.05224, 2019.
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. arXiv preprint
2014.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034 2016.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Advances in neural information processing systems, pp. 1043-1049, 1998.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 1006009700 2017.
Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts
Amherst, 2000.
Martin L Puterman. Markov Decision Processes.: Discrete Stochastic Dynamic Programming. John Wiley Sons, 2014.
Jürgen Schmidhuber. Planning simple trajectories using neural subgoal generators. From Animals to Animats 2: Proceedings of the Second International Conference Simulation of Adaptive Behavior, volume 2, pp. 196. MIT Press, 1993.
Alexander L Strehl, Lihong Li, and Michael L Littman. Reinforcement learning in finite mdps: Pac analysis. Journal of Machine Learning Research, 10(Nov):2413-2444, 2009.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-
211, 1999.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv: 2017.

---

Under review as a conference paper at ICLR 2020
A
TRAINING DETAILS
We provide a more detailed visualization and description of HRL (Figure
Hight-eww policy
ni
m
Low-level policy
17
1
1
50
a
Environment
R₂
R
c-1 R
high-level reward
Figure 6: A diagram showing the structural form of an HRL agent. Every steps, higher-level policy Thi chooses a high-level action Gt- In the options framework, this high-level action is an identifier, choosing which of m options to activate. In goal-conditioned HRL, the high-level action a goal-state. In either case, a lower-level policy Tio used to produce a sequence of atomic actions ag After steps, control is returned to the higher-level policy.
B EVALUATING THE BENEFITS OF MODULARITY
AmtMaz
AntPush
AntBlack
AntBlockMaee
1.0
BRL
Explore & Explost
Swisching Ensemble
HRL (Combined Networka)
E&B (Combined Networkss)
S6 (Corabnned Networkss)
Figure 7: We evaluate the importance of modularity in this case, using separate networks for sep- arate policies. We find that using separate networks is consistently better, suggesting that modularity important for both HRL and HRL-like methods.
We evaluate the merits of using modularity in HRL systems. We have already shown in the main text that a non-HRL agent can achieve performance similar to HRL. However, all of these non-HRL agents utilize multiple policies, similar how HRL agents have separate lower-level and higher- level policies.
Thus, we evaluate how important this modularity is. We evaluate HIRO well as the successful non-hierarchical methods from Section [.3] with and without separate networks. Specifically, for HIRO we combine the separate networks for lower and higher-level policies into single network with multiple heads. For Explore & Exploit and Five Exploit we combine the separate networks for each policy into single network with multiple heads. The results are presented Figure We see that combined networks consistently lead to worse performance than structurally separate networks. HIRO and Explore & Exploit are especially sensitive to this change, suggesting that Hypothesis H5 is true for settings using coal-conditioned hierarchy exploration. Overall, the use of separate networks for goal-reaching and task solving is beneficial to the performance of these methods in these settings.
12

---

Under review as a conference paper at ICLR 2020
C EXPERIMENT DETAILS
Our implementations are based on the open-source implementation of HIRO (Nachum et al. 201 8a). using default hyperparameters. HIRO uses TD3 for policy training (Fujimoto al., 2018). and so we train all non-hierarchical agents using TD3, with the same network and training hy iyperparameters as used by HIRO, unless otherwise stated.
Since the choice of с in goal-conditioned HRL can also impact low-level training, as the frequency of new goals in recorded experience can affect the quality of the learned low-level behavior. To neutralize this factor in our ablations, we modify transitions (St; It, dt, rt, St+1, 9c+1) used for low- level training by replacing the next goal g++1 with the current goal gr: in this way the lower-level policy is trained as if the high-level goal is never changed. This implementation modification has negligible effect on HRL's performance with otherwise default settings.
To implement HIRO with goal relabelling, augment the HIRO implementation with hindsight experience replay used for the lower-level policy. To implement Option-Crine Critii (Bacon et al. 2017) in this framework, create m 5 separate lower-level policies trained to maximize reward (using n-step returns, where n = 3). We replace the higher-level continuous-action policy with a discrete double DQN-based agent, with e-greedy exploration (c 0.5).
For our exploration alternatives (Explore & Exploit and Switching Ensemble), we utilize multi-step environment rewards with Grew We also found it beneficial to train at a lower frequency: we collect 2 environment steps per each training (gradient descent) step. To keep the comparisons fair, train these variants for 5M training steps (corresponding to 10M environment steps, equal to that used by HRI.).
which we found to work well in Section 5.2 (see Figure 3
D ADDITIONAL EXPERIMENTS
Mixing ratio 70: 30
AntMure
AntPush
AntBlock
AntllockMaze
64-
Mixing ratio 50 50
AntMuse
AntPush
AntBlock
Antil AoeMMookMazze
66-
66-
54
0.4
0:4
Mixing ratio 30
70
AntMuse
AntPuch
AntBlock
AutBlio MokMMae
0.:
66-
0.0
agent
Shadow,
Shadoow
Shadoww,
evaluating and comparing the performance of training non-hierarchical shadow agent trained on experience collected by hierarchical agent. Each row shows the results for a specific mixing ratio of experience between the shadow and HRL agents. We that regardless of the mixing ratio, the conclusions are mostly consistent.
Figure 8: We expand on the results from Figure 3
13

---

Under review as a conference paper at ICLR 2020
Comparison
AntMaze
AntPush
AntBlock
AntBlockMaze
Figure
Ctrain
vs. Ctrain 10
(- 00.0054
(~)0.0093
(-)0.35
(+)0.78
Figure
Cexpl 1
Cexpl 10
(-)0.78
(-)0.98
(-)0.017
(-)0.025
Figure
HRL shadow with Crew
(+)0.088
(-)0.73
(+)0.73
(-)0.60
Figure
shadow with Crew vs. Crew (-)0.014
(-)0.046
(-)0.022
(-)0.29
Figure 4
HRL vs. E&E with Cawitch
(-)0.099
0-)9
(-)0.98
(+)0.50
Figure 4
HRL vs. SE with Cswitch
(+)0.15
(+)0.17
(+)0.11
(+)0.21
Figure 4
E&E with Cswitch
VS. Cswitch 3
(- (-)020002 (- )0.023
(-)0.35
(+)0.29
Figure 4
SE with Cawitch
vS. Cawitch = 3 --0001
(-)0.014
(-)0.83
(- --33333
Figure 9: Results of running a two-sided t-test on our experimental results. The sign designates the direction of the t-test result; i.e., a negative sign for A B indicates that the mean of the results for is less than that B.
14


Reviews:
[{"review_id": "SyxiwxDPtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After Rebuttal:\nAfter the discussion with the other reviewers, I tend to agree with them  regarding the need for more experimental evidence given the strong title and claims of the paper. For this reason I lower my  score from weak accept to weak reject. Please note that the idea that the paper tries to explore is of great value to the community and I encourage the authors to further perform more experiments (more environments and more algorithms) to solidify their claims.  \n\n************************************\nThe paper aims to answer the question proposed in the title. The authors conduct a series of experiments using well known hierarchical and non-hierarchical algorithms in order to extract the what is it that makes hierarchical reinforcement learning (HRL) efficient. The conclusion is that the main benefits of HRL are due to 1) temporally extended training enabling multi-step rewards, 2) temporally extended and semantically meaningful exploration.\n\nI really enjoyed reading the paper and I believe it is an important contribution to the community. I believe the experiments are sufficient to support the hypothesis stated in the paper. Also, the authors state clearly the potential weak points of the paper in the discussion.  For the previous reasons I suggest the acceptance of this paper, although some improvements could be made. \n\nSupporting arguments:\n\nThe paper is clearly well written. There is a clear exposition of the ideas, hypothesis and experiments. \n\nThe sequence of experiments makes sense to me. For example, section 5.1 tries to disentangle H1 and H2 (concluding both are important to different degrees), then in 5.2 H1 and H3 are evaluated (concluding that H3 is non-beneficial) and 5.3 focuses on the most important effect (exploration) through H2 and H4. \n\nAs mentioned on the discussion, other hierarchical designs, environments and tasks might lead to different conclusions. Even though this is true, I think the paper correctly balanced breadth vs depth of experiments and conditions. I think this paper serves as a baseline and might inspire further similar research that may cover wider settings (more hierarchical designs, environments, tasks etc.) to see if the results still hold. \n\nThings to improve:\n\n\nThe data from all the figures presented in the paper could  be used to perform statistical tests to support the authors’ statements. For example, In Figure 2 (top) there is a varying parameter (c_train) and the authors claim that there is a noticeable effect on performance for c_train > 1. One can clearly see this in AntMaze and AntPush but it would be nice that when authors make such statements that they are backed by a significance test (e.g. analysis of variance, two-way anovas, t-tests, or any kind of test that the authors might deem suitable).  Similarly, one can do this for Figure 2(bottom),  and the remaining figures with their corresponding statements (e.g. c_switch >1 vs c_switch =1 in Figure 4).\n\nPerforming such statistical tests might require to run more than 5 seeds tough, but I believe the paper would be stronger with such statistical tests and I would increase the score accordignly.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJl7wJU6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper evaluates the benefits of using hierarchical RL (HRL) methods compared to regular shallow RL methods for fully observed MDPs. The goal of the work is to isolate and evaluate the benefits of using HRL on different control tasks (AntMaze, AntPush, AntBlock, AntBlockMaze). They find that the major benefit of HRL comes in the form of better exploration, compared to the ease of learning policies. They claim that the use of multi-step rewards alone is sufficient to provide the benefits associated with HRL. They also provide two exploration methods that are not hierarchical in nature but achieve similar performance:  a) Explore and Exploit and b) Switching Ensemble.\n\nI appreciate the effort to contribute to the field by understanding and highlighting why HRL works better than shallow RL for a few tasks when theoretically there is no such incentive. However, this work still has a few missing components that need to be addressed in order to fully support the claims. Given these clarifications in an author's response, I would be willing to increase the score.\n\n\n1) The claim needs more support.\nThe authors only compare policy-search based methods, on a very specific suite of control tasks, and then generalize the results. Maybe on these sets of tasks, exploration is the main issue, and not the training. This is without taking transfer or sparse-reward scenario into account. If noisy/sparse reward is added to the scenario, then training will also get difficult, and then maybe HRL methods can help with them too? Maybe not? But without considering that case, it seems wrong to come to a conclusion. \n\n2) The hypothesis is only valid for HIRO. \nIn Sec 5, the authors test their method on a specific form of HRL method, but from Introduction and Conclusion, it seems that they are generalizing this conclusion to all HRL methods. The authors need to be either explicit about that these results only hold for HIRO, or provide some evidence that supports this generalization to other HRL methods.\n\n3) Design choices not clear. \nThe following are the few choices that were made, but no argument or discussion regarding them was provided:\n- In Sec 5.2, for the training of the shadow agent, why the particular split ratio (7:3) was used. Wouldn’t change in this ratio, can also lead to a different result for that comparison (H1 and H3)?\n- In Sec 5.1, the train and explore hypothesis have c_{train} and  c_{explore} hyperparameters respectively, that control the multi-step horizon. However, how the exact decoupling is done is not clear. Is it that exploration sub-policies have a different horizon (c_{explore}) compared to how they are being updated, in terms of how the targets are calculated (c_{train}). With the nature of the algorithms based on policy-search methods, I don’t understand how this division induces decoupling between them. \n\n\n4) Missing supporting literature/ Novelty\nFor the Explore and Exploit method, the authors propose a new method by adding OU noise and randomly switch between explore and exploit phase with c_{switch} hyperparameters. The use of OU noise to have temporally correlated exploration for continuous control tasks is already known [1]. Instead of explicitly exploring or exploiting [2], they use a hyper-parameter for the same, but no ablation study or discussion about the effect of that parameter is included. The same holds true for the Switching Ensemble case. \n\n\n\n\nThings to improve the paper that did not impact the score:\nEq 3, shouldn’t there be discounting for g_{t} targets?\n\n\n\n\n\n\n\nReferences: \n\n[1] Lillicrap, Timothy P., et al. \"Continuous control with deep reinforcement learning.\" arXiv preprint arXiv:1509.02971 (2015).\n\n[2] Kearns, Michael, and Satinder Singh. \"Near-optimal reinforcement learning in polynomial time.\" Machine learning 49.2-3 (2002): 209-232.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1lg9SyKFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper, as it tries to understand the role of hierarchical methods (such as Options, higher level controllers etc) in RL. The core contribution of the paper is understand and evaluate the claimed benefits often proposed by hierarchical methods, and finds that the core benefit in fact comes from exploration. The paper studies hierarchical methods to eventually draw the conclusion that HRL in fact leads to better exploration based behaviour in complex tasks. \n\nWhile the conclusion suggesting HRL leading to better exploration seems interesting, I am not sure whether this is in fact too surprising? For example, the Options framework has been fundamentally proposed and shows benefits in terms of transfer learning and faster exploration. Options has been fundamentally argued to lead to faster exploration (for example, when the goal changes in the four rooms task). Therefore, isn't the conclusion that this paper draws already known? Maybe the paper studies HRL methods in different experimental settings, not considerered in existing HRL or Options based frameworks before, but I would imagine similar conclusions can be drawn if a pure HRL system is studied?\n\n\u000bThe paper mainly does an experimental ablation study for HRL systems, and draws the conclusion of faster exploration led by HRL methods. However, I am not sure whether this conclusion, while assummably true as argued by previous methods too, is convincing enough based on the proposed suite of tasks? \n\n\u000bIt might have been more useful if there were similar theoretical contributions or analysis were also made for a stronger claim for HRL methods. Without lack of theoretical analysis or proofs, and simply based on the ablation studies as in this paper, I am not sure whether this paper is yet ready for acceptance in a venue like ICLR. \n\n\u000bIn the tasks considered, also there is no termination condition being learnt, and in fact the HRL systems studied here terminates every c time steps (as in HIRO and most of other HRL papers). This might be a significant limitation though. If we learn the termination condition, as in Option-Critic, would we expect similar behaviour in performance and can attribute the benefits of HRL only to exploration?\n\n\u000bThere are a vast amount of approaches based on identifying bottleneck states, and often they have shown better performance in terms of transfer learning. While in these approaches, often the benefits are claimed to be in terms of both exploration and transfer learning, this paper seems to contradict that? \n\n\u000bExperimentally, there are only few specific tasks considered, like the variations of Ant tasks like AntPush and AntMaze. I am not convinced from the set of experimental results that they are sufficient enough to draw the conclusion that HRL methods only excel due to better exploration. \n\n\u000bWhy is the conventional Four Rooms domain ignored? What if we take the Four Rooms domain, change the goal states - I would expect the paper to do such analysis on the range of HRL methods, and then propose a convincing argument. \n\n\u000bI think overall the paper needs more work, before such conclusion can be drawn overall. It seems to me like a strong claim that the benefits of HRL is only due to exploration. The paper does not do enough experimental abltation studies to strengthen the claim, mainly lacking fundamental HRL task setups. It does not do a theoretical analysis studying HRL methods and their benefits either. Overall, I think with these contributions, if similar behaviour persists, then it would make a more convincing argument. \n\n\u000bThere are a lot of theoretical papers studying provably sufficient exploration methods. Perhaps such approaches can also be taken here to study HRL methods and whether they provably lead to faster exploration? \n\n\u000bExperimentally, I would expect a more wide range of task setups and domains to be studied - since this is mainly a paper based on experimental studies and trying to draw conclusion for existing HRL methods without proposing new approaches. I think without these carefully studied experiments, the conclusions are over-claimed. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper seeks to analyse the important question around why hierarchical reinforcement learning can be beneficial. The findings show that improved exploration is at the core of this improved performance. Based on these findings, the paper also proposes some simple exploration techniques which are shown to be competitive with hierarchical RL approaches.

This is a really interesting paper that could serve to address an oft speculated about result of the relation between HRL and exploration. While the findings of the paper are intuitive, it was agreed by all reviewers that the claims are too general for the evidence presented. The paper should be extended with a wider range of experiments covering more domains and algorithms, and would also benefit from some theoretical results.

As it stands this paper should not be accepted.
Decision: Reject