Hierarchical Reinforcement Learning by Discovering Intrinsic Options
Authors: Jesse Zhang
Haonan Yu
Wei Xu
Venue: ICLR 2021 Poster (2021)
Keywords: hierarchical reinforcement learning
reinforcement learning
options
unsupervised skill discovery
exploration

Abstract: We propose a hierarchical reinforcement learning method, HIDIO, that can learn task-agnostic options in a self-supervised manner while jointly learning to utilize them to solve sparse-reward tasks. Unlike current hierarchical RL approaches that tend to formulate goal-reaching low-level tasks or pre-define ad hoc lower-level policies, HIDIO encourages lower-level option learning that is independent of the task at hand, requiring few assumptions or little knowledge about the task structure. These options are learned through an intrinsic entropy minimization objective conditioned on the option sub-trajectories. The learned options are diverse and task-agnostic. In experiments on sparse-reward robotic manipulation and navigation tasks, HIDIO achieves higher success rates with greater sample efficiency than regular RL baselines and two state-of-the-art hierarchical RL methods. Code at: https://github.com/jesbu1/hidio.

Published as a conference paper at ICLR 2021
HIERARCHICAL REINFORCEMENT LEARNING BY DISCOVERING INTRINSIC OPTIONS
Jesse Zhang*11 Haonan Yu²?, Wei Xu²
University of Southern California, HHorioon Robotics
ABSTRACT
We propose a hierarchical reinforcement learning method, HIDIO, that can learn task-agnostic options in self-supervised manner while jointly learning utilize them to solve sparse-reward tasks. Unlike current hierarchical RL approaches that tend to formulate goal-reaching low-level tasks or pre-define ad hoc lower- level policies, HIDIO encourages lower-level option learning that is independent of the task at hand, requiring few assumptions or little knowledge about the task structure. These options are learned through an intrinsic entropy minimization ob- jective conditioned on the option sub-trajectorier The learned options are diverse and task aagnosicc In experiments igation tasks, HIDIO achieves higher success rates with greater sample efficiency than regular RL baselines and state-of-the-ar hierarchical RL methods. Code available at ht https: / /www. g t hhub. com/ jesbul hoauu//hhh /hidio.
sparse-rewe reward robotic manipulation and nav-
INTRODUCTION
Imagine wheeled robot learning to kick soccer ball into goal with sparse reward supervision. In order to succeed, must discover how to first navigate in its environment, then touch the ball, and finally kick it into the goal, only receiving a positive reward the end for completing the task. This is a naturally difficult problem for traditional reinforcement learning (RL) to solve, unless the task has been manually decomposed into temporally extended stages where each stage constitutes a much easier subtask. In this paper we ask, how do we learn to decompose the task automatically and utilize the decomposition solve sparse reward problems?
Deep RL has made great strides solving a variety of tasks recently, with hierarchical RL (bRL) demonstrating promise solving such sparse reward tasks (Sharma et al., 2019h; Le et al., 2018 88; Merel et al., 2019; Ranchod et al., 2015). In hRL, the task is decomposed into hierarchy of subtasks, where policies at the top of the hierarchy call upon policies below to perform actions to solve their respective subtasks. This abstracts away actions for the policies at the top levels of the hierarchy. hRL makes exploration easier by potentially reducing the number of steps the agent needs to take to explore its state space. Moreover, at higher levels of the hierarchy, temporal abstraction results in more aggressive, multi-step value bootstrapping when temporal-differene differennce (TD) learning is employed. These benefits are critical in sparse reward tasks as they allow an agent more easily discover reward signals and assign credit.
Many existing hRL methods make assumptions about the task structure (e.g., fetching an object involves three stages: moving towards the object, picking it up. and combing back), and/or the skills needed solve the task (e.g., pre-prreeeeee pre-programmed motor skills) (Florensa al., 2016; Riedmiller al., 2018; Lee et al., 2019; Hausman et al., 2018; Lee et al., 2020; Sohn et al., 2018; Ghavamzadeh & Mahadevan, 2003; Nachum et al., 2018). Thus these methods may require manually designing the correct task decomposition, explicitly formulating the option space, or programming pre-defined options for higher level policies to compose. Instead, we seek to formulate a general method that can learn these abstractions from scratch, for any task, with little manual design in the task domain.
The main contribution of this paper HIDIO (Hlerarchical RL by Discovering Intrinsic Options), a hierarchical method that discovers task-agnostic intrinsic options in a self-supervised manner while
DDnotes equal contribution. Email to jessez (ussezzusccedu, (haonan. yu, suuehooizznn.c Work done intern at Horizon Robotics.

---

Published as a conference paper at ICLR 2021
leamning schedule them to accomplish environment tasks. The latent option representation is uncovered as the option-conditioned policy is trained, both according to the same self-supervised worker objective. The scheduling of options is simultancuusly learned by maximizing environment reward collected by the option-conditioned policy. HIDIO can he easily applied to new sparse- reward tasks by simply re-discovering options. We propose and empirically evaluate various in- stantiations of the option discovery process, comparing the resulting options with respect to their final task performance. We demonstrate that HIDIO able efficiently learn and discover di- verse options to be utilized for higher task reward with superior sample efficiency compared to other hierarchical methods.
2 PRELIMINARIES
We consider the reinforcement learning (RL) problem in a Markov Decision Process (MDP). Let s E R be the agent state. We use the terms "state" and "observation" interchangeably to denote the environment input to the agent. A state can be fully or partially observed. Without loss of generality, we assume a continuous action space a E R.4 for the agent. Let (a(a))) ) be the policy distribution with learnable parameters 0, and P(s:+1|Ss [S,, a,) the transition probability that measures how likely the environment transitions to 5t+1 given that the agent samples an action by a ~ (-(ss))). After the transition to St+1, the agent receives deterministic scalar reward r(s,, a, Bt+1).
The objective of RL is to maximize the sum of discounted rewards with respect to 0:
E Ta.P
Σ r r'rs a sc+1)
1=0
where 7 E [0, 1] is a discount factor. We will omit P in the expectation for notational simplicity.
In the options framework (Sutton al., 1999), the agent can switch between different options during an episode, where an option is translated to a sequence of actions by an option-conditioned policy with termination condition. A of options defined over an MDP induces a hierarchy that models temporal abstraction. For a typical two-level hierarchy, a higher-level policy produces options, and the policy at the lower level outputs environment actions conditioned on the proposed options. The expectation in Eq. is taken over policies at both levels.
3 HIERARCHICAL RL BY DISCOVERING INTRINSIC OPTIONS
We now introduce our hierarchical method for solving sparse reward tasks. We as- sume little prior knowledge about the task structure, except that it can be learned through hierarchy of two levels. The higher-level policy (the scheduler TT), is trained to maximize environment reward, while the lower-level policy (the worker To) is trained in a self-supervised manner efficiently discover options that are uti- lized by Na to accomplish tasks. Impor- tantly, by self-supervision the worker gets access to dense intrinsic rewards regard- less of the sparsity of the extrinsic rewards.
R
Envirummen
sk
U
K
On
Time
Worker x day Box.Rat-s,
Discfminator City ao Raa))
The overall framework of HIDIO. The sched- uler лy samples an option प, every K (3 in this case) time steps, which is used to guide the worker Ts to di- rectly interact in the environment conditioned on u, and the current sub-trajectory T k The sched- uler receives accumulated environment rewards R,, while the worker receives intrinsic rewards rlo rh,++1 Re- fer to Eq. 2 for sampling and Eqs. 3 and 5 for training.
Figure 1:
Without loss of generality, we assume that each episode has a length of T and the scheduler outputs option every K steps. The scheduled option u E 1 (where D is pre-defined dimensional- ity), is a latent representation that will be leamned from scratch given the environment task. Modulated by u, the worker executes K steps before the scheduler outputs the next option. Let the time horizon of the scheduler be H = + Formally, define
2

---

Published as a conference paper at ICLR 2021
Scheduler policy:
To(-
O<h< 0 O<kcK 0 <<h<H,o<<>>
Worker policy:
an.k Tal Sh.kk un);
(2)
Environment dynamics: Sh,4+1 P( an,k),
where we denote Sp.k and a as the k-th state and action respectively, within the h-th option window of length K. Note that given this sampling process, have S S4+1,0- namely, the last state of the current option in is the initial state of the next option U The overall framework of our method is illustrated in Figure I.
3.1 LEARNING THE SCHEDULER
Every time the scheduler issues an option un it receives an reward Rt computed by accumulating environment rewards over the next K steps. Its objective is:
H-1 3h Rr
max E
Σ hm
where 3 = - and Rr Er
> (sn, an SA.,,) Sha+1)
(3)
Am0
This scheduler objective itself is not concept, similar ones have been adopted by other hRL methods (Vezhnevets et al., 2017; Nachum et al., 2018; Riedmiller et al., 2018). One significant difference between our option with that of prior work that our option u is simply a latent variable; there is no explicit constraint on what semantics u could represent. In contrast, existing methods usually require their options reside in subspace of the state space, to be grounded to the envi- ronment, or to have known structures, so that the scheduler can compute rewards and termination conditions for the worker. Note that our latent options can easily re-trained given a new task.
3.2 LEARNING THE WORKER
The main focus of this paper is to investigate how effectively learn the worker policy in self- supervised manner. Our motivation that it might be unnecessary to make an option dictate the worker reach some "c-space" of goals (Vezhnevets et al., 2017; Nachum et al., 2018). As long as the option can be translated to short sequence of primitive actions, it does not need to be grounded with concrete meanings such goal reaching. Below will treat the option that modulates the worker, and propose to learn its latent representation in a hierarchical setting from the environment task.
latent variable
3.2.1 WORKER OBJECTIVE
We first define new meta MDP top of the original task MDP that for any h, k, and to
1) FR, == (Sp.0r k
2) An. := (a),0 )
3) r(Sn,k an.k: Sh == r (S,,, ag.k, Sh. and
4) ((5 Kn. ab,k) == P(sh a),k).
This new MDP equips the worker with historical state and action information since the time (h, 0) when option h was scheduled. Specifically, each state S4,A action a,,A encodes the history from the beginning (h, 0) up to (h, k) within the option. In the following, we will call pairs (a,,e, Sh.k+1) option sub-trajectories. The worker policy now takes option sub-trajectories inputs: av, k (. an. Un), < k < K, whereas the scheduler policy still operates in the original MDP.
Denote Σn, Σı= Σ K-1 for simplicity. The worker objective, defined this new MDP, is minimize the entropy of the option u₁ conditioned the option sub-trajectory {an,k, 1
max
Σ
p(u aa,,, Sh,++1) 3 log Th (ar, (h Bh,k-1. Un
TONT
nogative conditional option cntropy
worker policy entropy
where the expectation over the current To and Fa but the maximization is only with respect to ø. Intuitively, the first term suggests that the worker is optimized to confidently identify option given
3

---

Published as a conference paper at ICLR 2021
sub-trajectory. However, it alone will not guarantee the diversity of options because potentially even very similar sub-trajectories can be classified into different options if the classification model has high capacity, in which case we say that the resulting sub-trajectory space has a very high "resolution". As a result, the conditional entropy alone might not be able to generate useful options to be exploited by the scheduler for task solving, because the coverage of the sub-trajectory space is poor. To combat this degenerate solution, we add a second term which maximizes the entropy of the worker policy. Intuitively, while the worker generates identifiable sub-trajectories corresponding to given option, it should act as randomly as possible to separate sub-tra sub-trajectories of different options, lowering the "resolution" of the sub-trajectory space to encourage its coverage.
Because directly estimating the posterior p(u, ā kt, is intractable, we approximate it with k S4,1+11 to obtain a lower bound (Barber Agakov, 2003), parameterized posterior log Ie (1, [,,,,,, where qg is a discriminator to be learned. Then we can maximize this lower bound instead:
max E 0.1
Σ A.k
log qos (un (a,k, Sa, B log To (a (S, a 1, Un..
The discriminator qu is trained by maximizing likelihoods of options given sampled sub- sub-trajectories. The worker T is trained via max-entropy RL (Soft Actor-Crite (SAC) (Haarnoja et al., 2018)) with the intrinsic reward r/o r,,,+1 = log g 3 log ne (+). B is fixed to 0.01 in our experiments.
Note that there are least four differences between Eq. and the common option discovery objective in either VIC (Gregor et al., 2016) or DIAYN (Eysenbach et al., 2019):
1. Both VIC and DIAYN assume that a sampled option will last through an entire episode, and the option is always sampled at the beginning of an episode. Thus their option trajec- tories "radiate" from the initial state set. contrast, our worker policy learns options that initialize every K steps within episode, and they can have more diverse semantics de- pending on the various states 8h,0 visited by the agent. This is especially helpful for some tasks where new options need to be discovered after the agent reaches unseen areas in later stages of training.
2. Actions taken by the worker policy under the current option will have consequences on the next option. This is because the final state S of the current option is defined to be the initial state 11,0 of the next option. So in general, the worker policy is trained not only to discover diverse options across the current K steps. but also to make the discovery casier in the future steps. In other words, the worker policy needs to solve the credit assignment problem across options, under the expectation of the scheduler policy.
3. To enable the worker policy learn from a discriminator that predicts based on option sub- trajectories (ān,k, S instead of solely on individual states SA,A, we have constructed a new meta MDP where each state Sn,a encodes history from the beginning (h, 0) up to (h, k) within an option h. This new meta MDP is critical, because otherwise one simply cannot learn a worker policy from a reward function that is defined by multiple time steps (sub-trajectories) since the learning problem is no longer Markovian.
4. Lastly, thanks to the new MDP, we are able to explore various possible instantiations of the discriminator (see Section 3.3). As observed in the experiments, individual states actually not the optimal features for identifying options.
These differences constitute the major novelty of our worker objective.
3.2.2 SHORTSIGHTED WORKER
It's challenging for the worker to accurately predict values over a long horizon, since its rewards are densely computed by a complex nonlinear function gø- Also each option only lasts at most K steps. Thus we set the discount 17 for the worker in two shortsighted ways:
1. Hard: setting 7 = every K-th step and 1 = otherwise. Basically this truncates the temporal correlation (gradients) between adjacent options. Its benefit might be faster and easier value learning because the value is bootstrapped over at most K steps (K & T). K. which considers rewards of roughly K steps ahead. The worker policy
2. Soft: still needs to take into account the identification of future option subtrraecctories but their importance quickly decays.
4

---

Published as a conference paper at ICLR 2021
We will evaluate both versions and compare their performance in Section 4.1.
3.3 INSTANTIATING THE DISCRIMINATOR
We explore various ways instantiating the discriminator qoi in order to compute useful intrinsic rewards for the worker. Previous work has utilized individual states (Eysenbach et al., 2019; Jabri et al., 2019) or full observation trajectories (Warde-Farles et al., 2019; Sharma et al., 2019a; Achiam et al., 2018) for option discrimination. Thanks to the newly defined meta MDP, discriminator is able to take option sub-trajee trajectories instead of current individual states for prediction. In this paper, investigate six sub-trajectory feature extractors fe=
Feature extractor
Name
Formulation
Explanation
State
MLP((S,,++1)
Next state alone
Action
MLP([SA.0: k))
Action in context
stateDiss
MLP(S Sh,
Difference between state pairs
(an,k. Sp,k11
StateAction
MLP( A, Sh,411)
Action and next state
StateConcat
MLP
(1,++1)
Concatenation of states
ActionConcat
MLP (A,0 k))
Concatenation of actions
where the operator [-] denotes concatenation and MLP denotes a multilayer perceptron's Our State feature extractor most similar to DIAYN (Eysenbach al., 2019), and StateConcat similar to Warde-Fariey et al., 2019; Sharma et al., 2019a; Achiam et al.. 2018). However we note that unlike these works, the distribution of our option sub-trajectories is also determined by the scheduler in the context of hRL. The other four feature extractors have not been evaluated before. With the extracted feature, the log- probability of predicting an option is simply computed as the negative squared L2 norm: log Qo Uu, [,, assume the discriminator's 's output distribution to be a (0, ID) multivariate Gaussian.
- fl (a kn, 12. by which we implicitly
3.4 OFF-POLICY TRAINING
The scheduler and worker objectives (Eq. 3 and Eq. 5) trained jointly. In principle, on-policy training such as A2C (Clemente et al., 2017) is needed due to the interplay between the scheduler and worker. However, to reuse training data and improve sample efficiency, employ off off-policy training (SAC (Haarnoja et al., 2018)) for both objectives with some modifications.
Modified worker objective In practice, the expectation over the scheduler Te in Eq. 5 is replaced with the expectation over its historical versions. Specifically, we sample options u, from a replay buffer, together with sub-trajectories (an,k, 8t,++1). J. This type of data distribution modification is conventional in off-policy training (Lillicrap et al., 2016).
Intrinsic reward relabeling We always recompute the rewards in Eq. using the up-to-date dis- criminator for every update of é, which can be trivially done without any additional interaction with the environment.
Importance correction The data in the replay buffer was generated by historical worker policies. Thus a sampled option sub-trajectory will be outdated under the same option, causing confusion to the scheduler policy. To resolve this issue, when minimizing the cemporal-differene (TD) еттor between the values of $6,0 and $4+1,0 for the scheduler, an importance ratio can multiplied: (.s, T (an,
A similar correction can also be applied to the discriminator loss. However, in practice we find that this ratio has a very high variance and hinders the training. Like the similar observations made Nachum al. (2018); Fedus et al. (2020), even without importance correction our method is able to perform well empirically?.
IIK-1
In this paper focus non-image observations that be processed with MI.s, although our method doesn't have any assumption about observation space.
"One possible reason is that the deep process is "highly non-stationary anyway, due to changing policies, state distributions and bootstrap targets" (Schaul al., 2016).
5

---

Published as a conference paper at ICLR 2021
Figure 2: The four tasks we evaluate on. From left to right: 7-DOF PUSHER, 7-DOF REACHER, GOALTASK, and KICKBALL The first two tasks simulate one- one-armed PR2 robot environment while the last two are the SOCIALROBOT environment. The final picture shows closeup of the PIONEER 2DX robot used in SOCTAL
4 EXPERIMENTS
Environments We evaluate success rate and sample efficiency across two environment suites, as shown in Figure 2. Important details are presented here with more information in appendix Section B. The first suite consists of two 7-DOF reaching and pushing environments evaluated in Chua et al. (2018). They both emulate a one-armed PR2 robot. The tasks have sparse rewards: the agent gets reward of 0 at every timestep where the goal is not achieved, and upon achieved. There is also a small L2 action penalty applied. In 7-DOF REACHER, the goal is achieved when the gripper reaches 3D goal position. In 7-DOF PUSHER, the goal is to push object to 3D goal position. Episodes have fixed length of 100; a success of an episode defined to be if the goal is achieved the final step of the episode.
We also propose another suite of environments called SOCIALROBOT 3. We construct two sparse reward robotic navigation and manipulation tasks, GOAL.TAKK and KICKBALI. In GOALTASK, the agent gets a reward of when it successfully navigates to goal, -1 if the goal becomes too far, -0.5 every time it is too close distractor object, and 0 otherwise. In KICKBALL, the agent receives reward of 1 for successfully pushing ball into the goal, 0 otherwise, and has the same distractor object penalty. At the beginning of each episode, both the agent and the ball are spawned randomly. Both environments contain small L2 action penalty, and terminate an episode upon success.
Comparison methods One baseline algorithm for comparison standard SAC (Haarnoja et al., 2018), the building block of our hierarchical method. To verify if our worker policy can just be re- placed with a naive action repetition strategy, we compare with SAC+ActRepaat with an action rep- ctition for the same length K as option interval. We also compare against HIRO (Nachum al., 2018), a data efficient hierarchical method with importance-based option relabeling, and HiPPO (Li et al., 2020) which trains the lower level and higher level policies together with one unified PPO-based objective. Both are state-of-the-ar hierarchical methods proposed to solve sparse re- ward tasks. Similar our work, HiPPO makes no assumptions about options, however it utilizes discrete option space and its options are trained with environment reward.
RL framework called ALF A comprehensive hyperparameter search is performed for every method, with far greater search space over HiPPO and HIRO than our method HIDIO ensure maximum fairness comparison; details are presented in Appendix D.
We implement HIDIO based
Evaluation For every evaluation point during training, we evaluate the agent with current determin- istic policies (by taking arg max of action distributions) for fixed number of episodes and compute the mean success rate. We plot the mean evaluation curve over 3 randomly seeded runs with standard deviations shown as the shaded area around the curve.
4.1 WORKER DESIGN CHOICES
We ask and answer questions about the design choices in HIDIO specific to the worker policy ma-
What sub-trajectory feature results in good option discovery? We evaluate all six features pro- posed in Section 3.3 in all four environments. These features are selected to evaluate how different types of subtrajectory information affect option discovery and final performance. They encom- pass varying types of both local and global subtrajectory information. We plot comparisons of
1.
Phttps://ge http:s///ggttbbtcooe www.tthooo.oo.oiii.cooooooo www.obttt.ooooooceeooooocococccc cs/ /Sociaccaaa Robot Ahttps:// :/g/ thub. com Hori www.orizznnobbbooooo zonRobet 1f

---

Published as a conference paper at ICLR 2021
Pusher Success Plot
Reacher Success Poct
Goe Task Success Plot
KackBall Success Poot
1.0
3.0
Acioo
0.8
0.0
0.8
0.6
0.4
4.4
0.2
02
0.5
0.0
0.0
3
3
2
Num Timesteps
Num Timeslaps.
Num Timeslaps
Nam Timeslaos
Figure 3: Comparison of all discriminator features against each other across the four environments. Solid lines indicate hard short-sighted workers (Hard), dotted lines indicated soft short-sighted workers (Soft).
Pucher Success Plot
Reacher Success Hot
GoolTzok Success Plot
cick Success Hot
1.0
0.8
0.8
0.8
LEPO
Action
0.4
0.6
2.4
Stateactoo
01
StateDar
0.6
6.0
Timesteps.
le6
Timesteps
166
Figure 4: Comparisons of the mean success rates of three features of HIDIO (Action, StateAction, StateDiff; solid lines) against other methods (dashed lines).
sample efficiency and final performance in Figure 3 across all environments (solid lines), find- ing that Action, ItateAction, and StateDi ff are generally among the top performers. StateAction ion includes the current action and next state, encouraging wa to differentiate its op- tions with different actions even at similar states. Similarly, Action in includes the option initial state and current action, encouraging option diversity by differentiating between actions conditioned on initial states. Meanwhile StateDi ff simply encodes the difference between the next and current state, encouraging TH to produce options with different state changes at each step.
2. How do soft shortsighted workers (So ft) compare against hard shortsighted workers rd)) In Figure 3, we plot all features with Soft in dotted lines. We can see that in general there is not much difference in performance between Hard and Soft except some extra instability of Soft in REACHER regarding the StateConcal and State features. One reason of this similar general performance could be that since our options very short-term in Hard, the scheduler policy has the opportunity of switching to good option before the current one leads to bad consequences. In a few cases, Ha rd seems better learned, perhaps due an easier value bootstrapping for the worker.
4.2 COMPARISON RESULTS
We compare our three best sub-trajectory features of Ha rd, in Section 4.1, against the SAC baselines and hierarchical RL. methods across all four environments in Figure 4. Generally we see that HIDIO (solid lines) achieves greater final performance with superior sample efficiency than the compared methods. Both SAC and SAC+ActRepeat perform poorly across all environments, and all baseline methods perform significantly worse than HIDIO on REACHER, GOALTASK, and KICKBALL.
In PUSHER, HiPPO displays competitive performance, rapidly improving from the start. How- ever, all three HIDIO instantiations achieve nearly 100% success rates while HiPPO unable to do so. Furthermore, HIRO and SAC+ActRepeat take much longer to start performing well, but never achieve similar success rates as HIDIO. HIDIO is able to solve REACHER while HiPPO achieves only about 60% success rate at best. Meanwhile, HIRO, SAC+ActRepeat, and SAC are unsta- ble or non-competitive. REACHER is difficult exploration problem as the arm starts far from the goal position, and we see that HIDIO's automatically discovered options ease exploration for the higher level policy to consistently reach the goal. HIDIO performs well on GOALTASK, achieving 60-80% success rates, while the task is too challenging for every other method. In KICKBALL, the most challenging task, HIDIO achieves 30-40% success rates while every other learns poorly again, highlighting the need for the intrinsic option discovery of HIDIO in these environments.
7

---

Published as a conference paper at ICLR 2021
Figure 5: Two example options from the StateAction ion instantiation on KICKBALL (top) and PUSHER (bottom). The top option navigates directly to the goal by bypassing obstructions along the way and the bottom option sweeps the puck towards one direction.
In summary, HIDIO demonstrates greater sample efficiency and final reward gains over all other baseline methods. Regular RL (SAC) fails all four environments, and while HiPPO is strong baseline on PUSHER and REACHER, is still outperformed in both by HIDIO. All other meth- ods fail GOALTASK and KICKBALI, while HIDIO is able learn and perform better in both. This demonstrates the importance of the intrinsic, short-term option discovery employed by HIDIO, where the options are diverse enough be useful for both exploration and task completion.
4.3 JOINT T AND TH TRAINING
We ask the next question: is jointly training Ty and To necessary? To answer this, we compare HIDIO against a pre-training baseline where we first pre-train Th. with uniformly sampled options for portion p of total numbers training time steps, and then fix To while training for the remaining (1 p) time steps. This essentially using pre-trained options for downstream higher-level tasks as demonstrated in DIAYN (Eysenbach et al., 2019). We conduct this experiment with the StateAction feature both KICKBALL and PUSHER, with [16 81 1}. The results are shown in Figure 6. We can see that in PUSHER, fewer pre-training time steps are more sample efficient, as the enviromment is simple and options can be learned from small amount of samples. The nature of PUSHER also only requires options that can be learned independent of the scheduler policy evolution. Nevertheless, the pre- training baselines seem less stable. In KICKBALL, the optimal pre- pre-training baseline on p & of the total time steps. However without the joint training scheme of HIDIO, the learned options are unable to used efficiently for the difficult obstacle avoidance, navigation, and ball manipulation subtasks required for performing well.
Pudder Survees Pae
KindW Sunneed ot
2.6
de
Nar Timakpp
Mar Timalapp
3M4
Figure 6: Pretraining baseline comparison fractions f 16 8 11 of the total number of training time steps.
4.4 OPTION BEHAVIORS
Finally, since options discovered by HIDIO in our sparse reward environments help it achieve supe- rior performance, we ask, what do useful options look like? To answer this question, after training. sample options from the scheduler ma to visualize their behaviors in different environments in Figure 5. For each sampled option u, we fix it until the end of an episode and use the worker To to output actions given We can that the options learned by HIDIO are low-level navigation and manipulation skills useful for the respective environments. We present more visualizations in Figure and more analysis in Section C.2 in the appendíx. Furthermore, present analysis of task performance for different option lengths in appendix Section C.I and Figures and 8.
5 RELATED WORK
Hierarchical RL Much of the previous work in hRL makes assumptions about the task structure and/or the skills needed to solve the task. While obtaining promising results under specific settings, they may have difficulties with different scenarios. For example, SAC-X (Riedmiller et al., 2018) quires manually designing auxiliary subtasks as skills to solve a given downstream task. SNN4HRL (Florensa et al., 2016) is geared towards tasks with pre-training and downstream components. Lee
8

---

Published as a conference paper at ICLR 2021
et al. (2019; 2020) learns to modulate compose given primitive skills that are customized for their particular robotics tasks. Ghavamzadeh & Mahadevan (2003) and Sohn et al. (2018) operate under the assumption that tasks can be manually decomposed into subtasks.
The feudal reinforcement learning proposal (Dayan & Hinton, 1993) has inspired another line of works (Vezhnevets et al., 2017; Nachum et al., 2018; Levy et al., 2019; Rafati & Noelle, 2019) which make higher-level manager policies output goals for lower-level worker policies to achieve. Usually the goal space is a subspace of the state space or defined according to the task so that lower-level rewards are easy to compute. This requirement of manually "grounding" goals in the environment poses generalization challenges for tasks that cannot be decomposed into state or goal-reaching.
The MAXQ decomposition (Dietterich, 2000) defines an hRL task decomposition by breaking up the target MDP into a hierarchy of smaller MDPs such that the value function in the target MDP is represented as the sum of the value functions of the smaller ones. This has inspired works that use such decompositions (Mehta et al., 2008; Winder et al., 2020; Li et al., 2017) to learn structured, hierarchical world models or policies to complete target tasks perform transfer learning. However, building such hierarchies makes these methods limited to MDPs with discrete action spaces.
Our method HIDIO makes few assumptions about the specific task at hand. It follows from the op- tions framework (Sutton et al., 1999), which has recently been applied to continuous domains (Bacon et al., 2017), spawning a diverse set of recent hierarchical options methods (Bagaria & Konidaris, 2020; Klissarov et al., 2017; Riemer et al., 2018; Tiwari & Thomas, 2019; Jain et al., 2018). HIDIO automatically learns intrinsic options that avoids having explicit initiation or termination policies de- pendent on the task at about the task, but does not employ self-supervised learning for training the lower-level policy.
hand. HiPPO (Li et al., 2020), like HIDIO, also makes no major assumptions
Self-supervised option/skill discovery There are also plenty of prior works which attempt to learn skills or options without task reward. DIAYN (Eysenbach et al., 2019) and VIC (Gregor al., 2016) learn skills by maximizing the mutual information between trajectory states and their corresponding skills. VALOR (Achiam et al., 2018) learns options by maximizing the probability of options given their resulting observation trajectory. DADS (Sharma et al., 2019a) learns skills that are predictable by dynamics models. DISCERN (Warde-Farke et al., 2019) maximizes the mutual information between goal and option termination states to learn a goal-conditioned reward function. Brunskill & Li (2014) learns options in discrete MDPs that are guaranteed to improve a measure of sample complexity. Portable Option Discovery (Topin et al., 2015) discovers options by merging options from source policies to apply to some target domain. Eysenbach et al. (2019); Achiam et al. (2018); Sharma al. (2019a); Lynch al. (2020) demonstrate pre-trained options be useful for hRL. These methods usually pre-train options in an initial stage separate from downstream task learning; few works directly integrate option discovery into a hierarchical setting. For higher dimensional input domains. Lynch et al. (2020) learns options from human-collected robot interaction data for image-based, goal-conditioned tasks, and Chuck et al. (2020) learns a hierarchy of options by dis- covering objects from environment images and forming options which can manipulate them. HIDIO can also be applied to image-based environments by replacing fully-connected layers with convolu- tional layers in the early stages of the policy and discriminator networks. However, we leave this to future work to address possible practical challenges arising in this process.
6 CONCLUSION
Towards solving difficult sparse reward tasks, we propose a new hierarchical reinforcement learn- ing method, HIDIO, which can learn task-agnostic options in a self-supervised manner and simul- taneously learn to utilize them to solve tasks. We evaluate several different instantiations of the discriminator of HIDIO for providing intrinsic rewards for training the lower-level worker policy. We demonstrate the effectiveness of HIDIO compared against other reinforcement learning methods in achieving high rewards with better sample efficiency across a variety of robotic navigation and manipulation tasks.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv, 2018. 5,
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, 2017. 9
Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. ICIR, 2020.
D. Barber and F. Agakov. The im algorithm: A variational approach to information maximization. In NeurlPS, 2003. 4
Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. volume 32 of Proceedings of Machine Learning Research, pp. 316-324, Bejing, China, 22-24 Jun 2014. PMLR. URL http:/ /ttp://e////ceeeee/e ngs mllr mir..n press / forrnsssill11 html.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurlPS, 2018.
Caleb Chuck, Supawit Chockchowwat, and Scott Niekum. Hypothesis-driven skill discovery for hierarchical deep reinforcement learning, 2020. 9
Alfredo V. Clemente, Humberto Nicolás Castejón Martinez, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. CoRR, ab4/1705.04862, 2017.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In NeurIPS, pp. 271-278, 1993.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. Journal of artificial intelligence research, 13:227-303, 2000. 9
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need:
Learning skills without a reward function. In ICLR, 2019. 4, 5, 8,
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In ICML, 2020. 5
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hicrarchical rein- forcement learning. In ICLR, 2016. 1,
Mohammad Ghavamzadch and Sridhar Mahadevan. Hierarchical policy gradient algorithms. ICML, 2003. 1,
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv, abs/1611.07507, 2016. 4,
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018. 4, 5,
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In ICLR, 2018. 1
Allan Jabri, Kyle Hsu, Ben Eysenbach, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Unsu- pervised curricula for visual meta-reiinorremmet learning. In NeurIPS, 2019. 5
Arushi Jain, Khimya Khetarpal, and Doina Precup. Safe option-critic: Learning safety in the option- critic architecture. arXiv, 2018. 9
Martin Klissarov, Pierre-Luc Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end for continuous action tasks. arXiv, 2017. 9
Hoang M. Le, Nan Jiang, Alckh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hier- archical imitation and reinforcement learning. In ICMI., 2018.
10

---

Published as a conference paper at ICLR 2021
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward Hu, and Joseph J. Lim. Com- posing complex skills by learning transition policies with proximity reward induction. In ICLR, 2019. 1,
Youngwoon Lee, Jingyun Yang, and Joseph J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. In ICLR, 2020. 1,
Andrew Levy, Robert Platt Jr., and Kate Saenko. Learning multi-level hierarchies with hindsight. In ICLR, 2019.
Alexander C. Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. In ICLR, 2020. 6, 9
Zhuoru Li, Akshay Narayan, and Tze- Yun Leong. An efficient approach to model-based hierarchical reinforcement learning. In Thirty- Firry- AAAI Conference on Artificial Intelligence, 2017. 9
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016. 5
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, pp. 1113- 1132, 2020. 9
Neville Mehta, Soumya Ray, Prasad Tadepalli, and Thomas Dietterich. Automatic discovery and transfer of maxq hierarchies. In Proceedings of the 25th international conference Machine learning, pp. 648-655, 2008. 9
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hicrarchical visuomotor control of humanoids. In ICLR, 2019.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In NeurlPS, 2018. 1, 6,
Jacob Rafati and David C Noelle. Learning representations in model-free hierarchical reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 10009 10010, 2019.
Pravesh Ranchod, Benjamin Rosman, and George Konidaris. Nonparametric bayesian reward seg- mentation for skill discovery using inverse reinforcement learning. In IROS, 2015.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In ICML, 2018. 1.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. In NeurlPS, 2018.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016. 5
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv, 0ttp//9770016777 2019a. 5,
Arjun Sharma, Mohit Sharma, Nicholas Rhinehart, and Kris M. Kitani. Directed-info gail: Learning hierarchical policies from unsegmented demonstrations using directed information. In ICLR, 2019b.
Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies. In NeurlPS, 2018. 1,9
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1 112(1):181 -211, 1999. 2,
Saket Tiwari and Philip S. Thomas. Natural option critic. In AAAI, 2019.

---

Published as a conference paper at ICLR 2021
Nicholay Topin, Nicholas Haltmeyer, S. Squire, J. Winder, M. desJardins, and J. MacGlashan. Portable option discovery for automated learning transfer in object-oriented markov decision pro- cesses. In IJCAI, 2015.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In ICML, 2017. 3,
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. ICLR, 2019. 5,
John Winder, Stephanie Milani, Matthew Landen, Erebus Oh, Shane Parr, Shawn Squire, Cynthia Matuszek, et al. Planning with abstract learned models while learning transferable subtasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9992-10000, 2020.
A
PSEUDO CODE FOR HIDIO
Algorithm 1: Hierarchical RL with Intrinsic Options Discovery
Input:
Episode length M
Batches per iteration
The (av. (8 us) qo un ( Bh.4+1) Te (n (8h.0)
Worker
B
a Option interval P(s (s.... a
Batch size
Learning rate
Discriminator Scheduler
K
Environment dynamics
Output: Learned parameters 6, é. and vb
Initialize: Random model parameters 0, o. and th; empty replay buffers D Decheduler and Dookerr-
while termination mel do x Data collection
for scheduler step h 0.. T 1 do
Sample an option un ~ ( (-10p..)) for worker step k = O..K
do
Sample an action an.x ~ T (- Un). Step through the environment s/,4+1 P( (s, an,k). al.k: 51,,+1 - b a (5, k/,+-1 Dworker Dworker U (u,, an,k 51,++1)
end
Rh. <- [K-1 r(sh,kk an.k: Sh,++1)
D Dxcheduler f- D Decheduler U (8h,0, Un 8h+1,0; RA)
end
/* Model training
for batch m m O..M - do
Scheduler training
Uniformly sample transitions ((s,, u, st41))
Decheduler.
Compute gradient AO according to Eq. 3.
Update models +- 0 aA0.
/* Worker training
Uniformly sample transitions {(u,, a k. bb D Dwerker- Compute intrinsic rewards r + qo (un ( kh,4+1).
Compute gradient A and A according to Eq. 5.
Update models <- + aAd and v t- s adr.
end
end
12

---

Published as a conference paper at ICLR 2021
Pusher Success Piot
Reacher Success Rot
Goe Task Success Plot
1dckSall Success Poot
3.0
0.0
0.4
0.2
02
0.5
0.0
0.0
0.0
0.5
3.0
3.5
0.0
0.5 3.0
1.5
3
Num Timesdeps
Num Timesaaps
Num Timeslaps
Nam Timesiaos
Figure 7: Comparisons of the mean success rates of three features of HIDIO (Action, StateAct i on,, StateDiff at different option lengths K. Dotted lines indicate K = 1, solid lines indicate K = 3, and dashed lines indicate K = 5. = was used across all environments for the results in the main text.
B MORE ENVIRONMENT DETAILS
B.0.1 PUSHER AND REACHER
These environments both have a time horizon of 100 with no early termination: each episode always runs for 100 steps regardless of goal achievement. For both, a success when the agent achieves the goal at the final step of an episode. In REACHER, observations are 17-dimensional, including the positions, angles, and velocities of the robot arm, and in PUSHER observations also include the 3D object position. Both include the goal position in the observation space. Actions are 7-dimensional vectors for joint velocity control. The action range is [-20, 20j in REACUER and [-2, 2] in PUSHER.
There is an action penalty in both environments: every timestep the squared L2 norm of the agent action is subtracted from the reward. In PUSHER, this penalty is multiplied by a coefficient of 0.001. In REACHER, it's multiplied by 0.0001.
B.0.2 GOALTASK AND KICKBALL
For both SOCIALROBOT environments, episode terminates early when either success is reached or the goal is out of range. For each episode, the positions of all objects (including the agent) are randomly picked. Observations centric positions, distances, and directions from the agent to different objects while in KICK BALL, they are absolute positions and directions. In KICKBALL, the agent receives a reward of for suc- cessfully pushing a ball into the goal (episode termination) and O otherwise. At the beginning of each episode, the ball spawned randomly inside the neighborhood of the agent. Three distractor objects are included on the ground to increase task difficulty. In GOALTASK, the number of dis- tractor objects increases to 5. Both environments contain a small L2 action penalty: at every time step the squared L2 norm of the agent action, multiplied by 0.01, is subtracted from the reward. GOALTASK has a time horizon of 100 steps, while KICKBALL'S 's horizon 200. Observations are 30-dimensional, including absolute poses and velocities of the goal, the ball, and the agent. Both GOALTASK and KICKBALL use the same navigation robot PIONEER2DX which has 2-dimensional actions that control the angular velocities (scaled to [-1, , 1)) of the two wheels.
18 18-dimensional. In GOALTASK, these observations include ego-
C OPTION DETAILS
C.1 OPTION LENGTH ABLATION
We ablate the option length K in all four environments on the three best HIDIO instantiations in Figure 7. K = 3,, 5} timesteps per option are shown, with K = 3 and K 5 performing similarly across all environments, but K provides no temporal abstraction, resulting in worse sample efficiency in PUSHER and REACHER, and failing to learn in GOALTASK and KICKBALL Although K 5 and K similar, we see in GOALTASK that K = results in better performance than K across all three instantiations, demonstrating the potential benefit of longer temporal abstraction lengths.
performing very poorly in comparison. K
are generally
13

---

Published as a conference paper at ICLR 2021
GoalTask Option Length Comparison
KickBall Option Length Comparison
1.5
7.5
VVelocity
0.3
Pozition
2.5
-0.5
-2.5
-10
-5.0
-20
-10.0
X Velecity
X Position
Figure 8: Trajectory distributions compared for different option lengths K for the StateActio ion HIDIO instantiation in both SOCiALROBOT environments. These obtained by randomly sam- pling an option uniformly in 1- l and keeping it fixed for the entire trajectory. 100 trajectories from each option are visualized and plotted in different colors.
We also plot the distribution of (z. 3) velocities in GOALTASK and (z. y) coordinates in KICK- BALL of randomly sampled options of different lengths in Figure 8. Despite the fact that these two dimensions only represent a small subspace of the entire (30-dimensional) state space, they still demonstrate difference in option behavior at different option lengths. We can that the option length K increases, the option behaviors become more consistent within a trajectory. Mcanwhile regarding coverage, K I's (blue) trajectory distribution in both environments is less concentrated near the center, while K = (green) the most concentrated at the center. K = 3 (orange) lies somewhere in between. We believe that this difference in behavior signifies trade off between the coverage of the state space and how consistent the learned options can be depending on the option length. Given the the discriminator can more easily discriminate the sub-trajectorie created by these options, so that their coverage does not have be as wide for the worker policy obtain high intrinsic rewards. Meanwhile, with shorter option lengths, the shorter jub-trajectories have to be more distinct for the discriminator be able to successfully differentiate between the options.
entropy coefficient (3 in Eq 5), with longer option lengths, is likely that
C.2 OPTION VISUALIZATIONS
We visualize more option behaviors in Figure 9, produced in the same way as Figure 5 and as detailed in Section 4.4. The top 4 picture reels are from KICKBAL.I We see that KICKBAL.L options lead to varied directional driving behaviors that can be utilized for efficient navigation. For example, the second, third, and fourth highlight options that produce right turning behavior, however at different speeds and angles. The option in the third reel is a quick turn that results the robot tumbling over into unrecoverable state, but the options in the second and fourth reels turn more slowly and do not result the robot flipping. The first option simply proceeds forward from the robot starting position, kicking the ball into the goal.
The bottom reels are from PUSHER. Each option results in different sweeping behaviors with varied joint positioning and arm height. These sweeping and arm folding behaviors, when utilized in short sub-trajectories, are useful for controlling where and how to move the arm to push the puck into the goal.
D HYPERPARAMETERS
To ensure a fair comparison across all methods, we perform a hyperparameter search over the fol- lowing values for each algorithm and suite of environments.
Velocities relative to the agent's yaw rotation. Because GOAL.TAS has egocentric inputs, the agent is of the absolute (ar y) coordinates in this task.
14

---

Published as a conference paper at ICLR 2021
Figure 9: Eight example options from the StateAct ion instantiation on KICKBALL (top 4) and PUSHER (bottom 4).
15

---

Published as a conference paper at ICLR 2021
D.1 PUSHER AND REACHER
Shared hyperparameters across all methods are listed below (where applicable, and except when overridden by hyperparameters listed for each individual method). For all methods, we take the hyperparameters that perform best across random seeds in terms of the area under the evaluation success curve (AUC) the PUSHER environment.
Number of parallel aetors/envirrnments per rollout: 20
Steps per episode: 100
Batch size: 2048
Learning rate: 10 for all network modules
Policy/Q network hidden layers: (256, 256, 256) with ReLU hon-linearities
Polyak averaging coefficient for target Q: 0.999
Target Q update interval (training iterations):
Training batches per iteration: 100
Episodes per evaluation: 50
Initial environment steps for data collection before training: 10000
Rollouts and training iterations are performed alternatively, one after the other. The rollout length searched below refers how many time steps in each environment are taken per rollout/training iteration, effectively controlling the ratio of gradient steps to environment steps. A smaller roll- length corresponds higher ratio. This ratio is also searched over for HIPPO and HIRO. Other hyperparameters searched separately for each algorithm are listed below, and selected ones bolded.
D.1.1
SAC
Target entropy min prob A: (0.1, 0.2, 0.3)
Replay buffer length per parallel actor: (50000, 200000)
Rollout Length: {12, 25, 50, 100)
D.1.2 SAC w/ ACTION REPETITION
Action repetition length?: 3
Rollout Length: {4, 8, 16, 33}
Other hyperparameters are kept the same as the optimal SAC ones.
1.3.3 HIDIO
The hyperparameters of HIDIO were mostly heuristically chosen due the hyperparameter search space being too large.
Latent option u vector dimension (D): {8. 12]
Policy/Q network hidden layers for गe : (128, 128, 128)
Steps per option (K):
T has a fixed entropy coefficient a of 0.01. Target entropy min prob A for Ty is 0.2.
Discriminator network hidden layers: (64, 64)
Replay buffer length per parallel actor: (50000, 200000)
Rollout Length: {25, 50, 100}
'The target entropy used for automatically adjusting calculated as: [In(M, mm)) + In where M//m; are maximium/minnmmmm value of action dim Intuitively, the target distribution concentrates on segment of length (M, m.) with a constant probability.
"Chosen match the option interval K of HIDIO.
16

---

Published as a conference paper at ICLR 2021
1.1.4
HIRO
Steps per option: 53 8}
Replay buffer size (total); {500000, 2000000}
Meta action space (actions relative, e.g. meta-action is current-obs + action):
(-np. ones (obs_space 3D.goal.pos) +2, np. (obs.space 3D.goal.pos) +2)
Policy stddev noise: {0.1, 0.3, 0.5)
Number of gradient updates per training iteration: {100, 200, 400)
D.1.5 НлРРО
For most hyperparameters, the search ranges chosen were derived after discussion with the first author of HiPPO.
Learning rate: 3 x 10-4
Policy network hidden layers: (256, 256)
Skill selection network hidden layers: ((32, 32), (128, 64))
Latent skill vector size: (5, 10, 15)
PPO clipping parameter: (0.05, 0.1)
Time commitment range: {(2, 5), (3, 7))
Policy training steps per epoch: {25, 50, 100)
D.2 ROBIALROBOT
For all methods, we select the hyperparameters with the best area under the evaluation success curve (AUC) in the KICKBALL environment, and apply them to both KICKBALL and GOALTASK. The shared hyperparameters are as follows (if applicable the algorithm, and except when overridden by the respective algorithm's 's list of hyperparameters):
Number of parallel actors/environments per rollout: 10
Steps per episode: 100 (GoALTAsk), 200 (KICKBALL)
Batch size: 1024
Learning rate: 5 x 10-4 for all network modules
Policy/Q network hidden layers: (256, 256, 256) with ReLU non-linearities
Polyak averaging coefficient for target Q: 0.95
Target Q update interval (training iterations):
Training batches per iteration: 100
Episodes per evaluation: 100
Evaluation interval (training iterations): 100
Initial environment steps for data collection before training: 100000
The training terminology here generally follows section D.1.
D.2.1
SAC
Target entropy min prob A: {0.1, 0.2, 0.3)
Replay buffer length per parallel actor: (20000, 100000)
Rollout length: (12, 25, 50, 100)
17

---

Published as a conference paper at ICLR 2021
D.2.2
SAC w/ ACTION REPETITION
Action repetition length":
Rollout Length: {4, 16, 33}
Other hyperparameters are kept the same as the optimal SAC ones.
D.2.3
HIDIO
Due to the large hyperparameter search space, we only search over the option vector size and rollout length, and select everything else heuristically.
Latent option u vector dimension (D): {4, 6}
Policy/Q network hidden layers for n. (128, 128, 128)
Steps per option (K):
T has fixed entropy coefficient of 0.01. Target entropy min prob A for TT is 0.2.
Discriminator network hidden layers: (32, 32)
Replay buffer length per parallel actor: 20000
Rollout Length: {50, 100}
D.2.4 HIRO
Learning rate: 3 x 14-4
Steps per option: 5.
Replay buffer size (total): (500000, 2000000)
Meta action space (actions are relative, e.g., meta-action is current-obs action):
GOALTASK:
(-np. ones (obs_space)
2, np.ones (obs_space)
2)
KICKBALL:
I-np.ones (obs space goal 1 spacee 2, np. ones (obs_space goal_sgaapeee __space) 2) (because the goal position is given but will not change in the observation space)
Policy stddev noise (0.1, 0.3, 0.5}
Number of gradient updates per training iteration: (100, 200, 400)
D.2.5 НıРPO
Learning rate: 3 x 10-4
Policy network hidden layers: {(64, 64), (256, 256))
Skill selection network hidden layers: ((32, 32), (128, 64))
Latent skill vector size: {4, 8)
PPO clipping parameter: (0.05, 0.1)
Time commitment range: ((2, 5), (3, 7))
Policy training steps per epoch: (25, 50, 100}
*Chosen match the option interval K of HIDIO.
18


Reviews:
[{"review_id": "fGyHGG_ZvV1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper develops a hierarchical reinforcement learning algorithm and analyzes its behaviour in four robotic manipulation and navigation tasks. The approach is based on a two-level hierarchy, *scheduler* at the top and *worker* at the bottom. This is similar to other approaches in the literature and the algorithm uses many ideas and elements from existing algorithms. However, these ideas and elements are combined in a novel and well-justified manner. The result is an algorithm that yields good results in a range of problems. The experiments are well done. The paper is generally organised well and written clearly. Relevant literature is reviewed well. \n\nThe paper can be improved by a more comprehensive and detailed analysis of the behaviour of the algorithm, in particular, the options that are found. Section 4.4 is useful but very short. It describes only two options. Perhaps such an analysis can be added to the appendix.\n\nIn the proposed algorithm, the scheduler outputs an option every K steps in the environment. It would be reasonable to question whether more flexibility would be useful here, one that allows for varying option durations.\n\nThe authors state in Section 2 that they 'will use the terms “goal”, “option”, and “skill” interchangeably.' In the literature, these terms refer to related but different concepts. Using them interchangeably is not good scientific practice.\n\nI did not find Figure 1 particularly useful. The simple and intuitive structure of the algorithm does not come through in the figure.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "T4Fi5cWSHrt", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper present HIDIO, a hierarchical RL method that leverages self-supervised losses to discover intrinsic options while learning a scheduler to leverage the learned options to optimize the accumulated reward. HIDIO differentiates from prior work through enabling the low-level network/worker to discover task-agnostic options that can be generalized to future tasks, thus requiring no pre-training of skills,  and at the same time makes minimal assumption about the task structure.\n\nThis paper is in general well-written with clarity. The proposed  method is technically sound and empirically outperforms existing methods. The experiments in four different tasks first demonstrate an ablation of different feature extractors used for the self-supervised loss and analyzed the each of their advantages/disadvantages; followed by experiments comparing HIDIO with existing hRL methods and show that HIDIO outperforms in all tasks. Although the experiments are conducted in two different domains and four different tasks, all four tasks seem to be similar in nature (i.e. all pushing and reaching tasks). It would be great to see how HIDIO compare with other methods in more complex task domains such as in the work of [1] and [2], where the skill discovery network need to work with high dimensional raw input data and computing state-similarity can be tricky.\n\n\n[1] Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., & Sermanet, P. (2020, May). Learning latent plans from play. In Conference on Robot Learning (pp. 1113-1132).\n[2]Chuck, C., Chockchowwat, S., & Niekum, S. (2020). Hypothesis-Driven Skill Discovery for Hierarchical Deep Reinforcement Learning. International Conference on Intelligent Robots and Systems, 2020.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bn2AziXMkRG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an interesting approach to self-supervised option learning for hierarchical reinforcement learning. In particular, the authors propose to learn options through an intrinsic entropy minimization objective conditioned on option trajectories. Empirical studies are performed to demonstrate the success rate and sample efficiency of this method over two state-of-the-art methods.\n\nThe key concern that I have with this paper is its organization and presentation. I am concerned that the paper takes too much space describing what other methods do not do (throughout the contributions sections), rather than concretely describing the approach. Furthermore, many sections seem to be an aggregation of contributions, related work, and results, which further makes the paper hard to follow. I believe that this does the paper a disservice by decreasing readability. Therefore, I am currently voting for rejecting the paper. \n\nQuestions and Comments:\n1) In the Introduction, the authors claim that their approach requires \"little\" manual design? Can this claim be better grounded? What design is required?\n2) There is extensive work on option discovery [1,2]. as well as learning decompositions for reinforcement learning [3-6]. I would have liked to see a more extensive literature review comparing and contrasting with these works.\n3) On page 2, the authors say, \"for the purpose of this paper, we consider a hierarchy of two levels\". However, it is not clear to me how this approach would extend to a hierarchy with more than two levels. Can the authors please clarify this?\n4) What is k? h? (page 2)\n5) On page 3, the authors claim that, unlike prior work, their formulation has no constraints on what the options can be. Can the authors comment on this claim in more detail?\n6) In section 3.2, the authors mention the use of goals. Are the authors considering goal-conditioned MDPs [7]? If so, this should be made more explicit.\n8) The MDP definition in section 3.2 is unclear. Could the authors please clarify this definition? \n9) What is meant by \"resolution\" (section 3.2)? \n10) Why does the objective defined by the authors enable skills that have more diverse semantics than previous approaches (claimed on page 4, section 3.2)?\n11) Page 5, section 3.4: is the importance correction not used at all (for either the scheduler or discriminator)? \n12) What assumptions are needed for this method to work well?\n13) In the experiment section, it says that in the pusher environment, success means that the goal is achieved in the final step of the episode. Does this mean that pushing the object to the position does not terminate the episode and, instead, the object must remain there until the episode terminates in a predetermined number of steps?\n14) Can the authors please clarify the purpose of the additional suite of environments? What do these environments enable that existing ones do not?\n\n[1] Brunskill, et al., PAC-inspired option discovery in lifelong reinforcement learning. ICML, 2014.\n[2] Topin, et al., Portable option discovery for automated learning transfer in object-oriented Markov decision processes. IJCAI, 2015.\n[3] Mehta et al., Automatic discovery and transfer of MAXQ hierarchies. ICML, 2008.\n[4] Winder et al., Planning with abstract learned models while learning transferable subtasks. AAAI, 2020.\n[5] Li, et al., An efficient approach to model-based hierarchical reinforcement learning. AAAI, 2017.\n[6] Rafati, et al., Learning representations in model-free hierarchical reinforcement learning. AAAI, 2019.\n[7] Nasiriany, et al., Planning with goal-conditioned policies. NeurIPS, 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9EzHASQRauK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#######################################################################\n\nSummary:\n\nIn this paper the authors present a new method for hierarchical reinforcement learning, demonstrated with a 2 layer architecture, in which the higher layer Scheduler policy choose lower level Worker policies at fixed intervals. The lower level Worker policies (options) are trained through an intrinsic entropy minimization.\n\n#######################################################################\n\nReasons for score:\n\nThis paper seems to be based strongly on the work in Sharma et al., 2019b and Eysenbach et al., 2019, and while it makes a number of interesting modifications to those methods there is in sufficient decoupling / ablation of the issues. I think the paper would be notably stronger with additional experimental evidence \n\n#######################################################################Pros:\n\n1. The paper makes some reasonably sensible extension to well-know methods.\n\n2. The authors treatment of hyper parameters and implementation details looks to be thorough\n\n#######################################################################\n\nCons:\n\n1. I found parts of the paper difficult to parse. For example it was not immediately clear to me what exactly an option $\\bf{u} \\in [-1,1]^D$ is... What values can / does D take? Another free parameter? Is $\\bf{u}$ a latent variable that yields an option through the worker policy? (this terminology would seem to better align with that in Eysenbach et al., 2019 and the broader literature around options). I'm also not sure $\\bar{\\bf{s}}_t$ (single subscript) is defined?\n\n2. As the authors note the paper introduces a number of new elements: the reset of the worker policy every k-step, the utilization of full sub-trajectories, and the various instantiations of the discriminator... but these are never unwound. It would seem to be the case that there would be a fair bit of interplay between these elements. Additionally it appears that there is some need to modify the implementation to support these choices - most notably the discounting for the \"shortsighted worker\"... is this still necessary if full sub-trajectories are not used?\n\n3. The experimentation did not address some fairly obvious enquires (what is the effect of different choices of K in your experiments? How would a practitioner approach choosing this value?)\n\n4. There is remarkably little in this paper that talks about the actual options uncovered via this method. Does the method uncover different options for different values K? What do they look like? etc.\n\n#######################################################################\n\nQuestions during rebuttal period:\n\nQ1: a fair amount of the focus of the paper appears to be on the instantiation of the discriminator - do you consider that to be a central contribution?\n\nQ2: you talk about the fact that a key divergence from priori work is that there is \"no constraint on the value on what $\\bf{u}$ could represent.\" Can you talk about why this is a good thing and have you explored when it might not be such a good thing?\n\n#######################################################################", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to hierarchical RL which automatically learns intrinsic task-agnostic options. The approach involves a two-level hierarchy, with policies learned by lower-layer Workers and selected by a higher-layer Scheduler. The approach is evaluated on four complex tasks and is shown to outperform existing methods.
There were initial concerns with this paper around clarity of a number of points. These included the contributions of this work and questions around the experimental results, such as  discussing the learned options themselves. The authors provided extensive responses to these concerns, and updated the paper accordingly, including addition results and analysis. I believe the paper is now much clearer with interesting contributions.
Decision: Accept (Poster)