Hierarchical Skills for Efficient Exploration
Authors: Jonas Gehring
Gabriel Synnaeve
Andreas Krause
Nicolas Usunier
Venue: NeurIPS 2021 Poster (2021)
Keywords: Hierarchical Reinforcement Learning
Skill Discovery

Abstract: In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensitivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest. In this work, we analyze this trade-off for low-level policy pre-training with a new benchmark suite of  diverse, sparse-reward tasks for bipedal robots. We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner. For utilization on downstream tasks, we present a three-layered hierarchical learning algorithm to automatically trade off between general and specific skills as required by the respective task. In our experiments, we show that our approach performs this trade-off effectively and achieves better results than current state-of-the-art methods for end-to-end hierarchical reinforcement learning and unsupervised skill discovery.

Hierarchical Skills for Efficient Exploration
Jonas Gehring! 1.2
Gabriel Synnaeve'
Andreas Krause?
Nicolas Usunier'
Facebook AI Research
"ETH Zürich
jgehr jgehringefb.com
Abstract
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task required to strike the right balance between generality fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensi- tivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest. In this work, we analyze this trade-off for low-level policy pre-training with benchmark suite of diverse, sparse-reward tasks for bipedal robots. We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner. For utilization on downstream tasks, we present a layee-layered hierarchical learning algorithm automatically trade off between general and specific skills as required by the respec- tive task. In our experiments, we show that our approach performs this trade-off effectively and achieves better results than current state-of saate-of-the of-the-art methods for end- to-end hierarchical reinforcement learning and unsupervised skill discovery. Code and videos are available at httph:::/ htttps://facebbookessell ch github. 3o/hsd3
Introduction
promising direction for improving the sample efficiency of reinforcement learning agents in com- plex environments is to pre-train low-level skills that are then used to structure the exploration in This has been studied in particular for the control of (simulated) robots, where there is natural hierarchical decomposition of the downstream tasks into low-level con- trol of the robot's actuators with skill policy, and a high-level control signal that specifies a direction or target robot configuration with coarser temporal resolution. The large body of work on unsuper- vised skill or option discovery hierarchical reinforcement learning (HRL) for continuous control prior knowledge that low-level skills should control the center of mass of the robot (26] [7 [3 45 This nicely fits wide range of benchmark tasks that are variants of navigation problems, but the benefit of such hierarchical setups outside this problem class is unclear.
downstream tasks P iV K 30
relies, explicitly or implicitly,
The prior knowledge embedded in pre-trained skill defines specific trade-off between sample efficiency and generality. Skills that severely constrain the high-level action space to elicit specific behavior (e.g., translation of the center of mass) likely provide the largest gains in sample effi- ciency, but are unlikely to be useful on diverse set of downstream tasks. Conversely, low-level skills that expose many degrees of freedom are more widely applicable but less useful for guiding explo- ration. There is, thus. no single universally superior pre-trained skill. Depending on the downstream task, different skills might also useful to efficiently explore in different parts of the environment.
In this paper, we aim to acquire skills that are useful for a variety of tasks while still providing strong exploration benefits. We propose to pre-train a hierarchy of skills of increasing complexity which can subsequently be composed with a high-level policy. In the context of simulated robots, each skill consists of controlling a part of the robot configuration over short time horizon, such
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

as the position of the left foot of humanoid, or the orientation of its torso. Skills of increasing complexity are constructed by jointly controlling larger portions of the configuration. These skills are modelled with a shared policy and pre-trained in an environment without rewards and containing the robot only. Subsequently, skills are used in downstream tasks within a three-level hierarchical policy: the highest level selects the skill (which specifies a goal space), the second level the target configuration within that skill (the goal), and the pre-trained skill performs the low-level control to reach the goal. Compared to standard approaches involving a single static pre-trained skill [13 [K]. our approach offers increased flexibility for structuring exploration and offloads the issue of selecting prior knowledge from pre-trate pre-training downstream task learning. As result, our skills can be acquired once per robot and applied to many different tasks.
We perform an experimental analysis of our hierarchical pre-training on a new set of challenging sparse-reward tasks with simulated bipedal robots. Our experiments show that each task is most efficiently explored by a distinct set of low-level skills, confirming that even on natural tasks, where locomotion is of primal importance, there no overall single best pre-trained skill. We further show that dynamic selection of goal spaces with a three-level hierarchy performs equally or better than generic skill on all tasks, and can further improve over the best single skills per task.
The main contributions of our work are summarized as follows:
We propose a novel unsupervised pre-training approach that produces a hierarchy of skills based on control of variable feature sets.
We demonstrate how to automatically select between different skills of varying complexity with a three-level hierarchical policy that selects skills, goals, and native actions.
We introduce a benchmark suite of sparse-reward tasks that allows for consistent and thorough evaluation of motor skills and HRI. methods beyond traditional navigation settings.
We study the implications of prior knowledge in skills experimentally and showease the
efficacy of our hierarchical skill framework on the proposed benchmark tasks, achieving superior results compared to existing skill discovery and HRL approaches
[3 33 53
2
Related Work
The success of macro-operators and abstraction in classic planning systems large body of works on hierarchical approaches to reinforcement learning 8 26
14
40
has inspired
50
3K
39
While the decomposition of control across multiple levels of abstraction provides intuitive benefits such easier individual learning problems and long-term decision making, recent work found that a primary benefit of HRL, in particular for modern, neural-network based learning systems, stems from improved exploration capabilities 211 B5 From a design perspective, HRL allows for separate acquisition of low-level policies (options: skills), which can dramatically accelcrate learning on downstream tasks. A variety of works propose the discovery of such low-level primitives from
random walks 26 52 mutual information objectives 17 13 + 3 datasets of agent or expert
traces
motion capture data 36
or from dedicated pre-training tasks 15
In order to live up to their full potential, low-level skills must be useful across a large variety of downstream tasks. In practice, however, a trade-off between generality (wide applicability) and specificity (benefits for specific tasks) arises. A large portion of prior work on option discovery and HRL resolved this trade-off, explicitly or implicitly, in favor of specificity. This can be exemplified by the choice of test environments, which traditionally revolve around navigation in grid-world mazes 46 JO.. In recent work on simulated robotics, navigation problems that are similar in spirit remain the benchmarks of choice [22 33 13, 27 22. In these settings, directed locomotion, i.c., translation of the robot's center of mass, is the main required motor skill, and the resulting algorithms require corresponding prior. This prior is made explicit with skills partitioning the or is implicitly realized by high contribution (Appendix G. Similarly, works that target non- navigation environments acquire skills with tailored pre-training tasks [SS] or in-domain motion capture data 36 31 32 In contrast, our work is concerned with learning low-level skills without extra supervision from traces or pre-training task design, and which do not prescribe a fixed tra towards a particular type of behavior.
state space according the agent's position B 43 of position features to reward signals 33
34
2

---

8, [s]
Pre- Training Environment
0r 8.4
Downstream Task
x
g
π
g
-6-1b....0-2) R(r-1i (0-1-1..-----1---- Rt-1) Sample pool spuce 6" with Sample goal E
R(s))
π
Figure 1: Illustration our proposed hierarchical skill learning framework. Left Low-level policies are learned in an empty pre-training environment, with the objective to reach random configurations (goal g) of a sampled skill (goal space G defined over a feature set F). Examples of goal space features are translation along the X-axis or the position of a limb. Right Learning downstream tasks with a three-level hierarchical policy to select a goal space, a goal and finally a native action a with the pre-trained low-level policy. The low-level policy acts on proprioceptive states s", while high-level policies x² and n. leverage extra task-specific information via s*
Hierarchical Skill Learning
3.1 Overview
In this work, we propose an approach where we first acquire low-level policies that are able to carry out a useful set of skills in an unsupervised manner, i.e., without reward signals from the main tasks of interest. We subsequently employ these skills in hierarchical reinforcement learning setting (Figure D The pre-training environment consists solely of the robot. In downstream tasks, additional objects may be present besides the robot; these may be fixed (e.g., obstacles) or only indirectly controllable (such as ball).
Crucially, the usefulness of skill policy stems from providing additional structure for effective exploration. In order to provide benefits across a wide range of tasks, skills need to support both fast, directed exploration (e.g., locomotion) well as precise movements (e.g.. lift the left foot while bending down). We propose to fulfill these requirements with short-horizon, goal goal-directed low-level policies that are trained achieve target configurations of robot-level state features such as the position or orientation of its torso or relative limb positions. We denote this feature space with S This allows the definition a hierarchy skills by controlling single features and their combinations, resulting in varying amounts of control exposed to high-level policies. Each skill is trained to reach goals in a goal space G defined over set of features F of St, yielding policies of the form
: SP x g + +A] FEF
with proprioceptive observations SD (Figure [ left). Goal-directed skill policies are trained without sask-specific rewards, relying solely on state features related to the specific robot and the prescribed hierarchy of goal spaces.
On downstream tasks, high-level policies operate with combinatorial action space the skill policy Io conditioned on both a feature set F and a concrete goal E GF_ State spaces are enriched with task-specific features St, s.t. S = SP S+, that contain information regarding additional objects in the downstream task. This enriched state space is only available to the high-level policy. We model the high-level policy in hierarchical fashion with policies
GF.
that prescribe a goal space and goal, respectively (Figure [ right). With task- and state-dependent policy πf, it is possible to not only select the required set of skills for a given environment, but also to switch between them within an episode. In this three-level hierarchy, the higher-level policy z¹ explores the hierarchy of goal spaces and dynamically trades off between generality and specificity. Temporal abstraction obtained by selecting new high-level actions at regular intervals but with reduced frequency compared to low-level actions. We term the resulting learning algorithm HSD-3, emphasizing the three different levels of control (goal spaces, goals and native actions) obtained after our hierarchical skill discovery phase.
3

---

3.2 Unsupervised Pre- Pre-Training
During pre-training, skill policies act in an MDP M with proprioceptive states s", native actions a, transition probabilities and initial states Sy E Si drawn from a distribution Po A5 for example], but without an extrinsic reward function. The skill policies are trained to reach goals goal space defined over 5 consisting of robot-level sensor readings, which may include pon-proprioceptive information.
Critically, each skill f aims to achieve goals defined over a variable-sized subset of features F of 89, To avoid learning combinatorial number of independent policies, we follow the principle of universal value function approximators 3 and share the parameters of each skill. We augment the policy's 's input with goal and goal space information, and learn single set of parameters 0, leading to i/ : SP x F x G 7 A. where F == {F F S S*} and g = {g" : F S S*). For ease of modeling, F is provided to Th as bag-of-wordk. input where a coordinate is set to I if its respective feature is included in F and to 0 otherwise. Also, the low-level policy receives goals relative to the current values of the s Thus, at each time step, the input goal to the policy is updated to reflect the progress towards the goal. As a learning signal, we provide distance- reward as R(s" a, s F. = || " (s9) - gll2 ||~ (s") glle. where w" S G" is a fixed transformation that selects the subset F of goal space features and applies a suitable normalization. The selection of features for S and transformations w represents the prior knowledge that we utilize for unsupervised pre-training. We discuss examples for bipedal robots in our experiments and in Appendix B
During training, features F and goals g are sampled anew for each episode in the environment. Episodes consist of several dozen steps only, reflecting the fact that we are interested in short-horizon skills. In-between episodes, we reset the simulation to states drawn from S only sporadically to encourage the resulting policies to be applicable in wide range of states, facilitating their sequencing under variable temporal abstractions. Likewise, rewards are propagated across episodes as long as the simulation is not reset. The low-level policy parameters 0 can be optimized with any reinforcement learning algorithm; here, we opt for Soft Actor- (SAC), ofte-offthheart method tailored to continuous action spaces [8.. Pseudo-code for the pre-training algorithm and further implementation details are provided in Appendix C
3.3 Hierarchical Control
After low-level policy x' has been obtained via unsupervised pre-training, we employ it to construct a full hierarchical policy <[ass) = π (F]s) (1ss F).). The hierarchical policy is applied to downstream tasks, where we are supplied with reward signal and state observations S = SP S+ While different frequencies may be considered for acting with n' and T5, in this work both high-level actions are selected in lock-step. For jointly learning xf and x8, we extend the SAC algorithm to incorporate the factored action space F x G. In particular, it is necessary to extend SAC to deal with a joint discrete and continuous action space, which we describe below (see Appendix D for further details and pseudo-code for high-level policy training).
We formulate our extension with shared critic Q(s, F, incorporating both factors of the action space. (s,, F.g) is trained to minimize the soft Bellman residual [[] Eq. 5]. with the soft value function Vss) including entropy terms for both policies. Since n' outputs distribution over discrete actions (to select from available goal spaces), we compute its entropy in closed form instead of using the log- log-likelihood of the sampled action as for T8 We employ separate temperatures a and 3F for the goal space policy as well as for each corresponding goal policy; 3 are normalized to account for varying action dimensions across goal spaces.
B
V(+)= = Σ (F)s)
Q(s.F.)) gITTH
log [
log 7 (1ls, + (H((T((1ss) - log (Fl).
FEF
Compared to the usual SAC objective for continuous actions, we subtract log [F] from the entropy to ensure a negative contribution to the reward. This matches the sign of the entropy penalty from n², which is negative in standard hyper-parameter setting!" [18] Otherwise, the entropy terms for discrete and continuous actions may cancel each other in -funccio targets computed from V(s).
'The entropy for continuous actions sampled from Gaussian distribution can be negative since it is based on probability densities.

---

The policy loss [8 Eq. 7] is likewise modified to include both entropy terms. Parameterizing the policies with & and &b and the Q-function with ps we obtain
B log क ( (
Σ T
(F(s)
E groet
H(
smw
where the replay buffer is denoted with B. The scalar-valued entropy temperatures are updated automatically during learning to maintain target entropies Ht' and 7²" for both of and n². The following losses are minimized at each training step:
a (H(T((1s) - -7')
J(a) = s~
J(3)
E B
Σ FEF
B (Fls) E
log r ( F) H.
FF
We implement temporal abstraction by selecting high-level actions with reduced frequency. This incurs a reduction in available training data since, when laking a high-level action every c steps in an N-step episode, we obtain only N/c high-level transitions. To leverage all available transitions gathered during training, we adopt the step-conditioned critic proposed by Whitney et al. (52) in our SAC formulation. The Q-function receives an additional input 0 < S marking the number of steps from the last high-level action, and trained to minimize a modified soft Bellman residual:
27
JQ(p)=
E Feighid Streorc fi Fomnt B
2 Qo(sn, Ft.
Σ
(Pr(s))),aH))) + + ((((t0--)))
As in 12.. F,, is used when computing V(s) and the policy loss J= o (0) vp).
The factorized high-level action space of HSD-3 has previously been studied as parameterized action spaces [28] [5], albeit with a small number of discrete actions and not the context of hierarchical RL. A possible application of Soft Actor-Critic is described in Delalleau et al. 19; our approach differs in that we (a) compute the soft value function as a weighted sum over all discrete actions, and (b) opt for two separately parameterized actors and shared critic. These design choices proved to be more effective in initial experiments.
Benchmark Environments
We propose a benchmark suite for comprehensive evaluation of pre-trained motor skills, tailored to bipedal robots (Figure 2 and implemented for the MuJoCo physics simulator 44 48]. We selected tasks that require diverse abilities such as jumping (Hurdles), torso control (Limbo), fine-grained foot control (Stairs, GoalWall) and body balance (PoleBalance). Besides PoleBalance, all tasks also require locomotion. The tasks are designed have a sparse reward that provided once an obstacle or step has been passed or a goal has been shot, emphasizing the need for good exploration. As an exception, PoleBalance provides constant reward during the entire episode, which ends when the pole placed on the robot falls over. PoleBalance requires precise motor control, and is, in combination with its reward structure, chiefly outside of the realm benchmarks that have been traditionally used to evaluate HRI. techniques. Obstacle position and step lengths are subject to random perturbations, sampled anew for each episode. We tailor several environment parameters to the concrete robot being controlled, c.g., the positions of Limbo bars. For the effectively two-dimensional Walker robot, limit the movement of objects (ball, polc) to translation along X and Z and rotation around the Y-axis. In all environments, we label simulation states that would cause the robot to fall over invalid; these will terminate the episode with a reward of -1. Appendix A contains full specifications for all tasks.
Variants of some of our benchmark tasks have been proposed in prior works: Hurdles 20 24 13 68
Stairs [6] Gaps [20 48, and PoleBalance is inspired by the classical CartPole problem 44 Our tasks are different from earlier published counterparts in parameterization and in the sparsity of rewards they provide. All environments are provided via a standard Gym interface with accompanying open-source code, enabling easy use and re-use.
5

---

(a) Hurdles
(b) Limbo
(c) Stairs
(d) Gaps
(c) GoalWall
(f) PoleBalance
Benchmark environments for evaluating motor skills for bipedal robots, pictured with the Walker robot. Hurdle and Limbo bar heights and spacing, as well as stair lengths, are sampled ran- domly from fixed distributions. We also include a combination of Hurdles and Limbo (HurdesLimbo), in which both obstacle types alternate.
Figure 2:
Experimental Results
For our experiments, we first pre-train skill policies as described in Section 3.2 in an empty environ- ment. The feature X-axis, its position on the Z-axis, torso rotation around the Y-axis and positions of each foot, relative 31 skills with a shared
describe target configurations consists of the robot's translation along the
the body (Appendix B). With these five features, we obtain a set of 2 set of policy parameters. We then train separate high-level policies for each benchmark environment, directing the same set of pre-trained skills. High-level actions are taken every environment steps; in the PoleBalance environment, which requires short reaction times, all three policies operate at the same frequency. All neural network policies employ skip connections as proposed for SAC by Sinha et al. (4). We show results over 9 seeds for each run; in experiments involving pre-training, we combine 3 seeds for pre-training with 3 seeds for high-level policy training. Pre-training takes approximately days on 2 GPUs (V 1000) and fine-tuning (downstream task training) takes 2 days to reach 5M samples on GPU. Further details regarding the training setup,
for skill and high-level policy training, well as for baselines, can be found in Appendix B
5.1 Trade-off's for Low-Level Skills
In first experiment with the Walker robot, we aim investigate the trade-off between specificity and generality of low-level skills. For each environment, we train high-level policies directing a fixed skill within its respective goal space, which for example consists of torso-related features only, of feet positions combined with forward translation. At the end of training, we measure average returns obtained for all 31 skills in deterministic evaluation setting, and plot quartiles over different seeds for the best skills in Figure 3 The performance for different fixed skills varies significantly across environments, and no single skill is able to obtain the best result (shaded) across all environments. While controlling X, Y and Z. features only produces the good hierarchical policies for most tasks, poor results are obtained in Gaps and GoalWall. Controlling feet positions (LF and RF) crucial for GoalWall, although close-to-optimal performance is only achieved in few runs across the respective
Linbo
HurdicaLimbo
Gaps
GollWall
X.Y
X.Y
X.RF
X.FF
X.2
ZLF
Goal Spaca
X.
X.Y.FF
X.
X.Y.F
X.F.RI
X.L..F
X.Y.2
X.Y.2
XZLLF
X.Z.F
X.Y.Z.F
X.Y.ZRRE
X.Y.A.FFF
X.Y.Z.F..F
15
10
20
10
20
50B
1000
Roturn
Ratum
Roturn
Return
Return
Retum
Return
Figure 3: Returns achieved after 5M samples on the benchmark tasks with the Walker robot with fixed low-level policy goal spaces (quartiles). Each row corresponds to a set of features for the respective goal space. Best skills (marked) differ significantly across tasks.

---

Hurdles
Limbo
HurdiesLimbo Limo
Stairs
15
10
Retum
10
0
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5 les
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
lc6
lcb
Samples
le6
GoalWall
Gaps
PoleBalance
HSD-3
15
750
HSD-Bandi
SD
Return
0.5
10
500
(hindsight)
SAC
250
Switching Ensemble
HIRO-SAC
0.0
HIDIO
0.0
2.5
5.0
7.5 166
0.0
2.5
5.0
7.5 166
0.0
2.5
5.0
7.5
DIAYN-C
Samples
Samples
Samples
165
Figure 4: Learning curves benchmark environments with the Walker robot. For clarity, plot mean performance over 9 seeds, averaged over 0.5M samples. Full learning curves including error bands are provided in Appendix F.1
skills. The most general skill, high-level control of all considered features (bottom row), also allows for learning progress on all tasks; however, is generally outperformed by other, more specific skills.
5.2 Skill Selection with HSD-3
We compare HSD-3 against number of base- and toplines to examine the effectiveness of our proposed framework for skill learning and hierarchical control. We include state-of-the-art non- hierarchical RL methods (SAC [18),, end-to-end hierarchical RI algorithms (HIRO 33 HIDIO (33)
and evaluate DIAYN as an alternative unsupervised pre-training algorithm [33 For HIRO, we use SAC for learning both high- and low-level policies (HIRO-SAC). We facilitate the comparison to DIAYN with a continuous-vale valued skill variable obtained from an embedding MM (DIAYN-C). For both HIRO and DIAYN-C, we provide the same goal space as in our pre-training stage, consisting of the full feature set, to ensure a fair comparison. We further compare to the Switching Ensemble proposed in [35 which does not learn high-level policy but has been shown improve exploration. Finally, we examine how our traitreined skills perform if we use single goal space only (SD). Without prior knowledge about the task at hand, this corresponds
the full goal space F {0, 1, .... dim(S*) settling for maximum generality. As topline, we select the best per-task goal spaces from Section 5.1 denoted with SD* which required exhaustive training and evaluation with all available goal spaces.
In Figure 4 we plot learning curves over 5M training interactions for HSD-3 as well as base- and toplines, and list final performances in Table D On all tasks, HSD-3 outperforms or matches (Limbo) the learning speed of a conventional hierarchical policy with a skill defined over the full goal space (SD). On GoalWall and Stairs, the three-level policy significantly exceeds the mean performance of single goal space base- and toplines. For Gaps, the variances of the best results in Table I (HSD-3,
Method
Hurdles
Limbo HurdlesL
Stairs
Gaps
GoalWall
PoleBalance
SAC
2.2.16.2
-0.1
-0.0=0.4
5.0
0.1
-0.2:1..3
804.
Switching Ensemble
0.7+3.0
1.944.3
1.6=3.6
7.4238
-0.2+0.33
-0.2+0.3
569.4+-30..1
HIRO-SAC
0.411..6
0,912,2
-0.0=0.1
-0.0=0.0
-0.110.2
-0.011.00
23.8
HIDIO
-0.1
.0.1
-0.2=0.1
-0.2=0.3
-0.2+:0.3
-0,3::0.3
117.6:133.8
DIAYN-C
9.5.1:2.5
7.511..
5.7
12,5=2.9
3.41:4.1
0.1:10.3
839.9158.4 4
SD
15.0++1.4
11.0+100
10.8=1.4
15.2=0.9
14.5+99.3
0.4+0.5
789.5=79.1
HSD-Bandit
1.3
1.4+177
2.0=1.5
0.5-0.8
-0.1+0.2
-0.1+0.2
61.0+
HSD-3
15.342.0
11.7409
11.8-1.3
17.2=0.7
15.148.9
0.940.1
876.0=36.99
SD*
15.0+1.0
12.2+110
12.7-1.3
16.2-1.3
14.0=11.3
0,5+0.5
868.2-191.1
Table 1: Final performance after 7.5M samples across benchmark tasks with the Walker robot. Mean and standard deviation reported for average returns over 9 seeds.
7

---

Foot
Rihht Foot
300 350
Envtroment Stop
Figure 5: Skills selected by HSD-3 during an episode on the Stairs task. Different skills are utilized for different sections of the environment: walking upstairs is achieved by controlling the torso position (X,Z) in a regular pattern, with goals for the rotation (Y) and the right foot being set occasionally. On the top section, different pattern is selected for quickly running forwards. When going downstairs, the right foot the robot is controlled explicitly to maintain balance.
SD, SD*) are high, hinting at especially challenging exploration problems. A closer look at per-seed performance (Appendix FT reveals learning failure for 2 out of 9 runs for these methods. For GoalWall, 4 and 5 runs for SD* and SD, respectively, achieve zero return. Per-task goal space selection with bandit algorithm (HSD-Bandit) fails most environments, and achieves low returns on Hurdles, Limbo and HlrdleesLimbo.
These results demonstrate that, without additional task-specific knowledge, our hierarchical skill framework able to automatically and efficiently trade off between skills of varying complexity at training time. In Figure 5 visualize the goal spaces that are selected across the course of an episode in the Stairs environment. During different stages of the environment (walking upstairs, downstairs, and on a planar surface), HSD-3 selects appropriate skills of varying complexity P
The performance of the remaining baselines underscores the challenges posed by our benchmark tasks. Standard SAC works well in PoleBalance, which has a dense reward structure compared to the remaining environments. On most other tasks, learning progress slow or non-existent, e.g., on Hurdles, only 2 out of 9 runs achieve positive return at the end of training. DIAYN-C exhibits subpar performance across compared our hierarchical pre-training scheme, with the exception of PoleBalance where learning is fast and returns are higher than for SD. The end-to-end HRL methods HIRO-SAC and HIDIO are unable to make meaningful progress on any task, which highlights the utility of skill pre-training in the absence of task-specific priors. The Switching Ensemble is the best baseline without pre-training (except for the PoleBalance task), but clear majority of individual runs do not obtain positive returns (Appendix F.I) On the Stairs task, both SAC and the Switching Ensemble manage to climb the flight of stairs in several runs, but fail to discover the second, downwards flight at the end of the platform.
We refer to the supplementary material for further ablation and analysis with the Walker robot. In Appendix F22 we demonstrate the efficacy of our proposed hierarchical pre-training method, compared to pre- vre-training a single skill policy on the full goal space exclusively. In Appendix F3 analyze the exploration the behavior of various methods from Table I We find that, in general, hierarchical methods visit more states compared to SAC. HIRO-SAC and DIAYN-C visit a higher number of states in the Hurdles, Limbo, HurdlesLimbo and Stairs environments but fail to leverage this experience for achieving higher returns.
5.3 Evaluation on a Humanoid Robot
To evaluate the scalability of HSD-3 to more complex robots, perform additional set of experiments with the challenging 21-joint Humanoid robot from Tassa et al. 48 For skill pre- training, we select goal space approximately matching the Walker setup, consisting of the translation along the X-axis of simulation, Z position of the torso, its rotation around the Y axis and three- dimensional feet positions relative the robot's hip location.
In Figure 0 we plot average learning progress across 100M environment steps, comparing HSD-3 to SD (the full goal space), SD* (the best goal space), and conventional, non-hierane non-hierarchical SAC setup. The results align with the observations for the Walker robot Section 5.2 in that HSD-3
Videos are available at alttpss//faccbbooeeseaddd wht.eeerwwwrrrr.r....bb.....ae.. .ith.b..ttb....... io/hsd3

---

Hurdles
Limbo
HurdlesLimbo
Stairs
PoleBalance
10
Return
2
5
500
0
0
0
1
Samples
1e8
Samples
1e8
Samples
Samples
Samples
1e8
HSD
SD
SD* (hindsight)
SAC
Figure 6: Results on four of the benchmark tasks with 21-joint Humanoid robot. We show mean performance over seeds, clipped to 0 and averaged over 2.5M samples. Appendix F.5 contains learning curves for all seeds and error bands.
matches surpasses the performance of the full goal space, and further outperform the best goal space selected in hindsight. Both HSD-3 and SD make good learning progress on three out of five tasks used for this evaluation (Hurdles, Stairs, PoleBalance); in fact the best single goal space. On HurdlesLimbo, which requires both jumping and crouching, HSD-3 outperforms both SD and SD* On the Limbo task, HSD-3 is able to learn faster than SD and achieves a higher return by utilizing the most effective single skills on this task, which control X translation and leaning forward or backward. This behavior was not discovered by the SD baseline, which exercises control over all goal space features. In this preliminary study, none of the hierarchical policies was able make significant progress on the WallWall and Gaps tasks, which were already shown to be difficult with the Walker robot in Section 5.2
Hurdles and Stairs, SD is
6 Conclusion
Our work is among the first to highlight the specific trade-offs when embedding prior knowledge in pre-trained skills, and to demonstrate their consequences experimentally in simulated robotic control settings. We describe hierarchical skill learning framework that, compared to existing approaches, allows high-level policies to perform the trade-off between directed exploration and fine-grained control automatically during training on downstream tasks. Our experiments with a bipedal Walker robot demonstrate HSD-3's 3's efficacy struggle. We further apply our framework to a challenging Humanoid robot, where learns effective policies the majority of tasks.
variety of sparse-reward tasks in which previous approaches
With this work, we provide a new set benchmark tasks that require diverse motor skills and pose interesting challenges for exploration. We release the tasks in a dedicated distribution, with the aim of spurring further research on motor skills continuous control settings, and to ultimately broaden the utility of hicrarchical approaches to control to tasks beyond navigation.
Limitations: Our investigations with the Humanoid robot have been performed in limited goal spaces and hence with significant priors. We believe that further work is required to successfully acquire large, exhaustive set of low-level skills in unsupervised environments that work on more and more complex morphologics.
thank Alessandro Lazaric for insightful discussions, and Franziska Meier, Ludovic Denoyer, and Kevin Lu for helpful feedback early versions of this paper.
Acknowledgements:

---

References
[T] Achiam, H. Edwards, D. Amodei, and Abbeel. Variational Option Discovery Algorithms. arXiv:1807.10299 [cs]. Jul 2018.
[2] A. Ajay. A. Kumar, P. Agrawal, S. Levine, and O. Nachum. OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning. In International Conference on Learning Representations, 2021.
[3] P.-L. Bacon, J. Harb, and D. Precup. The Option-Critic Architecture. In Thirty-First AAAI Conference on Artificial Intelligence, pages 1726-1734, 2017.
[4] A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cyberneties, 8MC-13665:334-866 1983.
[5] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAl Gym. arXiv:1606.01400 [cs]. Jun 2016.
[6] V. Campos, A. Trott, C. Xiong, Socher, X. Giro-i-Nieto, and Torres. Explore, Discover and Learn: Unsupervised Discovery of State- catee-Co Skills. In International Conference on Machine Learning, pages 1317-1327, 2020.
[7] M. S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing, STOC '02, pages 380-388, Montreal, Quebec, Canada, 2002. Association for Computing Machinery.
[8] Dayan and G. E. Hinton. Feudal Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 5, pages 27 1-278. Morgan-Kaufman 1992.
[9] O. Delalleau, M. Peter, E. Alonso, and A. Logut. Discrete and Continuous Action Representation for Practical RL in Video Games. In AAAI-20 Workshop Reinforcement Learning in Games, 2019.
[10] G. Dietterich. Hierarchical Reinforcement Learning with the MAXQ Value Function Decom- position. Journal of Artificiial Intelligence Research, 13:227-303, 2000.
[11] P. Dobrowolski. Swing-twis decomposition in Clifford algebra. 15X.v:1506.05443 [cs], Jun 2015.
[12] Y. Duan, X. Chen, R. Houthooft, Schulman, and Abbeel. Benchmarking Deep Reinforce- ment Learning for Continuous Control. In International Conference on Machine Learning, pages 1329-1338, 2016.
[13] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is All You Need: Learning Skills without a Reward Function. In International Conference on Learning Representations, 2019.
[14] R. E. Fikes, E. Hart, and N. J. Nilsson. Learning and executing generalized robot plans. Artificial Intelligence, 3:251-288 1972.
[15] C. Florensa, Y. Duan, and P. Abbeel. Stochastic Neural Networks for Hierarchical Reinforcement Learning. In International Conference on Learning Representations, 2017.
[16] D. Ghosh, A. Singh, A. Rajeswaran, V. Kumar, and S. Levine. Divide-and -onnuuee Reinforce- ment Learning. In International Conference on Learning Representations, 2018
[17] K. Gregor, D. J. Rezende, and D. Wierstra. Variational Intrinsic Control. In International Conference Learning Representations (Workshop). 2017.
[18] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and S. Levine. Soft Actor-Critic Algorithms and Applications. arXiv:1812.0590522 [cs), Dec 2018.
[19] K. Hausman, J. T. Springenberg. Z. Wang, N. Heess, and M. Riedmiller. Learning an Embedding Space for Transferable Robot Skills. In International Conference on Learning Representations, 2018.
10

---

[20] N. Heess, D. TB. S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, M. A. Eslami, M. Riedmiller, and D. Silver. Emergence of Locomotion Behaviours in Rich Environments. arXiv: /707.02286 [cs], Jul 2017.
[21] N. K. Jong. The utility of temporal abstraction in reinforcement learning. In 7th International Joint Conference on Autonomous Agents and Multiagent Systems, pages 299-306, 2008
[22] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
[23] G. Konidaris. Building Portable Options: Skill Transfer in Reinforcement Learning. In IJCAI, volume 7, pages 895-900, 2007.
[24] A. Li, C. Florensa, I. Clavera, and P. Abbeel. Sub-policy Adaptation for Hierarchical Reinforce- ment Learning. International Conference on Learning Representations, 2019.
[25] T. Li, R. Calandra, D. Pathak, Y. Tian, F. Meier, and A. Rai. Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion. 1XXi::000..11677 [undefined], (2):2682-2688)
Mar 2021.
[26] M. C. Machado, M. G. Bellemare, and M. Bowling. A Laplacian Framework for Option Discovery in Reinforcement Learning. In International Conference on Machine Learning, 2017.
[27] K. Marino, A. Gupta, R. Fergus, and A. Szlam. Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies. In International Conference on Learning Representations, 2019.
[28] W. Masson, P. Ranchod, and G. Konidaris. Reinforcement Learning with Parameterized Actions. Proceedings of the AAAI Conference on Artificial Intelligence, 30,1), 2016.
[29] Merel, Y. Tassa, D. TB, S. Srinivasan, J. Lemmon, 7. Wang, G. Wayne, and N. Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv: (000:0222000 [cs), Jul 2017.
[30] Merel, A. Ahuja, V. Pham, Tunyasuvumakool, S. Liu, D. Tirumala, N. Heess, and G. Wayne. Hicrarchical Visuomotor Control of Humanoids. In International Conference on Learning Representations, 2019.
[31] J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, W. Teh, and N. Heess. Neural Probabilistic Motor Primitives for Humanoid Control. In International Conference on Learning Representations, 2019.
[32] J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess. Catch & Carry: Reusable Neural Controllers for Guiied Whole-Body Tasks. ACM Transactions on Graphics, 39(4), 2020.
[33] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-Efficient Hierarchical Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 31, pages 3313. Curran Associates, Inc., 2018.
[34] O. Nachum, S. Gu, H. Lee, and S. Levine. Near-Optimal Representation Learning for Hier- archical Reinforcement Learning. In International Conference on Learning Representations, 2019.
[35] O. Nachum, H. Tang, X. Lu, S. Gu, H. Lee, and S. Levine. Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? In NeurlPS 2019 Deep Reinforcement Learning Workshop, 2019.
[36] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. MCP: Learning Composable Hicrarchical Control with Multiplicative Compositional Policies. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[37] K. Pertsch, Lee, and J. J. Lim. Accelerating Reinforcement Learning with Learned Skill Priors. In 4th Conference Robot Learning (CoRL), 2020.

---

[38] A. H. Qureshi, J. J. Johnson, Y. Qin, T. Henderson, B. Boots, and M. C. Yip. Composing Task- Agnosticc Policies with Deep Reinforcement Learning. In International Conference on Learning Representations, 2020.
[39] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by Playing Solving Sparse Reward Tasks from Scratch. In International Conference Machine Learning, pages 4344 4353. PMLR, 2018.
[40] E. D. Sacerdoti. Planning in hierarchy of abstraction spaces. Artificial Intelligence, 5(2): 115-135, 1974.
[41] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal Value Function Approximators. In International Conference on Machine Learning, pages 1312-1320, 2015.
[42] Shankar and Gupta. Learning Robot Skills with Temporal Variational Inference. In International Conference on Machine Learning.. 2020.
[43] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-Awar Unsupervised Discovery of Skills. In International Conference on Learning Representations, 2020.
[44] S. Sinha, H. Bharadhwaj. A. Srinivas, and A. Garg. D2RL: Deep Dense Architectures in Reinforcement Learning. In NeurlPS 2020 Deep RL Workshop, 2020.
[45] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-039224-.
[46] R. Sutton, D. Precup, and Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 11(11-2)181-211, 1999.
[47] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. #Exploration: A Study of Bount-Basee Exploration for Deep Reinforcement Learning. In arXiv:1611.0471 [Cs], Advances in Neural Information Processing Systems 30. 2017.
[48] Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, P. Trochim, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, and N. Heess. bon_controll Software and Tasks for Continuous Control. arXiv:2006. 12983 [undefined], 6: 100022, Nov 2020.
[49] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control, In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages
2012.
[50] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In International Conference Machine Learning, pages 3540-3549, 2017.
[51] E. Wei, D. Wicke, and S. Luke. Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space. In 20/8 AAA/ Spring Symposia. AAAI Press, Stanford University, Palo Alto, California, USA, 2018.
[52] W. Whitney, R. Agarwal, K. Cho, and A. Gupta. Dynamics-aware Embeddings. In International Conference on Learning Representations, 2020.
[53] Zhang, H. Yu, and Xu. Hierarchical Reinforcement Learning by Discovering Intrinsic Options. In International Conference on Learning Representations, 2020.
12


Reviews:
[{"review_id": "tSEJJQ_Rrh", "reviewer": "Reviewer_gM3y", "summary": "This paper argues that, in the context of pre-trained low level skills, there is a trade-off between generality and learning speed. It makes two contributions: firstly, it proposes a benchmarking suite of sparse reward tasks that require different motor skills to study this trade off. Secondly, the paper proposes a hierarchical skill learning algorithm that attempts to trade off generality and learning speed directly. The proposed algorithm outperforms reasonable baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper argues that, in the context of pre-trained low level skills, there is a trade-off between generality and learning speed. It makes two contributions: firstly, it proposes a benchmarking suite of sparse reward tasks that require different motor skills to study this trade off. Secondly, the paper proposes a hierarchical skill learning algorithm that attempts to trade off generality and learning speed directly. The proposed algorithm outperforms reasonable baselines.\n\nThe proposed benchmark environments are a number of obstacle courses (e.g. stairs, gaps, hurdles) as well as a pole balancing task. The tasks are set up for the 2d walker as well as a 3d humanoid.\n\nThe proposed algorithm pre-trains skills first defining a set of features. Then the agent is trained to control subsets of the full feature set. The subsets and target features are randomly sampled during pre-training. Selecting the feature set induces and the target features induce an inductive bias. I would have like to see more discussion of how the features and target feature distributions are selected. As an aside, I think for more complex embodiments, I think using demonstrations or mocap data to define the targets would be an interesting approach. To reuse the learned skills, the skills are frozen and a hierarchical policy is learned that first selects the subset of features to control and then specifies a target. \n\nI think the related works section is reasonably comprehensive but should cite\n\n    Heess et al.  “Learning and Transfer of Modulated Locomotor Controllers.” arXiv Preprint arXiv:1610. 05182.\n\nas an early example of low level controllers in deep RL. Another relevant paper is\n\n    Hasenclever et al. “CoMic: Complementary Task Learning & Mimicry for Reusable Skills.” ICML 2020. http://proceedings.mlr.press/v119/hasenclever20a.html. \n\nthat studies (among other things) the same trade-off between generality and learning speed in the context of skills acquired from mocap data.\n\nIn a first experiment, the performance with different fixed goal spaces is compared. The results indicate that no single goal space works best on all tasks: there is indeed a trade-off between speed and generality. However it's worth noting that the full goal space works pretty well across tasks. In the other experiments, the proposed methods performs favourably relative to baselines. \n\nOverall, I think this is a nice paper and I am leaning towards acceptance.\n\nMinor comments:\n- page 4, section 3.1 last paragraph: I'm a little confused by this. If you don't reset why call the experience segments episode? Isn't it just one long episode at that point?\n- Figure 1: I found the $s^+$ notation confusing. Why not $s^t$\n- I would quite like to see some videos of the resulting skills and policies. This is something that other papers in this space provide\n- Table 1: How many seeds and episodes do your experiments correspond to?\n- Figure 6: typo \"are now show\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The experimental results with the humanoid are limited in terms of the goal spaces and the full goal space baseline appears to work just as well if not better. This suggests that substantially more work is needed to scale up to more complex embodiments. This is acknowledged by the authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "tIkES810p5r", "reviewer": "Reviewer_tdmg", "summary": "This paper presents a hierarchical framework for training locomotion policies for tasks with sparse reward. Low level policies are trained to achieve a selection of subgoals and high level policies are trained by exploring in the goal spaces. Challenging locomotion tasks are introduced to show the effectiveness of the proposed methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper presents a three level hierarchical policy with a low level policy generated via training on general tasks. This is new to me.\n\nQuality: The paper is technically sound to me.\n\nClarity: Paper is well written.\n\nSignificance: The result is not very impressive since most tasks can be solved via a simple forward progress reward. But it presents an important step towards solving tasks with sparse reward using a hierarchical policy.\n\nPro:\n\n1. The training of low level policies based on goal spaces specified by position of torso and foot.\n\n2. A three level hierarchical policy and an extension of SAC to accommodate the resulting action distribution.\n\n3. Challenging sparse reward tasks for locomotion.\n\nCon:\n\n1.  The computation required for training even the low level skills are significant. It will be nice to provide more details on the low level training. The low level skills do not seem challenging to me (at least for the walker) so it seems 3 days on 2 GPUs is a lot.\n\n2. Tasks such that designing a simple dense reward (such as the forward progress reward) is none trivial will really demonstrate the strength of hierarchical framework. e.g., the GoalWall task presented. More in Question 2 below.\n\nQuestion:\n\n1. It is not clear to me why Gap will be more challenging than other tasks. Maybe the reward design needs to be modified? e.g., a negative reward for touching the gap can cause the robot want to terminate as soon as possible. Some failure mode of the gap task will help illustrate the issues.\n\n2. Most of the tasks can also be solved via a simple forward progress reward (with the exception of GoalWall, which is difficult to solve in the current framework anyway). It will be interesting to see what is the performance gain the proposed approach has over baseline methods under this setup.\n\n3. In Figure 6, SAC makes no progress on Stairs, but that is not the case in Table 5 in the Appendix. The video also shows SAC makes some progress on Stairs.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, limitation is discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "lyHk1Dz1QOZ", "reviewer": "Reviewer_ECxq", "summary": "This work proposes a benchmark task with bipedal robots instead of locomotion tasks, where the bipedal robots are low dimensional and perform various movements. It also proposes a hierarchical reinforcement learning method with three levels: a policy that specifies a goal space (a set of features to operate on), a policy to specify the goal configuration given that goal space, and a low level policy to reach the desired goal configuration. It learns the low level policies through unsupervised learning, and then optimizes off policy the high level options by optimizing the value function.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Abstract: It is not clear throughout the abstract what the \"inductive bias\" is. It is not clear how the \"the potential to facilitate exploration\" is an inductive bias. since this is the primary distinction on which this work centers, this makes the abstract confusing.\n\nIntroduction:\n24 While it is true that much of the work in exploration with hierarchical reinforcement learning deals in navigation or navigation-like domains where the center of mass of the agent is particularly useful, there also exists a large body of work which deals with exploration related to object manipulation where this assumption is not the case. Thus, the generalization expressed here is misleading, since it highlights that \"the work that benefits from center of mass knowledge has an implicit bias towards center of mass knowledge.\"\n\n30 While this is true of policies, it may not be the case that low level skills are more widely applicable, since individual skills, low level or high level might be limited to a a very small set of behavior. It would probably be more appropriate to refer to this in the context of policies.\n\nRelated work:\n77 While re-usability across different high level actions is a significant benefit, this underscores the idea that one of the large benefits of Hierarchical reinforcement learning is for exploration. Even if a primitive is only used for one task, it might be that the learning procedure is able to exhibit gains in performance or sample efficiency even without generalizability. While it is likely that navigation tasks as a test domain does introduce implicit biases, it is not clear from the description in this section why this is the case.\n\nHierarchical skill learning: \nThe \"introduction\" section for this component lacks a heading, and it is not clear exactly what it is expressing. Is this part a description of the low level policies? A description of the state space or of general terms? At this point it remains unclear what exactly the high level policy comprises, and why there are task specific features and additional objects separated out to be accessed only by the high level policy. While these components gain some clarity later, the ordering makes this confusing.\n\n107 It is not clear up to this point if the argument is being made for more specific, or less specific policies. However, while such a low-level policy as reaching a goal configuration after short time is not navigation related, it is certainly very specific to robotic motion.\n\n3.1 While it is reasonable to train low level skills over different sets of features, perhaps the most important property related to these would be how they are chosen. It is not clear if these features are selected as random samples or specified. However, random sampling seems like a suspect way, as if the number of features exceeds even a small amount the number of possible sets explodes exponentially.\n\nTraining the low level policies only by unsupervised pre-training also seems like it could introduce issues. In particular, while the low level space might afford many bad ways of controlling the agent, there are probably only a limited number of useful ways to control the agent. There should be a tradeoff between using the higher level policies to specify what is learned at the lower levels, and simply exploring with the lower level policies, but at present the former appears to be completely ignored.\n\n3.2 Equation numbers would be much appreciated in this section, especially since there are clear changes being made to the typical bellman equation/value function, not the least of which is in the notation of taking the feature mean. \n166 While matching sign seems like it should have an effect on optimization, by negating the log of the features, this seems to change the meaning of the equation. \n\n170 It is not entirely clear how the equation in line 164 arrives at the one in 170. In particular, it seems that the loss is the negative expected value, but then the log|F| component has disappeared.\n\n173 A more in-depth description of how the \\alpha and \\beta loss terms are defined is necessary. At this point, it is simply provided as a given without clear explanation, especially since the intuitive meaning of H^f, H^g is not made clear.\n\nBenchmark Environments:\nWhile these environments are provided as one of the clear contributions of this work, they are described fleetingly. It would be useful to note why properties like center of mass or other normal navigation \"implicit biases\" do not apply to these cases. \n\nExperimental results:\n211 With only 5 features, these proposed domains do not actually differ too significantly from other mujoco locomotion tasks that are more commonly given. As highlighted before, the number of features also seems necessary for this method to work since the subset sampling would explode exponentially otherwise.\n\n227 It is still unclear what emphasis is being made about \"no single skill\" In particular, it should be expected in any skill learning framework that one skill does not dominate, otherwise there would be no point in using the framework at all.\n\nFigure 4: it is hard to parse the results, in particular those of SD*. Is this meant to outperform the proposed method?\n\n5.2 The proposed baselines, while interesting, do not always capture a fair comparison. In particular they are all single level skill learning methods except for DIAYN, but in this case DIAYN-C does not appear to be ideal for this case.\n\nFigure 6: It is not entirely clear why SAC would completely fail for the given tasks. It would be useful to see a comparison against SAC where it is able to learn at least some useful behavior, or find another baseline that does give valid results. There should exist algorithms which function on humanoid walker.\n\nOverall, this work proposes an interesting way of selecting features to control over and an effective 3 level hierarchy which has encouraging results. While the idea of selecting features has been proposed, it has not been shown in a multi-level hierarchy. However, the writing is sufficiently difficult to parse such that it is difficult to determine how exactly this method is novel from existing work, except that it encodes more specific information for the tasks. It is also difficult to determine from the writing exactly what features of the proposed method contribute to the success. Furthermore, the experiments are limited because they test on a new domain against baselines that do not seem like fair comparison on that domain. \n\nOriginality: marginal\n\nQuality: marginal\n\nClarity: poor\n\nSignificance: below marginal\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work has a limited discussion of limitations, particularly how closely the method is tied to the task being performed. It has no discussion of societal impact, though robotics has a large variety of effects on modern society so a discussion could have occurred.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_WUliaG3gV", "reviewer": "Reviewer_opgb", "summary": "The authors propose a 3-level hierarchical method, one that operates on a feature space, another that operates on a feature-conditioned goal space, and finally a low level policy that outputs actions in the environment conditioned on everything above and the current state. They pretrain the policies in a pretraining environment before applying it to the task at hand. The authors also introduce a new suite of environments to test hierarchical RL. The issues brought up by the authors about current HRL methods are enlightening and the method is a novel contribution. Results are fine, with subtle/no improvements over the baselines on some tasks, and substantive improvements on others. I believe the paper has some issues with baseline comparisons and tasks, which I hope will be addressed in the rebuttal. As such, I am currently learning towards not accepting the paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Paper Strengths\n**Paper Framing/Originality**\nPointing out the issues with existing HRL algorithms’ focus on learning skills that are relevant mainly for controlling center of mass is an interesting contribution. Furthermore, the algorithm that results is novel, while also incorporating ideas from modern papers such as the DYNE step-conditioned critic. The inclusion (and supposed release) of the benchmark environments are a benefit to the community as some of them seem like solid tasks to test hierarchical algorithms on.\n\n**Results Significance**\nI think the paper does a good job of demonstrating the importance of goal-space separation with results that demonstrate significant improvements over baselines on some of the tasks. Furthermore, it’s rare to see evaluations over 3 seeds, while the authors evaluate on 9 seeds in total. The inductive bias experiment on HIRO in the appendix (sec G) was very enlightening.\nThe experiment shown in Figure 3 is also very comprehensive and informative.\n\n**Clarity**\nOverall, the writing is generally clear and mathematical derivations are pretty well explained. \n\n## Paper Weaknesses\n\n**Sparse-reward tasks**\nI think more analysis on similar environments with different robots should be given. Tmake general claims about this method, there should be examples of many robots, not just two. Furthermore, the demonstrating an experiment like Figure 3 for another robot (perhaps the Humanoid that’s already included) would be helpful.\n\n**Experiments**\nThere should be comparisons with some works which learn skills without explicit assumptions of skill/task types and do not require pre-training skills, e.g. HIDIO (Hierarchical Reinforcement Learning By Discovering Intrinsic Options, Zhang et al. 2021), HiPPO (Sub-policy Adaptation for Hierarchical Reinforcement Learning, Li et al. 2020). These comparisons would further demonstrate the advantages of having the initial pretraining environments (which the comparison to SAC-HIRO already contributes to) and explicit goal space learning when compared with more modern HRL methods, both of which have demonstrated improvements over HIRO in their experiments. In a sense, HIRO is the only true SOTA HRL baseline in this paper so this would need to be addressed.\n\nThere’s little explicit analysis that studies how exploration is explicitly affected by these skills, despite it being stressed in the introduction. This can be partially remedied by adding more analysis in Section 5.1 regarding exploration specific to each environment (I think Sec 5.1 is also just generally lacking more analysis). \n\n\n**Clarity**\nI think that an example of the feature set F should be given earlier in the paper, it’s confusing to learn bits and pieces about feature sets F throughout section 3 before getting an example (perhaps at L108). This also makes the distance-based reward (L135) clearer when introduced.\n\nYou should bold or highlight the best performances for each column in Table 1. That would improve clarity here greatly.\n\n\n**Minor Issues**\nSome grammatical hiccups throughout the 3rd paragraph of the introduction, making it a little harder to read.\n\n\nFigure 4: “averaged over 0.5M environment steps” this should be 5M steps\n\nL271: “for SG” -> “for SD”? Or for subgoals?\n\nAppendix L597, equation:  F -> |F|\n\n## Questions for the authors\n\nHow do you anticipate extending this to visual environments? \n\nWhy are you normalizing the entropy of $\\pi^g$ by $|F|$? Isn’t the size of $F$ fixed?\n\nIt seems that many of the fixed skill experiments achieve performance nearly on par with full HSD-3. Why is this the case when it seems in Figure 5 many skills are needed?\n\n\n--UPDATE--: raised score from 5 to 6 after response\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new way of hierarchical goal-based learning. There have been multiple examples of hierarchical strategies where higher levels set goals for the lower levels. However, often such goals are set in a quite low dimensional space, either because the state space is low dimensional to begin with or because a subset of dimensions, e.g. corresponding to the COM coordinates, are pre-specified to be the relevant ones. The paper proposes that which dimensions are relevant for the goal are task dependent, and so it should be up to the higher level policy to choose which dimensions are relevant for goal-setting. In this way, the higher level policy can make the goal for a skill more general or specific, allow a better trade-off between these factors. 

The reviewers had mixed opinions initially, but additional results from the authors convinced some reviewers to update their scores, resulting in a unanimous accept recommendation. Summarizing their opinions:
- Originality: the proposed approach is interesting & original. The authors also propose an original and interesting new suite of benchmarking tasks. 
- Technical quality: Initially, the opinions were mixed on quality, as some reviewers deemed important baselines to be missing. With the provision of additional HRL baseline, the reviewers were satisfied on quality. 
- Relevance and significance: The problem is relevant for NeurIPS. The results were not super surprising (e.g., the 'full goal space' baseline also worked pretty well across tasks), however, reviewers pointed out the paper might be an important step towards solving sparse reward task. 
- On clarity, the reviews were a bit mixed, from 'difficult to parse' to 'clear and well explained', with specific issues pointed out for possible improvement. 

Overall, the paper proposes an original new method and (taken into account the new results), sufficiently evaluates it in the context of relevant baselines. The paper could certainly be improved further, but I think as is it will be an interesting addition to the NeurIPS program. 

I had one additional minor comment: In the introduction, it is mentioned that "In the large body of work ... on HRL ... relies explicitly or implicitly on prior knowledge that low-level skills should control the center of mass" (lines 22-25). While I agree that this is a common assumption, I don't think it's true for the all current HRL methods as seems implied here (e.g. the option-critic lets the agent control any dimensions, feudal networks do use a subspace but the subspace is learned), and in particular, the reference given for this statement in line 25 are mostly methods where the state consists only of the COM, thus, it's inevitable that a HRL method would control the COM rather than a particular assumption. The HIRO paper, for example, would be a better example. 
HIRO: "Data efficient reinfocement learning", Nachum et al.
Option-critic: "The option-critic architecture", Bacon et al. 
Feudal Networks: "Feudal networks for reinforcement learning", Vezhnevets et al. 
 
Decision: Accept (Poster)