Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a benchmark task with bipedal robots instead of locomotion tasks, where the bipedal robots are low dimensional and perform various movements. It also proposes a hierarchical reinforcement learning method with three levels: a policy that specifies a goal space (a set of features to operate on), a policy to specify the goal configuration given that goal space, and a low level policy to reach the desired goal configuration. It learns the low level policies through unsupervised learning, and then optimizes off policy the high level options by optimizing the value function.
main_review: Abstract: It is not clear throughout the abstract what the "inductive bias" is. It is not clear how the "the potential to facilitate exploration" is an inductive bias. since this is the primary distinction on which this work centers, this makes the abstract confusing.

Introduction:
24 While it is true that much of the work in exploration with hierarchical reinforcement learning deals in navigation or navigation-like domains where the center of mass of the agent is particularly useful, there also exists a large body of work which deals with exploration related to object manipulation where this assumption is not the case. Thus, the generalization expressed here is misleading, since it highlights that "the work that benefits from center of mass knowledge has an implicit bias towards center of mass knowledge."

30 While this is true of policies, it may not be the case that low level skills are more widely applicable, since individual skills, low level or high level might be limited to a a very small set of behavior. It would probably be more appropriate to refer to this in the context of policies.

Related work:
77 While re-usability across different high level actions is a significant benefit, this underscores the idea that one of the large benefits of Hierarchical reinforcement learning is for exploration. Even if a primitive is only used for one task, it might be that the learning procedure is able to exhibit gains in performance or sample efficiency even without generalizability. While it is likely that navigation tasks as a test domain does introduce implicit biases, it is not clear from the description in this section why this is the case.

Hierarchical skill learning: 
The "introduction" section for this component lacks a heading, and it is not clear exactly what it is expressing. Is this part a description of the low level policies? A description of the state space or of general terms? At this point it remains unclear what exactly the high level policy comprises, and why there are task specific features and additional objects separated out to be accessed only by the high level policy. While these components gain some clarity later, the ordering makes this confusing.

107 It is not clear up to this point if the argument is being made for more specific, or less specific policies. However, while such a low-level policy as reaching a goal configuration after short time is not navigation related, it is certainly very specific to robotic motion.

3.1 While it is reasonable to train low level skills over different sets of features, perhaps the most important property related to these would be how they are chosen. It is not clear if these features are selected as random samples or specified. However, random sampling seems like a suspect way, as if the number of features exceeds even a small amount the number of possible sets explodes exponentially.

Training the low level policies only by unsupervised pre-training also seems like it could introduce issues. In particular, while the low level space might afford many bad ways of controlling the agent, there are probably only a limited number of useful ways to control the agent. There should be a tradeoff between using the higher level policies to specify what is learned at the lower levels, and simply exploring with the lower level policies, but at present the former appears to be completely ignored.

3.2 Equation numbers would be much appreciated in this section, especially since there are clear changes being made to the typical bellman equation/value function, not the least of which is in the notation of taking the feature mean. 
166 While matching sign seems like it should have an effect on optimization, by negating the log of the features, this seems to change the meaning of the equation. 

170 It is not entirely clear how the equation in line 164 arrives at the one in 170. In particular, it seems that the loss is the negative expected value, but then the log|F| component has disappeared.

173 A more in-depth description of how the \alpha and \beta loss terms are defined is necessary. At this point, it is simply provided as a given without clear explanation, especially since the intuitive meaning of H^f, H^g is not made clear.

Benchmark Environments:
While these environments are provided as one of the clear contributions of this work, they are described fleetingly. It would be useful to note why properties like center of mass or other normal navigation "implicit biases" do not apply to these cases. 

Experimental results:
211 With only 5 features, these proposed domains do not actually differ too significantly from other mujoco locomotion tasks that are more commonly given. As highlighted before, the number of features also seems necessary for this method to work since the subset sampling would explode exponentially otherwise.

227 It is still unclear what emphasis is being made about "no single skill" In particular, it should be expected in any skill learning framework that one skill does not dominate, otherwise there would be no point in using the framework at all.

Figure 4: it is hard to parse the results, in particular those of SD*. Is this meant to outperform the proposed method?

5.2 The proposed baselines, while interesting, do not always capture a fair comparison. In particular they are all single level skill learning methods except for DIAYN, but in this case DIAYN-C does not appear to be ideal for this case.

Figure 6: It is not entirely clear why SAC would completely fail for the given tasks. It would be useful to see a comparison against SAC where it is able to learn at least some useful behavior, or find another baseline that does give valid results. There should exist algorithms which function on humanoid walker.

Overall, this work proposes an interesting way of selecting features to control over and an effective 3 level hierarchy which has encouraging results. While the idea of selecting features has been proposed, it has not been shown in a multi-level hierarchy. However, the writing is sufficiently difficult to parse such that it is difficult to determine how exactly this method is novel from existing work, except that it encodes more specific information for the tasks. It is also difficult to determine from the writing exactly what features of the proposed method contribute to the success. Furthermore, the experiments are limited because they test on a new domain against baselines that do not seem like fair comparison on that domain. 

Originality: marginal

Quality: marginal

Clarity: poor

Significance: below marginal

limitations_and_societal_impact: This work has a limited discussion of limitations, particularly how closely the method is tied to the task being performed. It has no discussion of societal impact, though robotics has a large variety of effects on modern society so a discussion could have occurred.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper argues that, in the context of pre-trained low level skills, there is a trade-off between generality and learning speed. It makes two contributions: firstly, it proposes a benchmarking suite of sparse reward tasks that require different motor skills to study this trade off. Secondly, the paper proposes a hierarchical skill learning algorithm that attempts to trade off generality and learning speed directly. The proposed algorithm outperforms reasonable baselines.
main_review: This paper argues that, in the context of pre-trained low level skills, there is a trade-off between generality and learning speed. It makes two contributions: firstly, it proposes a benchmarking suite of sparse reward tasks that require different motor skills to study this trade off. Secondly, the paper proposes a hierarchical skill learning algorithm that attempts to trade off generality and learning speed directly. The proposed algorithm outperforms reasonable baselines.

The proposed benchmark environments are a number of obstacle courses (e.g. stairs, gaps, hurdles) as well as a pole balancing task. The tasks are set up for the 2d walker as well as a 3d humanoid.

The proposed algorithm pre-trains skills first defining a set of features. Then the agent is trained to control subsets of the full feature set. The subsets and target features are randomly sampled during pre-training. Selecting the feature set induces and the target features induce an inductive bias. I would have like to see more discussion of how the features and target feature distributions are selected. As an aside, I think for more complex embodiments, I think using demonstrations or mocap data to define the targets would be an interesting approach. To reuse the learned skills, the skills are frozen and a hierarchical policy is learned that first selects the subset of features to control and then specifies a target. 

I think the related works section is reasonably comprehensive but should cite

    Heess et al.  “Learning and Transfer of Modulated Locomotor Controllers.” arXiv Preprint arXiv:1610. 05182.

as an early example of low level controllers in deep RL. Another relevant paper is

    Hasenclever et al. “CoMic: Complementary Task Learning & Mimicry for Reusable Skills.” ICML 2020. http://proceedings.mlr.press/v119/hasenclever20a.html. 

that studies (among other things) the same trade-off between generality and learning speed in the context of skills acquired from mocap data.

In a first experiment, the performance with different fixed goal spaces is compared. The results indicate that no single goal space works best on all tasks: there is indeed a trade-off between speed and generality. However it's worth noting that the full goal space works pretty well across tasks. In the other experiments, the proposed methods performs favourably relative to baselines. 

Overall, I think this is a nice paper and I am leaning towards acceptance.

Minor comments:
- page 4, section 3.1 last paragraph: I'm a little confused by this. If you don't reset why call the experience segments episode? Isn't it just one long episode at that point?
- Figure 1: I found the $s^+$ notation confusing. Why not $s^t$
- I would quite like to see some videos of the resulting skills and policies. This is something that other papers in this space provide
- Table 1: How many seeds and episodes do your experiments correspond to?
- Figure 6: typo "are now show"

limitations_and_societal_impact: The experimental results with the humanoid are limited in terms of the goal spaces and the full goal space baseline appears to work just as well if not better. This suggests that substantially more work is needed to scale up to more complex embodiments. This is acknowledged by the authors.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a hierarchical framework for training locomotion policies for tasks with sparse reward. Low level policies are trained to achieve a selection of subgoals and high level policies are trained by exploring in the goal spaces. Challenging locomotion tasks are introduced to show the effectiveness of the proposed methods.
main_review: Originality: This paper presents a three level hierarchical policy with a low level policy generated via training on general tasks. This is new to me.

Quality: The paper is technically sound to me.

Clarity: Paper is well written.

Significance: The result is not very impressive since most tasks can be solved via a simple forward progress reward. But it presents an important step towards solving tasks with sparse reward using a hierarchical policy.

Pro:

1. The training of low level policies based on goal spaces specified by position of torso and foot.

2. A three level hierarchical policy and an extension of SAC to accommodate the resulting action distribution.

3. Challenging sparse reward tasks for locomotion.

Con:

1.  The computation required for training even the low level skills are significant. It will be nice to provide more details on the low level training. The low level skills do not seem challenging to me (at least for the walker) so it seems 3 days on 2 GPUs is a lot.

2. Tasks such that designing a simple dense reward (such as the forward progress reward) is none trivial will really demonstrate the strength of hierarchical framework. e.g., the GoalWall task presented. More in Question 2 below.

Question:

1. It is not clear to me why Gap will be more challenging than other tasks. Maybe the reward design needs to be modified? e.g., a negative reward for touching the gap can cause the robot want to terminate as soon as possible. Some failure mode of the gap task will help illustrate the issues.

2. Most of the tasks can also be solved via a simple forward progress reward (with the exception of GoalWall, which is difficult to solve in the current framework anyway). It will be interesting to see what is the performance gain the proposed approach has over baseline methods under this setup.

3. In Figure 6, SAC makes no progress on Stairs, but that is not the case in Table 5 in the Appendix. The video also shows SAC makes some progress on Stairs.
limitations_and_societal_impact: Yes, limitation is discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a 3-level hierarchical method, one that operates on a feature space, another that operates on a feature-conditioned goal space, and finally a low level policy that outputs actions in the environment conditioned on everything above and the current state. They pretrain the policies in a pretraining environment before applying it to the task at hand. The authors also introduce a new suite of environments to test hierarchical RL. The issues brought up by the authors about current HRL methods are enlightening and the method is a novel contribution. Results are fine, with subtle/no improvements over the baselines on some tasks, and substantive improvements on others. I believe the paper has some issues with baseline comparisons and tasks, which I hope will be addressed in the rebuttal. As such, I am currently learning towards not accepting the paper.
main_review: ## Paper Strengths
**Paper Framing/Originality**
Pointing out the issues with existing HRL algorithms’ focus on learning skills that are relevant mainly for controlling center of mass is an interesting contribution. Furthermore, the algorithm that results is novel, while also incorporating ideas from modern papers such as the DYNE step-conditioned critic. The inclusion (and supposed release) of the benchmark environments are a benefit to the community as some of them seem like solid tasks to test hierarchical algorithms on.

**Results Significance**
I think the paper does a good job of demonstrating the importance of goal-space separation with results that demonstrate significant improvements over baselines on some of the tasks. Furthermore, it’s rare to see evaluations over 3 seeds, while the authors evaluate on 9 seeds in total. The inductive bias experiment on HIRO in the appendix (sec G) was very enlightening.
The experiment shown in Figure 3 is also very comprehensive and informative.

**Clarity**
Overall, the writing is generally clear and mathematical derivations are pretty well explained. 

## Paper Weaknesses

**Sparse-reward tasks**
I think more analysis on similar environments with different robots should be given. Tmake general claims about this method, there should be examples of many robots, not just two. Furthermore, the demonstrating an experiment like Figure 3 for another robot (perhaps the Humanoid that’s already included) would be helpful.

**Experiments**
There should be comparisons with some works which learn skills without explicit assumptions of skill/task types and do not require pre-training skills, e.g. HIDIO (Hierarchical Reinforcement Learning By Discovering Intrinsic Options, Zhang et al. 2021), HiPPO (Sub-policy Adaptation for Hierarchical Reinforcement Learning, Li et al. 2020). These comparisons would further demonstrate the advantages of having the initial pretraining environments (which the comparison to SAC-HIRO already contributes to) and explicit goal space learning when compared with more modern HRL methods, both of which have demonstrated improvements over HIRO in their experiments. In a sense, HIRO is the only true SOTA HRL baseline in this paper so this would need to be addressed.

There’s little explicit analysis that studies how exploration is explicitly affected by these skills, despite it being stressed in the introduction. This can be partially remedied by adding more analysis in Section 5.1 regarding exploration specific to each environment (I think Sec 5.1 is also just generally lacking more analysis). 


**Clarity**
I think that an example of the feature set F should be given earlier in the paper, it’s confusing to learn bits and pieces about feature sets F throughout section 3 before getting an example (perhaps at L108). This also makes the distance-based reward (L135) clearer when introduced.

You should bold or highlight the best performances for each column in Table 1. That would improve clarity here greatly.


**Minor Issues**
Some grammatical hiccups throughout the 3rd paragraph of the introduction, making it a little harder to read.


Figure 4: “averaged over 0.5M environment steps” this should be 5M steps

L271: “for SG” -> “for SD”? Or for subgoals?

Appendix L597, equation:  F -> |F|

## Questions for the authors

How do you anticipate extending this to visual environments? 

Why are you normalizing the entropy of $\pi^g$ by $|F|$? Isn’t the size of $F$ fixed?

It seems that many of the fixed skill experiments achieve performance nearly on par with full HSD-3. Why is this the case when it seems in Figure 5 many skills are needed?


--UPDATE--: raised score from 5 to 6 after response

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper proposes a new way of hierarchical goal-based learning. There have been multiple examples of hierarchical strategies where higher levels set goals for the lower levels. However, often such goals are set in a quite low dimensional space, either because the state space is low dimensional to begin with or because a subset of dimensions, e.g. corresponding to the COM coordinates, are pre-specified to be the relevant ones. The paper proposes that which dimensions are relevant for the goal are task dependent, and so it should be up to the higher level policy to choose which dimensions are relevant for goal-setting. In this way, the higher level policy can make the goal for a skill more general or specific, allow a better trade-off between these factors. 

The reviewers had mixed opinions initially, but additional results from the authors convinced some reviewers to update their scores, resulting in a unanimous accept recommendation. Summarizing their opinions:
- Originality: the proposed approach is interesting & original. The authors also propose an original and interesting new suite of benchmarking tasks. 
- Technical quality: Initially, the opinions were mixed on quality, as some reviewers deemed important baselines to be missing. With the provision of additional HRL baseline, the reviewers were satisfied on quality. 
- Relevance and significance: The problem is relevant for NeurIPS. The results were not super surprising (e.g., the 'full goal space' baseline also worked pretty well across tasks), however, reviewers pointed out the paper might be an important step towards solving sparse reward task. 
- On clarity, the reviews were a bit mixed, from 'difficult to parse' to 'clear and well explained', with specific issues pointed out for possible improvement. 

Overall, the paper proposes an original new method and (taken into account the new results), sufficiently evaluates it in the context of relevant baselines. The paper could certainly be improved further, but I think as is it will be an interesting addition to the NeurIPS program. 

I had one additional minor comment: In the introduction, it is mentioned that "In the large body of work ... on HRL ... relies explicitly or implicitly on prior knowledge that low-level skills should control the center of mass" (lines 22-25). While I agree that this is a common assumption, I don't think it's true for the all current HRL methods as seems implied here (e.g. the option-critic lets the agent control any dimensions, feudal networks do use a subspace but the subspace is learned), and in particular, the reference given for this statement in line 25 are mostly methods where the state consists only of the COM, thus, it's inevitable that a HRL method would control the COM rather than a particular assumption. The HIRO paper, for example, would be a better example. 
HIRO: "Data efficient reinfocement learning", Nachum et al.
Option-critic: "The option-critic architecture", Bacon et al. 
Feudal Networks: "Feudal networks for reinforcement learning", Vezhnevets et al. 
 
Decision: Accept (Poster)