Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies

Abstract: In this paper we introduce a simple, robust approach to hierarchically training an agent in the setting of sparse reward tasks.
      The agent is split into a low-level and a high-level policy. The low-level policy only accesses internal, proprioceptive dimensions of the state observation. The low-level policies are trained with a simple reward that encourages changing the values of the non-proprioceptive dimensions. Furthermore, it is induced to be periodic with the use a ``phase function.'' The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time. Using this approach, we solve difficult maze and navigation tasks with sparse rewards using the Mujoco Ant and Humanoid agents and show improvement over recent hierarchical methods. 

Reviews and author responses:

We went ahead and uploaded a revised version making the couple minor changes we promised in our responses.

Thanks to everyone for their discussions thus far.
Do the authors wish to submit a revised version of the paper?
Do the reviewers have comments to the current author responses?
Thanks...   Area Chair

While I think the author responses are adequate and get the point across, I would still like to see more comparisons with prior work such as Florensa et al. 

I have increased my score by 1 point largely because we need a push for more practical and actually functional HRL methods in the community and this work gets at how to incorporate small amounts of knowledge to get there. But please do incorporate the changes suggested if accepted. 

Brief summary: 
HRL method which uses a 2 level hierarchy for sparse reward tasks. The low level policies are only provided access to proprioceptive parts of the observation, and are trained to maximize change in the non-proprioceptive part of the state as reward. The higher level policy is trained as usual by commanding lower level policies. 

Overall impression:
I think the paper has a major assumption about the separation of internal and external state, thereby setting the form of the low level primitives. This may not be fully general, but is particularly useful for the classes of tasks shown here as seen from the strong results. I would like to see the method applied more generally to other robotic tasks, and a comparison to Florensa et al. And perhaps the addition of a video which shows the learned behaviors. 

Introduction: 
the difficulty of learning a high-level controller when the low-level policies shifts -> look at “data efficient hierarchical reinforcement learning” (Nachum et al)

The basic assumption that we can separate out observations into proprioceptive and not proprioceptive can often be difficult. For example with visual inputs or entangled state representations, this might be very challenging to extract. This idea seems to be very heavily based on what is “internal” and what is “external” to the agent, which may be quite challenging to separate. 

The introduction of phase functions seems to be very specific to locomotion?
 Related work: 
The connection of learning diverse policies should be discussed with Florensa et al, since they also perform something similar with their mutual information term. DeepMimic, DeepLoco (Peng et al) also use phase information in the state, worthwhile to cite. 

Section 3.1:
The pros and cons of making the assumption that representation is disentangled enough to make this separation, should be discussed. 

Also, the internal and external state should be discussed with a concrete example, for the ant for example. 

Section 3.2:
The objective for learning diverse policies is in some sense more general than Florensa et al, but in the same vein of thinking. What are the pros and cons of this approach over that?  The objective is greedy in the change of external state. We’d instead like something that over the whole trajectory maximizes change?

Section 3.3:  How well would these cyclic objectives work in a non-locomotion setting? For example manipulation

Section 3.4: This formulation is really quite standard in many HRL methods such as options framework. The details can be significantly cut down, and not presented as a novel contribution. 

Experiments:
It is quite cool that Figure 2 shows very significant movement, but in some sense this is already supervised to say “move the CoM a lot”. This should be compared with explicitly optimizing for such an objective, as in Florensa et al. I’m not sure that this would qualify as “unsupervised” per se. As in it too is using a particular set of pre-training tasks, just decided by the form of choosing internal and external state.

all of the baselines fail to get close to the goal locations.-> this is a bit surprising? Why are all the methods performing this poorly even when rewarded for moving the agent as much as possible.

Overall, the results are pretty impressive. A video would be a great addition to the paper. 

Comparison to Eysenbach et al isn’t quite fair since that method receives less information. If given the extra information, the HRL method performs much better (as indicated by the ant waypoint plot in that paper).

We thank the reviewer for their time. We will try to answer your questions and concerns here.

Peng et al is definitely worth citing. We will add that citation.

Phase function
The phase training does also benefit the ant, although not as much perhaps as humanoid. For humanoid, there definitely have been works that have trained humanoid without phase information with techniques such as Soft-Actor Critic. In practice, this seems to be difficult to tune. We chose a widely used PPO implementation in Kostrikov (2018), but default parameters and a grid search over parameters, we were unable to train a humanoid that receives reasonable movement reward.
 
State space selection
It is possible that you could learn the high-level and low-level policy with the entire state space, although not necessarily as efficiently. But the idea of the paper is that we want to abstract away low-level details from the high-level controller so it can focus on the planning and high-level problems. Especially as RL starts to tackle more complicated problems, and as it moves to real-world robotics, this abstraction is very useful for efficiently learning difficult high-level policies.

Confusion in notation
b_phi is a learned parameter in our network, like a bias term, that depends on the phase index. These are input to our network and the value is updated by back-propogation. d is just the choice for the dimensionality of b_phi, in our case 16.
We will clarify this in revision.

Choice of K
K=10 is approximately the time a trained Mujoco ant model will take to make a complete cycle of action. The simulation step length is 0.05sec.

Energy consumption
As we say in S3.2, we do train with the Mujoco environment rewards including energy penalty.

We thank the reviewer for their time. We will try to answer your questions and concerns here.

Overall algorithm
We will put a high-level algorithm in the appendix to make this more clear. To summarize:

1. Train K low-level policies on our low-level objective using PPO/A2C/DQN
2. Train a high level policy on the task reward using PPO/A2C/DQN where the action space is choosing one of the K low-level policies to run for T timesteps.

We will also add an appendix where we can describe the algorithm in terms of pseudo-code for one of the algorithms.

Timescales
We have a timescale for the low-level policies and a time-scale for the high-level policy, which operates on a longer timescale since you don’t need to change skills as often. 

Generality
Please see the discussion with reviewer 1 on when our external/internal assumption holds, and when the periodic assumption holds. We believe the idea of abstracting away low-level details from the high-level controller so it can focus on the planning and high-level problems is quite general in nature. As RL starts to tackle more complicated problems, and as it moves to real-world robotics, this abstraction is very useful for efficiently learning difficult high-level policies

Fits into RL/control:
Hierarchical RL is important but data-inefficient; our separation into internal and external, our use of a skills framework, and the way we specifically train the low-level policies improves performance on sparse reward tasks in Mujoco. See related work where we discuss some prior work in hierarchical RL and how it compares to our method.

Data efficient hierarchical RL
We did not intend to suggest that it is impossible to learn hierarchical policies end to end, just that it is difficult. This argument is also made in Florensa et al. Nachum et al. attempt to learn the high & low level policies jointly, which is in contrast to our approach that side-steps the problem by learning them separately. 

Level of supervision
We agree that our low-level reward is not unsupervised. We state in the introduction that we use a weak form of supervision for training the low-level policy, since the reward does not come from the final task environment.

Eysenbach comparison
In the DIAYN+prior model (which is what we compare to in Figure 7A), they do use a center of mass reward to train, which is similar to our movement reward. If you mean that we give more information to our model in the form of the split between the external and internal states, we agree. Our intention was less to claim that we beat Eysenbach et al given exactly the same information and more to say that with the assumptions we make, we are in fact able to beat DIYAN. We will make this clearer in the paper.

Non-locomotion settings
We believe that locomotion by itself is an important class of problems, so having a method that is more aimed to those isn’t a bad thing. If the cyclic objectives are less effective in other domains (e.g., grasping), they can also be removed from our method. However, we would like to point that even in some manipulation tasks  such as stirring, or repeatedly performing a task on objects as they come through on an assembly line, the cyclic prior makes sense. 

Formulation 
We  will move formulation to appendix. Note, we did not claim this as a novel contribution. We did cite Frans et al. (2017) in this section, and we do make the connection to options in the related work, as well as make the connection to Eysenbach in a later section using the same high-level methodology. Overall, we will clarify this on revision.

Baseline failures
This was surprising to us too. In Haarnoja et al. (2018a), those baselines do eventually go toward the goal, but in that setting, the goals aren't changing between episodes. The worse results could be because we are changing the goal location randomly, and that makes it far more difficult to earn reward and learn about the different goals, despite the movement prior.

Other citations to add
Thank you for the additional citations for the phase information. We will add these to related work.

We thank the reviewer for their time. We will try to answer your questions and concerns here.

Internal/External assumption
While we agree that the separation of inputs is not fully general, we think it is appropriate in many reasonable settings.  In particular, for any actuated robot, it is only reasonable that the robot be designed to know which are proprioceptive sensors. More generally for agents whose action spaces are complex and have sensors to directly measure that action space, there is essentially no cost to provide this information to such agents. It is also common for researchers in robotics get this information with localization techniques such as visual odometry, SLAM or particle filters. 
One may argue that we measure success with HRL in these settings as a proxy task, and what we are really interested in is an HRL algorithm(s) that can learn anywhere; but in our view, clean, cheap, widely applicable assumptions are the best hope for real progress.  Moreover, even if one is searching for the primal-generic HRL algorithms, our work is useful: (i) because it shows that this simple assumption leads to good results on these tasks and (ii) because in the current literature, these tasks are the standard testbeds, this work allows a researcher to recognize a mechanism that a more general algorithm might be using to achieve success.
To answer your specific question about the ant: we use the center of mass position as external, and the body angles, as well as the joint configurations as internal. Velocities we consider to be the same as the positions for categorizing as internal/external.

“comparison to Florensa et al.”: the main differences between this work and that one are that we do not try to use stochastic neural networks, we do not try to compress the one-hot representation of the low level networks into a dense vector during low level training (instead, keeping them fully separate, what they call the “multi-policy” architecture), and we do not attempt to impose any regularization to encourage the low level networks to be diverse. These can be considered simplifications;  what we show is that the simple thing works quite well.   

However, one might argue that one of the main points of Florensa et. al. was to be able to save sample complexity via compressing the multiple policies at train time into a single network. Our empirical results suggest that the situation is not so clear cut.  First, we are able to do well on the Ant task, whereas they have trouble making the high level policy work well there (see appendix d in that work). We also do better on humanoid, which is yet more difficult.  Moreover, if we correctly understand their measurements of sample complexity, our method, even accounting for the multiple independent models, is using far fewer environmental interactions at both the high and low level. Thus while training multiple independent models may seem wasteful on paper, it seems to work well in practice, and has superior sample complexity for the low-level policy. 

We briefly discuss Florensa et al in our related work, but we can expand it to go into more detail in the comparison.

This paper proposes a method to train high- and low-level controllers to tackle hierarchical RL.  The novelty is framing hierarchical RL as a problem of training a diverse set of LL controllers such that they can be used by a HL controller to solve high-level tasks.  By dividing state representation into proprioceptive and task-specific, the reward used to train LL and HL controllers are simplified.  Experimental result shows that the method is effective at solving maze environments for both ant and humanoid.

The good parts:
- The method of training diversified LL controller and a single high-level controller seems to work unreasonably well.  And one benefit of this approach is that the rewards for both high (sparse) and low (difference) level controllers can be trivially defined.

- The separation of proprioceptive and task-specific states seems to be gaining popularity.  For the maze environment (and any task that involves locomotion), this can be done intuitively.

Place to improve:
- Terrain-Adaptive Locomotion (Peng et al. 2016) used a similar approach of phase-indexed motion, as well as selecting from a mixture of experts to generate action sequence for the next cycle.  Perhaps worthwhile to cite.

- In fact, it seems that phase in this work only benefited training of low-level controller for humanoid.  But it should be possible to train humanoid locomotion with using phase information.

- This hierarchical approach shouldn't depend on the selection of state space.  What would happen when LL and HL controllers all receive the same inputs?

- The paper is difficult to follow at places.  Ex. b_phi element of R^d in Section 3.3.  I'm still not sure what is b_phi, and what is d here.

- The choice of K = 10 feels arbitrary.  Since K corresponds to the length of a cycle, it should make sense to choose K such that the period is reasonable compared to average human stride period, etc.  What is the simulation step length?

- Since LL policies control the style of the motion and the only reward it gets is to keep moving, presumably the resulting motion would look unnatural or exhibit excessive energy consumption.  Does "keep moving" reward work with other common rewards like energy penalty, etc?


This paper presents an approach for hierarchical RL based on an ensemble of low-level controllers.
From what I can tell, you train K randomly initialized models to maximize displacement (optionally with a periodic implementation).
This ensemble of low-level models is then presented to a high-level controller, that can use them for actions.
When you do this, the resultant algorithm performs well on a selection of deep RL tasks.

There are several things to like about this paper:
- Hierarchical RL is an important area of research, and this algorithm appears to make progress beyond the state of the art.

- The ideas of using ensemble of low-level policies is intuitive and appealing.

- The authors provide a reasonable explanation of their "periodicity" ideas, together with evidence that it can be beneficial, but is not always essential to the algorithm.

- Overall the writing is good... but I did find the main statement of the algorithm confusing! I think this deserves a proper appendix with everything spelled out.


There are several places this paper could be improved:
- First, the statement of the *main* algorithm needs to be brought together so that people can follow it clearly. I understand one of the main reasons this is complicated is because the authors have tried to make this "general" or to be used with DQN/PPO/A3C... but if you present a clear implementation for *one* of them (PPO?) then I think this will be a huge improvement.

- Something *feels* a little hacky about this... why are there only two timescales? Is this a general procedure that we should always expect to work? *why* are we doing this... and what can its downsides be? The ablation studies are good, but I think a little more thought/discussion on how this fits in with a bigger picture of RL/control would be good.

Overall, I hope that I understood the main idea correctly... and if so, I generally like it.
I think it will be possible to make this much clearer even with some simple amendments.

Meta-review: Strengths 

The paper presents a method of training two-level hierarchies that is based on relatively intuitive ideas and that performs well.
The challenges of hierarchical RL makes this an important problem. The benefits of periodicity and the
separation of internal state from external state is a clean principle that can potentially be broadly employed. 
The method does well in outperforming the alternative baselines.

Weaknesses

There is no video of the results. There is related work, i.e., [Peng et al. 2016] (rev 4) uses 
a policy ensemble;  phase info is used in DeepLoco/DeepMimic; methods such as "Virtual Windup Toys for Animation" 
exploited periodicity (25y ago);  More comparisons with prior work such as Florensa et al. would help. 
The separation of internal and external state is an assumption that may not hold in many cases.
The results are locomotion focussed. There are only two timescales.

Decision

The reviewers are largely in agreement to accept the paper. 
There are fairly-simple-but-useful lessons to be found in the paper
for those working on HRL problems, particularly those for movement and locomotion. 
The AC sees the novely with respect to different pieces of related work is the weakest point of the paper.  
The reviews contain good suggestions for revisions and improvements;  the latest version may take care
of these (uploaded after the last reviewer comments). Overall, the paper will make a good contribution
to ICLR 2019.


Decision: accepted-poster-papers