Sub-policy Adaptation for Hierarchical Reinforcement Learning
Authors: Alexander Li
Carlos Florensa
Ignasi Clavera
Pieter Abbeel
Venue:  (2020)
Keywords: Hierarchical Reinforcement Learning
Transfer
Skill Discovery

Abstract: Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy jointly. Second, we propose a method of training time-abstractions that improves the robustness of the obtained skills to environment changes.  Code and videos are available at sites.google.com/view/hippo-rl.

Published as a conference paper at ICLR 2020
SUB-POLICY ADAPTATION FOR HIERARCHICAL REINFORCEMENT LEARNING
Alexander C. Li', Carlos Florensa", Ignasi Clavera, Pieter Abbeel University of California, Berkeley (a falexli,, florensa, iclavera, pabbeel @@berkeley..eeo edu
ABSTRACT
Hierarchical reinforcement learning is promising approach to tackle long-horizon pocision-mmking problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm discover set of skills and continuously adapt them along with the higher level even when training on new task. Our main contributions are two-fold. First, we derive new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy jointly. Second, we propose a method of training the obtained skills environment changes. Code and videos are available. |
that improves the robustness of
INTRODUCTION
Reinforcement learning (RL) has made great progress in a variety of domains, from playing games such as Pong and Go MMiih et al.] 2015 Silver et al. 2017) to automating robotic locomotion (Schulman et al. 2015 Heess et al. (017). dexterous manipulane lation (Florensa et al.) [201bb] OpenA] (ct 2011)))) and perception (Nair et al. 2018:) Florensa et al., 2018). Yct, most work in RL is still learning from scratch when faced with new problem. This is partieularly inefficient when tackling multiple related tasks that are hard to solve due to sparse rewards or long horizons.
A promising technique to overcome this limitation is hierarchical reinforcement learning (HRL) (Sutton et al.] [999). In this paradigm, policies have several modules of abstraction, allowing to reuse subsets of the modules. The most common consists of temporal hierarchies (Precup) 2000 Dayan & Hinton| 1993), where a higher-level policy (manager) takes actions at lower frequency, and its actions condition the behavior of some lower level skills or sub-policies. When transferring
knowledge to a new task, most prior works fix the skills and train a new manager on top. Despite having clear benefit in kick-starting the learning in the new task, having fixed skills can considerably cap the final performance on the new task (Florensa et al., 2017a). Little work has been done on adapting pre-trained trained sub-policies to be optimal for new task.
In this paper, we develop a new framework for simultancously adapting all levels of temporal hierarchies. First, we derive an efficient approximated hierarchical policy gradient. The key insight that, despite the decisions of the manager being unobserved latent variables from the point of view of the Markovian environment, from the perspective of the sub-policies they can be considered as part of the observation. We show that this provides decoupling of the manager and sub-policy gradients, which greatly simplifies the computation used in other prior works (Frans et al.) 2018). Second, we introduce sub-policy specific baseline for our hierarchical policy gradient. We prove that this baseline is unbiased, and our experiments reveal faster convergence, suggesting efficient gradient variance reduction. Then, we introduce more stable way of using this gradient, Hierarchical Proximal Policy Optimization (HiPPO). This method helps us take more conservative steps in our policy space (Schulman al. 2017). critical in hierarchies
principled way. It also theoretically justifies a technique
sites. www.ge. wtt. wwwogle..comvviw////iaee www.//wwwwwwppoo.z.
Equal Contribution

---

Published as a conference paper at ICLR 2020
because of the interdependence of each layer. Results show that HiPPO is highly efficient both when learning from scratch, i.c. adapting randomly initialized skills, and when adapting pretrained skills on new task. Finally, we evaluate the benefit of randomizing the commimemit of the sub-policies, and show it helps both in terms of final performance and zero-shot adaptation on similar tasks.
2 PRELIMINARIES
$0
81
Spo
We define discrete-time finite- horizon discounted Markov decision process (MDP) by a tuple M AS,APP, Po, 7, H), where S is state set, A an action set, p S x S x XxSS + R4 is the transition probability distribution, Y E [ is discount factor, and Hl the hori- zon. Our objective is to find a stochas- tie policy He that maximizes the ex- pected discounted return within the
a
a1
Opn
na (clser, 24) To, (-18s, Pa) Cat Prnx))
20
Po
Figure I: Temporal hierarchy studied in this paper. A latent code 2 is sampled from the manager policy ne, (2e(st) every time-steps, using the current observation Skp- The actions Or are sampled from the sub-policy Ta (ax] 8t, Ekp) condi- tioned on the same latent code from / t= kp to (I: + 1)p
MDP, =0) = E- E (r(s at)]. We use 7 = (Sp, ao ...: to denote the entire state-action trajectory, where
80 ~ (s0),, ar ~ Te (ar(ss)) and 8141 P($1411/8..1
In this work, we propose method to learn hierarchical policy and efficiently adapt all the levels in the hierarchy to perform new task. We study hierarchical policies composed of a higher level, or manager Ta. (#a(s)). and a lower level, or sub-policy To (at [2, sy)). The higher level does not take actions in the environment directly, but rather outputs command, conditions the behavior of the lower level. We focus on the common case where 2 = Z,, making the manager choose among n sub-policies, skills, to execute. The manager typically operates at lower frequency than the sub-policies, only observing the environment every time-steps. When the manager receives a new observation, it decides which low level policy to commit to for p environment steps by the means of a latent code 2. Figure depicts this framework where the high level frequency p is random variable, which is one of the contribution of this paper as described in Section 4.4 Note that the class of hierarchical policies we work with is more restrictive than others like the options framework, where the commitmettt is also decided by the policy. Nevertheless, we show that this loss in policy expressivity acts as a regularizer and does not prevent our algorithm from surpassing other state-of-the art methods.
latent variable E Z, that
3 RELATED WORK
There has been growing interest in HRI. for the past few decades (Sutton al. 1999: Precup 2000 but only recently has it been applied to high-dimensional continuous domains as we do in this work (Kulkarni al. 2016 Daniel al. 2016). To obtain the lower level policies, or skills, most methods exploit some additional assumptions, like access to demonstrations (Le et al. [0018] Merel et al. 2019; Ranchod et al. 2013; Sharma al. 2018), policy sketches (Andreas et al.) 2017). or task decomposition into sub-tasks (Ghavamzadeh & Mahadevan 2003 Sohn et al. 20188) Other methods use a different reward for the lower level, often constraining to be "goal reacher" policy, where the
signal from the higher level is the goal to reach (Nachum et al. 2018 Levy et al. [009 Vezhnevets al. 2017). These methods are very promising for state-reaching tasks, but might require access to goal-rcaching reward systems not defined in the original MDP, and are more limited when training on tasks beyond state-reaching. Our method does not require any additional supervision, and the obtained skills are not constrained to be goal-reaching.
When transferring skills to a new environment, most HRL methods keep them fixed and simply train new higher-level top (Hausman et al. 2018 Heess et al.) 20166). Other work allows for building on previous skills by constantly supplementing the set of skills with new ones (Shu et al. 2018). but they require hand-defined curriculum of tasks, and the previous skills never fine-tuned.
2

---

Published as a conference paper at ICLR 2020
Our algorithm allows for seamless adaptation of the skills, showing trade-off between leveraging the power of the hierarchy and the final performance in a new task. Other methods use invertible functions skills (Haarnoja et al. 2018), and therefore a fixed skill can be fully overwritten when new layer of hierarchy is added on top. This kind of "fine-tuning" is promising, although similar to other works (Peng et al. 2019). they do not apply it to temporally extended skills as we do here.
One of the most general frameworks to define temporally extended hierarchies is the options frame- work (Sutton al. 1999), and it has recently been applied continuous state spaces (Bacon et al.) 2017). One of the most delicate parts of this formulation is the termination policy, and it requires sev- eral regularizers to avoid skill collapse (Harb al. 2017; Vezhnevets et 2016). This modification of the objective may be difficult to tune and affects the final performance. Instead of adding such penalties, we propose to have skills of a random length, not controlled by the agent during training of the skills. The benefit is two-fold: no termination policy to train, and more stable skills that transfer better. Furthermore, these works only used discrete action MDPs. We lift this assumption, and show good performance of our algorithm in complex locomotion tasks. There are other algorithms recently proposed that go in the same direction, but we found them more complex, less principled (their per-action marginalization cannot capture well the temporal correlation within each option), and without available code or evidence of outperforming non-hierarchical methods (Smith et al. 2018).
The closest work to ours in terms of final algorithm structure is the proposed by Frans al. (2018). Their method can be included our framework, and hence henefits from our new theoretical insights. We introduce modification that is shown to be highly beneficial: the random time- commitment mentioned above, and find that our method can learn in difficult environments without their complicated training scheme.
4 EFFICIENT HIERARCHICAL POLICY GRADIENTS
When using hierarchical policy, the intermediate decision taken by the higher level is not directly applied in the environment. Therefore, technically it should not be incorporated into the trajectory description as an observed variable, like the actions. This makes the policy gradient considerably harder to compute. In this section we first prove that, under mild assumptions, the hierarchical policy gradient can be accurately approximated without needing to marginalize over this latent variable. Then, we derive an unbiased baseline for the policy gradient that can reduce the variance of its estimate. Finally, with these findings, we present method, Hierarchical Proximal Policy Optimization (HiPPO), an on-policy algorithm for hierarchical policies, allowing learning at all levels of the policy jointly and preventing sub-policy collapse.
4.1 APPROXIMATE HIERARCHICAL POLICY GRADIENT
Policy gradient algorithms are based on the likelihood ratio trick (Williams) 1992) to estimate the gradient of returns with respect to the policy parameters
Vor((ao) E. Vo log P(T) R(T) RJ
Vo log ((T.))((TT))
(1)
Ve log ((.
In temporal hierarchy, hierarchical policy with a manager Te, (ze(st) selects every p time-steps one of n sub-policies to execute. These sub-policies indexed by E,,, can be represented as a single conditional probability distribution over actions Te [ag|2), st). This allows us to not only use a given set of sub nol-policies, but also leverage skills learned with Stochastic Neural Networks (SNNs) (Florensal (et al. 2017a). Under this framework, the probability of a trajectory = written
(so, on $1...., 8t) can be
H/p II k=
(k+1)p-1 II t=kp
(a.ls, 2)) =0))) P (P(s0) P(00)
P(7)
Σ
no (z)l)kp)
II P( 118. [=1
The mixture action distribution, which presents itself as an additional summation over skills, prevents additive factorization when taking the logarithm, from Eq. to This can yield numerical
3

---

Published as a conference paper at ICLR 2020
instabilities due the product of the sub sub-policy probabilities. For instance, in the where all the skills are distinguishable all the sub- sub-policies' probabilities but one will have small values, resulting in exponentially small value. In the following Lemma, derive approximation of the policy gradient, whose error tends to zero as the skills become more diverse, and draw insights on the interplay of the manager actions.
Lemma 1. If the skills are sufficiently differentiated. then the latent variable be treated as part of the observation to compute the gradient of the trajectory probability. Let Te, (z|s) and ne, (a s, 2) be Lipschitz functions w.E.f. their parameters, and assume that O < To, (als, 25) Vjj + kp, then
H/p
H
Vo log P(T)= =
Σ
Vo log T0, (2kp) Skp)
Vo log no, (aols), =kp) + O(HHeP-)) (4)
K=
Proof. See Appendix.
Our assumption can be seen as having diverse skills. Namely, for each action there just one sub-policy that gives it high probability. In this case, the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability. Many algorithms to extract lower- level skills based promoting diversity among the skills (Florensa et al., [201aa] [Esenbbcch et al., 2019), therefore usually satisfying our assumption. We further analyze how well this assumption holds in our experiments section and Table 2
4.2 UNBIASED SUB-POLICY BASELINE
The policy gradient estimate obtained when applying the log- log-likelihood ratio trick as derived above known to have large variance. very common approach to mitigate this issue without biasing the estimate is to subtract a baseline from the returns (Peters & Schaal 2008). It is well known that such baselines can be made state-dependent without incurring any bias. However, is still unclear how formulate baseline for all the levels in hierarchical policy, since action dependent baseline does introduce bias in the gradient (Tucker et al., 2018). It has been recently proposed to use latent-conditioned baselines (Weber al. (019). Here go further and prove that, under the assumptions of Lemma we can formulate an unbiased latent dependent baseline for the approximate gradient (Eq.5))
Lemma 2. For any functions bn S + I and bj : S x Z + R we have:
H/p
H
E11
Vo log P(ekpl (h = and E,
Vo log no, (a. St, 2kp) =kp)) =
k=
Proof. See Appendix.
Now apply Lemma 1 and Lemma 2 to Eq. [ By using the corresponding value functions the function baseline, the return can be replaced by the Advantage function A(skp, Zkp) (see details in Schulman al. (2016), and we obtain the following approximate policy gradient expression:
and
H/v
= EE,
Vo log mo, (itplskp)AAsss )) + Vo log To, (ar(sts =kp) a, zxp))
0==
This hierarchical policy gradient estimate can have lower variance than without baselines, but using it for policy optimization through stochastic gradient descent still yields an unstable algorithm. In the next section, we further improve the stability and sample efficiency of the policy optimization by incorporating techniques from Proximal Policy Optimization (Schulman al., 2017
4.3 HIERARCHICAL PROXIMAL POLICY OPTIMIZATION
Using appropriate step size in policy space is critical for stable policy learning. Modifying the policy parameters in some directions may have a minimal impact on the distribution over actions, whereas small changes in other directions might change its behavior drastically and hurt training
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 HiPPO Rollout
Algorithm НiPPO
1: Input: skills (als, manager To, (z|s), time-
1: Input: skills no, (a|s, 8, z), man-
commitment bounds min and P maxx horizon
ager TT, (=|s), horizon H, learn- ing rate
2: Reset environment: So Po. 0.
3: while < H do
2: while not done do
4: Sample time-commitment p Cat([Pming Pass,PPaa Pinax)
3: for actor 1, ... N do
5: Sample skill 24 ~ Te, (-18:)
4: Obtain trajectory with
6: for 1' L.... (1+ pp) do
HiPPO Rollout
7: Sample action ay ~ To, (-18v, zx)
5: Estimate
advantages A(ar, st, xe) and st) end for
8: Observe new state s141 and reward ry
9: end for
- p
7: 0 + 0 + VV
HiPPO
11: end while
end while
Output: (So, 20, ag, $1, .1....,......................... 8H, 2n, ag 8H+11
efficiency (Kakade 2002). Trust region policy optimization (TRPO)
constraint on the KL-
divergence between the old policy and the new policy to prevent this issue (Schulman et al. (015). Unfortunately, hierarchical policies generally represented by complex distributions without closed form expressions for the Kl.-divergence. Therefore, to improve the stability of our hierarchical policy gradient we turn towards Proximal Policy Optimization (PPO) (Schulman et al. 2017). PPO is more flexible and compute-cfficient algorithm. In nutshell, it replaces the KL-divergence constraint with cost function that achieves the same trust region benefits, but only requires the computation of the likelihood. Letting wy (0)
motax]ss) the PPO objective is: ste (0r]=z)
CLIP (0) = E, min { (0) clip(w,((0), (1
We can adapt our approximated hierarchical policy gradient with the same approach by letting Wp kp
#ld (as]ar stp)' and using the super-index clip to denote the clipped objective version, we obtain the new surrogate objective:
ota.ota (Rtp lekp) and (1.)(0) ney (ar (epp))
Se, (#kpl ske))
H/p
CLIP LIMPPo(0)
= E,
Σ
min (w, (0) 2xp) w h.
cllp
(f)A(skpp 2kp))
k=
min { (0) az zkp), (0)AA(())((s)..)))) des akp)
0=0
We call this algorithm Hierarchical Proximal Policy Optimization (HiPPO). Next, we introduce a critical additions: switching of the time-commitment between skills.
4.4 VARYING
Most hierarchical methods either consider a fixed time-commitment to the lower level skills (Florensa al. 2017a Frans al. 2018), implement the complex options framework (Precupl 2000 Bacon et al. 2017). In this work we propose an in-between, where the time-commitment to the skills random variable sampled from a fixed distribution Categorical ca Tmux) just before the manager takes a decision. This modification does not hinder final performance, and we show improves zero-shot adaptation to new task. This approach to sampling rollouts detailed in The full algorithm is detailed in Algorithm
Algorithm [
5 EXPERIMENTS
We designed our experiments to answer the following questions: 1) How does HiPPO compare against flat policy when learning from scratch? 2) Does it lead to policies more robust to environment changes? 3) How well does it adapt already learned skills? and 4) Does our skill diversity assumption hold in practice?
5

---

Published as a conference paper at ICLR 2020
(a) Block Hopper
(b) Block Half Cheetah
(c) Snake Gather
(d) Ant Gather
Figure 2: Environments used to evaluate the performance of our method. Every episode has a different
configuration: wall heights for (a)-(b), ball positions for (c)-(d)
HiPPO
HiPPO p=10
HiPPO p=1
PPO action repeat
PPO
Average Relum
500
1000
0
250
500 Ilerstoo
750
1000
250
530
750
1000
250
600
750
1000
Ber elion
Hll alion
Beralion
(a) Block Hopper
(b) Block Half Cheetah
(c) Snake Gather
(d) Ant Gather
Figure 3: Analysis of different time-commitment strategies on learning from scratch.
5.1 TASKS
We evaluate our approach on variety of robotic locomotion and navigation tasks. The Block environments, depicted in Fig. 2a 26 have walls of random heights at regular intervals, and the objective is learn gait for the Hopper and observe the height of the wall ahead and their proprioceptive information (joint positions and velocities), receiving a reward of for each wall cleared. The Gather environments, described by Duan et al. (016)), require agents to collect apples (green balls, +1 reward) while avoiding bombs (red balls, 1 reward). The only available perception beyond proprioception is through a LIDAR-type sensor indicating at what distance are the objects in different directions, and their type, depicted in the bottom left corner of Fig. 2c 2d This is challenging hierarchical task with sparse rewards that requires simultaneously learning perception, locomotion, and higher-level planning capabilities. We use the Snake and Ant robots in Gather. Details for all robotic agents are provided in Appendix
robots jump over them. The agents
5.2 LEARNING FROM SCRATCH AND TIME-COM CMME-COMITTMET
In this section, we study the benefit of using our HiPPO algorithm instead of standard PPO on a flat policy (Schulman et 2017). The results, reported in Figure demonstrate that training from scratch with HiPPO leads to faster learning and better performance than flat PPO. Furthermore, we show that the benefit of HiPPO does not just come from having temporally correlated exploration: PPO with action repeat converges at a lower performance than our method. HiPPO leverages the time-commitment more efficiently, as suggested by the poor performance of the ablation where set 1, when the manager takes an action every environment step as well. Finally, Figure shows the effectiveness of using the presented skill-dependent baseline.
5.3 COMPARISON TO OTHER METHODS
We compare HiPPO to current state-of-the-art hierarchical methods. First, we evaluate HIRO (Nachum et al. 2018). an off-policy RL method based on training goal-reaching lower level policy,
Fig. 5 shows that HIRO achieves poor performance on our tasks. As further detailed in Appendix D this algorithm is sensitive access to ground-truth information. like the exact (æ, y) position of
the robot in Gather. In contrast, our method is able to perform well directly from the raw sensory
inputs described in Section [.1 We evaluate Option-Critic (Bacon al.) 2017), variant of the options framework (Sutton et al 1999) that can be used for continuous action fails to learn, and we hypothesize that their algorithm provides less time-correlated exploration and learns

---

Published as a conference paper at ICLR 2020
Manager and Skill Baselines (ouns)
Single Baseline
Average
500
1000
250
5:00
750
1000
0
250
500
1000
0
250
500 Iteration
750
1000
lteration
Ierrtion
Iterasion
(a) Block Hopper
(b) Block Half Cheetah
(c) Snake Gather
(d) Ant Gather
Figure 4: Using skill-condiiioned baseline, defined in Section 4.2 generally improves perfor-
mance of HiPPO when learning from scratch.
HiPPO (ours)
HieVVFG
Option Critic
MLSH
HIRO
Avarage
2
500 Iteration
1000
250
500
750
1000
0
250
500
750
1000
0
250
500
750
1000
Iooration
Iteration
Iteration
(a) Block Hopper
(b) Block Half Cheetah
(c) Snake Gather
(d) Ant Gather
Figure 5: Comparison of HiPPO and HierVPG to prior hierarchical methods on learning from scratch.
less diverse skills. We also compare against MLSH (Frans al. 2018), which repeatedly samples new environment configurations to learn primitive skills. We take these hyperparameters from their Ant Twowalk experiment: resetting the environment configuration every 60 iterations, a warmup period of 20 during which only the manager is trained, and joint training period of 40 during which both manager and skills are trained. Our results show that such a training scheme does not provide any benefits. Finally, we provide comparison direct application of our Hierarchical Vanilla Policy Gradient (HierVPG) algorithm, and we see that the algorithm is unstable without PPO's trust-region-like technique.
5.4 ROBUSTNESS TO DYNAMICS PERTURBATIONS
We investigate the robustness of HiPPO to changes in the dynamics of the environment. We per- form several modifications to the base Snake Gather and Ant Gather environments. One at a time, change the body mass, dampening of the joints, body inertia, and friction characteris- tics of both robots. The results, presented Table I show that HiPPO with randomized period Categori ca caccaa(2 c Tmax]) is able better handle these dynamics changes. In terms of the drop in policy performance between the training environment and test environment, it outperforms HiPPO with fixed period on 6 out of related tasks. These results suggest that the randomized period exposes the policy to wide range of scenarios, which makes it easier to adapt when the environment changes.
Gather
Algorithm
Initial
Mass
Dampening
Inertia
Friction
Flat PPO
2.72
3.16 (+16%)
2.75 (+1%)
2.11 (-22%)
2.75 (+1%)
Snake
HiPPO, p = 10
4.38
3.28 (-25%)
3.27 (-25%
3.03 (-31%)
3.27 (-25%)
HiPPO random p
5.11
4.09
(-20%)
4.03 (-21%)
3.21 (-37%)
4.03 (-21%)
Flat PPO
2.25
2.53 (+12%)
2.13 (-5%)
2.36 (+5%)
1.96 (-13%)
Ant
HiPPO, p 10
3.84
3.31 (-14%)
3.37 (-12%)
2.88 (-25%)
3.07 (-20%)
HiPPO random
3.22
3.37 (+5%)
2.57 (-20%)
3.36 (+4%)
2.84 (-12%)
Table Zero-shot transfer performance. The final return in the initial environment is shown, as well as the average return over 25 rollouts in each new modified environment.
5.5 ADAPTATION OF PRE-TRAINED SKILLS
For the Block task, use DIAYN (Eysenbach et al. 2019) to train 6 differentiated subpolicies in an environment without any walls. Here, we see if these diverse skills can improve performance on downstream task that's out of the training distribution. For Gather, take 6 pretrained
7

---

Published as a conference paper at ICLR 2020
HIPP Finetuning (purs)
PPO on faodd stdls
Option Chtt Finetuning
MUSH Finetuning
HIRO Fineburing
HPPPO from scratch
Avanage Retum
2
500
1000
250
500
750
1000
250
500
1000
250
500
750
1000
Iteration
Boration
Itanation
Iteration
(a) Block Hopper
(b) Block Half Cheetah
(c) Snake Gather
(d) Ant Gather
Figure 6: Benefit of adapting some given skills when the preferences of the environment are different from those of the environment where the skills were originally trained. Adapting skills with HiPPO has better learning performance than leaving the skills fixed or learning from scratch.
subpolicies encoded by Stochastic Neural Network (Tang & Salakhutdinov 2013) that trained in diverrity-promootie environment (Florensa et al. 2017a). We fine-tune them with HiPPO on the Gather environment, but with an extra penalty on the velocity of the Center of Mass. This can be understood as preference for cautious behavior. This requires adjustment of the sub-policies, which were trained with proxy reward encouraging them move as far as possible (and hence quickly). Fig. shows that using HiPPO to simultancously train a manager and fine-tune the skills achieves higher final performance than fixing the sub-policies and only training a manager with PPO. The two initially learn at the same rate, but HiPPO's ability to adjust to the new dynamics allows it to reach higher final performance. Fig. alaso shows that HiPPO can fine-tune the same given skills better than Option-Critic (Bacon et al. 2017). MLSH (Frans et al. 20118), and HIRO (Nachum et al. 2018).
5.6 SKILL DIVERSITY ASSUMPTION
In Lemma 1 we derived more efficient and numerically stable gradient by assuming that the sub-policies are diverse. this section, we empirically test the validity of our assumption and the quality of our approximation. We run the HiPPO algorithm Ant Gather and Snake Gather both from scratch and with given pretrained skills, as done in the previous section. In Table we report the average maximum probability under other sub-policies. corresponding In all settings, this is on the order of magnitude of 0.1. Therefore, under the p R: 10 that we use in our experiments, the term neglect has factor e 10-10 It is not surprising then that the average cosine similarity between the full gradient and our approximation is almost 1, as reported in Table
from the assumption.
Gather
Algorithm
Cosine Sim.
max
Ta. (a. St, x')
Te (ay St, 2kp)
HiPPO on given skills
0.98 + 0.01
0.09 = 0.04
0.44 = 0.03
Snake
HiPPO on random skills
0.97 + 0.03
0.12 = 0.03
0.32 + 0.04
HiPPO on given skills
0.96 + 0.04
0.11 = 0.05
0.40 0.08
Ant
HiPPO on random skills
0.94 + 0.03
0.13 = 0.05
0.31 + 0.09
Table 2: Empirical evaluation of Lemma 1 In the middle and right columns, we evaluate the quality of our assumption by computing the largest probability of a certain action under other skills (c), and the action probability under the actual latent. We also report the cosine similarity between our approximate gradient and the exact gradient from 3 The mean and standard deviation of these values are computed over the full batch collected iteration 10.
6 CONCLUSIONS AND FUTURE WORK
In this paper, we examined how to effectively adapt temporal hierarchies. We began by deriving hierarchical policy gradient and its approximation. We then proposed a new method, HiPPO, that can stably train multiple layers of a hierarchy jointly. The adaptation experiments suggest that we can optimize pretrained skills for downstream environments, and learn emergent skills without any unsupervised pre-traine pre-training. We also demonstrate that HiPPO with randomized period can learn from scratch on sparse-reward and long time horizon tasks, while outperforming non- hicrarchical methods on zero-shot transfer.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Jacob Andreas, Dan Klein, and Sergey Levine. Modular Multitask Reinforcement Learning with Policy Sketches. International Conference in Machine Learning, 2017. URL htt.httt://ggthu
com/
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option Critic Architecture. AAAI, pp. 1726- 1734, 2017. http::ttattp...rprr.tct/ abs/g//a 001116199.111000
Christian
Daniel, Herke van Hoof, Jan Peters, Gerhard Neumann, Thomas Gärtner, Mirco Nanni, Andrea Passerini, and Celine B Robardet Christian Daniel ChristianDaniel. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(104), 2016. doi:
10.1007/s10994-016--
Peter Dayan and Geoffrey E. Hinton. Feudal Reinforcement Learning. Advances in Neural Informa- fion Processing Systems, pp. 271 1993. ISSN 0143991X. doi: 10.1
http:://ww/ http://www...oooo.ooeoooo /ttpp//////ww.................a. wwttteeitddeee/eeeeeeeeeeeiatiee hbtp://aass////hh/ http:e////..ttttaaaasss/h//3.
Yan Duan, Xi Chen, John Schulman, and Pieter Abbeel. Benchmarking Deep Reinforcement
Learning for Continuous Control. International Conference in Machine Learning, 2016. URL
http:://axiit org/ aate//bs//eeee 0tbb//604.06777
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You Need:
Learning Skills without Reward Function. International Conference Learning Representations, 2019. /tRt: wtwp:/i..orrrroooo wtt.///././////...// 06070
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic Neural Networks for Hierarchical Re- inforcement Learning. International Conference in Learning Representations, pp. 1-17, 70 ISSN 14779129. doi: 10.1002/rem.7653 (tRL:/tt:///aat / www.//g////o/.ooooo.//./../...a/ 21004 03011
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse Curriculum Generation for Reinforcement Learning. Conference Robot Learning, pp. 1-16. 2017b. ISSN 1938-7228. doi: 10. 10.1080000081112889977777 URL
abs fbbs/107.. 05300
Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. Self-supervised Learning of Image Embedding for Continuous Control, In Workshop on Inference to Control at NeurlPS, 2.18. /BKl/tttpp//arsi..cc/////.
901. 00943
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta Learning Shared Hierarchies. International Conference in Learning Representations, pp. 1-11, 2018. ISSN 14639076. doi: 10. 10.10999/20355... httpht:::/ www.:xwwwwwrgg...ee 11700. 09767
Mohammad Ghavamzadeh and Sridhar Mahadevan. Hierarchical Policy Gradient Algorithms. In- ternational Conference in Machine Learning, 2003. URL http::tttthchheeeuu// lille
inria. fr/~ghavamza hatpiiiaaaammmmssbbee http://wwwatteee/oooaaiooooooooa
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine, Latent Space Policies for Hierarchical Reinforcement Learning. Internation Conference in Machine Learning, 2018. URL
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When Waiting is not an Option Learning Options with a Deliberation Cost. AAAI, 2017. http:::::/arxi org. abs/
1709.0457)
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning Embedding Space for Transferable Robot Skills. International Conference in Learning Representations, pp. 1-16, 2018
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap. Martin Riedmiller, David Silver, and Google Deepmind. Learning and Transfer Modulated Locomotor Controllers. 2016. URL www./oggg/////oooo 161 1.0.00551
https://arrs http:////xxxvv..o//o...o/.//

---

Published as a conference paper at ICLR 2020
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. Emergence of Locomotion Behaviours in Rich Environments. 7 2017. URL http://axxxvv..rr///./ 1707. 02286)
Sham Kakade. A Natural Policy Gradient. Advances in Neural Information Processing Systems, 2002.
Tejas D Kulkarni, Karthik R Narasimhan, Ardavan Saeedi CSAIL, and Joshua B Tenenbaum BCS. Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Moti- vation. Advances in Neural Information Processing Systems, pp. 1-13, 2016.
Hoang M Le, Nan Jiang, Alekh Agarwal, Miroslav Dud, and Yue Hal. Hierarchical Imitation and Reinforcement Learning. International Conference in Machine Learning, 2018.
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical Actor-Critic. arXiv:1712.00945, 12 2017.
URL http:h/ttxipiit// wwt.///i.ooooooooooo..o abs 00948
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical Reinforcement Learning with Hindsight. International Conference Learning Representations, 5 2019. URL http://arxiv.orgg 08100
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control humanoids. International Conference in Learning Representations, 2019,
org/
09656
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Ofir Nachum, Honglak Lee, Shane Gu, and Sergey Levine. DftaiEfficient Hierarchical Reinforcement Learning. Advances in Neural Information Processing Systems, 2018.
Ashvin Nair, Vitehyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual Reinforcement Learning with Imagined Goals. Adavances in Neural Information Processing Systems, 2018
OpenAl, Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, and Alex Ray. Learning Dexter- ous Hn- Manipulation. pp. 1-27, 2018.
Xue Bin Peng, Michael Chang, Grace Zhang, Picter Abbeel, and Sergey Levine. MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. 2019. URL 09808
http: wwt.//////.....
Jan Peters and Stefan Schaal. Natural Actor-Critic. Neurocomputingg 71(7-9):1180-11199 2008.
ISSN 09252312. doi: 10. n000m/..nuccmmm20077 10.2.1
Doina Precup. Temporal abstraction in reinforcement learning, 1 2000. URL https:/7
scholarworks. umass. https//aeec////ieerrrrrtraaa/aaa
Pravesh Ranchod, Benjamin Rosman, and George Konidaris. Nonparametric Bayesian Reward Segmentation for Skill Discovery Using Inverse Reinforcement Learning. 2015. ISSN 21530866.
doi: 10.1009/100.2011.73333.
John Schulman, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust Region Policy Optimization. International Conference in Machine Learning, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael I Jordan, and Pieter Abbeel. HIGH- DIMENSIONAL CONTINUOUS CONTROL USING GENERALIZED ADVANTAGE ESTIMA- TION. International Conference in Learning Representations, pp. 1-14, 2016.
10

---

Published as a conference paper at ICLR 2020
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Pol- icy Optimization Algorithms. 2017.
us-us -asts-wett222
amazonaws
Arjun Sharma, Mohit Sharma, Nicholas Rhinchart, and Kris M Kitani. Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information. Interna- tional Conference in Learning Representations, 2018. ULL/Etp:///aax...O////////
01266
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement Learning. International Conference in Learning Representations, 3:1-13, 2018. doi: 10.1 10.109/0WC.201..55553333
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature. 55017676) 359, 10 2017. ISSN 14764687. doi: 10. 1270////auu
00633
Matthew J. A. Smith, Herke van Hoof, and Joelle Pineau. An inference-based policy gradient method for learning options. 2018. URL httppsss//ppeeeeeeeeeeeeetee/e/e wwwp:/rrwieewweteeeeeeeeeeeeee 2t 21
Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Multitask Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies. Advances in Neural Information Processing Systems, 2018.
Richard S Sutton, Doina Precup. and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112: 181- 1999. URL http: http://wwwwawwwwww cs umass. edu/-barto courses/c36877 Sutton Prennp-reppppp Singh Manngh-A-J AIJ99 pdd
Yichuan Tang and Ruslan Salakhutdinov. Learning Stochastic Feedforward Neural Networks. Ad-
vances in Neural Information Processing Systems, 2:530-538, 2013. doi: 10.1.1.63.1777
Emanuel Todorov, Tom Frez, and Yuval Tassa. MuloCo A physics engine for model-based control. pp. 5026-5033, 2012.
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E Turner, Zoubin Ghahramani, and Sergey Levine. The Mirage of Action-Dependent Baselines in Reinforcement Learning. Internation Conference Machine Learning, 2018. /ttt://
org/abs 802.10031
Alexander Vezhnevets, Volodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, and Koray Kavukcuoglu Google DeepMind. Strategic Attentive Writer for Learning Macro-Actions. Advances in Neural Information Processing Systems, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal Networks for Hierarchical Reinforcement Leaminng International Conference in Machine Learning, 2017. URL https:tttta.tttt.t pettps://axxiv.oog//da 1703.01161.0
Théophane Weber, Nicolas Heess, Lars Buesing, and David Silver. Credit Assignment Techniques in Stochastic Computation Graphs. 2019. http:////aaa wwtp:ii.orggab///ooo abs/ /105//9911... 01 01761
Ronald J Williams. Simple Statistical Gradient- Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, 8(3-4):229-25 -26,, 1992.
11

---

Published as a conference paper at ICLR 2020
A HYPERPARAMETERS AND ARCHITECTURES
The Block environments used horizon of 1000 and batch size of 50,000, while Gather used a batch size of 100,000. Ant Gather has horizon of 5000, while Snake Gather has a horizon of 8000 due its larger size. For all experiments, both PPO and HiPPO used learning rate x 10 clipping parameter 0.1, 10 gradient updates per iteration, and discount 7 0.999. The learning rate, clipping parameter, and number of gradient updates come from the OpenAl Baselines implementation.
HiPPO used n sub-policies. HiPPO uses a manager network with 2 hidden layers of 32 units, and skill network with 2 hidden layers of 64 units. In order to have roughly the same number of parameters for each algorithm, flat PPO uses a network with hidden layers with 256 and 64 units respectively. For HiPPO with randomized period, resample manager network outputs a latent, and provide the number of timesteps until the next latent selection as an input into both the manager and skill networks. The single baselines and skill-dependent baselines used MLP with 2 hidden layers of 32 units to fit the value function. The skill-dependent baseline receives, addition the full observation, the active latent code and the time remaining until the next skill sampling. All runs used five random seeds.
Uniform (5, 15) every time the
B ROBOT AGENT DESCRIPTION
Hopper is 3-link robot with 14-dimensional observation space and 3-dimensional action space. Half-Cheetah has a 20-dimensional observation space and a 6-dimensional action space. We evaluate both of these agents and positions, they observe the height and length of the next wall, the x- x-position of the next wall, and the distance the wall from the agent. We also provide the same wall observations for the previous wall, which the agent can still interact with.
sparse block hopping task. In addition to observing their own joint angles
Snake is a 5-link robot with a 17-dimensional observation space and a 4-dimensional action space. Ant is a quadrupedal robot with 27 7-diiennsionl observation space and a 8-dimensional action space. Both Ant and Snake can move and rotate in all directions, and Ant faces the added challenge of avoiding falling over irrecoverably. In the Gather environment, agents also receive 2 sets of 10-dimensional lidar observations, wheih correspond to separate apple and bomb observations. The observation displays the distance to the nearest apple bomb in each 36 bin, respectively. All environments are simulated with the physics engine MuJoCo Tooooov et al. 2012).
C PROOFS
Lemma 1. If the skills are sufficiently differentiated, then the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability. Concretely, if To, (2|s) and Ta (als, =) are Lipschitz in their parameters, and 0 < To, (urlst, 2j) < Vj o kp, then
H/p
Vo log P(T) = Σ
Vo log no, (2kp|Skp)
Vo log ne, (arls), + O(nHe" (5)
0mm0
Proof. From the point of view of the MDP, trajectory is sequence 7 (8o, an; 81,41, "i,,..,,,a .,, 8H). Let's assume we use the hierarchical policy introduced above, with a higher-level policy modeled as parameterized discrete distribution with n possible outcomes ne, (2|s) = Categoricale, (n). We can expand P(T) into the product of policy and environment dynamics terms, with 2j denoting the jth possible value out the n choices.
Hip
(k+1)p-1
P(r)
|I k=0
Σ
n (2g 8kk))
II <=
no, (2)))1112)))
P(so) II ((sctals),aa) t=1
12

---

Published as a conference paper at ICLR 2020
Taking the gradient of log P(T) with respect to the policy parameters 0 [8x. 0t]. the dynamics terms disappear, leaving:
H/p
n
(A++ 1p=1
Va log P(T)
Σ
Ve log
Σ ==1
Til (2 skp)
II kpp
H/p
(k+1)p-1
Vo Ten (2)l8kp))
II t kp
To, (ael 23))
Ej=1 TB, (kylokp) II
1++1)p-1 tk= Te, (arl Rg) f=1
The sum over possible values of prevents the logarithm from splitting the product over the p-step sub-trajectories. This term is problematic, as this product quickly approaches as increases, and suffers from considerable numerical instabilities. Instead, want to approximate this sum of products by a single one the terms, which can then be decomposed into a sum of logs. For this we study each of the terms in the sum: the gradient of sub-trajectory probability under specific latent Vo To. (zy Skp) T
(0e|st; Now we can use the assumption that the skills are easy to distinguish, 0 < Te, (ax (4x]8t, zz) < v/j 3 kp. Therefore, the probability of the sub-trajectory under latent different than the one that was originally sampled 2j o Zkp- is upper bounded by eP, Taking the gradient, applying the product rule, and the Lipschitz continuity of the policies, we obtain that for
all 2j # Zkp-
(b+1)p-1
(k+1)p-1
Ve Toh (zgl (kkp)
II t kp
Te, (ayl(t)) = V VOTTe (2yl8
II kp
To (aelss, 2z)+
(k+1))-1
(k++1p--1
no,
(2) (alss)))(ooog (Vo (aclsc: 2j))
II
(arlar 23)
k=
t= d' t'#l
O(pe-11)
Thus, across the board replace the summation over latents by the single term corresponding to the latent that was sampled at that time.
H/P
(k+1)p-1
2kx) + nH
Vo log P(r)
Σ
Vo P( (kkp)
II
T0,
O
Top (2tplskp)
(1-1))=- 1) no,
(aelse, 2kp)
p
H/P
(k+1)x-1
Vo log
Tox (2kplskg)
II
To, (a se 2kp)) + OmHHet-1)
k=
t=
H/p
V. log To, (celses 2bp)) + OnHee-1)
E,
Vo log Ton (Zkplskp)
+
k=
Interestingly, this is exactly P 20, ag, $1,...) ). In other words, it's the gradient of the probabil- ity of that trajectory, where the trajectory now includes the variables z as if they were observed.
Lemma 2. For any functions by S - R and by S Z R we have:
Hip
E,
v log P(zkp (kp 16( )6(kk)) = 0
k=0
E
Vp log मe, (azlsr, Zkpp Zkp (b(se, =xy)]] 0
0
13

---

Published as a conference paper at ICLR 2020
Proof. We can the tower property well the fact that the interior expression only depends Skp
and Ekp
H/P Va log ((zzp|8kp)b(( ((spp)
H/p
E Eap [Br
[Brreep. Vo log P(zkpl Skp (kkk))]
k=
k=0
H/p
E Bapp
Va log (bpb))ss))))()ppp)
Then, write out the definition of the expectation and undo the gradient-log trick prove that the baseline unbiased.
H/p E,
H/P Σ
V o log To, (2kpl Skp (kkkp)
=
P(skpt 2kp)
Vo log Te, (#kplskp) (b(())((((p))) d2kk
R=0
(Ekp:Pkp)
H/p
P(skp)b(Skp)
Ten (Zkpl sky) Vo log To, (2kp Skp dssp
8kp
zxp
H/P
/ Tep (Zxplskp) To, (zkp) 8kp)
Σ
P(sun)b(skp)
VOTTH (Zkp skep) dss
Shp
H/P
Σ
P(sun)b((spp) Vo
/ zap
Tep (Eupplopp dEseddoR
R=U
Skp
H/P
Σ
((sep) Voldsnp
k=0
0
Subtracting a state- and subpolicy- dependent baseline from the second term also unbiased, i.e.
H
E
V log T Zkp 2bp) =kp)] = 0
We'll follow the same strategy to prove the second equality: apply the tower property, express the expectation an integral, and undo the gradient-log trick.
E,
V log no, (a, 8t, Zkp (6 [kp))]
Ex 0z Rkp
Er Er Vo log To.. (aclse, 2kp (b(()) spp)
B Bee Vo log nH, (aclst, Zkp (b(k)pppppppp 2kp)]
1=0
P(se, Zkp (bpp))((s..))))p))))))))) 2kp) / Te, (aelsee 2kp) Vo log no, (aclse 2kp das
[=() (Bt,8kp)
P( 2kp (6( 2kp) V oldzppds:
(s1,0ep)
0
14

---

Published as a conference paper at ICLR 2020
With (x.y)
Average Return
Without (xy)
3
2-
0
50
100
150
lteration
Figure 7: HIRO performance Ant Gather with and without access to the ground truth (z, y), which it needs to communicate useful goals.
D HIRO SENSITIVITY OBSERVATION-SPACE
In this section we provide more detailed explanation of why HIRO (Nachum et al. 2018 performs poorly under our environments. As explained in our related work section, HIRO belongs to the general category of algorithms that train goal-reaching policies lower levels of the hierarchy (Vezhnevets) jet al. 2017 Levy et al. 2017). These methods rely on having goal-space that is meaningful for the task hand. For example, in navigation tasks they require having access to the (I, y) position of the agent such that deltas in that space can be given as meaningful goals to move in the environment. Unfortunately, in many cases the only readily available information (if there's 's no GPS signal or other positioning system installed) are raw sensory inputs, like cameras or the LIDAR sensors we mimic in our environments. such cases, our method still performs well because it doesn't rely on the goal-reaching extra supervision that is leveraged (and detrimental in this case) in HIRO and similar methods. Figure 7 we show that knowing the ground truth location critical for success. We have reproduced the HIRO results in Fig. 7 Jusiin the published codebase, so we are convinced that our results showcase a failure mode HIRO.
E HYPERPARAMETER SENSITIVITY PLOTS
AntGather Sensbliity to
Sreihn HPPO Surslisity to
Anarape
2:35, 46
2-20 30
15
200
402
5003
800
1000
2002
4000
600
8000
1000
Bemstoo
Bersston
Figure 8: Sensitivity of HiPPO to variation in the time-commitment.
AntSather HPPO: Senstiity to number of skllss
SnakerGather HPPO: Sensibyity to number sisklss
Aowngg Retm
2
1000
200
400
800
1000
200
400
800
1003
Iberation
ftextion
Figure 9: Sensitivity of HiPPO to variation in the number of skills.
15


Reviews:
[{"review_id": "SkllWnFOKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main problem they try to tackle is to train agents for unseen tasks and environmental changes. They show that their method has a better performance and is more robust against sensor errors and physical parameter alterations.\n\nThe authors clearly position their work in the HRL paradigm and explain current limitations/challenges within that field. Alike other HRL agents, their method has two types of policies (manager and subpolicies), but different from other works they do not keep parameters fixed in post training for new tasks. In addition to the parameters, they do not fix the time length. \n\nThe paper is very well written, clearly stated the contributions. \n\nRemarks:\n- Fig 2 has no caption. How are the colors of balls obtained, since they only explain how the sensors (lidar) measure distances to balls (bombs/apples).\n- Fig 4/5a, some agents (blue) seems to have undesired behaviour (until half of the iterations). This behaviour is not described anywhere. \n- The URL of the website with code and videos does not have any code. \n\nQuestions to the authors:\n- Closest work is Frans et al. (2018). The experiments do not show Frans et al. as a benchmark method. Why?\n- HiPPO shows to have a higher robustness. Why are the results of different methods (p=10/random) in Table 1 for different environments (Snake/Ant) different?\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BylQQdyTYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is under the topic of hierarchical reinforcement learning. The motivation of this paper is \"most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task.\" The paper proposes a method to learn higher-level skill selection and lower-level skill improvement jointly.\n\nWhat I like in this paper:\n    1. The paper, in general, is well-written so that I can understand it well.\n    2. Experiments are question-driven and provide interesting results.\n    3. Theories are closely related to the algorithm.\n\nKey reasons for my rejection:\n    1. My biggest concern is the motivation of this paper. \n    The joint learning of higher-level policy and lower-level skill discovery is not rare in modern literature. Some works are even cited in this paper, for example, option-critic, feudal network, etc. These methods fix their skills in the new task, not because they are inherently not able to do so, but because they want to demonstrate that the learned skills can be reused in new tasks, even if there is no further adaptation. I agree with the author that the agent needs to adapt its skills when faced with new tasks. But I don't think most works are limited in this aspect, as claimed by the paper in the abstract.\n\n    2. I think the author didn't justify his key design choices well. \n    This paper is under the research area of \"hierarchical reinforcement learning.\" However, just like temporal abstraction, the HRL is a general idea instead of an existing problem formulation or a particular algorithm. It seems that the author is not aware of this point as the paper claims a particular way of achieving HRL is the HRL itself (in section 4.1 \"In the context of HRL, a hierarchical policy with a manager πθh(zt|st) selects every p time-steps one of n sub-policies to execute.\"). I would like to see the paper takes the responsibility to justify the reason it follows this particular way. There are two more key decisions the paper proposed but not fully justified and analyzed.\n        1. Why is random length a valid choice? The paper doesn't tell readers the consequence of this design choice. For example, what about the optimality of the solution? Since bounding the random length needs prior knowledge, how difficult is it to come up with the prior knowledge. Is the algorithm sensitive to prior knowledge?\n        2. Why is it fine to assume \"for each action, there is just one sub-policy that gives it high probability\"? What would be the consequence of this assumption? Well, the extreme case is each action is only being chosen by one sub-policy. Therefore, executing the sub-policy becomes executing a repeated sequence of the same action. Obviously, this is a kind of temporal abstraction but is a very limited one. \n\nOther small issues:\nSection 2: \"... maximize the discounted expected reward ...\" should be \"... maximize the discounted expected return ...\".\nSection 2: the horizon T in the definition in \\eta should be H.\nSection 4: the advantage function is not defined.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers hierarchical reinforcement learning, and specifically the case where the learning and use of lower-level skills should not be decoupled. To this end the paper proposes Hierarchical Proximal Policy Optimization (HiPPO) to jointly learn the different layers of the hierarchy. This is compared against other hierarchical RL schemes on several Mujoco domains.

The reviewers raised three main issues with this paper. The first concerns an excluded baseline, which was included in the rebuttal. The other issues involve the motivation for the paper (in that there exist other methods that try and learn different levels of hierarchy together) and justification for some design choices. These were addressed to some extent in the rebuttal, but I believe this to still be an interesting contribution to the literature, and should be accepted.

Decision: Accept (Poster)