Sub-policy Adaptation for Hierarchical Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has made great progress in a variety of domains , from playing games such as Pong and Go ( Mnih et al. , 2015 ; Silver et al. , 2017 ) to automating robotic locomotion ( Schulman et al. , 2015 ; Heess et al. , 2017 ) , dexterous manipulation ( Florensa et al. , 2017b ; OpenAI et al. , 2018 ) , and perception ( Nair et al. , 2018 ; Florensa et al. , 2018 ) . Yet , most work in RL is still learning from scratch when faced with a new problem . This is particularly inefficient when tackling multiple related tasks that are hard to solve due to sparse rewards or long horizons . A promising technique to overcome this limitation is hierarchical reinforcement learning ( HRL ) ( Sutton et al. , 1999 ) . In this paradigm , policies have several modules of abstraction , allowing to reuse subsets of the modules . The most common case consists of temporal hierarchies ( Precup , 2000 ; Dayan & Hinton , 1993 ) , where a higher-level policy ( manager ) takes actions at a lower frequency , and its actions condition the behavior of some lower level skills or sub-policies . When transferring knowledge to a new task , most prior works fix the skills and train a new manager on top . Despite having a clear benefit in kick-starting the learning in the new task , having fixed skills can considerably cap the final performance on the new task ( Florensa et al. , 2017a ) . Little work has been done on adapting pre-trained sub-policies to be optimal for a new task . In this paper , we develop a new framework for simultaneously adapting all levels of temporal hierarchies . First , we derive an efficient approximated hierarchical policy gradient . The key insight is that , despite the decisions of the manager being unobserved latent variables from the point of view of the Markovian environment , from the perspective of the sub-policies they can be considered as part of the observation . We show that this provides a decoupling of the manager and sub-policy gradients , which greatly simplifies the computation in a principled way . It also theoretically justifies a technique used in other prior works ( Frans et al. , 2018 ) . Second , we introduce a sub-policy specific baseline for our hierarchical policy gradient . We prove that this baseline is unbiased , and our experiments reveal faster convergence , suggesting efficient gradient variance reduction . Then , we introduce a more stable way of using this gradient , Hierarchical Proximal Policy Optimization ( HiPPO ) . This method helps us take more conservative steps in our policy space ( Schulman et al. , 2017 ) , critical in hierarchies 1sites.google.com/view/hippo-rl ∗ Equal Contribution because of the interdependence of each layer . Results show that HiPPO is highly efficient both when learning from scratch , i.e . adapting randomly initialized skills , and when adapting pretrained skills on a new task . Finally , we evaluate the benefit of randomizing the time-commitment of the sub-policies , and show it helps both in terms of final performance and zero-shot adaptation on similar tasks . 2 PRELIMINARIES We define a discrete-time finitehorizon discounted Markov decision process ( MDP ) by a tuple M = ( S , A , P , r , ρ0 , γ , H ) , where S is a state set , A is an action set , P : S × A × S → R+ is the transition probability distribution , γ ∈ [ 0 , 1 ] is a discount factor , and H the horizon . Our objective is to find a stochastic policy πθ that maximizes the expected discounted return within the MDP , η ( πθ ) = Eτ [ ∑H t=0 γ tr ( st , at ) ] . We use τ = ( s0 , a0 , ... , ) to denote the entire state-action trajectory , where s0 ∼ ρ0 ( s0 ) , at ∼ πθ ( at|st ) , and st+1 ∼ P ( st+1|st , at ) . In this work , we propose a method to learn a hierarchical policy and efficiently adapt all the levels in the hierarchy to perform a new task . We study hierarchical policies composed of a higher level , or manager πθh ( zt|st ) , and a lower level , or sub-policy πθl ( at′ |zt , st′ ) . The higher level does not take actions in the environment directly , but rather outputs a command , or latent variable zt ∈ Z , that conditions the behavior of the lower level . We focus on the common case where Z = Zn making the manager choose among n sub-policies , or skills , to execute . The manager typically operates at a lower frequency than the sub-policies , only observing the environment every p time-steps . When the manager receives a new observation , it decides which low level policy to commit to for p environment steps by the means of a latent code z . Figure 1 depicts this framework where the high level frequency p is a random variable , which is one of the contribution of this paper as described in Section 4.4 . Note that the class of hierarchical policies we work with is more restrictive than others like the options framework , where the time-commitment is also decided by the policy . Nevertheless , we show that this loss in policy expressivity acts as a regularizer and does not prevent our algorithm from surpassing other state-of-the art methods . 3 RELATED WORK . There has been growing interest in HRL for the past few decades ( Sutton et al. , 1999 ; Precup , 2000 ) , but only recently has it been applied to high-dimensional continuous domains as we do in this work ( Kulkarni et al. , 2016 ; Daniel et al. , 2016 ) . To obtain the lower level policies , or skills , most methods exploit some additional assumptions , like access to demonstrations ( Le et al. , 2018 ; Merel et al. , 2019 ; Ranchod et al. , 2015 ; Sharma et al. , 2018 ) , policy sketches ( Andreas et al. , 2017 ) , or task decomposition into sub-tasks ( Ghavamzadeh & Mahadevan , 2003 ; Sohn et al. , 2018 ) . Other methods use a different reward for the lower level , often constraining it to be a “ goal reacher ” policy , where the signal from the higher level is the goal to reach ( Nachum et al. , 2018 ; Levy et al. , 2019 ; Vezhnevets et al. , 2017 ) . These methods are very promising for state-reaching tasks , but might require access to goal-reaching reward systems not defined in the original MDP , and are more limited when training on tasks beyond state-reaching . Our method does not require any additional supervision , and the obtained skills are not constrained to be goal-reaching . When transferring skills to a new environment , most HRL methods keep them fixed and simply train a new higher-level on top ( Hausman et al. , 2018 ; Heess et al. , 2016 ) . Other work allows for building on previous skills by constantly supplementing the set of skills with new ones ( Shu et al. , 2018 ) , but they require a hand-defined curriculum of tasks , and the previous skills are never fine-tuned . Our algorithm allows for seamless adaptation of the skills , showing no trade-off between leveraging the power of the hierarchy and the final performance in a new task . Other methods use invertible functions as skills ( Haarnoja et al. , 2018 ) , and therefore a fixed skill can be fully overwritten when a new layer of hierarchy is added on top . This kind of “ fine-tuning ” is promising , although similar to other works ( Peng et al. , 2019 ) , they do not apply it to temporally extended skills as we do here . One of the most general frameworks to define temporally extended hierarchies is the options framework ( Sutton et al. , 1999 ) , and it has recently been applied to continuous state spaces ( Bacon et al. , 2017 ) . One of the most delicate parts of this formulation is the termination policy , and it requires several regularizers to avoid skill collapse ( Harb et al. , 2017 ; Vezhnevets et al. , 2016 ) . This modification of the objective may be difficult to tune and affects the final performance . Instead of adding such penalties , we propose to have skills of a random length , not controlled by the agent during training of the skills . The benefit is two-fold : no termination policy to train , and more stable skills that transfer better . Furthermore , these works only used discrete action MDPs . We lift this assumption , and show good performance of our algorithm in complex locomotion tasks . There are other algorithms recently proposed that go in the same direction , but we found them more complex , less principled ( their per-action marginalization can not capture well the temporal correlation within each option ) , and without available code or evidence of outperforming non-hierarchical methods ( Smith et al. , 2018 ) . The closest work to ours in terms of final algorithm structure is the one proposed by Frans et al . ( 2018 ) . Their method can be included in our framework , and hence benefits from our new theoretical insights . We introduce a modification that is shown to be highly beneficial : the random timecommitment mentioned above , and find that our method can learn in difficult environments without their complicated training scheme . 4 EFFICIENT HIERARCHICAL POLICY GRADIENTS . When using a hierarchical policy , the intermediate decision taken by the higher level is not directly applied in the environment . Therefore , technically it should not be incorporated into the trajectory description as an observed variable , like the actions . This makes the policy gradient considerably harder to compute . In this section we first prove that , under mild assumptions , the hierarchical policy gradient can be accurately approximated without needing to marginalize over this latent variable . Then , we derive an unbiased baseline for the policy gradient that can reduce the variance of its estimate . Finally , with these findings , we present our method , Hierarchical Proximal Policy Optimization ( HiPPO ) , an on-policy algorithm for hierarchical policies , allowing learning at all levels of the policy jointly and preventing sub-policy collapse . 4.1 APPROXIMATE HIERARCHICAL POLICY GRADIENT . Policy gradient algorithms are based on the likelihood ratio trick ( Williams , 1992 ) to estimate the gradient of returns with respect to the policy parameters as ∇θη ( πθ ) = Eτ [ ∇θ logP ( τ ) R ( τ ) ] ≈ 1 N n∑ i=1 ∇θ logP ( τi ) R ( τi ) ( 1 ) = 1 N n∑ i=1 1 H H∑ t=1 ∇θ log πθ ( at|st ) R ( τi ) ( 2 ) In a temporal hierarchy , a hierarchical policy with a manager πθh ( zt|st ) selects every p time-steps one of n sub-policies to execute . These sub-policies , indexed by z ∈ Zn , can be represented as a single conditional probability distribution over actions πθl ( at|zt , st ) . This allows us to not only use a given set of sub-policies , but also leverage skills learned with Stochastic Neural Networks ( SNNs ) ( Florensa et al. , 2017a ) . Under this framework , the probability of a trajectory τ = ( s0 , a0 , s1 , . . . , sH ) can be written as P ( τ ) = ( H/p∏ k=0 [ n∑ j=1 πθh ( zj |skp ) ( k+1 ) p−1∏ t=kp πθl ( at|st , zj ) ] ) [ P ( s0 ) H∏ t=1 P ( st+1|st , at ) ] . ( 3 ) The mixture action distribution , which presents itself as an additional summation over skills , prevents additive factorization when taking the logarithm , as from Eq . 1 to 2 . This can yield numerical instabilities due to the product of the p sub-policy probabilities . For instance , in the case where all the skills are distinguishable all the sub-policies ’ probabilities but one will have small values , resulting in an exponentially small value . In the following Lemma , we derive an approximation of the policy gradient , whose error tends to zero as the skills become more diverse , and draw insights on the interplay of the manager actions . Lemma 1 . If the skills are sufficiently differentiated , then the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability . Let πθh ( z|s ) and πθl ( a|s , z ) be Lipschitz functions w.r.t . their parameters , and assume that 0 < πθl ( a|s , zj ) < ∀j 6= kp , then ∇θ logP ( τ ) = H/p∑ k=0 ∇θ log πθh ( zkp|skp ) + H∑ t=0 ∇θ log πθl ( at|st , zkp ) +O ( nH p−1 ) ( 4 ) Proof . See Appendix . Our assumption can be seen as having diverse skills . Namely , for each action there is just one sub-policy that gives it high probability . In this case , the latent variable can be treated as part of the observation to compute the gradient of the trajectory probability . Many algorithms to extract lowerlevel skills are based on promoting diversity among the skills ( Florensa et al. , 2017a ; Eysenbach et al. , 2019 ) , therefore usually satisfying our assumption . We further analyze how well this assumption holds in our experiments section and Table 2 .

Summary: This paper is under the topic of hierarchical reinforcement learning. The motivation of this paper is "most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task." The paper proposes a method to learn higher-level skill selection and lower-level skill improvement jointly.