Compositional Transfer in Hierarchical Reinforcement Learning

The successful application of flexible , general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency . To address the challenge , domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time . To this end , we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning ( RL ) . We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference . Furthermore , we demonstrate the benefits of additional incentives to efficiently decompose task solutions . Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives . We design an RL algorithm that enables stable and fast learning of hierarchical policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting for complex , real-world domains . Finally , we evaluate our algorithm in simulated environments as well as physical robot experiments and demonstrate substantial improvements in data data-efficiency over competitive baselines . 1 INTRODUCTION . While recent successes in deep ( reinforcement ) learning for computer games ( Atari ( Mnih et al. , 2013 ) , StarCraft ( Vinyals et al. , 2019 ) ) , Go ( Silver et al. , 2017 ) and other high-throughput domains , e.g . ( OpenAI et al. , 2018 ) , have demonstrated the potential of these methods in the big data regime , the high cost of data acquisition has so far limited progress in many tasks of real-world relevance . Data efficiency in machine learning generally relies on inductive biases to guide and accelerate the learning process ; e.g . by including expert domain knowledge of varying granularity . Incorporating such knowledge can accelerate learning – but when inaccurate it can also inappropriately bias the space of solutions and lead to sub-optimal results . Robotics represents a domain in which data efficiency is critical , and human prior knowledge is commonly provided . However , for scalability and reduced dependency on human accuracy , we can instead utilise an agent ’ s permanent embodiment and shared environment across tasks . Intuitively , such a scenario suggests the natural strategy of focusing on inductive biases that facilitate the sharing and reuse of experience and knowledge across tasks while other aspects of the domain can be learned . As a general principle this relieves us from the need to inject detailed knowledge about the domain , instead we can focus on general principles that facilitate reuse ( Caruana , 1997 ) . Successes for transfer learning have , for example , built on optimizing initial parameters ( e.g . Finn et al. , 2017 ) , sharing models and parameters across tasks either in the form of policies or value functions ( e.g . Rusu et al. , 2016 ; Teh et al. , 2017 ; Galashov et al. , 2018 ) , data-sharing across tasks ( e.g . Riedmiller et al. , 2018 ; Andrychowicz et al. , 2017 ) , or through the use of task-related auxiliary objectives ( Jaderberg et al. , 2016 ; Wulfmeier et al. , 2017 ) . Transfer between tasks can , however , lead to either constructive or destructive transfer for humans ( Singley and Anderson , 1989 ) as well as for machines ( Pan and Yang , 2010 ; Torrey and Shavlik , 2010 ) . That is , jointly learning to solve different tasks can provide both benefits and disadvantages for individual tasks , depending on their similarity . Finding a mechanism that enables transfer where possible but avoids interference is one of the long-standing research challenges . In this paper we explore the benefits and limitations of hierarchical policies in single and multitask reinforcement learning . Similar to Mixture Density Networks ( Bishop , 1994 ) our models represent policies as state-conditional Gaussian mixture distributions , with separate Gaussian mixture components as low-level policies which can be selected by the high-level controller via a categorical action choice . In the multitask setting , to obtain more robust and versatile low-level behaviors , we additionally shield the mixture components from information about the task at hand . In this case , task information is only communicated through the choice of mixture component by the high-level controller , and the mixture components can be seen as domain-dependant , task-independent skills although the nature of these skills is not predefined and emerges during end-to-end training . We implement this idea by building on three forms of transfer : targeted exploration via the concatenation of tasks within one episode ( Riedmiller et al. , 2018 ) , sharing transition data across tasks ( Andrychowicz et al. , 2017 ; Riedmiller et al. , 2018 ) , and reusing low-level components of the aforementioned policy class . To this end we develop a novel robust and data-efficient multitask actor-critic algorithm , Regularized Hierarchical Policy Optimization ( RHPO ) . Our algorithm uses the multitask learning aspects of SAC ( Riedmiller et al. , 2018 ) to improve data-efficiency and robust policy optimization properties of MPO ( Abdolmaleki et al. , 2018a ) in order to optimize hierarchical policies . We furthermore demonstrate the generality of hierarchical policies for multitask learning via improving results also after replacing MPO as policy optimizer with another gradient-based , entropy-regularized policy optimizer ( Heess et al. , 2015 ) ( see Appendix A.10 ) . We demonstrate that compositional , hierarchical policies – while strongly reducing training time in multitask domains – can fail to improve performance in single task domains if no additional inductive biases are given . While multitask domains provide sufficient pressure for component specialization , and the related possibility for composition , we are required to introduce additional incentives to encourage similar developments for single task domains . In the multitask setting , we demonstrate considerably improved performance , robustness and learning speed compared to competitive continuous control baselines demonstrating the relevance of hierarchy for data-efficiency and transfer . We finally evaluate our approach on a physical robot for robotic manipulation tasks where RHPO leads to a significant speed up in training , enabling it to solve challenging stacking tasks on a single robot 1 . 2 PRELIMINARIES . We consider a multitask reinforcement learning setting with an agent operating in a Markov Decision Process ( MDP ) consisting of the state space S , the action space A , the transition probability p ( st+1|st , at ) of reaching state st+1 from state st when executing action at at the previous time step t. The actions are drawn from a probability distribution over actions π ( a|s ) referred to as the agent ’ s policy . Jointly , the transition dynamics and policy induce the marginal state visitation distribution p ( s ) . Finally , the discount factor γ together with the reward r ( s , a ) gives rise to the expected reward , or value , of starting in state s ( and following π thereafter ) V π ( s ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , at ∼ π ( ·|st ) , st+1 ∼ p ( ·|st , at ) ] . Furthermore , we define multitask learning over a set of tasks i ∈ I with common agent embodiment as follows . We assume shared state , action spaces and shared transition dynamics across tasks ; tasks only differ in their reward function ri ( s , a ) . Furthermore , we consider task conditional policies π ( a|s , i ) . The overall objective is defined as J ( π ) = Ei∼I [ Eπ , p ( s0 ) [ ∞∑ t=0 γtri ( st , at ) |st+1 ∼ p ( ·|st , at ) ] ] = Ei∼I [ Eπ , p ( s ) [ Qπ ( s , a , i ) ] ] , ( 1 ) where all actions are drawn according to the policy π , that is , at ∼ π ( ·|st , i ) and we used the common definition of the state-action value function – here conditioned on the task – Qπ ( s , a , i ) = Eπ [ ∑∞ t=0 γ tri ( st , at ) |a0 = a , s0 = s , at ∼ π ( ·|st , i ) , st+1 ∼ p ( ·|st , at ) ] . 1Additional videos for task and component visualization are provided under https : //sites.google . com/view/rhpo/ 3 METHOD . This section introduces Regularized Hierarchical Policy Optimization ( RHPO ) which focuses on efficient training of modular policies by sharing data across tasks ; extending the data-sharing and scheduling mechanisms from Scheduled Auxiliary Control with randomized scheduling ( SAC-U ) ( Riedmiller et al. , 2018 ) . We start by introducing the considered class of policies , followed by the required combination – and extension – of MPO ( Abdolmaleki et al. , 2018a ) and SAC-U ( Riedmiller et al. , 2018 ) for training structured hierarchical policies in a multitask , off-policy setting . 3.1 HIERARCHICAL POLICIES . We start by defining the hierarchical policy class which supports sharing sub-policies across tasks . Formally , we decompose the per-task policy π ( a|s , i ) as πθ ( a|s , i ) = M∑ o=1 πLθ ( a|s , o ) πHθ ( o|s , i ) , ( 2 ) with πH and πL respectively representing a “ high-level ” switching controller ( a categorical distribution ) and a “ low-level ” sub-policy ( components of the resulting mixture distribution ) , where o is the index of the sub-policy . Here , θ denotes the parameters of both πH and πL , which we will seek to optimize . While the number of components has to be decided externally , the method is robust with respect to this parameter ( Appendix A.8.3 ) . Note that , in the above formulation only the high-level controller πH is conditioned on the task information i ; i.e . we employ a form of information asymmetry ( Galashov et al. , 2018 ; Tirumala et al. , 2019 ; Heess et al. , 2016 ) to enable the low-level policies to acquire general , task-independent behaviours . This choice strengthens decomposition of tasks across domains and inhibits degenerate cases of bypassing the high-level controller . Intuitively , these sub-policies can be understood as building reflex-like low-level control loops , which perform domain-dependent but task-independent behaviours and can be modulated by higher cognitive functions with knowledge of the task at hand . 3.2 DATA-EFFICIENT MULTITASK POLICY OPTIMIZATION . In the following sections , we present the equations underlying RHPO . For the complete pseudocode algorithm the reader is referred to the Appendix A.2.1 . To optimize the policy class described above we build on the MPO algorithm ( Abdolmaleki et al. , 2018a ) which decouples the policy improvement step ( optimizing J independently of the policy structure ) from the fitting of the hierarchical policy . Concretely , we first introduce an intermediate non-parametric policy q ( a|s , i ) and consider optimizing J ( q ) while staying close , in expectation , to a reference policy πref ( a|s , i ) max q J ( q ) = Ei∼I [ Eq , s∼D [ Q̂ ( s , a , i ) ] ] , s.t . Es∼D , i∼I [ KL ( q ( ·|s , i ) ‖πref ( ·|s , i ) ) ] ≤ , ( 3 ) where KL ( ·‖· ) denotes the Kullback Leibler divergence , defines a bound on the KL , D denotes the data contained in a replay buffer , and assuming that we have an approximation of the ground-truth state-action value function Q̂ ( s , a , i ) ≈ Qπ ( s , a , i ) available ( see Equation ( 4 ) for details on learning Q̂ from off-policy data ) . Starting from an initial policy πθ0 we can then iterate the following steps to improve the policy πθk : Policy Evaluation : Update Q̂ such that Q̂ ( s , a , i ) ≈ Q̂πθk ( s , a , i ) , see Equation ( 4 ) . Policy Improvement : – Step 1 : Obtain qk = arg maxq J ( q ) , under KL constraints with πref = πθk ( Equation ( 3 ) ) . – Step 2 : Obtain θk+1 = arg minθ Es∼D , i∼I [ KL ( qk ( ·|s , i ) ‖πθ ( ·|s , i ) ) ] , under additional regularization ( Equation ( 6 ) ) . Multitask Policy Evaluation For data-efficient off-policy learning of Q̂ we build on scheduled auxiliary control with uniform scheduling ( SAC-U ) ( Riedmiller et al. , 2018 ) which exploits two main ideas to obtain data-efficiency : i ) experience sharing across tasks ; ii ) switching between tasks within one episode for improved exploration . Formally , we assume access to a replay buffer containing data gathered from all tasks , which is filled asynchronously to the optimization ( similar to e.g . Espeholt et al . ( 2018 ) ) where for each trajectory snippet τ = { ( s0 , a0 , R0 ) , . . . , ( sL , aL , RL ) } we record the rewards for all tasks Rt = [ ri1 ( st , at ) , . . . , ri|I| ( st , at ) ] as a vector in the buffer . Using this data we define the retrace objective for learning Q̂ , parameterized via φ , following ( Munos et al. , 2016 ; Riedmiller et al. , 2018 ) as min φ L ( φ ) = ∑ i∼I Eτ∼D [ ( ri ( st , at ) + γQ ret ( st+1 , at+1 , i ) − Q̂φ ( st , at , i ) ) 2 ] , ( 4 ) where Qret is the L-step retrace target ( Munos et al. , 2016 ) , see the Appendix A.2.2 for details . Multitask Policy Improvement 1 : Obtaining Non-parametric Policies We first find the intermediate policy q by maximizing Equation ( 3 ) . We obtain a closed-form solution with a non-parametric policy for each task , as qk ( a|s , i ) ∝ πθk ( a|s , i ) exp ( Q̂ ( s , a , i ) η ) , ( 5 ) where η is a temperature parameter ( corresponding to a given bound ) which is optimized alongside the policy optimization ( see Appendix A.1.1 for a detailed derivation of the multitask case ) . As mentioned above , this policy representation is independent of the form of the parametric policy πθk ; i.e . q only depends on πθk through its density . This , crucially , makes it easy to employ complicated structured policies ( such as the one introduced in Section 3.1 ) . The only requirement here , and in the following steps , is that we must be able to sample from πθk and calculate the gradient ( w.r.t . θk ) of its log density ( but the sampling process itself need not be differentiable ) . Multitask Policy Improvement 2 : Fitting Parametric Policies In the second step we fit a policy to the non-parametric distribution obtained from the previous calculation by minimizing the divergence Es∼D , i∼I [ KL ( qk ( ·|s , i ) ‖πθ ( ·|s , i ) ) ] . Assuming that we can sample from qk this step corresponds to maximum likelihood estimation ( MLE ) . Furthermore , we can regularize towards smoothly changing distributions during training – effectively mitigating optimization instabilities and introducing an inductive bias – by limiting the change of the policy ( a trust-region constraint ) . The idea is commonly used in on- as well as in off-policy RL ( Schulman et al. , 2015 ; Abdolmaleki et al. , 2018b ; a ) . The application to hierarchical policy classes highlights the importance of this constraint as investigated in Section 4.2 . Formally , we aim to obtain the solution θk+1 = arg min θ Es∼D , i∼I [ KL ( qk ( ·|s , i ) ‖πθ ( ·|s , i ) ) ] = arg max θ Es∼D , i∼I [ Eπθk [ exp ( Q̂ ( s , a , i ) /η ) log M∑ o=1 πLθ ( a|s , o ) πHθ ( o|s , i ) ] ] , s.t . Es∼D , i∼I [ KL ( πHθk ( o|s , i ) ‖π H θ ( o|s , i ) ) + 1 M M∑ o=1 KL ( πLθk ( a|s , o ) ‖π L θ ( a|s , o ) ) ] < m , ( 6 ) where m defines a bound on the change of the new policy . Here we drop constant terms and the negative sign in the second line ( turning min into max ) , and explicitly insert the definition πθ ( a|s , i ) = ∑M o=1 πL ( a|s , o ) πH ( o|s , i ) , to highlight that we are marginalizing over the high-level choices in this fitting step ( since q is not tied to the policy structure ) . Hence , the update is independent of the specific policy component from which the action was sampled , enabling joint updates of all components . This reduces the variance of the update and also enables efficient off-policy learning . Different approaches can be used to control convergence for both the “ high-level ” categorical choices and the action choices to change slowly throughout learning . The average KL constraint in Equation ( 6 ) is similar in nature to an upper bound on the computationally intractable KL divergence between the two mixture distributions and has been determined experimentally to perform better in practice than simple bounds . In practice , in order to control the change of the high level and low level policies independently we decouple the constraints to be able to set different for the means ( µ ) , covariances ( Σ ) and the categorical distribution ( α ) in case of a mixture of Gaussian policy . To solve Equation ( 6 ) , we first employ Lagrangian relaxation to make it amenable to gradient based optimization and then perform a fixed number of gradient descent steps ( using Adam ( Kingma and Ba , 2014 ) ) ; details on this step , as well as an algorithm listing , can be found in the Appendix A.1.2 .

Summary: This paper introduces a hierarchical policy structure for use in both single task and multitask reinforcement learning. The authors then assess the usefulness of such a structure in both settings on complex robotic tasks. These tasks include the stacking and reaching of blocks using a robotic hand, as an example. In addition to carrying out these experiments on simulated robots, the authors have also carried out experiments on a real Sawyer robotic arm.