Title: COMPOSITIONAL TRANSFER IN HIERARCHICAL REINFORCEMENT LEARNING

Abstract: The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency. To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time. To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL). We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference. Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions. Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives. We design an RL algorithm that enables stable and fast learning of hierarchical policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting for complex, real-world domains. Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and demonstrate substantial improvements in data data-efficiency over competitive baselines.

1. INTRODUCTION:

While recent successes in deep (reinforcement) learning for computer games (Atari (Mnih et al., 2013) , StarCraft (Vinyals et al., 2019) ), Go (Silver et al., 2017) and other high-throughput domains, e.g. (OpenAI et al., 2018) , have demonstrated the potential of these methods in the big data regime, the high cost of data acquisition has so far limited progress in many tasks of real-world relevance. Data efficiency in machine learning generally relies on inductive biases to guide and accelerate the learning process; e.g. by including expert domain knowledge of varying granularity. Incorporating such knowledge can accelerate learning -but when inaccurate it can also inappropriately bias the space of solutions and lead to sub-optimal results. Robotics represents a domain in which data efficiency is critical, and human prior knowledge is commonly provided. However, for scalability and reduced dependency on human accuracy, we can instead utilise an agent's permanent embodiment and shared environment across tasks. Intuitively, such a scenario suggests the natural strategy of focusing on inductive biases that facilitate the sharing and reuse of experience and knowledge across tasks while other aspects of the domain can be learned. As a general principle this relieves us from the need to inject detailed knowledge about the domain, instead we can focus on general principles that facilitate reuse (Caruana, 1997) .

Successes for transfer learning have, for example, built on optimizing initial parameters (e.g. Finn et al., 2017) , sharing models and parameters across tasks either in the form of policies or value functions (e.g. Rusu et al., 2016; Teh et al., 2017; Galashov et al., 2018) , data-sharing across tasks (e.g. Riedmiller et al., 2018; Andrychowicz et al., 2017) , or through the use of task-related auxiliary objectives (Jaderberg et al., 2016; Wulfmeier et al., 2017) . Transfer between tasks can, however, lead to either constructive or destructive transfer for humans (Singley and Anderson, 1989) as well as for machines (Pan and Yang, 2010; Torrey and Shavlik, 2010) . That is, jointly learning to solve different tasks can provide both benefits and disadvantages for individual tasks, depending on their similarity. Finding a mechanism that enables transfer where possible but avoids interference is one of the long-standing research challenges.

In this paper we explore the benefits and limitations of hierarchical policies in single and multitask reinforcement learning. Similar to Mixture Density Networks (Bishop, 1994) our models represent policies as state-conditional Gaussian mixture distributions, with separate Gaussian mixture components as low-level policies which can be selected by the high-level controller via a categorical action choice. In the multitask setting, to obtain more robust and versatile low-level behaviors, we additionally shield the mixture components from information about the task at hand. In this case, task information is only communicated through the choice of mixture component by the high-level controller, and the mixture components can be seen as domain-dependant, task-independent skills although the nature of these skills is not predefined and emerges during end-to-end training.

We implement this idea by building on three forms of transfer: targeted exploration via the concatenation of tasks within one episode (Riedmiller et al., 2018) , sharing transition data across tasks (Andrychowicz et al., 2017; Riedmiller et al., 2018) , and reusing low-level components of the aforementioned policy class. To this end we develop a novel robust and data-efficient multitask actor-critic algorithm, Regularized Hierarchical Policy Optimization (RHPO). Our algorithm uses the multitask learning aspects of SAC (Riedmiller et al., 2018) to improve data-efficiency and robust policy optimization properties of MPO (Abdolmaleki et al., 2018a) in order to optimize hierarchical policies. We furthermore demonstrate the generality of hierarchical policies for multitask learning via improving results also after replacing MPO as policy optimizer with another gradient-based, entropy-regularized policy optimizer (Heess et al., 2015) (see Appendix A.10) .

We demonstrate that compositional, hierarchical policies -while strongly reducing training time in multitask domains -can fail to improve performance in single task domains if no additional inductive biases are given. While multitask domains provide sufficient pressure for component specialization, and the related possibility for composition, we are required to introduce additional incentives to encourage similar developments for single task domains. In the multitask setting, we demonstrate considerably improved performance, robustness and learning speed compared to competitive continuous control baselines demonstrating the relevance of hierarchy for data-efficiency and transfer. We finally evaluate our approach on a physical robot for robotic manipulation tasks where RHPO leads to a significant speed up in training, enabling it to solve challenging stacking tasks on a single robot1 .

2. PRELIMINARIES:

We consider a multitask reinforcement learning setting with an agent operating in a Markov Decision Process (MDP) consisting of the state space S, the action space A, the transition probability p(s t+1 |s t , a t ) of reaching state s t+1 from state s t when executing action a t at the previous time step t. The actions are drawn from a probability distribution over actions π(a|s) referred to as the agent's policy. Jointly, the transition dynamics and policy induce the marginal state visitation distribution p(s). Finally, the discount factor γ together with the reward r(s, a) gives rise to the expected reward, or value, of starting in state s (and following π thereafter)

V π (s) = E π [ ∞ t=0 γ t r(s t , a t )|s 0 = s, a t ∼ π(•|s t ), s t+1 ∼ p(•|s t , a t )]

. Furthermore, we define multitask learning over a set of tasks i ∈ I with common agent embodiment as follows. We assume shared state, action spaces and shared transition dynamics across tasks; tasks only differ in their reward function r i (s, a). Furthermore, we consider task conditional policies π(a|s, i). The overall objective is defined as

J(π) = E i∼I E π,p(s0) ∞ t=0 γ t r i (s t , a t ) |s t+1 ∼ p(•|s t , a t ) = E i∼I E π,p(s) Q π (s, a, i) ,

(1) where all actions are drawn according to the policy π, that is, a t ∼ π(•|s t , i) and we used the common definition of the state-action value function -here conditioned on the task -

Q π (s, a, i) = E π [ ∞ t=0 γ t r i (s t , a t ) |a 0 = a, s 0 = s, a t ∼ π(•|s t , i), s t+1 ∼ p(•|s t , a t )].

3. METHOD:

This section introduces Regularized Hierarchical Policy Optimization (RHPO) which focuses on efficient training of modular policies by sharing data across tasks; extending the data-sharing and scheduling mechanisms from Scheduled Auxiliary Control with randomized scheduling (SAC-U) (Riedmiller et al., 2018) . We start by introducing the considered class of policies, followed by the required combination -and extension -of MPO (Abdolmaleki et al., 2018a) and SAC-U (Riedmiller et al., 2018) for training structured hierarchical policies in a multitask, off-policy setting.

3.1. HIERARCHICAL POLICIES:

We start by defining the hierarchical policy class which supports sharing sub-policies across tasks. Formally, we decompose the per-task policy π(a|s, i) as

EQUATION

with π H and π L respectively representing a "high-level" switching controller (a categorical distribution) and a "low-level" sub-policy (components of the resulting mixture distribution), where o is the index of the sub-policy. Here, θ denotes the parameters of both π H and π L , which we will seek to optimize. While the number of components has to be decided externally, the method is robust with respect to this parameter (Appendix A.8.3).

Note that, in the above formulation only the high-level controller π H is conditioned on the task information i; i.e. we employ a form of information asymmetry (Galashov et al., 2018; Tirumala et al., 2019; Heess et al., 2016) to enable the low-level policies to acquire general, task-independent behaviours. This choice strengthens decomposition of tasks across domains and inhibits degenerate cases of bypassing the high-level controller. Intuitively, these sub-policies can be understood as building reflex-like low-level control loops, which perform domain-dependent but task-independent behaviours and can be modulated by higher cognitive functions with knowledge of the task at hand.

3.2. DATA-EFFICIENT MULTITASK POLICY OPTIMIZATION:

In the following sections, we present the equations underlying RHPO. For the complete pseudocode algorithm the reader is referred to the Appendix A.2.1. To optimize the policy class described above we build on the MPO algorithm (Abdolmaleki et al., 2018a) which decouples the policy improvement step (optimizing J independently of the policy structure) from the fitting of the hierarchical policy. Concretely, we first introduce an intermediate non-parametric policy q(a|s, i) and consider optimizing J(q) while staying close, in expectation, to a reference policy π ref (a|s, i)

EQUATION

where KL(• •) denotes the Kullback Leibler divergence, defines a bound on the KL, D denotes the data contained in a replay buffer, and assuming that we have an approximation of the ground-truth state-action value function Q(s, a, i) ≈ Q π (s, a, i) available (see Equation ( 4) for details on learning Q from off-policy data). Starting from an initial policy π θ0 we can then iterate the following steps to improve the policy π θ k :

Policy Evaluation: Update Q such that Q(s, a, i) ≈ Qπ θ k (s, a, i), see Equation (4).

Policy Improvement:

-Step 1: Obtain q k = arg max q J(q), under KL constraints with

π ref = π θ k (Equation (3)).

-Step 2: Obtain θ k+1 = arg min θ E s∼D,i∼I KL q k (•|s, i) π θ (•|s, i) , under additional regularization (Equation ( 6)).

Multitask Policy Evaluation:

For data-efficient off-policy learning of Q we build on scheduled auxiliary control with uniform scheduling (SAC-U) (Riedmiller et al., 2018) which exploits two main ideas to obtain data-efficiency: i) experience sharing across tasks; ii) switching between tasks within one episode for improved exploration.

Formally, we assume access to a replay buffer containing data gathered from all tasks, which is filled asynchronously to the optimization (similar to e.g. Espeholt et al. (2018) ) where for each trajectory snippet τ = {(s 0 , a 0 , R 0 ), . . . , (s L , a L , R L )} we record the rewards for all tasks R t = [r i1 (s t , a t ), . . . , r i |I| (s t , a t )] as a vector in the buffer. Using this data we define the retrace objective for learning Q, parameterized via φ, following (Munos et al., 2016; Riedmiller et al., 2018) as

EQUATION

)

where Q ret is the L-step retrace target (Munos et al., 2016) , see the Appendix A.2.2 for details.

Multitask Policy Improvement 1: Obtaining Non-parametric Policies We first find the intermediate policy q by maximizing Equation (3). We obtain a closed-form solution with a non-parametric policy for each task, as

EQUATION

)

where η is a temperature parameter (corresponding to a given bound ) which is optimized alongside the policy optimization (see Appendix A.1.1 for a detailed derivation of the multitask case). As mentioned above, this policy representation is independent of the form of the parametric policy π θ k ; i.e. q only depends on π θ k through its density. This, crucially, makes it easy to employ complicated structured policies (such as the one introduced in Section 3.1). The only requirement here, and in the following steps, is that we must be able to sample from π θ k and calculate the gradient (w.r.t. θ k ) of its log density (but the sampling process itself need not be differentiable).

Multitask Policy Improvement 2: Fitting Parametric Policies In the second step we fit a policy to the non-parametric distribution obtained from the previous calculation by minimizing the divergence E s∼D,i∼I [KL(q k (•|s, i) π θ (•|s, i))]. Assuming that we can sample from q k this step corresponds to maximum likelihood estimation (MLE). Furthermore, we can regularize towards smoothly changing distributions during training -effectively mitigating optimization instabilities and introducing an inductive bias -by limiting the change of the policy (a trust-region constraint). The idea is commonly used in on-as well as in off-policy RL (Schulman et al., 2015; Abdolmaleki et al., 2018b; a) . The application to hierarchical policy classes highlights the importance of this constraint as investigated in Section 4.2. Formally, we aim to obtain the solution where m defines a bound on the change of the new policy. Here we drop constant terms and the negative sign in the second line (turning min into max), and explicitly insert the definition π θ (a|s, i) = M o=1 π L (a|s, o) π H (o|s, i), to highlight that we are marginalizing over the high-level choices in this fitting step (since q is not tied to the policy structure). Hence, the update is independent of the specific policy component from which the action was sampled, enabling joint updates of all components. This reduces the variance of the update and also enables efficient off-policy learning. Different approaches can be used to control convergence for both the "high-level" categorical choices and the action choices to change slowly throughout learning. The average KL constraint in Equation ( 6) is similar in nature to an upper bound on the computationally intractable KL divergence between the two mixture distributions and has been determined experimentally to perform better in practice than simple bounds. In practice, in order to control the change of the high level and low level policies independently we decouple the constraints to be able to set different for the means ( µ ), covariances ( Σ ) and the categorical distribution ( α ) in case of a mixture of Gaussian policy. To solve Equation (6), we first employ Lagrangian relaxation to make it amenable to gradient based optimization and then perform a fixed number of gradient descent steps (using Adam (Kingma and Ba, 2014) ); details on this step, as well as an algorithm listing, can be found in the Appendix A.1.2.

4. EXPERIMENTS:

In the following sections, we investigate the effects of training hierarchical policies in single and multitask domains, finally demonstrating how RHPO can provide compelling benefits for multitask learning in real and simulated robotic manipulation tasks and significantly reduce platform interaction time. In the context of single-task domains from the DeepMind Control Suite (Tassa et al., 2018) , we first demonstrate how this type of hierarchy on its own fails to improve performance and that for the model to exploit compositionality, additional incentives for component specialization are required. Subsequently, we introduce suited incentives leading to improved performance and demonstrate that the variety of objectives in multitask domains can serve the same purpose. The evaluation includes experiments on physical hardware with robotic manipulation tasks for the Sawyer arm, emphasizing the importance of data-efficiency.

More details on task hyperparameters as well as the results for additional ablations and all tasks from the multitask domains are provided in the Appendix A.4. Across all tasks, we build on a distributed actor-critic framework (similar to (Espeholt et al., 2018) ) with flexible hardware assignment (Buchlovsky et al., 2019) to train all agents, performing critic and policy updates from a replay buffer, which is asynchronously filled by a set of actors. In all figures with error bars, we visualize mean and variance derived from 3 runs. We consider two high-dimensional tasks for continuous control: humanoid-run and humanoid-stand from Tassa et al. (2018) and compare a flat Gaussian policy to a hierarchical policy, a mixture of Gaussians with three components. We align the update rates of all approaches for fair comparison and to focus the comparison of the algorithm and not its specific implementation2 . Figure 1 visualizes the results in terms of the number of actor episodes.

4.1. SIMULATED SINGLE TASK EXPERIMENTS:

As can be observed, the hierarchical policy performs comparable to a flat policy with well aligned means and variances for all components as the model fails to decompose the problem. While both the flat and hierarchical policy are initialized with means close to zero, we now include another hierarchical policy with distributed initial means for the three components ranging for all dimensions from minimum to maximum of the allowed action range (here: -1, 0, 1). This simple change suffices to enable component specialization and significantly improved performance.

4.2. SIMULATED MULTITASK EXPERIMENTS:

We use three simulated multitask scenarios with the Kinova Jaco and Rethink Robotics Sawyer robot arms to test in a variety of conditions. Pile1: Here, the seven tasks of interest range from simple reaching for a block over tasks like grasping it, to the final task of stacking the block on top of another block. In addition to the experiments in simulation, which are executed with 5 actors in a distributed setting, the same Pile1 multitask domain (same rewards and setup) is investigated with a single, physical robot in Section 4.3. We further extend the evaluation towards two more complex multitask domains in simulation. The first extension includes stacking with both blocks on top of the respective other block, resulting in a setting with 10 tasks (Pile2). And a last domain including harder tasks such as opening a box and placing blocks into this box, consisting of a total of 13 tasks (Cleanup2).

We compare RHPO for training hierarchical policies against a flat, monolithic policy shared across all tasks which is provided with the additional task id as input (displayed as Monolithic in the plot) as well as policies with task dependent heads (displayed as Independent in the plots) following (Riedmiller et al., 2018) -both using MPO as the optimizer and a re-implementation of SAC-U using SVG (Heess et al., 2015) (which is related to a version of the option critic (Bacon et al., 2017) without temporal abstraction). The baselines provide the two opposite, naive perspectives on transfer: by using the same monolithic policy across tasks we enable positive as well as negative interference and independent policies prevent policy-based transfer. After experimentally confirming the robustness of RHPO with respect to the number of low-level sub-policies (see Appendix A.8.3), we set M proportional to the number of tasks in each domain. to the performance of the SVG-based implementation of SAC-U. From left to right: Pile1, Pile2, Cleanup2. We show averages over 3 runs each, with corresponding standard deviation. RHPO outperforms both baselines across all tasks with the benefits increasing for more complex domains.

Figure 2 demonstrates that the hierarchical policy (RHPO) outperforms the monolithic as well as the independent baselines. For simple tasks such as Pile1, the difference is smaller, but the more tasks are trained and the more complex the domain becomes (cf. Pile2 and Cleanup2), the greater is the advantage of composing learned behaviours across tasks. Compared to SVG (Heess et al., 2015) , we observe that the baselines based on MPO already result in an improvement, which becomes even bigger with the hierarchical policies. The results across all domains exhibit performance gains for the hierarchical model without the additional incentives from Section 4.1, demonstrating the sufficiency of variety in the training objectives to encourage component specialization and problem decomposition.

4.3. PHYSICAL ROBOT EXPERIMENTS:

For real-world experiments, data-efficiency is crucial. We perform all experiments in this section relying on a single robot (single actor) -demonstrating the benefits of RHPO in the low data regime. The performed task is the real world version of the Pile1 task described in Section 4.2. The main task objective is to stack one cube onto a second one and move the gripper away from it. We introduce an additional third cube which serves purely as a distractor. The setup for the experiments consists of a Sawyer robot arm mounted on a table, equipped with a Robotiq 2F-85 parallel gripper. A basket of size 20cm 2 in front of the robot contains the three cubes. Three cameras on the basket track the cubes using fiducials (augmented reality tags). As in simulation, the agent is provided with proprioception information (joint positions, velocities and torques), a wrist sensor's force and torque readings, as well as the cubes' poses -estimated via the fiducials. The agent action is five dimensional and consists of the three Cartesian translational velocities, the angular velocity of the wrist around the vertical axis and the speed of the gripper's fingers.

Figure 4 plots the learning progress on the real robot for two (out of 7) of the tasks, the simple reach tasks and the stack task -which is the main task of interest. Plots for the learning progress of all tasks are given in the appendix A.6. As can be observed, all methods manage to learn the reach task quickly (within about a few thousand episodes) but only RHPO with a hierarchical policy is able to learn the stacking task (taking about 15 thousand episodes to obtain good stacking success), which takes about 8 days of training on the real robot with considerably slower progress for all baselines. In addition, we compute distributions for each component over the tasks which activate it, as well as distributions for each task over which components are being used. For each set of distributions, we determine the Battacharyya distance metric to determine the similarity between tasks and the similarity between components in Figure 4 (right). The plots demonstrate how the components specialise, but also provide a way to investigate our tasks, showing e.g. that the first reach task is fairly independent and that the last four tasks are comparably similar regarding the high-level components applied for their solution.

4.3.1. ADDITIONAL ABLATIONS:

We perform a series of ablations based on the earlier introduced Pile1 domain, providing additional insights into benefits and shortcomings of RHPO and important factors for robust training. The algorithm is well suited for sequential transfer learning based on solving new tasks with pre-trained low-level components (Appendix A.9). We demonstrate the robustness of RHPO with respect to the number of sub-policies as well as importance of choice of regularization respectively in Appendix A.8.1 and A.8.3. Finally, we ablate over the number of data-generating actors to evaluate all approaches with respect to data rate and illustrate how hierarchical policies are particularly relevant at lower data rates such as given by real-world robotics applications in Appendix A.8.2.

5. RELATED WORK:

Transfer learning, in particular in the multitask context, has long been part of machine learning (ML) for data-limited domains (Caruana, 1997; Torrey and Shavlik, 2010; Pan and Yang, 2010; Taylor and Stone, 2009) . Commonly, it is not straightforward to train a single model jointly across different tasks as the solutions to tasks might not only interfere positively but also negatively (Wang et al., 2018) . Preventing this type of forgetting or negative transfer presents a challenge for biological (Singley and Anderson, 1989) as well as artificial systems (French, 1999) . In the context of ML, a common scheme is the reduction of representational overlap (French, 1999; Rusu et al., 2016; Wang et al., 2018) . Bishop (1994) utilize neural networks to parametrize mixture models for representing multi-modal distributions thus mitigating shortcomings of non-hierarchical approaches. Rosenstein et al. (2005) demonstrate the benefits of hierarchical classification models to limit the impact of negative transfer.

Hierarchical approaches have a long history in the reinforcement learning literature (e.g. Sutton et al., 1999; Dayan and Hinton, 1993) . Prior work commonly benefits from combining hierarchy with additional inductive biases such as (Vezhnevets et al., 2017; Nachum et al., 2018a; b; Xie et al., 2018) which employ different rewards for different levels of the hierarchy rather than optimizing a single objective for the entire model as we do. Other works have shown the additional benefits for the stability of training and data-efficiency when sequences of high-level actions are given as guidance during optimization in a hierarchical setting (Shiarlis et al., 2018; Andreas et al., 2017; Tirumala et al., 2019) . Instead of introducing additional training signals, we directly investigate the benefits of compositional hierarchy as provided structure for transfer between tasks.

Hierarchical models for probabilistic trajectory modelling have been used for the discovery of behavior abstractions as part of an end-to-end reinforcement learning paradigm (e.g. Teh et al., 2017; Igl et al., 2019; Tirumala et al., 2019; Galashov et al., 2018) where the models act as learned inductive biases that induce the sharing of behavior across tasks. In a vein similar to the presented algorithm, (e.g Heess et al., 2016; Tirumala et al., 2019) share a low-level controller across tasks but modulate the low-level behavior via a continuous embedding rather than picking from a small number of mixture components. In related work Hausman et al. (2018) ; Haarnoja et al. (2018) learn hierarchical policies with continuous latent variables optimizing the entropy regularized objective. Similar to our work, the options framework (Sutton et al., 1999; Precup, 2000) supports behavior hierarchies, where the higher level chooses from a discrete set of sub-policies or "options" which commonly are run until a termination criterion is satisfied. The framework focuses on the notion of temporal abstraction. A number of works have proposed practical and scalable algorithms for learning option policies with reinforcement learning (e.g. Bacon et al., 2017; Zhang and Whiteson, 2019; Smith et al., 2018; Riemer et al., 2018; Harb et al., 2018) or criteria for option induction (e.g. Harb et al., 2018; Harutyunyan et al., 2019) . Rather than the additional inductive bias of temporal abstraction, we focus on the investigation of composition as type of hierarchy in the context of single and multitask learning while demonstrating the strength of hierarchical composition to lie in domains with strong variation in the objectives -such as in multitask domains. We additionally introduce a hierarchical extension of SVG (Heess et al., 2015) , to investigate similarities to work on the option critic (Bacon et al., 2017) .

With the use of KL regularization to different ends in RL, work related to RHPO focuses on contextual bandits (Daniel et al., 2016) . The algorithm builds on a 2-step EM like procedure to optimize linearly parametrized mixture policies. However, their algorithm has been used only with low dimensional policy representations, and in contextual bandit and other very short horizon settings. Our approach is designed to be applicable to full RL problems in complex domains with long horizons and with high-capacity function approximators such as neural networks. This requires robust estimation of value function approximations, off-policy correction, and additional regularization for stable learning.

6. DISCUSSION:

We introduce a novel framework to enable robust training and investigation of hierarchical, compositional policies in complex simulated and real-world tasks as well as provide insights into the learning process and its stability. In simulation as well as on real robots, RHPO outperforms baseline methods which either handle tasks independently or utilize implicit sharing. Especially with increasingly complex tasks or limited data rate, as given in real-world applications, we demonstrate hierarchical inductive biases to provide a compelling foundation for transfer learning, reducing the number of environment interactions significantly and often leading to more robust learning as well as improved final performance. For single tasks with a single training objective all components can remain aligned, preventing problem decomposition and the hierarchical policy replicates a flat policy. Performance improvements appear only when the individual components specialize, either via variety in the training objectives or additional incentives. Furthermore, as demonstrated in Appendix A.9, a pre-trained set of specialized components can notably improve performance when learning new tasks. One important next step is identifying how to optimize a basis set of components which transfers well to a wide range of tasks Since with mixture distributions, we are able to marginalize over components when optimizing the weighted likelihood over action samples in Equation 6, the extension towards multiple levels of hierarchy is trivial but can provide a valuable direction for practical future work. While this approach partially mitigates negative interference between tasks in a parallel multitask learning scenario, addressing catastrophic inference in sequential settings remains a challenge.

We believe that especially in domains with consistent agent embodiment and high costs for data generation learning tasks jointly and information sharing is imperative. RHPO combines several ideas that we believe will be important: multitask learning with hierarchical and compositional policy representations, robust optimization, and efficient off-policy learning. Although we have found this particular combination of components to be very effective we believe it is just one instance of -and step towards -a spectrum of efficient learning architectures that will unlock further applications of RL both in simulation and, importantly, on physical hardware.

A APPENDIX:

A.1 ADDITIONAL DERIVATIONS In this section we explain the detailed derivations for training hierarchical policies parameterized as a mixture of Gaussians.

A.1.1 OBTAINING NON-PARAMETRIC POLICIES:

In each policy improvement step, to obtain non-parametric policies for a given state and task distribution, we solve the following program:

max q E µ(s),i∼I E q(a|s,i) Q(s, a, i)] s.t.E µ(s),i∼I KL(q(•|s, i), π(•|s, i, θ t )) < s.t.E µ(s),i∼I E q(a|s) 1 = 1.

To make the following derivations easier to follow we open up the expectations, writing them as integrals explicitly. For this purpose let us define the joint distribution over states s ∼ µ(s) together with randomly sampled tasks i ∼ I as µ(s, i) = p(s|D)U(i ∈ I), where U denotes the uniform distribution over possible tasks. This allows us to re-write the expectations that include the corresponding distributions, i.e. E µ(s),i∼I [1] = E µ(s,i) [1], but again, note that i here is not necessarily the task under which s was observed. We can then write the Lagrangian equation corresponding to the above described program as L(q,η, γ) = µ(s, i) q(a|s, i) Q(s, a, i) da ds di + η -µ(s, i) q(a|s, i) log q(a|s, i) π(a|s, i, θ t ) da ds di + γ 1 -µ(s, i)q(a|s, i) da ds di .

Next we maximize the Lagrangian L w.r.t the primal variable q. The derivative w.r.t q reads, ∂qL(q, η, γ) = Q(a, s, i) -η log q(a|s, i)

+ η log π(a|s, i, θ t ) -η -γ.

Setting it to zero and rearranging terms we obtain

q(a|s, i) = π(a|s, i, θ t ) exp Q(s, a, i) η exp - η + γ η .

However, the last exponential term is a normalization constant for q. Therefore we can write,

EQUATION

Now, to obtain the dual function g(η), we plug in the solution to the KL constraint term (second term) of the Lagrangian which yields L(q, η, γ) = µ(s, i) q(a|s, i)Q(s, a, i) da ds + η -µ(s, i) q(a|s, i) log π(a|s,i,θ t ) exp ( Q(s,a,i) η

) exp ( -η+γ η )

π(a|s,i,θ t ) da ds di + γ 1 -µ(s, i)q(a|s, i) da ds di .

After expanding and rearranging terms we get L(q, η, γ) = µ(s, i) q(a|s, i)Q(s, a, i) da ds di -η µ(s, i) q(a|s, i) Q(s, a, i) η + log π(a|s, i; θ t ) -η + γ η da ds di + η + η µ(s, i) q(a|s, i) log π(a|s, i; θ t ) da ds di + γ 1 -µ(s, i)q(a|s, i) da ds di .

Most of the terms cancel out and after rearranging the terms we obtain

L(q, η, γ) = η + η µ(s, i) η + γ η ds di.

Note that we have already calculated the term inside the integral in Equation 7. By plugging in equation 7 we obtain the dual function

EQUATION

which we can minimize with respect to η based on samples from the replay buffer.

A.1.2 EXTENDED UPDATE RULES FOR FITTING A MIXTURE OF GAUSSIANS:

After obtaining the non parametric policies, we fit a parametric policy to samples from said nonparametric policies -effectively employing using maximum likelihood estimation with additional regularization based on a distance function T , i.e,

EQUATION

where D is an arbitrary distance function to evaluate the change of the new policy with respect to a reference/old policy, and denotes the allowed change for the policy. To make the above objective amenable to gradient based optimization we employ Lagrangian Relaxation, yielding the following primal:

EQUATION

We solve for θ by iterating the inner and outer optimization programs independently: We fix the parameters θ to their current value and optimize for the Lagrangian multipliers (inner minimization) and then we fix the Lagrangian multipliers to their current value and optimize for θ (outer maximization).

In practice we found that it is effective to simply perform one gradient step each in inner and outer optimization for each sampled batch of data.

The optimization given above is general, i.e. it works for any general type of policy. As described in the main paper, we consider hierarchical policies of the form

π θ (a|s, i) =

This decoupling allows us to set different values for the change of means, covariances and categorical probabilities, i.e., µ , Σ , α . Different lead to different learning rates. We always set a much smaller epsilon for the covariance and categorical than for the mean. The intuition is that while we would like the distribution to converge quickly in the action space, we also want to keep the exploration both locally (via the covariance matrix) and globally (via the high level categorical distribution) to avoid premature convergence.

A.2 ALGORITHMIC DETAILS:



A.2.1 PSEUDOCODE FOR THE FULL PROCEDURE:

We provide a pseudo-code listing of the full optimization procedure -and the asynchronous data gathering -performed by RHPO in Algorithm 1 and 2. The implementation relies on Sonnet (Reynolds et al., 2017) and TensorFlow (Abadi et al., 2015) .

Algorithm log π θ (a j |s t , i) following Eq. 6

δ η ← ∇ η g(η) = ∇ η η + η st∈τ log 1 Ns Ns j=1 exp Q(st,aj ,i) η

following Eq. 8

δ Q ← ∇ φ i∼I (st,at)∈τ Qφ (s t , a t , i) -Q ret 2 with Q ret following Eq. 4 // apply gradient updates π θ k+1 = optimizer_update(π, δ π ), η = optimizer_update(η, δ η ) Qφ = optimizer_update( Qφ , δ Q ) k = k + 1 end for // update target networks π = π, Q = Q end while

A.2.2 DETAILS ON THE POLICY IMPROVEMENT STEP:

As described in the main paper we consider the same setting as scheduled auxiliary control setting (SAC-X) (Riedmiller et al., 2018) to perform policy improvement (with uniform random switches between tasks every N steps within an episode, the SAC-U setting).

Given a replay buffer containing data gathered from all tasks, where for each trajectory snippet τ = {(s 0 , a 0 , R 0 ), . . . , (s L , a L , R L )} we record the rewards for all tasks R t = [r i1 (s t , a t ), . . . , r i |I| (s t , a t )] as a vector in the buffer, we define the retrace objective for learning Q, parameterized via φ, following Riedmiller et al. (2018) as

EQUATION

)

where the importance weights are defined as c k = min (1, π θ k (a k |s k ,i) /b(a k |s k )), with b(a k |s k ) denoting an arbitrary behavior policy; in particular this will be the policy for the executed tasks during an architecture in all multitask experiments, we investigate variations of the policy architecture (left sub-figure) in this paper (see Figure 6 ). append the task-id as a one-hot encoded vector to the input. For the hierarchical architecture, we build on the same torso and create a set of networks parameterizing the Gaussians which are shared across tasks and a task-specific network to parameterize the categorical distribution for each task.

The final mixture distribution is task-dependent for the high-level controller but task-independent for the low-level policies.

A.2.4 ALGORITHM HYPERPARAMETERS:

In this section we outline the details on the hyperparameters used for RHPO and baselines in both single task and multitask experiments. All experiments use feed-forward neural networks. We consider a flat policy represented by a Gaussian distribution and a hierarchical policy represented by a mixture of Gaussians distribution. The flat policy is given by a Gaussian distribution with a diagonal covariance matrix, i.e, π(a|s, θ) = N (µ, Σ). The neural network outputs the mean µ = µ(s) and diagonal Cholesky factors A = A(s), such that Σ = AA T . The diagonal factor A has positive diagonal elements enforced by the softplus transform A ii ← log(1 + exp(A ii )) to ensure positive definiteness of the diagonal covariance matrix. Mixture of Gaussian policy has a number of Gaussian components as well as a categorical distribution for selecting the components. The neural network outputs the Gaussian components based on the same setup described above for a single Gaussian and outputs the logits for representing the categorical distribution. Tables 1 show the hyperparameters we used for the single tasks experiments. We found layer normalization and a hyperbolic tangent (tanh) on the layer following the layer normalization are important for stability of the algorithms. For RHPO the most important hyperparameters are the constraints in Step 1 and Step 2 of the algorithm. For the SAC-U baseline we used a re-implementation of the method from (Riedmiller et al., 2018) using SVG (Heess et al., 2015) for optimizing the policy. Concretely we use the same basic network structure as for the "Monolithic" baseline with MPO and parameterize the policy as

π θ = N (µ θ (s, i), σ 2 θ (s, i)I)

, where I denotes the identity matrix and σ θ (s, i) is computed from the network output via a softplus activation function.

Together with entropy regularization, as described in (Riedmiller et al., 2018) the policy can be optimized via gradient ascent, following the reparameterized gradient for a given states sampled from the replay:

EQUATION

which can be computed, using the reparameterization trick, as

EQUATION

where g θ (s, i, ζ) = µ θ (s, i) + σ θ (s) * ζ is now a deterministic function of a sample from the standard multivariate normal distribution. See e.g. Heess et al. (2015) (for SVG) as well as Kingma and Welling (2014) (for the reparameterization trick) for a detailed explanation.

A.4 DETAILS ON THE EXPERIMENTAL SETUP A.4.1 SIMULATION (SINGLE-AND MULTITASK)

For the simulation of the robot arm experiments the numerical simulator MuJoCo3 was used -using a model we identified from the real robot setup.

We run experiments of length 2 -7 days for the simulation experiments (depending on the task) with access to 2-5 recent CPUs with 32 cores each (depending on the number of actors) and 2 recent NVIDIA GPUs for the learner. Computation for data buffering is negligible.

A.4.2 REAL ROBOT MULTITASK:

Compared to simulation where the ground truth position of all objects is known, in the real robot setting, three cameras on the basket track the cube using fiducials (augmented reality tags).

For safety reasons, external forces are measured at the wrist and the episode is terminated if a threshold of 20N on any of the three principle axes is exceeded (this is handled as a terminal state with reward 0 for the agent), adding further to the difficulty of the task.

The real robot setup differs from the simulation in the reset behaviour between episodes, since objects need to be physically moved around when randomizing, which takes a considerable amount of time.

To keep overhead small, object positions are randomized only every 25 episodes, using a hand-coded controller. Objects are also placed back in the basket if they were thrown out during the previous episode. Other than that, objects start in the same place as they were left in the previous episode. The robot's starting pose is randomized each episode, as in simulation.

A.5 TASK DESCRIPTIONS A.5.1 PILE1

Figure 7 : Sawyer Set-Up.

For this task we have a real setup and a MuJoCo simulation that are well aligned. It consists of a Sawyer robot arm mounted on a table and equipped with a Robotiq 2F-85 parallel gripper. In front of the robot there is a basket of size 20x20 cm which contains three cubes with an edge length of 5 cm (see Figure 7 ).

The agent is provided with proprioception information for the arm (joint positions, velocities and torques), and the tool center point position computed via forward kinematics. For the gripper, it receives the motor position and velocity, as well as a binary grasp flag. It also receives a wrist sensor's force and torque readings. Finally, it is provided with the cubes' poses as estimated via the fiducials, and the relative distances between the arm's tool center point and each object. At each time step, a history of two previous observations is provided to the agent, along with the last two joint control commands, in order to account for potential communication delays on the real robot. The observation space is detailed in Table 4 . Let d(o i , o j ) be the distance between the reference of two objects (the reference of the cubes are the center of mass, TCP is the reference of the gripper), and let d A be the distance only in the dimensions denoted by the set of axes A. We can define the reward function details by: 16)

stol(v, , r) = 1 iff |v| < 1 -tanh 2 ( atanh( √ 0.95) r |v|) else (

EQUATION

A.5.2 PILE2

Figure 8 : The Pile2 set-up in simulation with two main tasks: The first is to stack the blue on the red cube, the second is to stack the red on the blue cube.

For the Pile2 task, taken from Riedmiller et al. ( 2018), we use a different robot arm, control mode and task setup to emphasize that RHPO's improvements are not restricted to cartesian control or a specific robot and that the approach also works for multiple external tasks.

Here, the agent controls a simulated Kinova Jaco robot arm, equipped with a Kinova KG-3 gripper.

The robot faces a 40 x 40 cm basket that contains a red cube and a blue cube. Both cubes have an edge length of 5 cm (see Figure 8 ). The agent is provided with proprioceptive information for the arm and the fingers (joint positions and velocities) as well as the tool center point position (TCP) computed via forward kinematics. Further, the simulated gripper is equipped with a touch sensor for each of the three fingers, whose value is provided to the agent as well. Finally, the agent receives the cubes' poses, their translational and rotational velocities and the relative distances between the arm's tool center point and each object. Neither observation nor action history is used in the Pile2 experiments. The cubes are spawned at random on the table surface and the robot hand is initialized randomly above the table-top with a height offset of up to 20 cm above the table (minimum 10 cm). The observation space is detailed in Table 6 .

The robot arm is controlled in raw joint velocity mode at 20 Hz. The action space is 9-dimensional as detailed in Table 5 . There are no virtual walls and the robot's movement is solely restricted by the velocity limits and the objects in the scene.

that is always closed initially (see Figure 9 ). The agent's goal is to clean up the scene by placing the cubes inside the box. In addition to the observations used in the Pile2 task, the agent observes the lid's angle and it's angle velocity. Move the blue cube.

• NO_TOUCH = 1 -GRASP Sparse binary reward, given when neither of the touch sensors is active.

• LIFT(R) = btol(pos z (R), 0.05) Increase the z-coordinate of the red cube to more than 5cm relative to the table.

• LIFT(B) = btol(pos z (B), 0.05) Increase the z-coordinate of the blue cube to more than 5cm relative to the table.

• OPEN_BOX = slin(angle(lid), 0.01, 1.5) Open the lid up to 85 degrees. Coordinating convergence progress in hierarchical models can be challenging but can be effectively moderated by the KL constraints. We perform an ablation study varying the strength of KL constraints on the high-level controller between prior and the current policy during training -demonstrating a range of possible degenerate behaviors.

As depicted in Figure 14 , with a weak KL constraint, the high-level controller can converge too quickly leading to only a single sub-policy getting a gradient signal per step. In addition, the categorical distribution tends to change at a high rate, preventing successful convergence for the low-level policies. On the other hand, the low-level policies are missing task information to encourage decomposition as described in Section 3.2. This fact, in combination with strong KL constraints, can prevent specialization of the low-level policies as the categorical remains near static, finally leading to no or very slow convergence. As long as a reasonable constraint is picked (here a range of over 2 orders of magnitude), convergence is fast and the final policies obtain high quality for all tasks. We note that no tuning of the constraints is required across domains and the range of admissible constraints is quite broad. A.8.2 IMPACT OF DATA RATE Evaluating in a distributed off-policy setting enables us to investigate the effect of different rates for data generation by controlling the number of actors. Figure 15 demonstrates how the different agents converge slower lower data rates (changing from 5 to 1 actor). These experiments are highly relevant for the application domain as the number of available physical robots for real-world experiments is typically highly limited. To limit computational cost, we focus on the simplest domain from Section 4.2, Pile1, in this comparison. Similarly to the experiments in the main paper, we can see that the hierarchical policy leads to better final performance -here for a gradient-based approach. All plots are generated by running 5 actors in parallel.

:

Additional videos for task and component visualization are provided under https://sites.google. com/view/rhpo/

In asynchronous RL systems, the update rate of the learner can have a significant impact on the performance if evaluated over actor steps

MuJoCo: see www.mujoco.org

Figures and Tables:

Figure 1: Using a hierarchical policy with different component initialization (red curve) demonstrates benefits over homogeneous initialization as well as the flat Gaussian policy. The plot shows that the simple change in initialization is sufficient to enable component specialization and the correlated improvement in performance.

Figure 2: Results for the multitask robotic manipulation experiments in simulation. The dashed line corresponds

Figure 3: Left: Overview of the real robot setup with the Sawyer robot performing the Pile1 task. Screen pixelated for anonymization. Middle: Simulated Sawyer performing the same task. Right: Cleanup2 setup with the Jaco.

Figure4: Robot Experiments. Left: While simpler tasks such as reaching are learned with comparable efficiency, the later, more complex tasks are acquired significantly faster with a hierarchical policy. Right: Similarities between tasks (based on their distribution over components) and similarities between components (based on the distribution over tasks which apply them).

Figure 5: Schematics of the fully connected networks in SAC-X. While we use the Q-function (right sub-figure)

Figure 6: Schematics of the alternative multitask policy architectures used in this paper. Left sub-figure: the monolithic architecture; Right sub-figure: the hierarchical architecture.

Figure 9: The Clean-Up task set-up in simulation. The task is solved when both bricks are in the box.

Figure 10: Complete results for all tasks from the Pile1 domain. The dotted line represents standard SAC-U after the same amount of training. Results show that using hierarchical policy leads to best performance.

Figure 14: Complete results for sweeping the KL constraint between 1e-6 and 1. in the Pile1 domain. For very weak constraints the model does not converge successfully, while for very strong constraints it only converges very slowly.

Figure15: Complete results for ablating the number of data-generating actors in the Pile1 domain. We can see that the benefit of hierarchical policies is stronger for more complex tasks and lower data rates. However, even with 20 actors we see better final performance and stability

Figure 18: Complete results for evaluating SVG with and without hierarchical policy class in the Pile1 domain.

1 Asynchronous Learner Input: N steps number of update steps, N targetUpdate update steps between target update, N

Table content: <table/>

Hyperparameters -Multitask.  Values are taken from the single task experiments with the above mentioned changes.A.3 ADDITIONAL DETAILS ON THE SAC-U WITH SVG BASELINE

Table content: <table><tr><td colspan="2">Hyperparameters</td><td>Hierarchical</td><td>Single Gaussian</td></tr><tr><td>Policy net</td><td/><td>200-200-200</td><td>200-200-200</td></tr><tr><td colspan="2">Number of actions sampled per state</td><td>10</td><td>10</td></tr><tr><td colspan="2">Q function net</td><td>500-500-500</td><td>500-500-500</td></tr><tr><td colspan="2">Number of components</td><td>3</td><td>NA</td></tr><tr><td/><td/><td>0.1</td><td>0.1</td></tr><tr><td>µ</td><td/><td>0.0005</td><td>0.0005</td></tr><tr><td>Σ</td><td/><td>0.00001</td><td>0.00001</td></tr><tr><td>α</td><td/><td>0.0001</td><td>NA</td></tr><tr><td colspan="2">Discount factor (γ)</td><td>0.99</td><td>0.99</td></tr><tr><td colspan="2">Adam learning rate</td><td>0.0002</td><td>0.0002</td></tr><tr><td colspan="2">Replay buffer size</td><td>2000000</td><td>2000000</td></tr><tr><td colspan="2">Target network update period</td><td>250</td><td>250</td></tr><tr><td>Batch size</td><td/><td>256</td><td>256</td></tr><tr><td colspan="2">Activation function</td><td>elu</td><td>elu</td></tr><tr><td colspan="2">Layer norm on first layer</td><td>Yes</td><td>Yes</td></tr><tr><td colspan="2">Tanh on output of layer norm</td><td>Yes</td><td>Yes</td></tr><tr><td colspan="2">Tanh on input actions to Q-function</td><td>Yes</td><td>Yes</td></tr><tr><td colspan="2">Retrace sequence length</td><td>10</td><td>10</td></tr><tr><td colspan="3">Table 1: Hyperparameters -Single Task</td><td/></tr><tr><td>Hyperparameters</td><td colspan="2">Hierarchical</td><td colspan="2">Independent Monolith</td></tr><tr><td>Policy torso</td><td/><td/><td/></tr><tr><td>(shared across tasks)</td><td/><td colspan="2">400-200</td></tr><tr><td>Policy task-dependent heads</td><td colspan="2">100 (high-level controller)</td><td>100</td><td>NA</td></tr><tr><td>Policy shared heads</td><td colspan="2">100 (per mixture component)</td><td>NA</td><td>200</td></tr><tr><td>Number of action samples</td><td/><td>20</td><td/></tr><tr><td>Q function torso</td><td/><td/><td/></tr><tr><td>(shared across tasks)</td><td/><td colspan="2">400-400</td></tr><tr><td>Q function head</td><td/><td/><td/></tr><tr><td>(per task)</td><td/><td>300</td><td/></tr><tr><td>Number of components</td><td colspan="2">number of tasks</td><td>NA</td><td>NA</td></tr><tr><td>Discount factor (γ)</td><td/><td>0.99</td><td/></tr><tr><td>Replay buffer size</td><td/><td colspan="2">1e6 * number of tasks</td></tr><tr><td>Target network update period</td><td/><td>500</td><td/></tr><tr><td>Batch size</td><td/><td colspan="2">256 (512 for Pile1)</td></tr></table>

Action space used in the experiments with the Kinova Jaco Arm.

Table content: <table><tr><td>Entry</td><td>Dimensions</td><td>Unit</td><td>Range</td></tr><tr><td>Joint Velocity (Arm)</td><td>6</td><td colspan="2">rad/sec [-0.8, 0.8]</td></tr><tr><td>Joint Velocity (Hand)</td><td>3</td><td colspan="2">rad/sec [-0.8, 0.8]</td></tr></table>

Peer review (iclr2020):
The paper is rather interesting and is able to solve some difficult tasks. A combination of different learning techniques for acquiring structure and learning with asymmetric data are used. While the combination of methods is new I am not sure that this particular combination of methods to train an HRL policy is sufficiently novel. Is the authors can highlight the effects and contribution of how these methods are combined to indicate this better it would be good. More detailed comments: - In section 3.2 you mention a reference policy? Can you provide more details on this reference policy? - IN the paper it is mentioned that the method collects data, including the reward for each task on a single state, action transition. This assumption seems rather strong. Earlier in the paper, the authors discussed the motivation for learning transferable sub-policies. In the real world, it may not be possible to collect the reward for every kind of task simultaneously. - The first evaluation in Section 4.1 uses two humanoid environments. While these environments can be considered difficult that does not seem like the multi-task type environment the method is motivated to work well on. There is little sub-task transfer in this task. - In section 4.1 it is noted that the version that initialized with different policy means works best. How are these means initialized? - Is the Pile1 collection of tasks really separate tasks? It would be good to have some more details on how these are organized. There may not be a clear definition in the community what is considered a specific task but I am not overly convinced that these *different* tasks are separate. Most of them look like a similar version of pick and place. - In Figure 2 the hierarchical method is similar in performance on the stack and leave (Pile1) set of tasks and marginally better in the Pile2 set yet does far better on the Cleanup2 set. While these are all simulations with multiple tasks is there some reasoning to why each method looks similar on the Pile1 set of tasks? - For the robotic tasks, it is noted that again the baseline methods do well on the *Reach* task. It is shown that the RHPO does much better on the Stack task. It would be great if the authors can describe the interesting differences between the tasks. It is not clear how difficult the Stack task is and why it is largely different from Reach + Grasping. - For the images on the right of Figure 4, It shows a comparison between the tasks and some *components* Are these components the states? The *components* are not explained well in the papers. - There is no algorithm in the main paper which makes it a little difficult to understand the operation of the learning method. For example how exactly is the learning of the two different levels compared? It seems like they are trained together. If they are they should be compared to HIRO or HAC. How is temporal abstraction handled between the two policy layers if they are trained together?