Compositional Transfer in Hierarchical Reinforcement Learning
Authors: Markus Wulfmeier
Abbas Abdolmaleki
Roland Hafner
Jost Tobias Springenberg
Michael Neunert
Tim Hertweck
Thomas Lampe
Noah Siegel
Nicolas Heess
Martin Riedmiller
Venue:  (2020)
Keywords: Multitask
Transfer Learning
Reinforcement Learning
Hierarchical Reinforcement Learning
Compositional
Off-Policy

Abstract: The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency. To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time. To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL). We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference. Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions. Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives. We design an RL algorithm that enables stable and fast learning of structured policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting. Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and  demonstrate substantial improvements in data data-efficiency over competitive baselines.

Under review as a conference paper at ICLR 2020
COMPOSITIONAL TRANSFER IN HIERARCHICAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency. To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time. To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL). We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference. Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions. Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives. We design an RL algorithm that enables stable and fast learning of hierarchical policies and the effective reuse both behavior components and transition data across tasks in an off- setting for complex, real-world domains. Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and demonstrate substantial improvements data dafa-efficencyy over competitive baselines.
INTRODUCTION
While recent successes in deep (reinforcement) learning for computer games (Atari (Mnih et al. 2013), StarCraft (Vinyals et al., 2019), Go (Silver al. (017) and other high-throughput domains, e.g. (OpenAl et al.] 2018). have demonstrated the potential of these methods in the big data regime, the high cost of data acquisition has so far limited progress in many tasks of real-world relevance. Data efficiency in machine learning generally relies on inductive biases to guide and accelerate the learning process; e.g. by including expert domain knowledge of varying granularity. Incorporating such knowledge can accelerate learning- but when inaccurate it can also inappropriately bias the space of solutions and lead to sub-optimal results.
Robotics represents a domain in which data efficiency is critical, and human prior knowledge is commonly provided. However, for scalability and reduced dependency human accuracy, we can instead utilise an agent's permanent embodiment and shared environment across tasks. Intuitively, such a scenario suggests the natural strategy of focusing on inductive biases that facilitate the sharing and reuse of experience and knowledge across tasks while other aspects of the domain can be learned. As general principle this relieves us from the need to inject detailed knowledge about the domain, instead we can focus on general principles that facilitate reuse (Caruana 1997).
Successes for transfer learning have, for example, built on optimizing initial parameters (c.g. Finn al. 2017), sharing models and parameters across tasks either in the form of policies or value Functions (c.g. Rusu et al. (2016; Teh et al. 2017 Galashov et al. 2018)) data-sharing across tasks (e.g. Riedmiller al. [2018] Andrychowicz et al. 2017). or through the use of task-related auxiliary objectives (Jaderberg et al. 2016, Wulfmeier el al.) 2017). Transfer between tasks can, however, lead to either constructive or destructive transfer for humans (Singley and Anderson 1989) well as for machines (Pan and Yang, 2010; Torrey and Shavlik, 2010). That is, jointly learning to solve different tasks can provide both benefits and disadvantages for individual tasks, depending on their similarity. Finding mechanism that enables transfer where possible but avoids interference is one of the long-standing research challenges.

---

Under review as a conference paper at ICLR 2020
In this paper explore the benefits and limitations of hierarchical policies in single and multitask reinforcement learning. Similar to Mixture Density Networks (Bishop) 1994) our models represent policies as statecconditional Gaussian mixture distributions, with separate Gaussian mixture com- ponents as low-level policies which can be selected by the high-level controller via a categorical action choice. In the multitask setting, to obtain more robust and versatile low-level behaviors, additionally shield the mixture components from information about the task hand. this case, task information is only communicated through the choice of mixture component by the high-level controller, and the mixture components can be seen as domain -emain-dependant, fask-independent skills although the nature of these skills is not predefined and emerges during end-to-end training.
We implement this idea by building on three forms of transfer: targeted exploration via the con- catenation of tasks within one episode (Riedmiller et al.] 2018), sharing transition data across tasks (Andrychowicz et al.] 2017; Riedmiller et al.] 2018). and reusing low-level components of the atorementioned policy class. To this end we develop a novel robust and data-efficient multitask actor-critic algorithm, Regularized Hierarchical Policy Optimization (RHPO). Our algorithm uses the multitask learning aspects of SAC (Riedmiller et al. (208)) to improve data-efficiency and robust policy optimization properties of MPO (Abdolmaleki et 20.]20018) order to optimize hierarchical policies. We furthermore demonstrate the generality of hierarchical policies for multitask learning via improving results also after replacing MPO as policy optimizer with another gradient-based entropy-regularizes policy optimizer (Heess et al. 2015) (see Appendix 10).
We demonstrate that compositional, hierarchical policies while strongly reducing training time in multitask domains can fail to improve performance in single task domains if no additional inductive biases are given. While multitask domains provide sufficient pressure for component specialization, and the related possibility for composition, we are required to introduce additional incentives to encourage similar developments for single task domains. In the multitask setting, we demonstrate considcrably improved performance, robustness and learning speed compared to competitive continuous control baselines demonstrating the relevance of hierarchy for data-efficiency and transfer. We finally evaluate our approach on a physical robot for robotic manipulation tasks where RHPO leads to significant speed up in training, enabling it to solve challenging stacking tasks on a single robot
2 PRELIMINARIES
We consider multitask reinforcement learning setting with an agent operating in a Markov De- cision Process (MDP) consisting of the state space S, the action space A, the transition proba- bility p(s (s, 04) of reaching state st+1 from state St when executing action a, at the previ- ous time step t. The actions are drawn from a probability distribution over actions <(als) re- ferred to as the agent's policy. Jointly, the transition dynamics and policy induce the marginal state visitation distribution p(s). Finally, the discount factor r together with the reward r(s. a) gives rise to the expected reward, or value, of starting in state (and following π thereafter) (5(s) = E,, [Ern 7'((s,, ss0 = s, s,a, fine multitask learning over set of tasks e I with common agent embodiment as follows. We assume shared state, action spaces and shared transition dynamics across tasks; tasks only differ in their reward function rs a). Furthermore, we consider task conditional policies (alss, i). The overall objective is defined as
r--[s;), $1+1 p(-|st, a.). Furthermore, we de-
Σ r'v (st, ar) 18441 ~ p(-1 an) Ei- [Bm.pCo) [Q (s, 1)]]
((T) = Ei-r Er. Ex,p(so)
(1)
where all actions are drawn according to the policy n, that is, de ~ [[1 i) and we used the common definition of the state-action value function here conditioned the task (* a) E, |Ete r'ra (St, ax) lao = a, 80 = 8, a, ~ (-(19,, 8t+1 ~ p(-18,, az)]-
Addiiiioal videos for task and component visualization provided under com/ vin ew/rhpo/
2

---

Under review as a conference paper at ICLR 2020
3 METHOD
This section introduces Regularized Hierarchical Policy Optimization (RHPO) which focuses on efficient training of modular policies by sharing data across tasks; extending the data-sharing and scheduling mechanisms from Scheduled Auxiliary Control with randomized scheduling (SAC-U) (Riedmiller al. 2018). We start by introducing the considered class of policies, followed by the required combination and extension - of MPO (Abdolmaleki et al., 200 and SAC-U (Riedmiller al. 2018 for training structured hierarchical policies in multitask, off-policy setting.
3.1
HIERARCHICAL POLICIES
We start by defining the hierarchical policy class which supports sharing sub-policies across tasks. Formally, we decompose the per-task policy π (aass, as
Ta(als, i) Σ T (als, 0) n (ols,
with n and n' respectively representing a "high-level" switching controller (a categorical distribu- tion) and "low-level" sub-policy (components of the resulting mixture distribution), where o the index of the sub-policy. Here, 0 denotes the parameters of both x and nL, which we will seek to optimize. While the number of components has be decided externally, the method is robust with respect to this parameter (Appendix A.8.3).
Note that, in the above formulation only the high-level controller TH is conditioned the task information i; i.c. we employ a form information asymmetry (Galashov et al., [2018] Tirumala ct al. 2019 Heess et al.] 2016) enable the low-level policies to acquire general, tusk-independent behaviours. This choice strengthens decomposition of tasks across domains and inhibits degenerate cases of bypassing the high-level controller. Intuitively, these sub-policies building reflex-like low-level control loops, which perform domain dependend but inskinndepnndent behaviours and can be modulated by higher cognitive functions with knowledge of the task at hand.
be understood as
3.2 DATA-EFFICIENT MULTITASK POLICY OPTIMIZATION
In the following sections, present the equations underlying RHPO. For the complete pscudocode algorithm the reader is referred to the Appendix A.2.I To optimize the policy class described above we build on the MPO algorithm (Abdolmaleki al. 201 Sa) which decouples the policy improvement step (optimizing independently of the policy structure) from the fitting of the hierarchical policy, Concretely, we first introduce intermediate non-parametric policy q(a)s, i) and consider optimizing [g) while staying close, in expectation, to reference policy Fref(ales (8,1)
max J(g)= = Esnr [ ((s, a, ()] s.t. BEDD,i
[L 2.) (|) Se, (3)
where KL(-II-) denotes the Kullback Leibler divergence, e defines a bound on the KL. D denotes the data contained in replay buffer. and assuming that we have an approximation of the ground-truth state-action value function ((s a, a) z Q (s, a, i) available (see Equation (4) for details on learning Q from off policy data). Starting from an initial policy ne, we can then iterate the following steps to improve the policy meL
Policy Evaluation: Update Q such that Q(s, a) x TT a, see Equation 4
Policy Improvement:
Step 1: Obtain d arg max, [g), under KL constraints with Kref Te, (Equation 3
Step 2: Obtain 0:-1 = arg ming Eg-D, (TT((g (0)0))))[=000))00000))) 1))] under additional
regularization (Equation (6).
Multitask Policy Evaluation For data-efficient off-policy learning of Q we build on scheduled auxiliary control with uniform scheduling (SAC-U) (Riedmiller al. 2018) which exploits two main
3

---

Under review as a conference paper at ICLR 2020
ideas to obtain data-efficiency: i) experience sharing across tasks; ii) switching between tasks within one episode for improved exploration.
Formally, we assume access to replay buffer containing data gathered from all tasks, which filled asynchronously to the optimization (similar to e-g. [Espeoott et al. (2018)) where for each trajectory snippet = {(80, ao Ro)...., (81, at, Rr)) we record the rewards for all tasks Re [ra, (s), de)....,?. 1 (8t, a+)] as a vector in the buffer. Using this data we define the retrace
objective for learning Q, parameterized via o, following (Munos et al. 2016 Riedmiller et al. 2018
as
min L( = Σ ErND (ri($1,a)) + + A4+1, 1) Qo(s,, ac, at. 0))²
(4)
where Qrat is the L-step retrace target (Munos et al. 2016), see the Appendix A.2.2 for details.
Multitask Policy Improvement 1: Obtaining Non-parametric Policies We first find the interme-
diate policy q by maximizing Equation 33. We obtain a closed-form solution with non- -araamamcttic policy for each task,
((.) 77
Th (als, i) x To, (als, exp exp
(5)
where 17 a temperature parameter (corresponding a given bound e) which is optimized alongside the policy optimization (see Appendix [.I.I for a detailed derivation of the multitask case). As mentioned above, this policy representation is independent of the form of the parametric policy me, iee g only depends on Te, through its density. This, crucially, makes it easy to employ complicated structured policies (such the one introduced in Section 3.I The only requirement here, and in the following steps, is that we must be able to sample from Té, and calculate the gradient (w.r.t. 0g) of its log density (but the sampling process itself need not be differentiable).
Multitask Policy Improvement 2: Fitting Parametric Policies In the second step we fit a pol- icy to the non-parametric distribution obtained from the previous calculation by minimizing the divergence EEwD, . [KL(gx ( (-Is, i) (-]s, i))l- Assuming that we can sample from this step corresponds maximum likelihood estimation (MLE). Furthermore, we can regularize towards smoothly changing distributions during training- effectively mitigating optimization instabilities and introducing an inductive bias by limiting the change of the policy (a trust-region constraint). The idea is commonly used in on- as well as in off-policy RL (Schulman et al. 2015) Abdolmaleki et al., 2018b ab. The application to hierarchical policy classes highlights the importance of this constraint as investigated in Section 4.2 Formally, we aim to obtain the solution
0 0++1 arg min E. D
KL (qu(-1s, T-
M TØ (als, o) th (ols, 1)
arg max E BiNd
Bmo, exp(((e...))))) log
(6)
oml
1
s.t. D.DDD
KL(TH H (ols, (ols, 1 (ols, + M Σ KL (als o (als,
Cms
where cm defines a bound on the change of the new policy. Here we drop constant terms and the negative sign in the second line (turning min into max), and explicitly insert the definition Te(a|s, i) = EM aL (als, o) πH (ols, i), to highlight that we are marginalizing over the high-level choices in this fitting step (since is not tied to the policy structure). Hence, the update is independent of the specific policy component from which the action was sampled, enabling joint updates of all components. This reduces the variance of the update and also enables efficient off--oliccy learning. Different approaches can be used control convergence for both the "high-level" categorical choices and the action choices to change slowly throughout learning. The average KL constraint in Equation (6) is similar in nature to an upper bound on the computationally intractable KL divergence between the two mixture distributions and has been determined experimentally to perform better in practice than simple bounds. In practice, in order control the change of the high level and low level policies independently we decouple the constraints to be able to set different for the means (e,)), covariances (es)) and the categorical distribution (c,,) in case of mixture of Gaussian policy. To solve Equation
M
4

---

Under review as a conference paper at ICLR 2020
(6), first employ Lagrangian relaxation make it amenable to gradient based optimization and then perform fixed number of gradient descent steps (using Adam (Kingma and Ba 2014)); details on this step, as well as algorithm listing, can be found in the Appendix A.1.2
4 EXPERIMENTS
In the following sections, we investigate the effects of training hierarchical policies in single and multitask domains, finally demonstrating how RHPO can provide compelling benefits for multitask learning in real and simulated robotic manipulation tasks and significantly reduce platform interaction time. In the context of single-task domains from the DeepMind Control Suite (Tasa et al. 2018), we first demonstrate how this type of hierarchy on its own fails to improve performance and that for the model to exploit compositionality. additional incentives for component specialization are required. Subsequently, we introduce suited incentives leading to improved performance and demonstrate that the variety of objectives multitask domains can serve the same purpose. The evaluation includes experiments on physical hardware with robotic manipulation tasks for the Sawyer arm, emphasizing the importance of data-efficiency.
More details on task hyperparameters as well as the results for additional ablations and all tasks from the multitask domains provided in the Appendix A.4 Across all tasks, build dis- tributed actor-critic framework (similar to (Espeholt et al. 2018) with flexible hardware assignment (Buchlovsky et al. 2019) to train all agents, performing critic and policy updates from replay buffer, which is asynchronously filled by a set of actors. In all figures with error bars, we visualize mean and variance derived from 3 runs.
4.1 SIMULATED SINGLE TASK EXPERIMENTS
We consider two high-dimensional tasks for continuous control: humanoid-run and humanoid-stand from Tassa al. (018)) and compare a flat Gaussian policy to a hierarchical policy, mixture of Gaussians with three components. We align the update rates of all approaches for fair comparison and to focus the comparison of the algorithm and not Figure
Sanar //anerrrrrra FNLTNT
Jontal www.rnuuaaaaaa.. raaeendard
-00
meglenei eodh
SpRCERS
4000200
Figure 1: Using a hierarchical policy with different component initialization (red curve) demonstrates benefits over homogeneous initialization as well the flat Gaussian policy. The plot shows that the simple change in initialization is sufficient to enable component specialization and the correlated improvement in performance.
its specific implementation visualizes the results in terms of the number of actor episodes.
As can be observed, the hierarchical policy performs comparable to flat policy with well aligned means and variances for all components as the model fails to decompose the problem. While both the flat and hierarchical policy are initialized with means close to zero, we now include another hierarchical policy with distributed initial means for the three components ranging for all dimensions from minimum to maximum of the allowed action range (here: -1, 0, 1). This simple change suffices to enable component specialization and significantly improved performance.
4.2 SIMULATED MULTITASK EXPERIMENTS
We use three simulated multitask scenarios with the Kinova Jaco and Rethink Robotics Sawyer robot arms to test in a variety of conditions. Pilel: Here, the seven tasks of interest range from simple reaching for a block over tasks like grasping it, to the final task of stacking the block on top of another block. In addition to the experiments in simulation, which are executed with 5 actors in distributed setting, the same Pilel multitask domain (same rewards and setup) is investigated with a single, physical robot in Section 4.3 We further extend the evaluation towards two more complex multitask domains in simulation. The first extension includes stacking with both blocks on top of the respective
In asynchronous RL systems, the update rate of the learner can have significant impact on the performance if evaluated over actor steps
5

---

Under review as a conference paper at ICLR 2020
other block, resulting in setting with 10 tasks (Pile2). And last domain including harder tasks such as opening box and placing blocks into this box, consisting of a total of 13 tasks (Cleanup2).
We compare RHPO for training hierarchical policies against flat, monolithic policy shared across all tasks which provided with the additional task id as input (displayed as Monolithic in the plot) as well as policies with task dependent heads (displayed as Independent in the plots) following (Riedmiller et al., 2018) both using MPO as the optimizer and re-implementation of SAC-U using SVG (Heess et al. 2013) (which is related to a version of the option critic (Bacon et al. (017)) without temporal abstraction). The baselines provide the two opposite, naive perspectives on transfer: by using the same monolithic policy across tasks we enable positive as well as negative interference and independent policies prevent policy-based transfer. After experimentally confirming the robustness of RHPO with respect to the number of low-level sub-policies (see Appendix A.8.3 we set M proportional the number of tasks in each domain.
Figure Results multitask robotic manipulation experiments in simulation. The dashed line corresponds the performance of the SVG-hased implementation of SAC-U. From left right: Pile1, Pile2, Cleanup2. show averages over 3 runs each, with corresponding standard deviation. RHPO outperforms both baselines across all tasks with the benefits increasing for more complex domains.
Figure 2 demonstrates that the hierarchical policy (RHPO) outperforms the monolithic as well as the independent baselines. For simple tasks such Pilel, the difference is smaller, but the more tasks are trained and the more complex the domain becomes (cf. Pile2 and Cleanup2), the greater is the advantage of composing learned behaviours across tasks. Compared to SVG (Heess et al. 2015), we observe that the baselines based on MPO already result in an improvement, which becomes even bigger with the hierarchical policies. The results across all domains exhibit performance gains for the hierarchical model without the additional incentives from Section 4. demonstrating the sufficiency of variety in the training objectives to encourage component specialization and problem decomposition.
4.3 PHYSICAL ROBOT EXPERIMENTS
For real-world experiments, data-efficiency is crucial. We perform all experiments in this section relying single robot (single actor) demonstrating the benefits of RHPO in the low data regime. The performed task is the real world version the Pilel task described Section The main task objective is to stack one cube onto second one and move the gripper away from it. We introduce an additional third cube which serves purely as a distractor.
Figure 3: Left: Overview of the real robot setup with the Sawyer robot performing the Pilel task. Screen pixelated for anonymization. Middle: Simulated Sawyer performing the same task. Right: Cleanup2 setup with the Jaco.
The setup for the experiments consists of Sawyer robot mounted on a table, equipped with Robotiq 2F-85 parallel gripper. A basket of size 20cm² in front of the robot contains the three cubes. Three cameras on the basket track the cubes using fiducials (augmented reality tags). As in simulation,

---

Under review as a conference paper at ICLR 2020
the agent is provided with proprioception information (joint positions, velocities and torques), wrist sensor's 's force and torque readings, as well as the cubes' poses estimated via the fiducials. The agent action is five dimensional and consists of the three Cartesian translational velocities, the angular velocity of the wrist around the vertical axis and the speed the gripper's fingers.
Figure 4 plots the learning progress on the real robot for two (out of 7) of the tasks, the simple reach tasks and the stack task which is the main task of interest. Plots for the learning progress of all tasks are given in the appendix [A.6] As can be observed, all methods manage to learn the reach task quickly (within about few thousand episodes) but only RHPO with hierarchical policy is able to learn the stacking task (taking about 15 thousand episodes to obtain good stacking success), which
takes about days of training on the real robot with considerably slower progress for all baselines.
Reach
Stack
150
Herarchical
Monolith
150
Avaraga Return
Independent
8.00
Average
2.00
Hierarchical Monolith
Independent
15
135
Task Similarity
Component Similarity
Episodes
164
Episodes
164
Reach
Stack
Figure 4: Robot Experiments. Left: While simpler tasks such as reaching are learned with comparable efficiency, the later, more complex tasks are acquired significantly faster with a hierarchical policy. Right: Similarities between tasks (based on their distribution over components) and similarities between components (based on the distribution over tasks which apply them).
In addition, we compute distributions for each component over the tasks which activate it, as well as distributions for each task over which components are being used. For each set of distributions, we determine the Battacharyya distance metric to determine the similarity between tasks and the similarity between components in Figure 4 (right). The plots demonstrate how the components specialise, but also provide way to investigate our tasks, showing e.g. that the first reach task is fairly independent and that the last four tasks are comparably similar regarding the high-level components applied for their solution.
4.3.1 ADDITIONAL. ABLATIONS
We perform series of ablations based on the earlier introduced Pilel domain, providing additional insights into benefits and shortcomings of RHPO and important factors for robust training. The algorithm is well suited for sequential transfer learning based on solving new tasks with pre-trained low-level components (Appendix (.9). We demonstrate the robustness of RHPO with respect to the number of sub-policies as well importance of choice of regularization respectively in Appendix A.8. and [A8.3) Finally, we ablate over the number of data- data-generating actors to evaluate all approaches with respect to data rate and illustrate how hierarchical policies are particularly relevant at lower data rates such as given by real-world robotics applications in Appendix A.8.2
RELATED WORK
Transfer learning, in particular in the multitask context, has long been part of machine learning (ML) for data-limited domains (Caruana 1997 Torrey and Shavlik 2010 Pan and Yang 2010 Taylor and Stone 2009). Commonly, it is not straightiorward to train single model jointly across different tasks as the solutions tasks might not only interfere positively but also negatively (Wang al. 2018).
Preventing this type of forgetting or negative transfer presents a challenge for biological (Singley and Anderson 1989) well as artificial systems (French) 1999). In the context of ML, common scheme is the reduction of representational overlap (French 1999 Rusu et al.) 2016 Wang et al. 2018)
Bishop 1994) utilize neural networks parametrize mixture models for representing multi-modal distributions thus mitigating shortcomings of non non-hierarchical approaches. |oosesttei et al. (0005)
demonstrate the benefits of hierarchical classification models limit the impact of negative transfer.
7

---

Under review as a conference paper at ICLR 2020
Hierarchical approaches have long history in the reinforcement learning literature (e.g. Sutton et al. 1999] Dayan and Hinton 1993). Prior work commonly benefits from combining hierarchy with additional inductive biases such as (Vezhnevets et al. 2017 Nachum al.) 20188 b Xie et al., 2018) which employ different rewards for different levels of the hierarchy rather than optimizing a single objective for the entire model as we do. Other works have shown the additional benefits for the stability of training and data-efficiency when sequences of high-level actions are given as guidance
during optimization in hierarchical setting (Shiarlis et al. 2018: Andreas et al. 2017 Tirumala (et al., 2019). Instead of introducing additional training signals, we directly investigate the benefits of compositional hierarchy as provided structure for transfer between tasks.
Hierarchical models for probabilistic trajectory modelling have been used for the discovery of behavior abstractions as part of an end-to-end reinforcement learning paradigm (e.g. [Teh et al. [017]] Igl et al.] 2019; Tirumala et al. 2019: Galashov al., 2018) where the models act as learned inductive biases that induce the sharing of behavior across tasks. In vein similar to the presented algorithm, (e.g[Heess et al.] 2016 Tirumala al. 2019) share a low-level controller across tasks but modulate the low-level behavior via a continuous embedding rather than picking from small number of mixture components. In related work Hausman et al. (2018); HHarmoja al. (2018) learn hierarchical policies with continuous latent variables optimizing the entropy regularized objective.
Similar to our work, the options framework (Sutton al. 1999 Precup 2000) supports behavior hierarchies, where the higher level chooses from a discrete set of sub-policies or "options" which commonly
until termination criterion is satisfied. The framework focuses the notion of temporal abstraction. A number of works have proposed practical and scalable algorithms for learning option policies with reinforcement learning (e.g. Bacon et al., [217)] Zhang and Whiteson |019| (Smihh et al. 2018), Riemer et al. 2018 Harb et al. 2018) or criteria for option induction (e.g. Harb et al. 2018 Harutyunyan et al. 2019). Rather than the additional inductive bias of temporal abstraction, we focus on the investigation of composition as type of hierarchy in the context of single and multitask learning while demonstrating the strength of hierarchical composition to lie in domains with strong variation in the objectives such as in multitask domains. We additionally introduce a hierarchical extension of SVG (Heess et al. 2015), to investigate similarities to work on the option critic (Bacon et al.) 2017).
With the use of KL regularization to different ends in RL, work related to RHPO focuses on contextual bandits (Daniel et al. 2016). The algorithm builds on a 2-step EM like procedure to optimize linearly parametrized mixture policies. However, their algorithm has been used only with low dimensional policy representations, and in contextual bandit and other very short horizon settings. Our approach is designed to be applicable to full RI. problems in complex domains with long horizons and with high-capacity function approximators such as neural networks. This requires robust estimation of value function approximations, off-policy correction, and additional regularization for stable learning.
DISCUSSION
We introduce a novel framework to enable robust training and investigation of hicrarchical, com- positional policies in complex simulated and real-world tasks as well as provide insights into the learning process and its stability. In simulation as well as on real robots, RHPO outperforms base- line methods which either handle tasks independently or utilize implicit sharing. Especially with increasingly complex tasks or limited data rate, as given in real-world applications, we demonstrate hierarchical inductive biases to provide a compelling foundation for transfer learning, reducing the number of environment interactions significantly and often leading to more robust learning as well as improved final performance. For single tasks with a single training objective all components can remain aligned, preventing problem decomposition and the hierarchical policy replicates a flat policy. Performance improvements appear only when the individual components specialize, either via variety in the training objectives or additional incentives. Furthermore, as demonstrated in Appendix A.9 pre-trained set of specialized components can notably improve performance when learning new tasks. One important next step is identifying how to optimize basis set of components which transfers well to a wide range of tasks
Since with mixture distributions, we are able to marginalize over components when optimizing the weighted likelihood over action samples in Equation the extension towards multiple levels of hierarchy is trivial but can provide valuable direction for practical future work. While this approach partially mitigates negative interference between tasks in a parallel multitask learning scenario, addressing catastrophic inference in sequential settings remains a challenge.
8

---

Under review as a conference paper at ICLR 2020
We believe that especially in domains with consistent agent embodiment and high costs for data generation learning tasks jointly and information sharing is imperative. RHPO combines several ideas that we believe will be important: multitask learning with hierarchical and compositional policy representations, robust optimization, and efficient off-policy learning. Although we have found this particular combination of components to be very effective believe it is just instance of and step towards spectrum of efficient learning architectures that will unlock further applications of RL both in simulation and, importantly, on physical hardware.
REFERENCES
Volodymyr Mnih. Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13/2.5602, 2013.
Oriol Vinyals. Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Woj- ciech M. Czarnekki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Junhhyk Oh, Valentin Dal- ibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang. Tobias Pfaff, Toby Pohlen, Yuhuai Wu, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https: http/eeepmminnneeeee com/ wtt.ccoooooooooo 2019.
David Silver, Julian Schrittwieser Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, el. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
OpenAl, Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. CoRR, Abs/1808.00117 2018. URL http:
//arxiv.org/aaaee wttp/////////r/e///r///////////e 00177
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 1126-1135. JMLR. org, 2017.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 01..:..6600077 2016.
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. CoRR, abs/1707. 04175, 2017. (Rtp:://://axx wttp:///.rrr/bb///////// 04175
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenelever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins. Wojeiech M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry kl-regularized rl. 2018.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint arXiv:1802.10560 2018.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048-5058, 2017.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 161 1.5397, 2016.

---

Under review as a conference paper at ICLR 2020
Markus Wulfmeier. Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. arXiv preprint Stt.:1707.07907,, 2017.
Mark K Singley and John Robert Anderson. The transfer of cognitive skill. Number 9. Harvard University Press, 1989.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):135--135) 2010.
Lisa Torre and Jude Shavlik. Transfer learning. In Handbook of research machine learning applications and trends: algorithms, methods, and techniques, pages 242-264. IGI Global, 2010.
Christopher M Bishop. Mixture density networks. Technical report, Citeseer, 1994.
Abbas Abdolmaleki, Jost Tobias Springenberg. Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint arx.v:1812.002256 2018a.
Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, 2015.
Dhruva Tinumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer kl-regularized rl. arXiv preprint 100.003903.07338 2019.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:/6/0.05//2 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. International Conference on Machine Learning. pages 1406-1415, 2018
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, 2016.
John Schulman, Sergey Levine, Philipp Moritz. Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, pages 1889-1897. JMLR. org, 2015.
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and 2018b.
Martin A. Riedmiller. Maximum posteriori policy optimisation. CoRR,
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXivv:412.6650 2014.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018
Peter Buchlovsky, David Budden, Dominik Grewe, Chris Jones, John Aslanides, Frederic Besse, Andy Brock, Aidan Clark, Sergio Gómez Colmenarejo, Aedan Pope, Fabio Viola, and Dan Belov. Tf-replicator: Distributed machine learning for researchers. arXiv preprint arXiv:1902.00 0002,00465, 2019.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 1633-1)1633-1665, 2009.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. arXiv preprint arXiv: 1811:981 2018.
10

---

Under review as a conference paper at ICLR 2020
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-132) 1999.
Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas Dietterich. To transfer or not to transfer. In NIPS 2005 workshop on transfer learning, volume 898, pages 1-4, 2005.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)1811-11, 1999.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural information processing systems, pages 271-278, 1993.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volee Volume 70, pages 3540- 3549. JMLR. org, 2017.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pages 3303-3313 2018a.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:/810.01255,, 2018b.
Saining Xie, Alexandre Galashov, Siqi Liu, Shaobo Hou, Razvan Pascanu, Nicolas Heess, and Yee Whye Teh. Transferring task goals via hierarchical reinforcement learning, 2018. URL
htttps://ppenreviie
net. worfroorrmiirre Y6TEJJG
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference Machine Learning, pages 4661-4670, 2018.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning- Voluuee 70, pages 166-175. JMLR. org, 2017.
Maximilian Igl, Andrew Gambardella, Nantas Nardelli, N Siddharth, Wendelin Böhmer, and Shimon Whiteson. Multitask soft option learning. arXiv preprint arXiv: 2019.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference Learning Representations, 2018
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hicrarchical reinforcement learning. In International Conference on Machine Learning, pages 1846-1855, 2018
Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. arXiv preprint arXiv: 12691, 2019.
Matthew Smith, Herke Hoof, and Joelle Pineau. An inference- based policy gradient method for learning options. In International Conference on Machine Learning, pages 4710-4719, 2018.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. In Advances in Neural Information Processing Systems, pages 10424-10434, 2018.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an op- tion: Learning options with a deliberation cost. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Rémi Munos, and Doina Precup. The termination critie. CoRR, abs/1902.09996, 2019. URL http/arwwww..r//aaaa/aa/re///aaa 09996
org 902
11

---

Under review as a conference paper at ICLR 2020
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. The Journal of Machine Learning Research, 17(1):310-3223 2016.
Malcolm Reynolds, Gabriel Barth-Maron, Frederic Besse, Diego de Las Casas, Andreas Fidjeland, Tim Green, Adrià Puigdomenech, Sébastien Racanière, Jack Rae, and Fabio Viola. Open sourcing Somnet a new library for constructing neural networks. https: http:::://deemmind.cm / 2017.
open-sourcing--sonnee
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. URL //teso///eeoo..oo.... org/ Software available from tensorflow.org.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning. (1):9 14:9-44,11 1988.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv/6611.007/2 2016.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.00 144, 2016
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient
estimation in machine learning. arXiv preprint 1906. 2019.
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1
ADDITIONAL DERIVATIONS
In this section we explain the detailed derivations for training hierarchical policies parameterized as mixture of Gaussians.
A.1.1 OBTAINING NON-PARAMETRES POLICIES
In each policy improvement step, to obtain non-parametric policies for given state and task distribution, we solve the following program:
max (ss) (a)))..)) [Q(s, a.)]
8. ((+) I~
(L 1s, 1), (,1,, 0.)) <6
s.t. i (i((
[1] 11
To make the following derivations easier to follow we open up the expectations, writing them as integrals explicitly. For this purpose let us define the joint distribution over states 8 ~ p(s) together with randomly sampled tasks / as (ts i)
(i)(( E I), where U denotes the uniform distribution over possible tasks. This allows us to re-write the expectations that include correpooniing distributions, i.e. En(s),a>Z [1] = Eyys.s) [1], but again, note that here is not necessarily
the task under which & was observed. We can then write the Lagrangian equation corresponding to the above described program as
L(gm,7) u(s,.) /
da ds di +1 E 6- u(s,i) a(als, i) <alss, 0,) ds
q(ass,)) 1)
+7 (1- /I u(s, Oa(als, da
Next we maximize the Lagrangian L w.r. the primal variable q. The derivative w.r. q reads,
0qL(g. 122) =Q s, y
+ r log π i, 02) 7.
Setting it to zero and rearranging terms we obtain
(Q(3,a,)) n
exp (- (-7t7)
a(a)s, (alss, 0.) exp
7
However, the last exponential term is normalization constant for q. Therefore we can write,
17 1
(alss, (() exp [(s. a,
exp
17 da
11
+ Y = log
(fm(alss,, 02) exp (Q(s.a,)) da
(7)
13

---

Under review as a conference paper at ICLR 2020
Now, to obtain the dual function (()), we plug in the solution to the KL constraint term (second term) of the Lagrangian which yields
[( / M(s,0)
da
+ <- u(s. f a(alss i) log ((alasii)) expl (le expl mt2
da ds
+7 (1- /1 ((s, de
After expanding and rearranging terms we get
L(a,7.7) u(s.i) p(als, da
- M(3,i) a(als, Q(s,a,i) (ogs(a)s;i;;), 7 シ + "
da ds di
+ me + v m(s,i) / alals, i) log rals, i; da
+ (1- /f m(s ((s,l) dads
Most of the terms cancel out and after rearranging the terms we obtain
L(g, 7)7 me + 7) p(s, "
+ ds dí.
Note that we have already calculated the term inside the integral in Equation 7 equation 7 we obtain the dual function
By plugging in
g( me + 17 pls, i) log (fT(als;,00) exp Q(s, da ds di,
(8)
")
which we can minimize with respect to based on samples from the replay buffer.
A.1.2 EXTENDED UPDATE RULES FOR FITTING MIXTURE GAUSSIANS
After obtaining the non parametric policies, we parametric policy samples from said non- parametric policies effectively employing using maximum likelihood estimation with additional regularization based on a distance function T, i.e,
@(k+1) = arg min E.-D KL (g:((g--s 2)
| (o(ss 0))
arg min E
lo
(9)
arg max E D. log log
E.-D,WI T(TO, To <.
where D is an arbitrary distance function evaluate the change of the new policy with respect to reference/old policy, and e denotes the allowed change for the policy. To make the above objective amenable to gradient based optimization employ Lagrangian Relaxation, yielding the following primal:
minT.(0,a) BaND BewDiiiaaaaggeeee log (a(ass aa(c- BanDDiin (T(,, (118,), f), (-(- (s,))))) a20
(10)
14

---

Under review as a conference paper at ICLR 2020
We solve for / by iterating the inner and outer optimization programs independently: We fix the pa- rameters 9 to their current value and optimize for the Lagrangian multipliers (inner minimization) and then fix the Lagrangian multipliers to their current value and optimize for 0 (outer maximization). In practice we found that it is effective to simply perform one gradient step each in inner and outer optimization for each sampled batch of data.
The optimization given above is general, i.e. it works for any general type of policy. As described in the main paper, we consider hierarchical policies of the form
To (a) - Σ Te (als, 0) me (o]s,
(1I)
In particular, in all experiments we made use of a mixture of Gaussians parametrization, where the high level policy nh is categorical distribution low level T Gaussian policies, i.e.
no(als, = T(alfag ngg (s)=
Σ ((dd))))
M a = 1, and, (8,1)> >0
where j denote the index of components and is the high level policy π assigning probabilities to each mixture component for a state given the task and the low level policies are all Gaussian. Here a's are the probabilities for categorical distribution over the components.
also define the following distance function between and new mixture of Gaussian policies T T(T (-1s, t), =0(-|s, i)) = (s, i) + (s())
(s = KLLCCa(ffa (L (IC (1))_.))
M Σ j=1
Tr((8)
KL(N (6), E (o))) (x),
M
where Ti evaluates the KL between categorical distributions and Tr corresponds to the average KL Gaussian components, as also described in the main paper (c.f. Equation 5 in the main paper).
In order to bound the change of categorical distributions, means and covariances of the components independently- which makes it easy control the convergence of the policy and which can prevent premature convergence as argued in AAdolmaleki et al. (2018a) we separate out the following three intermediate policies
(als, i) = =(alfo/ Hb, E// i).==I...))
i) =(alfag. 1b. E%, (s, i))=
(als,) <<alla M/.. Σ/ (s, i)}=
Which yields the following final optimization program
((++1) -arg max E, log (als, + log n (als, i) + log ~8 (a]s,
s.t. EaND.N
(t i))
Ens
(12)
s.t. EAnDD
i))
< cx
s.t. EAND,i
< tay
15

---

Under review as a conference paper at ICLR 2020
This decoupling allows to set different values for the change of means, covariances and categorical probabilities, i.e., Sp. EE, ca. Different e lead to different learning rates. We always set a much smaller epsilon for the covariance and categorical than for the mean. The intuition is that while would like the distribution to converge quickly in the action space, we also want to keep the exploration both locally (via the covariance matrix) and globally (via the high level categorical distribution) to avoid premature convergence.
A.2 ALGORITHMIC DETAILS
A.2.1 PSEUDOCODE FOR THE FULL PROCEDURE
We provide pseudo-code listing of the full optimization procedure and the asynchronous data gathering - performed by RHPO in Algorithm [ and 2
The implementation relies on Sonnet
(Reynolds al. 2017) and TensorFlow (Abadi et al. 2015
Algorithm 1 Asynchronous Learner
Input: N Nateps number of update steps, NitareeUppaase update steps between target update, N, number of action samples per state, KL regularization parameters e, initial parameters for x, 7) and 6 initialize N =
while & < Nateps do
for in [o...NMargstUddnne] do
sample a batch of trajectories from replay buffer B
sample N, actions from ne, estimate expectations below // compute mean gradients over batch for policy, Lagrangian multipliers and Q-function
6 - -o Emer EN exp (Q().a))))
log To(ay)st, i) following Eq.
6, + vng(n) = Vn + Eete log 1 ΣN, exp ((()a)))) following Eq.
N.
6Q V Eins E(e,,aner (3, Qret)² with Q following Eq.
I/ apply gradient updates
- -optimizer update(T, &z),
7 = optimizer update(1, -naitteeerupdatt(,, 8,))
Q
cetimizer_updated 6Q)
k = k +
end for // update target networks
ा'' = n. Q' Q
end while
A.2.2 DETAILS ON THE POLICY IMPROVEMENT STEP
As described in the main paper consider the same setting scheduled auxiliary control setting (SAC-X) (Riedmiller et al. 2018 to perform policy improvement (with uniform random switches between tasks every N steps within an episode, the SAC-U setting).
Given a replay buffer containing data gathered from all tasks, where for each trajectory snippet T {(So, ag, Ro),.... (SL, aL, R2)} we record the rewards for all tasks R; [ri, (8t, az),... rijn (St, [t)] vector in the buffer, we define the retrace objective for learning Q parameterized via Φ. following Riedmiller et al. (2018) as
min L(Ø) = Σ ENED (r:(st, a:) + (s411,
(es(s,,a,11 1))²] with
Q (2 des i)
Σ
II ck (00(s))
(13)
j=t
(g(3),8511) (e(0); + BBo, (als; 1.1)> 12 (s541, d')] Qur (0j. (sj. aj,
where the importance weights are defined as ca = min (1, mo, (o)lo lsx)), with b(ag(ss) (8x) denot- ing an arbitrary behavior policy; in partieular this will be the policy for the executed tasks during an
16

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Asynchronous Actor
Input: Nimjectoriss number of total trajectories requested, 7 steps per episode, & scheduling period
initialize N 0
while N < Mmjeztories do
fetch parameters
// collect new trajectory from environment
-= (
for in [0..T]] do
if (mod =0 then
// sample active task from uniform distribution fact
end if
0r ~ TT(-118; fact)
// execute action and determine rewards for all tasks
f = [ra (12, ..),...,,e Tain Sun (5 at)]
I +- U {(s,, ae, F, (a St, Pact))
end for
send batch trajectories replay buffer
N = N 1+1
end while
episode as in (Riedmiller et al.) 2018). Note that, in practice, truncate the infinite sum after L steps, bootstrapping with Q. We further perform optimization of Equation (4) via gradient descent and make use of a target network (Mnih et al. 2015), denoted with parameters o', which we copy from & after a couple of gradient steps. We reiterate that, as the state-action value function Q remains independent of the policy's 's structure, TD(0) (uttoon 1988).
able utilize any other off-the-shelf Q-learning algorithm such as
Given that we utilize the same policy evaluation mechanism SAC-U is worth pausing here to identify the differences berween SAC-U and our approach. The main difference in the policy parameterization: SAC-U used a monolithic policy for each task π (a|s, i) (although a neural network with shared components, potentially leading some implicit task transfer, was used). Furthermore, we perform policy optimization based on MPO instead of using stochastic value gradients (SVG (Heess al. (0166). We can thus recover a variant of plain SAC-U using MPO we drop the hierarchical policy parameterization, which we employ in the single task experiments in the main paper.
A.2.3 NETWORK ARCHITECTURES
To represent the Q-function in the multitask case we use the network architecture from SAC-X (see right sub-figure in Figure 5). The proprioception of the robot, the features of the objects and the actions are fed together in torso network. At the input 200 units, followed by a layer normalization operator, an optional tanh activation and another fully connected layer of 200 units with an ELU activation function. The output of this torso network is shared by independent head networks for each of the tasks (or intentions, as they are called in SAC-X paper). Each head has two fully connected layers and outputs Q-value for this task, given the input of the network. Using task identifier we then can compute the Q value for given sample by discrete selection of the according head output.
fully connected first layer of
While we use the network architecture for the function for all multitask experiments, we investigate different architectures for the policy in this paper. The original SAC-X policy architecture is shown in Figure (1eft sub-figure). The main structure follows the same basic principle that we use in the Q function architecture. The only difference is that the heads compute the required parameters for the policy distribution we want use (see subsection A.2.4). This architecture referenced as the independent heads (or task dependent heads).
The alternatives investigate in this paper are the monolithic policy architecture (see Figure 6 left sub-figure) and the hierarchical policy architecture (see Figure right sub-figure). For the monolithic policy architecture reduce the original policy architecture basically to one head and
17

---

Under review as a conference paper at ICLR 2020
obsee vetervaatoon
observation
nom GPr
policy
functio
300
Wit
action
taski
taskiD
Figure 5: Schematics of the fully connected networks in SAC-X. While we the Q-function (right sub-figure architecture in all multitask experiments, investigate variations of the policy architecture (left sub-figure) in this paper (see Figure 6
observation
S.. comvouution StOck
abseevv
f parariss
scale
norm disttr
pollcyy
layer nor
policy
4000
200
tanh
task
Figure 6: Schematics of the altemaatve multitask policy architectures used in this paper. Left sub-figure: the monolithic architecture; Right sub-figure: the hicrarchical architecture.
append the task-id one-hot encoded vector the input. For the hierarchical architecture, build on the same torso and create a set of networks parameterizing the Gaussians which are shared across tasks and rask-specific network to parameterize the categorical distribution for each task. The final mixture distribution task-dependent for the high-level controller but task-independent for the lowelevel policies.
A.2.4 ALGORITHM HYPERPARAMETERS
In this section we outline the details on the hyperparameters used for RHPO and baselines both single task and multitask experiments. All experiments use feed-forward neural networks. We consider flat policy represented by a Gaussian distribution mixture of Gaussians distribution. The flat policy is given by Gaussian distribution with a diagonal covariance matrix, i.e, x (a|s; 0) N (μ, E). The neural network outputs the mean f4 = ((()) and diagonal Cholesky factors diagonal elements enforced by the softplus transform A,; < log(1 (xp():)) to ensure positive definiteness the diagonal covariance matrix. Mixture of Gaussian policy has number of Gaussian components as well as a categorical distribution for selecting the components. The neural network outputs the Gaussian components based the same setup described above for single Gaussian and outputs the logits for representing the categorical distribution. Tables [ show the hyperparameters used for the single tasks experiments. We found layer normalization and hyperbolic tangent (Lanh) on the layer following the layer normalization are important for stability of the algorithms. For RHPO the most important hyperparameters the constraints Step and Step 2 of the algorithm.
a hierarchical policy represented by
A(s), such that Σ AAT", The diagonal factor has positive
18

---

Under review as a conference paper at ICLR 2020
Hyperparameters
Hierarchical
Single Gaussian 200-200-200)
Policy net Number of actions sampled per state Q function net Number of components
200-200-200
10
10
500-500-500
500-500-500
NA
0.1
0.1
0.0005
0.0005
0.00001
0.00001
6ox
0.0001
NA
Discount factor (7)
0.99
0.99
Adam learning rate
0,0002
0.0002
Replay buffer size
2000000
2000000
Target network update period
250
250
Batch size
256
256
Activation function
elu
elu
Layer norm on first layer Tanh on output of layer norm Tanh on input actions to Q-function
Yes
Yes
Yes
Yes
Yes
Yes
Retrace sequence length
10
10
Table I: Hyperparameters Single Task
Hyperparameters
Hierarchical
Independent
Monolith
Policy torso
(shared across tasks)
400-200
Policy task-dependent heads
100 (high-level controller)
100
NA
Policy shared heads
100 (per mixture component)
NA
200
Number of action samples
20
Q function torso
(shared across tasks)
400-400
Q function head (per task)
300
Number of components
number tasks
NA
NA
Discount factor (r)
0.99
Replay buffer size
1e6 number of tasks
Target network update period
500
Batch size
256 (512 for Pile1)
Table 2: Hyperparameters Multitask. Values are taken from the single task experiments with the above mentioned changes.
A.3 ADDITIONAL DETAILS ON THE SAC-U WITH SVG BASELINE
For the SAC-U baseline we used immlementation the method from (Riedmiller al. 2018 using SVG (Heess et al. 2015) for optimizing the policy. Concretely we use the same basic network structure as for the "Monolithic" baseline with MPO and parameterize the policy as
Tg = N (Ho (s, i), (s, i))
where 7 denotes the identity matrix and oo (8, i) is computed from the network output via softplus activation function.
Together with entropy regularization, described in (Riedmiller et al. 2018) the policy can be optimized via gradient ascent, following the reparameterized gradient for a given states sampled from the replay:
(Q(,,,,,)))aaa,,,,0,, + all (Ta(a)s))).
(14)
which can be computed, using the reparameterization trick,
EG~D( Vega ( + VoH mT((()s))).
(15)
19

---

Under review as a conference paper at ICLR 2020
where g, () (o(ss,) + tg now a deterministic function of sample from the standard multivariate normal distribution. See e.g. Heess et al. (2015) (for SVG) as well as Kingma and Welling 2014) (for the reparameterization trick) for a detailed explanation.
A.4 DETAILS ON THE EXPERIMENTAL SETUP
A.4.1 SIMULATION (SINGLE- AND MULTITASK)
For the simulation of the robot arm experiments the numerical simulator MuloCo was used - using model identified from the real robot setup.
run experiments of length 2 2-7 days for the simulation experiments (depending on the task) with access to 2-5 recent CPUs with 32 cores each (depending on the number of actors) and recent NVIDIA GPUs for the learner. Computation for data buffering is negligible.
A.4.2 REAL ROBOT MULTITASK
Compared to simulation where the ground truth position of all objects is known, in the real robot setting, three cameras on the basket track the cube using fiducials (augmented reality tags).
For safety reasons, external forces are measured at the wrist and the episode is terminated if threshold of 20N on any the three principle axes is exceeded (this handled as a terminal state with reward 0 for the agent), adding further to the difficulty of the task.
The real robot setup differs from the simulation in the reset behaviour between episodes, since objects need to be physically moved around when randomizing, which takes a considerable amount of time. To keep overhead small, object positions are randomized only every 25 episodes, using a hand-coded controller. Objects are also placed back in the basket if they were thrown out during the previous episode. Other than that, objects start the same place as they were left in the previous episode. The robot's 's starting pose is randomized each episode, as in simulation.
A.5
TASK DESCRIPTIONS
A.5.1
PILEI
Figure 7: Sawyer Set-Up.
For this task have real setup and MuJoCo simulation that are well aligned. It consists of Sawyer robot arm mounted on a table and equipped with Robotiq 2F-85 parallel gripper. In front of the robot there is basket of size 20x20 em which contains three cubes with edge length of 5 em (see Figure 7.
The agent is provided with proprioception information for the arm (joint positions, velocities and torques), and the tool center point position computed via forward kinematics. For gripper, receives the motor position and velocity, as well as binary grasp flag. It also receives a wrist sensor's force and torque readings. Finally, it is provided with the cubes' poses as estimated via the fiducials, and the relative distances between the arm's 's tool center point and each object. At each time step, history of two previous observations is provided to the agent, along with the last two joint control commands, in order to account for potential communication delays on the real robot. The observation space detailed Table
MuloCaa www.mujoco.org
20

---

Under review as a conference paper at ICLR 2020
Table 3: Action space for the Sawyer experiments.
Entry
Dimensions
Unit
Range
Translational Velocity x, y;
m/s
1-0.07, 0.071
Wrist Rotation Velocity
rad/s
[-1. 1]
Finger speed
tics/s
[-255, 255]
Table 4: Observation used in the experiments with the Sawyer arm. An object's pose is represented its world coordinate position and quaternion. the table, denotes meters, rad denotes radians, and refers to quaternion in arbitrary units (au).
Entry
Dimensions
Unit rad rad/s Nm
Joint Position (Arm)
Joint Velocity (Arm)
Joint Torque (Arm)
Joint Position (Hand)
rad tics/s
Joint Velocity (Hand)
Force- Torque (Wrist)
N, Nm au au rad/s m, au
Binary Grasp Sensor
TCP Pose
7
Last Control Command (Joint Velocity)
Green Cube Pose
Green Cube Relative Pose
m, au
Yellow Cube Pose
m. au
Yellow Cube Relative Pose
au
Blue Cube Pose
Blue Cube Relative Pose
The robot arm is controlled in Cartesian mode at 20Hz. The action space for the agent is 5-dimensional, as detailed in Table 3 The gripper movement also restricted to cubic volume above the basket using virtual walls.
For the Pilel experiment use 7 different task learn, following the SAC-X principles. The first 6 tasks are seen as auxiliary tasks that help to learn the final task (STACK LECK_AND___C LND_LEAVE(G, Y)) of stacking the green cube top of the yellow cube. Overview of the used tasks:
REACH(G): stol(d[TCP.. G), 0.02, 0.15):
Minimize the distance of the TCP the green cube.
GRASP:
Activate grasp sensor of gripper ("inward grasp signal" of Robotiq gripper)
LIFT(G):
slin(G, 0.03, 0.10)
Increase Z coordinate of an object more than 3em relative to the table.
Y): stol, Y + [0, 0, 0.05)), 0.01, 0.20) Bring green cube to position 5cm above the yellow cube.
PLACE_NARROW(C Y): (d Y + [0, 0, 0.05)), 0.00, 0.01): Y) but more precise.
Like
STACKiG, Y): (lol Y), 0.03) blol(d Y) + 0.05, 0.01) (1 GRASP) Sparse binary reward for bringing the green cube on top of the yellow one (with 3cm tolerance horizontally and lcm vertically) and disengaging the grasp sensor.
BECCKANDD_LLAEEER Y): sd (TCP, G) + 0.10, 0.03, 0.10) * STACK(G, Y) Like STACK(G, Y), but needs to move the arm 10em above the green cube.
21

---

Under review as a conference paper at ICLR 2020
Table 5: Action space used in the experiments with the Kinova Jaco Arm.
Entry
Dimensions
Unit
Range
Joint Velocity (Arm)
6
rad/sec
1-0.8, 0.8]
Joint Velocity (Hand)
3
rad/sec [-0.8, 0.8]
Let d(o;, 0ÿ) be the distance between the reference of two objects (the reference of the cubes are the center of mass, TCP is the reference of the gripper), and let d be the distance only in the dimensions denoted by the set of axes A. We can define the reward function details by:
stol(u,cr)) [
iff |u| <E
(16)
1 tanh? 2 aatan(
(ol) else
iff < Emin
slin(U, Emins Eman)
iff >> mman
(17)
else
Cmaz Encin
{1
iff |u| <e
bol(v, 6)
(18)
0
else
A.5.2
PILE2
The Pile2 set-up simulation two main tasks: The first is to stack the blue on the cube, the second is to stack the red on the blue cube.
Figure 8:
For the Pile2 task, taken from Riedmiller et al. (2018), we use a different robot arm, control mode and task setup to emphasize that RHPO's improvements are not restricted to cartesian control or specific robot and that the approach also works for multiple external tasks.
Here, the agent controls a simulated Kinova Jaco robot arm, equipped with a Kinova KG-3 gripper. The robot faces 40 x 40 em basket that contains red cube and blue cube. Both cubes have an edge length of 5 cm (see Figure B). The agent is provided with proprioceptive information for the arm and the fingers (joint positions and velocities) as well as the tool center point position (TCP) computed via forward kinematics. Further, the simulated gripper is equipped with touch sensor for each of the three fingers, whose value is provided to the agent as well. Finally, the agent receives the cubes" poses, their translational and rotational velocities and the relative distances between the arm's tool center point and each object. Neither observation nor action history used in the Pile2 experiments. The cubes are spawned at random on the table surface and the robot hand is initialized randomly above the table-top with height offset of up 20 cm above the table (minimum 10 cm). The observation space is detailed in Table 6
The robot arm is controlled in raw joint velocity mode 20 Hz. The action space is 9-dimensional as detailed in Table 'here are no virtual walls and the robot's movement is solely restricted by the velocity limits and the objects in the scene.
22

---

Under review as a conference paper at ICLR 2020
Table Observation used the experiments with the Kinova Jaco Arm. An object's pose is represented its world coordinate position and quaternion. The lid position and velocity are only used in the Clean- Up task. In the table, m denotes meters, rad denotes radians, and g refers to quaternion in arbitrary units (au).
Entry
Dimensions
Unit
Joint Position (Arm)
Joint Velocity (Arm)
6
rad/s
Joint Position (Hand)
3
rad rad/s
Joint Velocity (Hand)
3
TCP Position
3
Touch Force (Fingers)
3
Red Cube Pose
7
m, au m/s, da/dt
Red Cube Velocity
6
Red Cube Relative Position
3
Bluc Cube Pose
7
Blue Cube Velocity
6
m/s, da/dt
Blue Cube Relative Position
3
Lid Position
1
rad rad/s
Lid Velocity
Analogous Pilel and the SAC-X setup, seen as auxiliary tasks, that the agent uses to learn the main two tasks PILE_RED and PILE_BLUE, which represent stacking the red cube on the blue cube and stacking the blue cube on the red cube respectively. The tasks used the experiment are:
10 different task for Pile2. The first 8 tasks
REACH(R) stol(d[TCP,, R). 0.01, 0.25):
Minimize the distanee of the TCP to the red cube.
REACH(B) (dTP B), 0.01, 0.25):
Minimize the distance of the TCP the blue eube.
MOVE(R) slin(1 lineel (R) 0,
Move the red cube.
MOVE(B) slin linvel [,,
Move the blue cube.
LIFT(R) (tol (R), 0.05)
Increase the z-coordinate of the red cube to more than 5cm relative to the table.
LIFT(B) (pol( (B)) 0.05)
Increase the z-coordinate of the blue cube to more than Sem relative the table. [OOVE_ B) = above R, B) * (d) R, B). 0.05, 0.2)
Bring the red cube to a position above of and close the blue cube.
ABOVE_ B R) above(B B, R) stol R, B), 0.05, 0.2) Bring the blue cube to a position above of and close to the red cube.
PILE(R):
Place the red cube on another object (touches the top). Only given when the cube doesn't ' touch the robot or the table.
PILE(B):
Place the blue cube on another object (touches the top). Only given when the cube doesn't 't touch the robot the table.
The sparse reward above(A, B) is given by comparing the bounding boxes of the two objects A and B. the bounding box object A is completely above the highest point of object B's bounding box, above(A, B) is 1, otherwise above(A, B) is O.
A.5.3 CLEAN-UP
The Clean-Up Up task is also taken from Riedmiller et al. 2018) and builds on the setup described for the Pile2 task. Besides the two cubes, the work-spacc contains additional box with moveable lid,
23

---

Under review as a conference paper at ICLR 2020
that is always closed initially (see Figure 9. The agent's goal is to clean up the scene by placing the cubes inside the box. In addition to the observations used in the Pile2 task, the agent observes the lid's s angle and it's angle velocity.
Figure The Clean-Up Up task set-up simulation. The task solved when both bricks are box.
Analogous to Pile2 and the SAC-X setup, we use 13 different task for Clean-Up. The first 12 tasks seen as auxiliary tasks, that the agent uses to learn the main task CLL_INSIDE_BR The tasks used this experiments are:
REACH(R) stol(d(TCP,, R), 0.01, 0.25): Minimize the distance of the TCP to the red cube.
REACH( = sto((d(TCP, B), 0.01, 0.25):
Minimize the distance of the TCP to the blue cube.
MOVE(R)= slin() lineel R) 1, 1):
Move the red cube.
MOVE(B) slin linvel ( B) |, 1): Move the blue cube.
NO yo__OUCH!== 1 GRASP
Sparse binary reward, given when neither of the touch sensors is active.
LIFT(R) (B). 0.05)
Increase the z-coordinate of the red cube more than 5cm relative the table.
LIFT(B) (poo 0.05)
Increase the z-coordinate the blue cube to more than Sem relative to the table.
OPEN BOX
OPEN_BOX-T = slin slin(angle(lid), 0.01, 1.5)
Open the lid to 85 degrees.
ABOVE_CLOSEIC BOX) = above Rt BOX) blol(Id(RR R, BOX) 0.2) Bring the red eube to position above of and close the box.
(BOVE_CLOSERR BOX) above(B. BOX) btol[dd(BB _OX)BOO))... 0.2) Bring the blue cube to a position above of and close to the box.
INSIDE(R. BOX) inside( R, BOX)
Place the red cube inside the box.
INSIDE(B, BOX) = inside R, BOX)
Place the blue cube inside the box.
INSIDE(ALL, BOX) INSTDE BOX) * IBS Place the all cubes inside the box.
The sparse reward inside(A, BOX) is given by comparing the bounds of the object and the box. If the bounding box of object A is completely within the box's bounds inside(A, BOX) is 1, otherwise inside(A, BOX) is O.
24

---

Under review as a conference paper at ICLR 2020
A.6 MULTITASK RESULTS
A.6.1 PILEI ALL TASKS
100
indoinder
indeenden
14.
A/
W
Indoordew
Figure 10: Complete results all tasks from the Pile1 domain. The dotted represents standard SAC-U after the same amount of training. Results show that using hierarchical policy leads to best performance.
25

---

Under review as a conference paper at ICLR 2020
A.6.2 PILE2 ALL TASKS
BENOL
KEVOL.2
Heraerted aitee
Perelinc
Pewitty:
Parsiti
Figure 11: Complete results for all tasks from the Pile2 domain. Results show that using hierarchical policy leads best performance. The dotted line represents standard SAC-U after the same amount total training time.
26

---

Under review as a conference paper at ICLR 2020
A.6.3 CLEANUP2 ALL TASKS
PANOLSS
PANOL
Neraccic Henoliic
Hereccic Heaclty:
Revecto Passiiic
02
04
54
02
MOWK
SCIDICH
LP1
C0000
C00000
43
0.4
Figure 12: Complete results for all tasks from the Cleanup2 domain. Results show that using hierarchical policy leads best performance. The dotted line represents standard SAC-U after the same amount of total training time.
27

---

Under review as a conference paper at ICLR 2020
A.7
PHYSICAL ROBOT PILE1 ALL TASKS
Gramp
209
Average Return
109
62-
Herranhicc Monolith
Monolthh
Hierarchical
Indep on donz
Indesendent
lel
Lit
Place Widd
350
150
Hierarchiral
Monolkh
Average Retum
independent
Independent
100
5.0
0.5
1.0
0.
0.5
1.0
3.5
2.0
Eppodes
Episodes
Stack
152
Hllrarchical Monolthh
150
Hicrarchical Morolith Independent
Average Retum
100.
100.
oyrre mape
50
0.5
Episdds
Episades
Stack and Leave
Hierarchical independent
Arerags AverogsRetum Retum
Figure 13: Complete results for all tasks on the real robot Pilel domain. Results show that using hierarchical policy leads to best performance.
28

---

Under review as a conference paper at ICLR 2020
A.8 ADDITIONAL MULTITASK ABLATIONS
A.8.1 IMPORTANCE OF REGULARIZATION
Coordinating convergence progress in hierarchical models can be challenging but can be effectively moderated by the KL constraints. We perform an ablation study varying the strength of KL constraints on the high-level controller between prior and the current policy during training demonstrating range of possible degenerate behaviors.
As depicted in Figure 14 with a weak KL constraint, the high-level controller can converge too quickly leading to only a single sub-policy getting a gradient signal per step. In addition, the categorical distribution tends to change at high rate, preventing successful convergence for the low-level policies. On the other hand, the low-level policies are missing task information to encourage decomposition as described in Section 3.2 This fact, in combination with strong KL constraints, can prevent specialization of the low-level policies as the categorical remains near static, finally leading to very slow convergence. As long reasonable constraint is picked (here range of over 2 orders of magnitude), convergence is fast and the final policies obtain high quality for all tasks. We note that no tuning of the constraints is required across domains and the range of admissible constraints quite broad.
140
oxterriaal ozapeicall Lo-d
en comeriiaa EL upi calo sica
api calagg
122
ait caape
npt ciligiinall
ogi célgge forl bac
calecetea
caloge fna
60
Crtoddo
Figure 14: Complete results for sweeping the constraint between 1e-6 and 1. in Pilel domain. For very weak constraints the model does not converge successfully, while for very strong constraints only converges very slowly.
A.8.2 IMPACT OF DATA RATE
Evaluating in a distributed off-policy setting enables us to investigate the effect of different rates for data generation by controlling the number of actors. Figure 15 demonstrates how the different agents converge slower lower data rates (changing from 5 to actor). These experiments are highly relevant for the application domain as the number of available physical robots for real-world experiments is typically highly limited. To limit computational cost, we focus on the simplest domain from Section Pile1, in this comparison.
4.2
29

---

Under review as a conference paper at ICLR 2020
Monolithic
Independent
0.9
Hicrarchical-
0..8
Number of Actors:
Mo Monalitic
-0.6
Independent
Hierar aeearchical-
-04
Number of Acters: 20
-03
Monolithic Independent
-02
0.1
Hiler alccicai
2500
5000
7500
10000
12500
15000
17500
20000
Reach
moote 5 Somumaaa reward
Monolithic
Independent
-0.9
Hierarchical
00.8
Number of Actors:
Monoliticc
-06
Independent
-05
Hier archical
004
Number of Actors: 20
-03
Monohihic
-02
Independent
0.1
2500
5000.
7500
10000
12500
15000
17500
20000
Grasp
Monalithic
Number of Acters:
1.0
TeWard
Independent
-0.9
Hierarchical
008
Number Actors:
Monolitic
-06
Independent
Hier archical
0.4
Number of Actors: 20
-03
Monohihiic
Independent
0.1
2500
5000
7500
10000
12500
15000
17500
20000
Lift
Monolithic
Number of Actors:
-10
man g Badeaa reward
Independent
-0.9
008
Number Actors:
-06
Independent
Hier Herarchical
0.4
Number of Actors: 20
-03
Mo nolithic
Independentt
0.1
2500
5000
7500
10000
12500
15000
17500
20000
Place Wide
Monolithic
Number of Actors:
-10
Saruaata reword
Independent
-0.9
0.
Number Actors:
00.
-06
Independent
Heeaarchcca
0.4
Number of Actors: 20
-03
Mo nolithic
Indespendent
0.1
5
0.0
2500
5000
7500-
10000
12500
15000
17500-
20000
Place Narrow
30

---

Under review as a conference paper at ICLR 2020
Monolithic
Independent
0.9
Hicrarchical
0.8
Number of Actors:
Mo Monalitic
0.6
Independent
Hier aacharai
-04
Number of Actors: 20
-03
Monolitiic
-02
Independent
0.1
Hieer archical
2500
5000
7500
10000
12500
15000
17500
20000.
Stack
reward
Monolithic
Independent
Hierarchical
0.88
Number of Actors:
Monoliticc
0.6
Independent
Hier archical
0.4
Number of Acters: 20
-03
Monohtiic
Independent
archical
0.1
fracto
2500
5000
7500-
10000
12300
15000
17500
20000
Stack and Leave
Figure 15: Complete results for ablating the number of data generating actors in the Pilel domain. We can see that the henefit of hicrarchical policies is stronger for more complex tasks and lower data rates. However, even with 20 actors see better final performance and stability
A.8.3 NUMBER OF COMPONENT POLICIES
100
196
Kerrece keuum
Orvorerr
2 Cenoomo Eimoneni
50
164
304
140
130
Gaoer Grepenerh
Genevess
60
82
Stack Labis
50
Figure 16: Complete results 2,4.8 and 16 low-level policies in the Pilel domain. The approach is robust with respect the number of sub-policics and will build all further experiments setting the components equal the number tasks.
31

---

Under review as a conference paper at ICLR 2020
A.9 SEQUENTIAL TRANSFER EXPERIMENTS
To additionally investigate performance in adapting trained multitask policies to novel tasks, we train agents to fulfill all but the final task in the Pilel and Cleanup2 domains and subsequently evaluate training the models on the final task. We consider two settings for the final policy by introducing only a new high-level controller SSequeniia--Only-E a new high-level controller (Sequential). Figure 17 displays that in the sequential transfer setting, starting from policy trained on a set of related tasks results in up to 5 times more data-effice data-efficiency in terms of actor episodes on the final task than training the same policy from scratch. We observe that the final task can be solved by only reusing low-level components from previous tasks if the final task is the composition of previous tasks. This is the case for the final task in Cleanup2 which can be completed by sequencing the previously learned components and in contrast to Pilel where the final letting go of the block after stacking is not required for earlier tasks.
or both an additional shared component as well as
Stack
INSIDEBoXALL
70-
Average Return
Avcrage Return
50-
20
15
30
20
Sequential Sequential _nll__H HL From, Scratch
10-
Sequential only Hl.
Sequential
From, Scratch
1.5 Episodes
2.0
0.0
1.5
2.5 3.0
4.0
Episodes
Figure 17: Sequential transfer experiments: the models are first trained with all but the final task in the Pilel and Cleanup2 domains, and finally train the models to adapt the final task by either training 1- only high-level controller or 2-a high-level controller as well an additional component.
A.10 HIERARCHICAL POLICIES IN REPARAMETERIZATION GRADIENT-BASED RL
To test whether the benefits of a hierarchical policy transfer to a setting where a different algorithm is used to optimize the policy we performed additional experiments using SVG (Heess et al. 2015) in place of MPO. For this purpose we use the same hierarchical policy structure as for the MPO experiments but change the categorical to an implementation that enables reparameterization with the Gumbel-Softmax trick (Maddison et al. 2016 Jang et al., 2016). We then change the entropy regularization from Equation (15) to a KL towards a target policy (as entropy regularization did not give stable learning this setting) and use regularizer equivalent to the distance function (per component KL's from Equation (12) using multiplier of 0.05 for the regularization multiplier was found to be the best setting via coarse grid search. This similar to previous work on hierarchical RI. with SVG (Tirumala et al. 2019))
This extension of SVG is conceptually similar to a single-step-option version of the option-critic (Bacon et al., 2017). Simplified, SVG is an off-policy actor-critic algorithm which builds on the reparametrisation instead of likelihood ratio trick (commonly leading to lower variance (Mohamed ct al. 2019)). Since we do not build on temporally extended sub-policies, the algorithm simplifies to using a single critic (see Section 3.2p.
The results of this experiment are depicted in Figure 18 as can be seen, for this simple domain results
in mild improvements over standard SAC-U.
32

---

Under review as a conference paper at ICLR 2020
Croca
100
DD- FIPECTT
2X0 nareethe
SVC Hameeccaa
09
30
240-2rGa GUAEE
Figure 18: Complete results for evaluating SVG with and without hierarchical policy class in the Pilel domain.
Similarly to the experiments in the main paper, we can see that the hierarchical policy leads to better final performance here for madient-basee approach. All plots generated by running 5 actors in parallel.
33


Reviews:
[{"review_id": "HJxjYHATtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a hierarchical policy structure for use in both single task and multitask reinforcement learning. The authors then assess the usefulness of such a structure in both settings on complex robotic tasks. These tasks include the stacking and reaching of blocks using a robotic hand, as an example. In addition to carrying out these experiments on simulated robots, the authors have also carried out experiments on a real Sawyer robotic arm. \n\nThe particular form of their hierarchical policy for the multitask case is as follows. The policy, which is conditioned on the current state and task index consists of a gaussian mixture, where the individual gaussian densities are conditioned on the state and a context variable. The weights of this mixture are then dependent on a density on this context variable, which is conditioned on the state and task index. The intuition behind this is that the weight portion, which is called the high level component identifies task specific information, while the low level policy learns general, shareable knowledge of the different problems. \n\nThe authors adapt the Multitask Policy Optimisation algorithm for their use by introducing an intermediate non-parametric policy, which is derived by setting KL bounds on the policy w.r.t to a reference policy. Having derived a closed-form solution to this, they go on to learn the parametric policy of interest. \n\nThe authors consider 3 settings of experiments. Firstly, they assess the benefits of the hierarchical structure for single task settings in a simulated environment. For the most part, they find that compared to a flat policy, the hierarchical structure shows benefits only if the initial means of the high-level components are sampled to be different. While the experimental results are shown to support this, further discussion of why this is the case would have been welcome. \n\nThe main benefits of the hierarchical policy are shown in the multitask case, in both simulated and real situations. In fact, the authors have shown that the hierarchical case often shows major benefits in difficult, more complicated tasks (reach vs stacking for example).\n\nI think that the paper was very well written. It is nicely structured, with easy to read language, and without unnecessary jargon or clutter. Where necessary, the relevant extra details were provided in the Appendices.\n\nThe following are some additional notes:\n1) It would have been interesting to see how the hierarchal policy faired in new tasks that were not a part of the original training set, compared to a flat multitask policy.  \n2) Further details about how each task is differentiated from each other in the experiments. That is, what are their different goals, which are reflected by the reward functions. \n\nAs such I recommend this paper to be weak accepted.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ezMRHRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While this paper has some interesting experiments. I am quite confused about what exactly the author are claiming is the core contribution of their work. To me the proposed approach does not seem particularly novel and the idea that hierarchy can be useful for multi-task learning is also not new. While it is possible that I am missing something, I have tried going through the paper a few times and the contribution is not immediately obvious. The two improvements in section 3.2 seem quite low level and are only applicable to this particular approach to hierarchical RL. Additionally, it is very much not clear why someone, for example, would select the approach of this paper in comparison to popular paradigms like Option-Critic and Feudal Networks. \n\nThe authors mention that Feudal approaches \"employ different rewards for different levels of the hierarchy rather than optimizing a single objective for the entire model as we do.\" Why reward decomposition at the lower levels is a problem instead of a feature isn't totally clear, but this criticism does not apply to Option-Critic models. For Option-Critic models the authors claim that \"Rather than the additional inductive bias of temporal abstraction, we focus on the investigation of composition as type of hierarchy in the context of single and multitask learning while demonstrating\nthe strength of hierarchical composition to lie in domains with strong variation in the objectives such as in multitask domains.\" First of all, I should point out that [1] looked at applying Option-Critic in a many task setting and found both that there was an advantage to hierarchy and an advantage to added depth of hierarchy. Additionally, it is well known that Option-Critic approaches (when unregularized) tend to learn options that terminate every step [2].  So, if you generically apply Option-Critic, it would in fact be possible to disentangle the inductive bias of hierarchy from the inductive bias of temporal abstraction by using options that always terminate. \n\nIn comparison to past frameworks, the approach of this paper seems less theoretically motivated. It certainly does not seem justified to me to just assume this framework and disregard past successful approaches even as a comparison. While the experiments show the value of hierarchy, they do not show the value of this particular method of creating hierarchy. The feeling I get is that the authors are trying to make their experiments less about what they are proposing in this paper and more about empirical insights about the nature of hierarchy overall. If this is the case, I feel like the empirical results are not novel enough to create value for the community and too tied to a particular approach to hierarchy which does not align with much of the past work on HRL. \n\n[1] \"Learning Abstract Options\". Matthew Riemer, Miao Liu, and Gerald Tesauro. NeurIPS-18. \n[2] \"When Waiting is not an Option: Learning Options with a Deliberation Cost\" Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. AAAI-18. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkxMSq-nKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is rather interesting and is able to solve some difficult tasks. A combination of different learning techniques for acquiring structure and learning with asymmetric data are used. While the combination of methods is new I am not sure that this particular combination of methods to train an HRL policy is sufficiently novel. Is the authors can highlight the effects and contribution of how these methods are combined to indicate this better it would be good.\n\nMore detailed comments:\n- In section 3.2 you mention a reference policy? Can you provide more details on this reference policy? \n- IN the paper it is mentioned that the method collects data, including the reward for each task on a single state, action transition. This assumption seems rather strong. Earlier in the paper, the authors discussed the motivation for learning transferable sub-policies. In the real world, it may not be possible to collect the reward for every kind of task simultaneously.\n- The first evaluation in Section 4.1 uses two humanoid environments. While these environments can be considered difficult that does not seem like the multi-task type environment the method is motivated to work well on. There is little sub-task transfer in this task.\n- In section 4.1 it is noted that the version that initialized with different policy means works best. How are these means initialized?\n- Is the Pile1 collection of tasks really separate tasks? It would be good to have some more details on how these are organized. There may not be a clear definition in the community what is considered a specific task but I am not overly convinced that these \"different\" tasks are separate. Most of them look like a similar version of pick and place.\n- In Figure 2 the hierarchical method is similar in performance on the stack and leave (Pile1) set of tasks and marginally better in the Pile2 set yet does far better on the Cleanup2 set. While these are all simulations with multiple tasks is there some reasoning to why each method looks similar on the Pile1 set of tasks?\n- For the robotic tasks, it is noted that again the baseline methods do well on the \"Reach\" task. It is shown that the RHPO does much better on the Stack task. It would be great if the authors can describe the interesting differences between the tasks. It is not clear how difficult the Stack task is and why it is largely different from Reach + Grasping.\n- For the images on the right of Figure 4, It shows a comparison between the tasks and some \"components\" Are these components the states? The \"components' are not explained well in the papers.\n- There is no algorithm in the main paper which makes it a little difficult to understand the operation of the learning method. For example how exactly is the learning of the two different levels compared? It seems like they are trained together. If they are they should be compared to HIRO or HAC. How is temporal abstraction handled between the two policy layers if they are trained together?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is concerned with improving data-efficiency in multitask reinforcement learning problems. This is achieved by taking a hierarchical approach, and learning commonalities across tasks for reuse. The authors present an off-policy actor-critic algorithm to learn and reuse these hierarchical policies.

This is an interesting and promising paper, particularly with the ability to work with robots. The reviewers did however note issues with the novelty and making the contributions clear. Additionally, it was felt that the results proved the benefits of hierarchy rather than this approach, and that further comparisons to other approaches are required. As such, this paper is a weak reject at this point.
Decision: Reject