Composing Task-Agnostic Policies with Deep Reinforcement Learning
Authors: Ahmed H. Qureshi
Jacob J. Johnson
Yuzhe Qin
Taylor Henderson
Byron Boots
Michael C. Yip
Venue:  (2020)
Keywords: composition
transfer learning
deep reinforcement learning

Abstract: The composition of elementary behaviors to solve challenging transfer learning problems is one of the key elements in building intelligent machines. To date, there has been plenty of work on learning task-specific policies or skills but almost no focus on composing necessary, task-agnostic skills to find a solution to new problems. In this paper, we propose a novel deep reinforcement learning-based skill transfer and composition method that takes the agent's primitive policies to solve unseen tasks. We evaluate our method in difficult cases where training policy through standard reinforcement learning (RL) or even hierarchical RL is either not feasible or exhibits high sample complexity. We show that our method not only transfers skills to new problem settings but also solves the challenging environments requiring both task planning and motion control with high data efficiency.

Published as a conference paper at ICLR 2020
COMPOSING TASK-AGNOSTLE POLICIES WITH DEEP REINFORCEMENT LEARNING
Ahmed H. Qureshi UC San Diego a alquresh @ucsd.
Jacob J. Johnson
Yuzhe Qin UC San Diego y q nêeng ucsd edu
UC San Diego
ijj025@eng ucsd. edu
Taylor Henderson
Byron Boots
Michael C. Yip UC San Diego yipêucsd. edu
UC San Diego jjwest@ucsd.edi
University of Washington bboots washington. edu
ABSTRACT
The composition of elementary behaviors to solve challenging transfer learning problems is of the key elements in building intelligent machines. To date, there has been plenty of work on learning task-specific policies or skills but almost no focus on composing necessary, task-agnostic skills find solution to new problems. In this paper, we propose a novel deep reinforcement learning basedd skill transfer and composition method that takes the agent's 's primitive policies solve unseen tasks. We evaluate our method in difficult cases where training policy through standard reinforcement learning (RL) or even hierarchical RL is either not feasible exhibits high sample complexity. We show that our method not only transfers skills to new problem settings but also solves the challenging environ- ments requiring both task planning and motion control with high data efficiency.
INTRODUCTION
Compositionality is the integration of primitive functions into new complex functions that can fur- ther be composed into even more complex functions to solve novel problems (Kaelbling & Lozano- Pérez 2017). Evidence from neuroscience and behavioral biology research shows that humans and animals have the innate ability to transfer their basic skills to new domains and compose them hi- erarchically into complex behaviors (Rizzolatti al., (001). In roboties, the primary focus is acquiring new behaviors rather than composing and re-using the already acquired skills to solve novel, unseen tasks (Lake al. 2017).
In this paper, we propose a novel policy ensemble composition method 1 that takes the basic, task- agnostic robot policies, transfers them new complex problems, and efficiently learns composite model through standard- or hierarchical-Pe RL (Schulman et al. 2015 2017 Haarnoja et al. 20180 8b] Vezhnevets et al. 2017] Florensa et al., 2017 Nachum et al. 2018). Our model has an encoder- decoder architecture. The encoder is a bidirectional recurrent neural network that embeds the given skill set into latent states. The decoder is feed-forward neural network that takes the given task information and latent encodings of the skills to output the mixture weights for skill set compo- sition. We show that our composition framework can combine the given skills both concurrently (and -operation) and sequentially (or -operation) per the need of the given task. We evalu- ate our method in challenging scenarios including problems with sparse rewards and benchmark it against the state-of-the-art standard- and hierarchical- RL methods. Our results show that the proposed composition framework able to solve extremely hard RL-problems where standard- and hierarchical-RR methods sample inefficient and either fail yield unsatisfactory results.
"Supplementary material and videos available http://sstteeeggeecommmowwooooom

---

Published as a conference paper at ICLR 2020
2 RELATED WORK
In the past, robotics research has been primarily focused on acquiring new skills such as Dynamic Movement Primitives (DMPs) (Schaal et al. 2005) or standard reinforcement learning policies. A lot of research in DMPs revolves around learning compact, parameterized, and modular representa- tions of robot skills (Schaal et al., 2005 Lispeert et al., 2013: Paraschos et al., 2013] Matsubara et al.) 2011 However, there have been quite a few approaches that address the challenge of composing DMPs in an efficient, scalable manner. To date, DMPs are usually combined through human -uman-deffind heuristics, imitation learning or planning (Konidaris et al. 2012) Muelling al.] 2010 Arie al., 2012 Veeraraghaan & Veloso 2008). Likewise, RL (Sutton & Barto 2018)) research is also central- ized around learning new policies (?Schulman et al. 2015 Haarnoja et al. (0118)
8b) for complex
decision-making tasks by maximizing human-defined rewards or intrinsic motivations (Silver et al.,
2016
Qureshi et al. 2017
2018 Levine al. 2016).
To the best of the authors' knowledge, there hardly exists approaches that simultaneously combine and transfer past skills into new skills for solving new complicated problems. For instance. [Toorov [Todorow (2099), |Haarnoja et al.] (201 8a) and (Sahni et al. (2017) require humans to decompose high-level tasks into intermediate objectives for which either Q-functions or policies are obtained via learning- The high-level task is then solved by merely maximizing the average intermediate Q-functions or combining intermediate policies through temporal-logic. Note that these approaches do not combine task-agnostic skills thus lack generalizability and the ability to transfer skills to the new domains. A recent and similar work to ours is multiplicative composition policies (MCP) framework (Peng et al. 2019). MCP comprises i) a set of trainable Gaussian primitive policies that take the given state and proposes the corresponding set of action distributions and ii) a gating function that takes the extra goal information together with the state and outputs the mixture weights for composition. The primitive policies and a gating function trained concurrently using reinforcement learning. In their transfer learning tasks(Peng et al., 2019), the primitive polices parameters are kept fixed, and the gating function is trained to output the mixture weights according to the new goal information. In our ablation studies, we show that training an MCP like-gating function that directly outputs the mixture weights without conditioning on the latent encoding of primitive actions gives inferior performance compared to our method. Our method utilizes all information (states, goals, and primitive skills) in structured way through attention framework, and therefore, leads to better performance.
Recent advancements lead to Hierarchical RL (HRL) that automatically decomposes the complex tasks into subtasks and sequentially solves them by optimizing the given objective function (' Veze nevets et al., 2017 Nachum et al. 2018). In a similar vein, the options framework (Sutton et al.) 1999 Precup 2000) proposed that solves the given task through temporal abstraction. Recent methods such as option-critic algorithm (Bacon et al. 2017) simultaneously learns a set of sub-level policies (options), their termination functions, and a high-level policy over options to solve the given problem. Despite being an exciting step, the option-criti algorithm is hard to train and requires reg- ularization (Vezhnevets et al. 2016) Harb al., 2018), else it ends up discovering options for every time step or a single option for the entire task. In practice, the sub-level options or objectives obtained via HRL are inherently task-specific and therefore cannot be transferred to new domains.
3 BACKGROUND
We consider a standard RL formulation based on Markov Decision Process (MDP) defined by tuple (S. A, P, R), where S and A represent the state and action space, P is the set of transition probabilities, and R denotes the reward function. At time I. 2 0, the agent observes a state E S and performs an action a, E A. The agent's 's action o, transitions the environment state from 8; E S to 8t+1 E S with respect to the transition probability P(s (5c az) and leads to reward rt E R.
For compositionality, we extend the standard RL framework by assuming that the agent has ac- cess to the finite set of primitive policies II = { J.N that could correspond to agent's skills, controller, or motor- priorr-rimitiies. Our composition model is agnostic to the structure of primitive policy functions, but for the sake of this work, we assume that each of the sub-policies ( solves the MDP defined by a tuple (Š, A, P. Re.. Therefore, S, P and र$ are the state-space, transition probabilities and rewards of the primitive policy n; respectively. Each of the primitive policies n. $ A [0, Vi E [0, ,N], takes a state å E S and outputs distribu-
2

---

Published as a conference paper at ICLR 2020
[wo, Mi. Myy]
Decoder
hp
hD
4:
wy W 9)
hf
'Encader
9cl
ITx
Primitive Policies
Figure 1: Policy ensemble composition model that takes the state information & and a set of primi- tive policies' output {å, [ to compute a composite action 0g.
tion the agent's action space A. We define composition model composite policy TÕ : x A x A + [0, 1J, parameterize by 0, that outputs a distribution over the action space conditioned on the environment's current state s E S and the primitive policies {å; E AA) ~ II. The state space of the composite model is S = [$, g]. The space G could include any task spe- eific information such as target locations. Hence, our framework, the state inputs to the primitive policies II and composite policy z need not to be the same.
In remainder of this section, we show that our composition model solves an MDP problem. To avoid clutter, we assume that both primitive policy ensemble and composite policy have the same state space S, i.e., g 0. The composition model samples an action from a distribution parameterized by the actions of sub-level policies and the state 8 E S of the environment. Therefore, we can augment the given state space S as S N+1 S, where AN+ {Ai) are the outputs of sub-level policies. Hence, compositional MDP defined as (S", A,, AA,Po, R) where S A S is the new composite state-space, A is the action-space, Pe S x S x A - [0, 1] is the transition probability function, and is the reward function for the given task.
POLICY ENSEMBLE COMPOSITION
In this section, we present our policy ensemble composition framework, shown in Fig. 1 Our composition model consists of i) the encoder network that takes the outputs primitive policies and embeds them into latent spaces; ii) the decoder network that takes current state s, of the environment and the latent embeddings from the encoder network to parameterize the attention network; ii) the attention network that outputs the probability distribution over the primitive low-level policies representing their mixture weights. The remainder of the section explains the individual models of composition framework and the overall training procedure.
4.1 ENCODER NETWORK
Our encoder is bidirectional recurrent neural network (BRNN) that consists of Long Short-Term Memory units (Hochreiter & Schmidhuber 1997). The encoder takes the outputs of the policy ensemble {ā:}-= and transform them into latent states of forward and backward RNN, denoted as {hr N and (h Ji=0 N41. respectively, where h. hy E Rd; Vi E [0, 1. N + 1]. The N + states of forward and backward RNN corresponds to their last hidden states denoted as hf and h², respectively, in Fig.
4.2 DECODER NETWORK
Our decoder is simple feed-forward neural network that takes the last hidden states of the forward and backward encoder network, iee. {he, h*). and the current state of the environment to map them into latent space h RRd, The state input the decoder network is defined [8.g].
3

---

Published as a conference paper at ICLR 2020
where & E S is the state imput to the low-level policy ensemble and g 9 could be any additional
information related to the given task, e.g., goal position of the target to be reached by the agent.
4.3 ATTENTION NETWORK
The composition weights (see Fig. 1 fwo E [0, 10 are determined by the attention network as follows:
W tan bf + W6 hy Wa h); Vi E [0
(1)
where W,, Wr Wa E Raxd and W E Rd. The weights {w:}i- for the composite policy are computed using gumbel-softmax denoted as softmax(q/T)) where T is the temperature term (Jang et al. 2016).
4.4 COMPOSITE POLICY
Given the primitive policy ensemble { the composite action is the weighted sum of all primitive policies outputs, i.c., T Σi WyTT,, Since, we consider the primitive policies to be Gaussian distributions, the output of each primitive policy is parameterized by mean μ and variance a, i.e., {ā: ~ ( o; o. < { Hence, the composite policy can be represented as TH Σ (p,, o;), where (.)) denotes Gaussian distribution, and Σ, Wy 1. Given the mixture weights, other types of primitive policies, such as DMPs (Schaal et al. 2005). can also be composed together by the weighted combination of their normalized outputs.
4.5 COMPOSITE MODEL TRAINING OBJECTIVE
The general objective RL methods is maximize the cumulative expected reward, i.c., (TT) Eng - r'rr), where 7 (0, 1] is discount factor. We consider the policy gradient methods to update the parameters 9 of our composite model, i.e., 0 + 0 + 17 V J(z)), where is the learning rate. We show that our composite policy can be trained through standard RL and HRL methods, described as follow.
4.5.1 STANDARD REINFORCEMENT LEARNING
In standard RL, the policy gradients are determined by either on-policy off-policy updates ('Schulman et al. 2015 2017; Haarnoja et al., 2018b) and any them could be used to train composite model. However, this paper, consider off-policy soft-actor critic (SAC) method (Haarooa et al., (018b) for the training of our policy function. SAC maximizes the expected entropy H(-) in addition the expected reward, i.c.,
T
(( = Σ Englr(s az) ))++ + XH((T((-1s:))
t=
where À is a hyperparameter. We use SAC as motivates exploration and has been shown to capture the underlying multiple modes of an optimal behavior. Since there no direct method estimate a low-variance gradient of Eq (2), we use off off-policy value canction-based optimization algorithm (for details refer to Appendix of supplementary material).
4.5.2 HIERARCHICAL REINFORCEMENT LEARNING
In HRL, there are currently two streams task decomposition through sub-goals (Nachum et al., 2018) and option framework (Bacon et al. 2017) that learns temporal abstractions. In the options framework, the options can be composite policies that are acquired with their termination functions. In task decomposition methods that generate sub-goal through high-level policy, the low-level policy can be replaced with our composite policy. In our work, we use the latter approach (Nachum et al. 2018), known HIRO algorithm, train policy function.
Like, standard HIRO, we use two level policy structure. At each time step t, the high-level policy The with parameters 0', observes state s, and takes action by generating a goal gi E S in the state-space S for the composite low-level policy rolow to achieve. The welow takes the state
4

---

Published as a conference paper at ICLR 2020
+
(a) Ant Random Goal
(h) Ant Cross Maze
(c) Pusher
(d) HalfCheetah Hurdle
(e) Ant Maze
(f) Ant Push
(g) Ant Fall
Figure 2: Benchmark control and manipulation tasks requiring an agent to reach or move the object the given targets (shown in red for pusher and green for rest).
St. the goal St. and the primitive actions {ā. to predict a composite action or through which an agent interacts with the environment. The high-level policy is trained to maximize the expected task rewards given by the environment whereas the composite low-level policy is trained to maximize the expected intrinsic reward defined the negative of distance between current and goal states, i.e., perform off-policy correction of the high- level policy experiences and we train both high- and low-level policies via TD3 algorithm (Fujimoto et al. 2018) (for details refer to Appendix A.2 of supplementary material).
|ss + St. 8t+1 To conform with HIRO settings,
EXPERIMENTS AND RESULTS
We evaluate and compare our method against standard RL, and HRL approaches in challenging environments (shown in Fig. 2 that requires complex task planning and motion control. The imple- mentation details of all presented methods and environment settings are provided in Appendix C of supplementary material. We also do an ablative study in which take away different components of our composite model to highlight their importance. Furthermore, we depict attention weights of our model navigation task highlight ability of concurrent and sequential composition.
We consider the following seven environments for our analysis: (1) Pusher: A simple manipulator has to push an object to given target location. (2) Ant Random Goal: this environment, quadruped-Ant is trained to reach the randomly sampled goal location in the confined circular region. (3) Ant Cross Maze: The cross-maze contains three target locations. The task for quadruped Ant to reach any of the three given targets by navigating through 3D maze without collision. (4) HalfCheetah Hurdle: In this problem, the task for a halfcheetah is to run and jump over the three barriers reach the given target location. (5) Ant Maze: A D-shaped maze poses challenging navigation task for a quadruped-Ant. In this task, the agent is given random targets along the maze reach while training. However, during the evaluation, we test the agent for reaching the farthest end of the maze.
(6) Ant Push: A challenging environment that requires both task and motion planning. The environment contains movable block, and the goal region located behind that block. The task for an agent is to reach the target by first moving to the left of the maze so that it can move up and right push the block of the way for reaching the target. (7) Ant Fall: A navigation task where the target is located across the rift in front of the agent's initial position. There also happen to be a moveable block, so the agent has to move to the right, push the block forward, fill the gap, walk across, and move to the left reach the target location. (8) Multi-goal Point Mass: this scenario, the task is to navigate point-mass to one of the four goals located diagonally to agent initial position.
In all tasks, we also acquire primitive skills of the agent for our composite policy. For Ant, we use four basic policies for moving left, right, up. and down. The pusher uses two primitive policies that are to push an object to the left and down. HalfChectah hurdle environment, the low-level policies include jumping and running forward. Finally fot the point-mass robot, the composition model takes
5

---

Published as a conference paper at ICLR 2020
HerCwesk Horts
Puter
Zcm
PPO TAPO
Compostton
Figure 3: Comparison results of our method against several standard RL methods averaged over ten trials in a set of difficult tasks. The vertical and horizontal axis represents the distance of the agent/object from the target and environment steps in millions, respectively. Note that compo- sition framework learns to solve the task with high samples efficiency, whereas other benchmark methods either fail or perform poorly.
four policies for moving in the up, down, left and right directions. Furthermore, in all environments, except pusher, the primitive policies were agnostic of high-level tasks such as target locations) that were therefore provided separately to our composite model via decoder network. This highlights the ability of our model to transfer basic robot skills to novel problems.
Environments
Methods SAC TRPO
Ant Random Goal
Ant Cross Maze
Pusher
HalfCheetah Hurdle
0.21 + 0.08
0.78 = 0.06
0.17 + 0.02
0.79 + 0.01
1.09 = 0.15
0.85 = 0.15
0.64 + 0.09
0.87 = 0.05 0.88 + 0.04
PPO Our Method
1.06 + 0.11
0.95 + 0.07
0.71 + 0.06
0.11 + 0.05
0.11 + 0.02
0.14 + 0.02
0.27 + 0.22
Table 1: Performance comparison of our model against SAC (Haamooj et al. 2018b). TRPO (Schul-
man et al., 2215), and PPO (Schulman et al. (017)) on benchmark control tasks in ferms of distance (lower the better) of an agent from the given target. The mean final distances with standard devia- tions over ten trials are reported. We also normalize the reported values by the agent initial distance from the goal so values close bold) accomplishes the tasks by reaching goals whereas other methods fail except for SAC in simple Pusher and Ant Random Goal environments.
or higher show failure. can be seen that our method (shown in
5.1 COMPARATIVE STUDY
In our comparative studies, we divide our test environments into two groups. The first group includes Pusher, Random Goal Ant, Ant Cross Maze, and HalfCheetah-Hurrde environments, whereas the second group comprises the remaining environments that require task and motion planning under weak reward signals.
In the first group of settings, we compare our composite model trained with SAC (Haarnoja et al. 2018b) against the standard Gaussian policies obtained using SAC (Haarnoja al., (0088)), PPO (Schulman et al. 2017), and TRPO (Schulman et al. 2015). We exclude HRL methods in these cases as the environment rewards sufficiently represent the underlying task, whereas HRL approaches are applicable in cases that have a weak reward signal or require task and motion planning. Table presents the mean and standard deviation of the agent's 's final distance from the given targets after the end of an evaluation rollout over the ten trials. Fig. 3 shows the mean learning performance over all trials during the three million training steps. In these set of problems, TRPO and PPO entirely fail to reach the goal, and SAC performs reasonably well but only in simple Ant Random Goal and Pusher environments as fails in other cases. Our composite policy obtained using SAC successfully solves all tasks and exhibit high data-efficiency by learning in merely few thousand training steps.
In our second group of environments, we use distance-based rewards that are weak signals as greed- ily following them does not lead to solving the problem. Furthermore, in these environments, poli- cies trained with standard RL, including our composite policy, failed to solve the problem even after 20 million training steps. Therefore, trained composite policy with HIRO (Nachum et al. 2018) and compared its performance against standard HIRO formulation (Nachum et al. 2018). We also tried to include option-critic framework (Bacon al. 2017, but were unable get any

---

Published as a conference paper at ICLR 2020
Maze
Push
Compostiion BRO Standvii-ee
HRD Standert Art
HRR Tortue Art
HRR (Re Tortus Fertriieed
Figure 4: Performance comparison of our composition model trained with HIRO on standard Ant (150 units torque limit) against three variants of standard HIRO formulation in three challenging environments. The pretrained HIRO model undergoes 4 million steps of extra training to counter for the training time utilised by premitive skills of our composition model. The low-torque-Ant has 30 units toruqe limit. We report mean and standard error, over ten trials, of agent's final distances from the given goals, normalized by their initial distance, over 10 million steps.
2--
Ztack (MSBO0
Composition
Composition Wwthhtttttthrrrrteeee
Composition Witho Without-Aree AMBBRNN
Composition Without Without.BRNN
BRNN
Ablative Study: Performance comparison, averaged over ten trials, of our composite model against its ablated variations that lack attention model, bidirectional-RA. (BRNN) or both attention and BRNN (AtBBRNN) in three different environments.
Figure 5:
considerable results with their online implementation despite several attempts with the parameter tuning. One of the reasons option-critic fails is because relies purely on task rewards to learn, which makes them inapplicable for cases with weak reward signals (Nachum al. 2018).
Nachum et al. (2018) use a modified Ant in their paper that has 30 units joint torque limit (low- torque-Ant). For composition method, use Mujoco standard Ant that has torque limit of 150 units which makes the learning even harder as the Ant is now more prone to instability than towttorgue-Att. For standard HIRO formulation, on standard Mujoco Ant, ii) HIRO trained on low-torque Ant (as in (Nachum et al.) 2018);; iii) HIRO pretrained for the equal number of training steps used to train the primitive policies of our composition method on a low-torque Ant (For more details, refer to Appendix C.I).
trained three variants i) HIRO trained
Fig. 4 shows the learning performance, averaged over trials, during 10 million steps. It can be seen that the composite policy with HIRO outperforms standard HIRO (Nachum et al. 2018) by significant margin. It also certifies the utility of solving RL tasks using composition by leveraging basic pre-acquired skills. Furthermore, HIRO performs poorly with standard Ant, even pretraining did not improve the performance, as imposes harder control problem.
5.2 ABLATIVE STUDY
We remove bidirectional RNN (BRNN) (similar (Peng et al. 2019)). attention-network, and both attention-networt and BRNN (AuBRNN) from our composition model to highlight their importance in the proposed architecture in solving complex problems. We train all models with SAC (Haarnoja et al. 2018b). The first model is our composite policy without attention in which the decoder network takes the state information and last hidden states of the encoder (BRNN) to directly output actions rather than mixture weights The second model is without attention network and BRNN, a feed-forward neural network that takes the state information and the primitive actions and predicts the action to interact with the environment. The third model is without BRNN, it is feed- forward neural network that takes the state and goal information, and output the mixture weights. The mixture weights then used to combine the actions from primitive policies. This setting is
7

---

Published as a conference paper at ICLR 2020
same (Peng et al., 2019) for their transfer learning problems. Fig. shows the mean performance comparison. over ten trials, of our composite model against its ablated versions on a Ant Random Goal, Cross Maze Ant, and Pusher environment. We exclude remaining test environments in this study as ablated models completely failed to perform or show any progress.
Note that the better performance of our method compared to ablated versions highlight the mer- its of our architecture design. Intuitively, BRNN allows the dynamic encoding of skill set that could be of variable lengthe And our decoder network uses the encoded skills together with given state and goal information (states, goals, and primitive skills) in a structured way using at- tention framework and provides significantly better performance than other ablated models, includ- Furthermore, another merit of using the attention network is that it bypasses or actions and
ing (Peng et al., 20112)
the complex transformation of action embeddings (composition- wearttio-wihhoutatteenntn state-informatio (composition-withe without-AAtBRNN directly to action space. Hence, the proposed architecture crucial for the composition of task agnostic sub-level policies to solve new problems.
5.3 DEPICTION OF ATTENTION WEIGHTS
In order to further assess the merit of utilizing an attention network, we apply our model to simple 2D multi-goal point-mass environment as shown in Fig. 6 The point-mass is initial- ized around the origin (with variance o² 0.1) and must randomly choose one of four goals reach. For this experiment we use dense re- wards with both positional and actuation cost. Primitive policies of up (+y), down (-y), left (-x), and right (+-r) were trained and com- posed to reach goals, represented here as red dots, in the "diagonal" directions where com- bination two or more primitive policies are required to reach each goal.
UOAR
Figure 6: Each path corresponds to its adja- cent attention weight mapping. The weighting "strength" of each primitive policy is depicted for each step (i.e. up (U), down (D), left (L), and right (R)). Each path begins the origin and ends when the point-mass is within one unit of goal. The plot contours represent the position cost.
The four mappings in the figure give us insight into how the attention network is utilizing the given primitives to achieve the desired task. At each step in given path, the weights {w, j for each low-level policy are assigned and com- posed together move the point-mass in the desired direction. We see here that even with some noise and short-term error, the attention weights are strongest for primitive policies that move the point-mass its chosen goal. We also see that multiple policies are activated at once to achieve more direct movements toward the goal, op- posed to "stair-stepping" where only one primitive activated time. Both of these observations point to the concurrent and sequential nature of this composition model. Please refer to Appendix D for depiction of attention weights in other complicated environments.
CONCLUSIONS AND FUTURE WORK
We present a novel policy ensemble composition method that combines a set of independent and task-agnostic primitive policies through reinforcement learning to solve the given tasks. We show that our method can transfer the given skills to novel problems and can compose them both sequen- tially (or -operation) and concurrently (and -operation) to find a solution for the task in hand. Our experiments highlight that composition is vital for solving problems requiring complex motion skills and decision- decision-making where standard reinforcement learning and hierarchical reinforcement learn- ing methods either fail or need a massive number of interactive experiences to achieve the desired results.
In our future work, we plan to extend our method to automatically acquire the missing skills in the given skillset that necessary to solve the specific problems (refer to Appendix B for prelimi-
However, this paper we only consider primitive skill of fixed size
8

---

Published as a conference paper at ICLR 2020
nary results). We also aim towards system that learns the hierarchies of composition models by combining primitive policies into complex policies that would further be composed together for combinatorial outburst in the agent's 's skillset.
ACKNOWLEDGMENTS
We would like thank Ben Eysenbach for sharing his halfcheetah-hurrdee environment implemen- tation.
REFERENCES
Hiroaki Arie, Takafumi Arakaki, Shigeki Sugano, and Jun Tani. Imitating others by composition of primitive actions: neuro-dynamic model. Robotics and Autonomous Systems, 6055);72--74,, 2012.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint arXiv:/704.03000 2017.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1802.09477 2018.
Tuomas Haarnoja, Vitchyr Pong. Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. arXiv preprint 1r3iv:1803.06773, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01299. 2018b.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting not an op- tion: Learning options with deliberation cost. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Auke Jan ljspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2): 328-373, 2013.
Eric Jang. Shixiang Gu and Ben Poole. Categorical reparametterization with gumbel-softmax. arXiv preprint arXiv:1611.0114 2016.
Lestie Pack Kaelbling and Tomás Lozano-Pérez. Learning composable models of parameterized skills. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 886- 893. IEEE, 2017.
George Konidaris, Scott Kuindersma, Roderie Grupen, and Andrew Barto. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research, 31(3): 360-375, 2012.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17()):3334-333, 2016.
Takamitsu Matsubara, Sang-Hu Hyon, and Jun Morimoto. Learning parametric dynamic movement primitives from multiple demonstrations. Neural networks, 24(5):493-500, 2011.

---

Published as a conference paper at ICLR 2020
Katharina Muelling, Jens Kober, and Jan Peters. Learning table tennis with mixture of motor primitives. In Humanoid Robots (Humanoids), 2010 10th IEEE-RAS International Conference on, pp. 411-416. 2010.
Ofir Nachum, Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. arXiv preprint /80.: 08886, 2018.
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilisticc move- ment primitives. In Advances in neural information processing systems, pp. 2616-2624, 2013.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mep: Learn- ing composable hierarchical control with multiplicative compositional policies. arXiv preprint (00::09050000 2019.
Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000.
Ahmed Hussain Qureshi, Yutaka Nakamura, Yuichiro Yoshikawa, and Hiroshi Ishiguro. Show, tend and interact: Perceivable human-robot social interaction through neural attention q-network. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1639-1645. IEEE, 2017.
Ahmed Hussain Qureshi, Yutaka Nakamura, Yuichiro Yoshikawa, and Hiroshi Ishiguro. Intrinsically motivated reinforcement learning for human-robot interaction the real-world. Neural Networks, 2018.
Giacomo Rizzolatti, Leonardo Fogassi, and Vittorio Gallese. Neurophysiological mechanisms un- derlying the understanding and imitation of action. Nature reviews neuroscience, 2(9):661, 2001.
Himanshu Sahni, Saurabh Kumar, Farhan Tejani, and Charles Isbell. Learning to compose skills. 2017.
arXiv preprint
Stefan Schaal, Jan Peters, Jun Nakanishi. and Auke Ijspeert. Learning movement primitives. In Robotics research. the eleventh international symposium, pp. 561-572. Springer, 2005.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-
211. 1999.
Emanuel Todorov. Compositionality of optimal control laws. In Advances in Neural Information Processing Systems, pp. 1856-1866 2009.
Harini Veeraraghavan and Manuela Veloso. Teaching sequential tasks with repetition through demonstration. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volune Volume 3. pp. 1357-1360. International Foundation for Autonomous Agents and Multiagent Systems, 2008.
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Aga- piou, et al. Strategic attentive writer for learning macro-actions. In Advances in neural information processing systems, pp. 3486-3494, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv: 2017.
10

---

Published as a conference paper at ICLR 2020
Algorithm 1: Composition model training using SAC
Initialize parameter vectors Φ, 6', 0, Input: Primitive policies II = { for each iteration do
for each environment step do Compute primitive policies state $t t 9: 99: Sample primitive actions {ār, ~ (
Sample composite action at ~ 1g(ay|s1, (
Sample next state St+1 p(st+1ls1,
M M U {(st, 0t, (ār, J.N i= rr; 81+1)
for each gradient step do
Update value function @ + o - Vo V.
Update Q-function & - Vi (o(E)
Update policy 0 + 1) Vo (0)
TO + (1 -)o''
A COMPOSITE MODEL TRAINING ALGORITTMM
A.1 TRAINING WITH SOFT ACTOR-CRITIC
In this section, we briefly describe the procedure to train our composition model using SAC (Haarnoj et al. 2018b). Although any RL method can be used to optimize our model, we use SAC as it is reported to perform better than other training methods. Our composite policy is tractable function (a 8t,, (Te ) parameterized by The composite policy update through SAC requires the approximation of Q- and value-functions. The parametrized value- and Q-function are denoted as Vo(#; (5c) with parameters d, and (st, ae) with parameters E, respectively. Since, SAC algorithm build the soft-policy iteration, the soft value-function (s:) and soft Q-function (st,, a.) are learned by minimizing the squared residual error Jv (Ø) and squared Bellman error JE(E), respec- tively, i.e.,
(v(Ø) Bar (Vo (ss)))
(3)
Jo(E) = E(s (000mm/((00 ar) - Q( ac))
(4)
where M is replay buffer, ((s)) Earwa [g ai) - log (a((s))) and Q is the Bellman target computed as follows:
( a.) r(st, a.) + + VVr + 1)]
(5)
The function (8t is the target value function with parameters ø'. The parameters o' are the mov- ing average of the parameters e computed as Th (1 T)0', where is the smoothing coefficient. Finally the policy parameters are updated by minimizing the following expected KL-divergence.
exp(Qe(st)))
J (0) E.,-M PK
(T(-|s))]1
66)
Ze(st)
where Zę is a partition function that normalizes the distribution. Since, just- SAC, our Q-function is differentiable, the above cost function can be determined through simple reparametization trick, see Haarnoja et al. (2018b) for details. Like SAC, we also maintain two Q-functions that are trained independently, and we use the minimum of two Q-functions to compute Eqn. and Eqn. 6 This way of using two Q-function has been shown to alleviate the positive biasness problem in the policy improvement step. The overall training procedure is summarized in Algorithm 1.
A.2 TRAINING WITH HIRO
In this section, we outline the algorithm to train composite policy through HIRO that employs the two level policy structure. The high-level policy generates the sub-goals for the low-level composite
11

---

Published as a conference paper at ICLR 2020
Algorithm 2: Composition model training using HIRO
Initialize parameter vectors Φ, 6', 0, Input: Primitive policies II = { for each iteration do
for each environment step do Compute primitive policies state $t t 9t 99:
Sample primitive actions (âr, ~ (T{)))
Sample high-level action 9r ~ ($r)
Sample composite action at ~ Tz(atlst, (ār))
Sample next state St+1 ~ p(s2++1 at)
M M U {(st: It, at, (i re, 81+1)
for each gradient step do Sample mani-bbtchh with c-step transitions IN {(g", sh.tte f- fa ( j= reete--)) 1) ~ M Ji=0 &- (rr . Di, 8
Compute rewards for low-level policy (1
B
Update c:lo w.r.t Q using {(*k; Ik, ax, (āi,e) 1= r 8k41 k=
Nachum et al.
2018
Update hh w.r.t Qhi using {(*,, Sks Einh Tki 8k+c)
Nachum et 2018
policy achieve. The low-level composite policy also have access to the primitive policy actions. Like HIRO, we use TD3 algorithm (Fujimoto et al., 2018) to train both high-level and low-level policies with their corresponding Q-functions, QtT and Qt respectively. The low-level policy To Toloow with parameters 0, is trained to maximize the Q-values from the low-level Q-function Q for the given state-goal pairs. The Q-function (Q')) parameters & are optimized by minimizing cemporal-differenc error for the given transitions, i.e.,
(L) = (f' 9t, S1+1) + (St11, cito (s4+1.9411 ( Q (s,, (7)
where ( (t, sc+1) -1|s; It st+1l| and g++1 T hii (st+1).
The high-level policy Thi with parameters 0', is trained to maximize the values of Q The Q- function (Qht) parameters ç' are trained through minimizing the following loss for the given transi- tions.
-= (st, as, $141) + Q (Sttes The (st+c)) - (se.
(8)
During training, the continuous adaptation of low-level policy poses non-stationery problem for the high-level policy. To mitigate the changing behavior of low-level policy, Nachum et al. (2018) introduced off off-policy correction of the high-level actions. During correction, the high-level policy action g is usually re-labeled with I that would induce the same low-level policy behavior as was previously induced by the original high-level action (for details, refer to (Nachum et al. 2018 Algorithm presents the procedure to train our composite policy with HIRO.
B FUTURE RESEARCH DIRECTIONS & DISCUSSION
In this section, we present preliminary results to highlight the potential research avenues to further extend our method. Apart from compositionality, another research problem in the way of building intelligent machines is the autonomous acquisition of new skills that were lacking in the system and therefore, hindered it from the solving the given task (Lake et al. 2017). demonstrate that composition model holds the potential for building such system that can simultaneously learn new missing skills and compose them together with the existing skill set to solve the given problem.
this section, we
Fig. 7 shows a simple ant-maze environment which the 3D quadruped-ae has to reach the target, indicated as a green region. In this problem, we equip our composition model with single primitive
12

---

Published as a conference paper at ICLR 2020
New skill acquisition task. The composite model has access to a primitive policy for moving right and trainable policy function. Our method trained the new function to move in the upward direction to reach the given goal.
Figure 7:
policy for walking in the right direction and a trainable, parametric policy function. The trainable policy function takes the state without goal information and outputs the parameters for Gaussian distribution to model the robot's action space. Note that the composite model requires a skill of walking upward in addition moving in the right direction reach the given goal. To determine our composition model can teach the new parametric policy function to move upward, we trained composition model along with the new policy using the shared Q- and trained, we observed that our composition model learned the parametric policy function for walking in an upward direction with slight right-turning behavior.
Once
The plot in Fig. shows the performance of our composition model and standard RL policy obtained with SAC in this environment. The vertical axis indicates the distance from the target location averaged over five randomly seeded trials. The horizontal axis shows the number of environment steps in millions. It can be observed that our method converged faster than standard RL policy. shows that our approach is also utilizing the existing skills and therefore learns faster than a standard policy function that solves the entire task from scratch. Furthermore, not impose any constraint on the new policy function that would make it learn only the missing skills rather than replicating existing skills or solving the given task entirely by itself. We leave the formulation of such constraint for autonomous acquisition of new, missing skills to future works.
current settings, we do
C IMPLEMENTATION DETAILS
C.I PRETRAINING BENCIMARK METHOP
In our results, also consider the pretraining of benchmark methods for equivalent amount of time as used to train the primitive policies for our composition model for a better analysis. The total number of steps needed to train primitive skills of Ant (up, down, left, and right), halfcheetah (run and jump), and pusher (down and left) were 1, O.5, and 0.1 million, respectively. In Fig. 4 HIRO was pretrained for both standard (30 units) and low-torque-ant (150 units). The results of HIRO standard- ant are excluded in Fig to avoid clutter as they led to no improvement in the performance. For the non-hierarchical tasks presented in Fig. 3 the benchmark methods are not pretrained. However, to account for pretraining, the performance of other methods (TRPO, PPO, SAC) can be accessed after 1, 0.5, and 0.1 million for the ant, halfcheetah, and pusher environments, respectively. Also, notice that in Fig. 3 the pretraining will have no significant effect on the performance of TRPO and PPO all environments and SAC in half-cheetah hurdle and cross-maze-ant environments. Furthermore, since pusher and random- random-goal-an goal-ant are relatively simple environments (due to no obstacles or maze around), pretrained SAC can perform similar to our composition method.
C.2 ENVIRONMENT DETAILS
In this section, we present the environment details including reward functions, primitive policies, and state space information. The reward functions presented in the Table 3 together with the overall reward scaling values. For Ant and halfcheetah--urre environments, the primitive policies trained in free-space that had mazes, hurdle or fixed targets in the surrounding.
13

---

Published as a conference paper at ICLR 2020
C.2.1 ANT ENVIRONMENTS
In these environments, we use 8 DOF four-legged Ant with 150 units torque limit. The primitive policies of moving left, right, down and up were shared across all these tasks. These sub-level- policies were obtained using SAC and each policy was trained for about 0.25 million steps. these environments, the information g in the state [g.g] corresponds to the target location. Let us introduce the notation to defined reward function. Let Tays 9zy. u, and Se denote xy-position of the robot's s torso, xy-position of the goal, joint torques, and contact-cost, respectively. The scaling factors are defined as A. The reward function for the following environments is defined as with reward scaling of 5 units:
->gl 9ryl1 + Uxy + (II - >alluul Aafe
In this environment, the ant has to navigate to any randomly sampled target within the confined circular region of radius units. The goal radius is defined to be 0.25 units. The reward function coefficients Ag. Au Ag Act, and X are 0.3, 0.0, 0.05, 0.01, and 0.001, respectively.
Ant Random Goal:
In this environment, the ant has to navigate through the 3D maze to reach any the target sampled from the three targets. The goal radius is defined to be 1.0 units. The reward function parameters are same as for the random-goal ant environment.
Ant Cross Maze:
For the remaining environment (Ant Maze, Ant Push and Ant Fall), we use the following reward function with no reward scaling:
XyI1rzy= - Izyz 112 Acal|ul| - Xefe where coefficients Xg. Act and de are set to be 1.0, 0.05, and 0.5 x 10-4
(10)
Ant Maze: In this environment, we place the Ant in a D-shaped maze for a navigation task between given start and goal configurations. The goal radius is defined to be 5 units. During training, the goal is uniformly sampled from [-4, 20] x [-4, 20] space, and the Ant initial location is always fixed at (0, 0). During testing, the agent evaluated to reach the farthest end of the maze located at (0, 19) within L2 distance of
Ant Push: In this environment, the Ant is initially located at (0, 0) coordinate, the moveable block is at (0, 8). and the goal is at (0, 19). The agent is trained reach randomly sampled tar- gets whereas during testing, we evaluate the agent to reach the goal at (0, 19) within L2 distance of 5.
Ant Fall: In this environment, the Ant has to navigate in a 3D maze. The initial agent loca- tion (0, 0), and a movable block is at (8, 8) at the same elevation as Ant. Their is a rift in the the other side of the rift, the Ant must push the block down into the rift, and then step on it to get to the goal position.
region [-4, 12] x [12, 20). To reach the target
Parameters
SAC
HIRO
TRPO
PPO
Learning rate (7)
x 10
1 10
Discount factor (r))
0,99
0.99
0.99
0.99
Nonlinearity in feedforward networks
ReLU
ReLU
ReLU
ReLU
Minibatch samples
256
128
Replay butler size
10
2 10
Batch
1000
Target parameters smoothing enefficient (r)
0.005
0.005
Target parameters update interval
Gradient steps
0.01
0.01
Gumbel-softme temperature (T)
0.5
0.5
Table 2: Hyperparameters
C.2.2 PUSHER
In pusher environment, a simple manipulator has to move an object to the target location. The primitive policies were to push the object to the bottom and left. These low-level policies were obtained via SAC after 0.1 million training steps. In this environment, the state information for both primitive policies and the composite policy include the goal location. Therefore, G, in this case, is null. The reward function is given as:
-> -Agllozy Gzyl
Aollrzy -oyyl >ccl1u|1
(11)
14

---

Published as a conference paper at ICLR 2020
Model Architectures
Hidden units
Hevel Policy: Three layer feed forward network
300
Encoder Network: Bidirectional RNN with LSTMs
128
Composition-HHle HIRO
Decoder Network (Single layer feed forward netwark)
128
Attention Network: Wy. Ws, Wa R dd R
Encoder Network: Bidirectional RNN with LSTMs
128
Composition-SAC
Decoder Network (Single layer food forward network)
128
Atiention Network: Wy, Ws. Wa Re W R
128
High-level Policy: Three layer food forward netwark
300
HIRO
Low-level Policy: layer forward network
Standard RI. policy
Two layer feed forward network
255
Table 3: Network Architectures. The right most column shows the hidden units per layer.
where Owys Izy- Tay. and u are xy-position of object, xy-position of goal, xy-position of arm, and joint-torques. The coefficients Ag. λo and Aet are 1.0, 0.1, and 0.1, respectively.
C.2.3 MALFCHEETAH-HHRRDLE
In haffcheetah-hurdle environment, a 2D cheetah has to jump over the three hurdles to reach the target. The primitive policies include forward and jump where each poilcy was trained with SAC for 0.25 million steps. Furthermore, in this environment, the information in the state [8, g] corresponds to the x-position of the next nearest hurdle in front of the agent as well as the distance from that hurdle. The reward function is defined as:
-Agllray - -gyll Ancho(-) + Arg (goal) + Aa + A, - ).co(.)
(12)
where rzys Gry. Uz and Vs are xy-position of robot torso, xy-position of goal, velocity along z-axis, and velocity along x-axis, respectively. The function he(-) returns a count indicating the number of hurdles in front of the robot. The indicator function / (goal) returns target otherwise O. The function cc(-) is collision checker which returns if the agent collides with the hurdle otherwise O. The reward function coefficients Xg. Apes Arg. Ag des and de are 0.1, 1.0, 1000, 0.3, 1.0 and 2, respectively.
the agent has reached
C.3 HYPERPARAMETER AND NETWORK ARCHITECTURES
Table
summarizes the hyperparameters used to train policies with SAC (Haamoja et al. 2018b TRPO [Schulman et al. 2015). PPO (Schulman al. 2017), and HIRO (Nachum 2018
Table 3
summarizes the network architectures.
D DEPICTION OF ATTENTION WEIGHTS
In this section, we illustrate the attention weights of our composite policy in halfcheetah-hurdle (Fig- 9). pusher (Fig. [0), and cross-maze-ant (Fig. TT] environments. Our results highlight the ability of our model to compose a given set of primitive polices, both sequentially and concurrently, solve the given problem. Fig. shows the color-coding for the scale of attention weight value given by the composite policy.
X
Small
Large
Figure 8: The attention weights: The blue and yellow colors show low and high values, respectively
15

---

Published as a conference paper at ICLR 2020
300
MTP
Figure 9: The depiction of attention weights for the halfcheetah-hurdae environment. The weighting "strength" of each primitive policy (i.e., run and jump) for each step is shown. There are three hur- dles in front of the agent. The red-marked can be seen that the agent concurrently mixes the jump and run primitives before approaching the next hurdle to gain momentum and switches jump primitive when passing the hurdles.
the attention plot shows the position of those hurdles.
6e
Figure 10: The depiction attention weights for the pusher environment. The weighting "strength" of each primitive policy (i.e., Bottom and Left) for each step is shown. It can be seen that the agent concurrently as well sequentially compose the push-to-the-botton and push-to-the-letf primitives to reach the target.
150
The depiction of attention weights for the cross-maze environment. The weighting "strength" of each primitive policy (i.e., up (U), down (D), left (L.), and right (R)) for each step shown. The top and bottom rows show the agent reaching the goals on the top and right, re- spectively. In the top row, the agent emphasizes on the upward primitive and learns to ignore the remaining primitives. In the bottom row, the agent first focuses the upward primitive reach the cross- cross-point and then switches to fight-directio motion primitive to reach the given goal.
Figure 11:
16


Reviews:
[{"review_id": "ryefEkSAFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the specific question/problem tackled by the paper?\nThis paper addresses the hierarchical RL problem of combining multiple primitive policies (pi_1, …, pi_K) into policies for more complex tasks. Given a number of primitive skills and a new task within an environment, the paper aims to learn to pick and combine the primitives as needed to solve the new task. This problem statement is interesting and the method performs well on difficult tasks. \n\nHowever, I argue for rejecting this paper because it lacks meaningful contributions to the field. I do not see how the method presented in the paper is more than RL over  hand engineered action spaces that are better for the tasks. While this improves results, we already know that for any task, there is some best action space for performing that task. This is why most HRL work aims to also find the primitive policies in additional to composing them. Is this any better than option-critic if the options are hardcoded to be the primitives? The experiment state that the option-critic method did not work, but did you give it access to the same primitives?\n\nSummary:\nThe method presented in the paper is as follows: at each state s_t, the K primitives are queried for their action a_k ~ pi_k(s_t). Then, a biRNN reads in the actions in order from 1 to k. In parallel, the state and a goal are encoded by a network named “Decoder”. The encoded state and the hidden states of the RNN are used to output an attention weight over each primitive. Finally, the output action is the weighted combination of all the actions. The encoders and attention weights are trained with RL. \n\nThis method is evaluated on several mujoco tasks, such as making a cheetah jump hurdles by combining “forward” and “jump” primitives. Each environment has predefined primitives such as “forward” “left” “right” etc. \n\nThis method is compared against HIRO, which does not have access to the primitive policies. It is not surprising that hand engineering primitives helps performance. \n\nIs the approach well motivated?\nThe general idea behind the approach is well motivated: using primitive skills to learn complex skills is a useful goal. The details of the method are strange.\nI would like to see a better motivation and empirical justification for the biRNN. Why should the primitive’s action be encode in order? The ordering of the primitives is arbitrary and constant: a fully connected network could be used, or the attentions could be output entirely independently per primitive.\nIn fact, I do see not why the primitives’ actions need to be encoded at all. It would be much simpler for the encoder to look at (s_t, g_t) and output a discrete probability over the K primitives.\nThe ablations in 5.2 are for outputting actions directly rather than mixture weights. The paper would benefit from ablations where mixture weights are output but without the biRNN or without passing in the primitive’s actions.\n \n\nIs the method well placed in the literature? \nThe main idea of predicting weights over multiple experts is not novel (see \"Adaptive mixtures of local experts” from 1991). In the context of RL literature, we can interpret the primitive skills as actions directly, and then the method is performing basic RL over a better action space (the better actions being “go left”, “jump” etc. We can also interpret these as options, but unlike options a single primitive is not followed for multiple time steps with a termination condition. Functionally, this is equivalent to regular RL using domain knowledge to engineer the action space. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJeMg4_gqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach in which new tasks can be solved by an attention model that can weigh the contribution of different base policies conditioned on the current state of the environment and task-specific goals. The authors demonstrate their method on a selection of RL tasks, such as an ant maze navigation task and a more complicated “ant fall” task, which requires the agent to first move a block to fill a gap in the environment before it is able to reach the goal. \n\nI found the paper interesting and well written. My primary concern is that the primitive policies are learned independently of the composite policies, which might limit the application of this approach to more complex problems. Additionally, it would be great to also see the concurrent and sequential form of skill combination for the more complex tasks, and not just the point navigation task shown in Figure 7. \n\nStandard errors on Figures 5 and 6 seem to be missing. Additionally, I was curious that in Figure 4a and Figure 6a, the composite’s performance is already a lot better than the other methods after 0 training steps. Maybe the authors can elaborate on that. Maybe the performance at step 0 is just hard to make out in the graphs?\n\nI would suggest accepting the paper but there could be a more detailed analysis of how the pre-trained sub-modules are used and learning both composite and sub-policies together would make the paper stronger. \n\nAdditional comments:\n- Where is the training graph for the Composite-SAV applied to the “ant fall” task? Maybe I missed it?\n- Algorithm 1 should probably be moved to the main text. \n\n####After rebuttal####\nThe authors' response and the revised paper address my already minor concerns. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkx4LknOKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I think the method has some great merit but I am not overly confident in the reproducibility of the method. Some of the comparisons (HIRO) do not agree with the results in the HIRO paper. Also, more description is needed to describe how the baselines were used in the analysis. Were they also given the pre-trained sub-policies? A more fair comparison might be to give those baseline methods no sub-policies but let them pre-train for an equivalent amount of time as the sub-policies are trained.\n\n Here are some more detailed comments: \n- Figure 1 is not very clear and does not appear to add much to the explanation of the method. More detail should be included in the caption.\n - In the paper, it is noted that HRL has high sample complexity and needs lots of training data. I find the comment about how HRL requires many more training steps than regular RL very odd. The purpose of HRL is to have better sample efficiency and learn strong polices faster. Has the author observed different? The purpose of HRL is to reduce sample complexity and search in a well suited and structured way.\n- The assumption that the sub-policies solve the underlying MDP is rather strong. How are these policies going to be trained to guarantee this? \n- I like the idea of using an attention model to help pick learn a weighting for the combination of a number of sub-policies. I am not sure if using a bi-directional LSTM is the best or simplest method to accomplish this. The authors can look at \"MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies\" NeurIPS 2019 for a recent work that is similar to theirs. \n- For Figure 4: Do these methods also get to use the sub-policies that have pre-trained some version of tasks? Also, how are these component policies trained? Over what tasks are they trained? This information is very important to make sure the method is not overly biased to the composition of them. \n- I find the results in Figure 5 very odd. The baseline shows that HIRO does not learn to perform well on these tasks even though these are the tasks from the HIRO paper that it learned to solve rather well. Can this contradiction be explained? \n- For the HIRO comparison was the system also using the composite policies there were pretrained? HIRO is designed to learn the sub-policies concurrently but it seems in this case the authors are using the outputs of the composition policies as input to the HIRO low policy. \n- I do not understand the visualization in Figure 7. How to the colored paths for the agent represent the weights for the compose policy?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers deep reinforcement learning skill transfer and composition, through an attention model that weighs the contributions of several base policies conditioned on the task and state, and uses this to output an action. The method is evaluated on several Mujoco tasks.

There were two main areas of concern. The first was around issues with using equivalent primitives and training times for comparison methods. The second was around the general motivation of the paper, and also the motivation for using a BiRNN. These issues were resolved in a comprehensive discussion, leaving this as an interesting paper that should be accepted.
Decision: Accept (Poster)