Learning to Coordinate Manipulation Skills via Skill Behavior Diversification
Authors: Youngwoon Lee
Jingyun Yang
Joseph J. Lim
Venue:  (2020)
Keywords: reinforcement learning
hierarchical reinforcement learning
modular framework
skill coordination
bimanual manipulation

Abstract: When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together. Similarly, a robot with multiple end-effectors can perform complex tasks by coordinating sub-skills of each end-effector. To realize temporal and behavioral coordination of skills, we propose a modular framework that first individually trains sub-skills of each end-effector with skill behavior diversification, and then learns to coordinate end-effectors using diverse behaviors of the skills. We demonstrate that our proposed framework is able to efficiently coordinate skills to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside a container while pushing the container with two robot arms, and pushing a box with two ant agents. Videos and code are available at https://clvrai.com/coordination

Published as a conference paper at ICLR 2020
LEARNING TO COORDINATE MANIPULATION SKILLS VIA SKILL BEHAVIOR DIVERSIFICATION
Youngwoon Lee, Jingyun Yang, Joseph J. Lim
Department of Computer Science
University of Southern California
(1ee504, jingyuny, 1 limjj}e eusccccooo edu
ABSTRACT
When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together. Similarly, a robot with multiple end-effectors can perform complex tasks by coordinating sub-skills of each end-effector. To realize temporal and behavioral coordination of skills, we propose a modular framework that first individually trains sub-skills of each end-effector with skill behavior diversification, and then learns to coordinate end-effectors using diverse behaviors of the skills. We demonstrate that our proposed framework is able to efficiently coordinate skills to solve challenging collaborative control tasks such as picking up long bar, placing a block inside container while pushing the container with robot arms, and pushing a box with two ant agents. Videos and code are available at https:://// //vaaai....i.../...o com coordi nation
INTRODUCTION
Imagine you wish to play Chopin's Fantaisie Impromptu on the piano. With little prior knowledge about the piece, you would first practice playing the piece with each hand separately. After inde- pendently mastering the left and right hand parts, you would move to practicing with both hands simultaneously. To find the synchronized and non-interfering movements of two hands, you would try variable ways of playing the same melody with each hand, and eventually create complete piece of music. Through the decomposition of skills into sub-skills of two hands and learning variations of sub-skills, humans make the learning process of manipulation skills much faster than learning everything at once.
Can autonomous agents efficiently learn complicated tasks with coordination of different skills from multiple end-effectors like humans? Learning to perform collaborative and composite tasks from scratch requires huge amount of environment interaction and extensive reward engineering, which often results in undesired behaviors (Riedmiller et al.
2018) Hence, instead of learning a task at once,
modular approaches (Andreas et al. Oh17] al.
2017 Frans al. 2018 Lee et al.) 2019 Peng
et al. 2019; Goyal et al.) (0200) suggest to learn reusable primitive skills and solve more complex tasks by recombining the skills. However, all these approaches either focus on working with single end-effector manipulation or single agent locomotion, and these do not scale to multi-agent problems.
To this end, we propose modular framework that learns coordinate multiple end-effectors with their primitive skills for various robotics tasks, such as bimanual manipulation. The main challenge is that naive simultaneous execution of primitive skills from multiple end-effectors can often cause unintended behaviors (e.g. collisions between end-effectors). Thus, as illustrated in Figure | our model needs to learn appropriately coordinate end-effectors; and hence needs a way to obtain, represent, and control detailed behaviors of each primitive skill. Inspired by these intuitions, our method consists of two parts: (1) acquiring primitive skills with diverse behaviors by mutual information maximization, and (2) learning meta policy that selects a skill for each end-effector and coordinates the chosen skills by controlling the behavior of each skill.
The main contribution of this paper is modular and hierarchical approach that tackles cooperative manipulation tasks with multiple and-effectors by (1) learning primitive skills of each end-effector independently with skill behavior diversification and (2) coordinating end-effectors using diverse

---

Published as a conference paper at ICLR 2020
1
Pick (21)
Place (23)
Push (22)
Push (2:1)
Figure 1: Composing complex skills using multiple agents' primitive skills requires proper coordina- tion between agents since concurrent execution of primitive skills requires temporal and behavioral coordination. For example, to move a block into container on the other end of the table, the agent needs to not only utilize pick, place, and push primitive skills at the right time but also select the appropriate behaviors for these skills, represented as latent vectors 21, 22, 23, and 24 above. Naive methods neglecting either temporal or behavioral coordination will produce unintended behaviors, such collisions between end-effectors.
behaviors of the skills, Our empirical results indicate that our proposed method is able to efficiently learn primitive skills with diverse behaviors and coordinate these skills to solve challenging collabo- rative control tasks such as picking up a long bar, placing a block inside the container on the right side, and pushing a box with two ant agents. We provide additional qualitative results and code at https: http///cvaaaiaa wom/coordination
2 RELATED WORK
Deep reinforcement learning (RL) for continuous control is an active research area. However, learning a complex task either from a sparse reward or a heavily engineered reward becomes computationally impractical as the target task becomes complicated. Instead of learning from scratch, complex tasks can be tackled by decomposing the tasks into easier and reusable sub-tasks. Hierarchical reinforcement learning temporally splits task into sequence of temporally extended meta actions. It often consists of one meta policy (high-level policy) and a set of low-level policies, such as options framework (Sutton al. 1999). The meta policy decides which low-level policy activate and the chosen levelevel policy generates an action sequence until the meta policy switches it to another low- level policy. Options can be discovered without supervision (Schmidhuber 1990; Bacon et al. 2017)] Nachum et al. 2018 Levy et al. 2019), meta-learned (Frans et al.) 2018)) pre-defined (Kulkarni et al. 2016 et al. Merel al. 2019 Lee al., 2019), or attained from additional supervision signals (Andreas et al.) [017] Ghosh et al. 2018). However, option frameworks are not flexible to solve a task that requires simultancous activation or interpolation of multiple skills since only one skill can be activated at each time step.
To solve composite tasks multiple policies can be simultaneously activated by adding Q- functions (Haarooa et al. [00 8a), additive composition (Qureshi et al. 2020 Goyal et al. 2020). or multiplicative composition (Peng et al.) 2019). As each policy takes the whole observation as input and controls the whole agent, it is not robust to changes in unrelated parts of the observation. For example, a left arm skill can be affected by the pose change the right arm, which is not relevant to the left arm skill. Hence, these skill composition approaches can fail when an agent encounters a new combination of skills or a new skill is introduced since the agent will experience unseen observations.
Instead of having policy with the full observation and action space, multi-agent reinforcement learning (MARL) suggests to explicitly split the observation and action space according to agents (e.g. robots or end-effectors), which allows efficient low-level policy training as well flexible skill composition. For cooperative tasks, communication mechanisms (Sukhbaatar et al.) [2166] Peng et al. 2017 Jiang & Lu. 2018). sharing policy parameters (Gupta et al. (017). and decentralized actor with centralized critic (Lowe et al. 2017 Foerster et al. 2018) have been actively used. However, these approaches suffer from the credit assignment problem (Sutton 1984) among agents and the lazy agent problem (Sunehag et al.) 2018). As agents have more complicated morphology and larger observation space, learning a policy for a multi-agent system from scratch requires extremely long training time. Moreover, the credit assignment problem becomes more challenging when the complexity of cooperative tasks increases and all agents need to learn completely from scratch. To resolve these issues, we propose to first train reusable skills for each agent in isolation, instead of
2

---

Published as a conference paper at ICLR 2020
Fimela
Meta Policy
Selected Index
Behavior Embeddling m
Behavior Embedding x
State 8)
Pick
Pick
Primitive Skills
Primitive Skills
Pick
Place
Push
Place
Pick
Place
Push
Place
Cl
R5
Action
Agent/End-offector
Agent/End-effector
Next State
Figure 2: Our method is composed of two components: a meta policy and set of agent-specific primitive policies relevant to task completion. The meta policy selects which primitive skill to run for each agent as well as the behavior embedding (i.e. variation in behavior) of the chosen primitive skill. Each selected primitive skill takes as input the agent observation and the behavior embedding and outputs action for that agent.
learning primitive skills of multiple agents together. Then, we recombine these skills (Maes & Brooks 1990) to complete more complicated tasks with learned coordination of the skills.
To coordinate skills from multiple agents, the skills have be flexible; hence, a skill can be adjusted to collaborate with other agents' skills. Maximum entropy policies (Haarnoja et al., [017 2018at 8a b) can learn diverse ways to achieve a goal by maximizing not only reward but also entropy of the policy. In addition, Eysenbach et al. (2019) proposes to discover diverse skills without reward by maximizing entropy as well as mutual information between resulting states and latent representations of skills (i.e. skill embeddings). Our method leverages the maximum entropy policy (Haarnoja et al.) 2018 with the discriminability objective (Eysenbach et al. 2019) to learn a primitive skill with diverse behaviors conditioned on controllable skill embedding. This controllable skill embedding will be later used as a behavior embedding for the meta policy to adjust primitive skill's behavior for coordination.
METHOD
In this paper, we address the problem of solving cooperative manipulation tasks that require collab- oration between multiple end-effectors agents. Note that use the terms "end-effector" and "agent" interchangeably this paper. Instead of learning a multi-agent task from scratch (Lowe et al.
Gupta al.) 2017: Sunehag al. 2018 Foerster et al. 2018). modular approaches (Andreas) et al. 2017 Frans et al.j 2018: Peng et al. 20199) suggest to learn reusable primitive skills and solve
2017
complex tasks by recombining these skills. However, concurrent execution of primitive skills of multiple agents fails when agents never experienced a combination of skills during the pre-training stage, or skills require temporal or behavioral coordination.
Therefore, we propose a modular and hierarchical framework that learns to coordinate multiple agents with primitive skills perform complex task. Moreover, during primitive skill training, we propose to learn latent behavior embedding, which provides controllability of each primitive skill to the meta policy while coordinating skills. In Section 3.2] we describe our modular framework in detail. Next, in Section 3.3] we elaborate how controllable primitive skills can be acquired. Lastly, we describe how the meta policy learns to coordinate primitive skills in Section 3.4
3.1 PRELIMINARIES
We formulate our problem as a Markov decision process defined by a tuple (S, (S,ATT, R. p. p.7) of states, actions, transition probability, reward, initial state distribution, and discount factor. In our formulation, we assume the environment includes N agents. To promote consistency in our terminology, superscripts to denote the index of agent and subscripts to denote time or primitive skill index. Hence, the state space and action space for agent can be represented as St and AA' where each element of St a subset of the corresponding element in and A = A' x A2 x ... x A respectively. For each agent i, we provide set of m² skills, IT' {mi -.. me J. A policy of an agent is represented as
use

---

Published as a conference paper at ICLR 2020
(a) RI.
(b) MARI.
Modular
(d) RL-SBD
MARL--SD (f) Modular-SBD
Figure 3: Different multi-agent architectures. (a) The vanilla RL method considers all agents monolithic agent; thus a single policy takes the full observation as input and outputs the full action. (b) The multi-agent RL method (MARL) consists of N policies that operate on the observations and actions of corresponding agents. (c) The modular network consists of N sets of skills for the N agents trained in isolation and meta policy that selects a skill for each agent. (d-f) The RL, MARL, and modular network methods augmented with skill behavior diversification (SBD) has a meta policy
that outputs a skill behavior embedding vector for each skill.
Ty (af(ss) IT', where of a skill index, sf E St is a state, and of E A' an agent action at time t.
a An initial state so sampled from p. and then, N agents take actions aj at. ....,...0..... a. sampled from
composite policy π T(a), at at....,................ a, [st, c), .......... c²") = (T (a (st), T (afls)).................... ce (a (st)
and receive single reward rr. The performance evaluated based on discounted return R
ET y'r,, where T is the episode horizon.
3.2 MODULAR FRAMEWORK
As illustrated Figure our model is composed of two components: meta policy Tmata and set of primitive skills of N agents IT ,I...IIA Note that each primitive skill T E II' contains variants of behaviors parameterized by Na-dimensional latent behavior embedding z (see Section 3.3). The meta policy selects a skill to execute for each agent, rather than selecting one primitive skill for the entire multi-agent system to execute. Also, give the meta policy the capability to select which variant of the skill to execute (see Section 3.4.. Then, the chosen primitive skills are simultancously executed for Tiow time steps.
The concurrent execution of multiple skills often leads to undesired results and therefore requires coordination between the skills. For example. naively placing block in the left hand to a container being moved by the right hand can cause collision between the two robot arms. The arms can avoid collision while performing the skills by properly adjusting their skill behaviors (e.g. the left arm leaning to the left side while placing the block and the right arm leaning to the right side while pushing the container) as shown in Figure [ In our method, the meta policy learns coordinate multiple agents' skills by manipulating the behavior embeddings (i.c. selecting a proper behavior from diverse behaviors of each skill).
3.3 TRAINING AGENT-SPECIFC PRIMITIVE SKILLS WITH DIVERSE BEHAVIORS
To adjust a primitive skill collaborate with other agents' skills in new environment, the skill needs to support variations of skill behaviors when executed at a given state. Moreover, behavioral variation of skill should be controllable by the meta policy for skill coordination. order make our primitive skill policies generate diverse behaviors controlled by latent vector we leverage the entropy and mutual information maximization objective introduced in Eysenbach et al. (2019).
More specifically, primitive policy of an agent outputs an action a E A conditioned on the current state s E S and a latent behavior embedding z ~ p(=), where the prior distribution p(z) is Gaussian (we omit agent in this section for the simplicity of notations). Diverse behaviors conditioned on a random sample 2 can be achieved by maximizing the mutual information between behaviors and states MI (ss 2). while minimizing the mutual information between behaviors and actions given the state MI (a, 2/(s), together with maximizing the entropy of the policy H((as) to encourage diverse behaviors. The objective can be written as follows (we refer the readers to Eysenbach et al. 2019)
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 ROLLOUT
Input: Meta policy Tmeta. sets primitive policies IT', IT and meta horizon Tiom
2: Initialize episode / <- 0 and receive initial state 80
3: while episode not terminated do
4: Sample skill indexes and behavior embeddings (c...... (2 (Teta(
5: T +
6: while Tlow and episode is not terminated do
7: Octr = (a++T:..., a(.)) ~ (m))) ( 2/),..., xt (s1+T,
8:
9:
T + ++1
end while
11: Add transition $t, ( setT, Pratr-1 to the rollout buffer B
12: +- T+T
13:
end while
for derivation):
F(0) MI(s, 2) MI(a, 2|s) H(lls) H(als, H((ss) H(2)
(1)
= H(als,2] + (==p(2).s=7())) (lllg((((s)] E (o)g)))))ogg(()))
(2)
2 8, 2) + (xx (+ log p()))
where the learned discriminator (=|s) approximates the posterior p(z[s).
To achieve primitive skill with diverse behaviors, augment Equation to the enviromment reward:
+ x H((as, (= (log (2(s) log [o)](
(4)
where A is the entropy coefficient and A2 is the diversity coefficient which corresponds identifia- bility behaviors. Maximizing Equation (3) encourages multi-modal exploration strategies while maximizing the reward T, forces to achieve its own goal. Moreover, by maximizing identifiability of behaviors, the latent vector 2, named behavior embedding, can represent a variation of the Icarned policy and thus can be used to control the behavior of the policy. For example, when training robot to move an object, a policy learns to move the object quickly as well as slowly, and these diverse behaviors map different latent vectors We empirically show that the policies with diverse behaviors achieve better compositionality with other agents in our experiments.
3.4 COMPOSING PRIMITIVE SKILLS WITH META POLICY
We denote the meta policy as (e C. ,.,...., <Mss),, where cf E m²] represents skill index of an agent i E [1, N] and =² E N. represents behavior embedding of the skill. Every Tew time steps, the meta policy chooses one primitive skill T E II for each agent i. Also, the meta policy outputs a set latent behavior embeddings (a 2,,....,, and feeds them to the corresponding skills (i.e. (aa)s"" 2²) for agent i). Once set of primitive skills ( x chosen be executed, each primitive skill generates an action a' T/. (a*)s", 2²) based on the current state s and the latent vector z'. Algorithm illlstrates the overall rollout process.
Since there are finite number of skills for each agent to execute, the meta action space for each agent [L, m²] is discrete, while the behavior embedding space for each agent N is continuous. Thus, the meta policy is modeled as a (2 x N)ehead neural network where the first N heads represent m-'way categorical distributions for skill selection and the last N heads represent -dddiennsi Gaussian distributions for behavior control of the chosen skill.
3.5 IMPLEMENTATION
We model the primitive policies and posterior distributions ds as neural networks. We train the primitive policies using soft actor-critic (Haamoja al. 2018b). When train a primitive policy, we use a unit Gaussian distribution as the prior distribution of latent variables p(z). We use 5 as the size of latent behavior embedding N.. Each primitive policy outputs the mean and standard
5

---

Published as a conference paper at ICLR 2020
(a) JACO PICK-PUSEE PLACE
(b) JACO BAR-MOVING
(c) ANT PUSH
Figure The composite tasks pose a challenging combination of object manipulation and locomotion skills, which requires coordination of multiple agents and temporally extended behaviors. (a) The left Jaco arm needs to pick up a block while the right Jaco arm pushes a container, and then it places the block into the container. (b) Two Jaco arms required pick and place bar-shaped block together. (c) Two ants push the red box to the goal location (green circle) together.
deviation of Gaussian distribution over an action space. For primitive policy, we apply tanh activation to normalize the action between [-1, We model the meta policy as neural network with multiple heads that output the skill index c and behavior embedding 2 for each agent. The meta policy is trained using PPO (Schulman et al. 2017: 2016 Dhariwal et al. 2017). All policy networks this paper consist of 3 fully connected layers of 64 hidden units with ReLU nonlinearities. The discriminator ds in Equation (4) is 2-layer fully connected network with 64 hidden units.
4 EXPERIMENTS
To demonstrate the effectiveness of our framework, we compare method prior methods in the field of multi-agent RL and ablate the components of our framework to understand their importance. We conducted experiments on of challenging robot control environments that require coordination of different agents to complete collaborative robotic manipulation and locomotion tasks.
Through experiments, aim answer the following questions: (1) can framework efficiently learn to combine primitive skills to execute complicated task; (2) can our learned agent exhibit collaborative behaviors during task execution; and (3) can our framework leverage the controllable behavior variations of the primitive skills to achieve better coordination?
For details about environments and training, please refer the supplementary material. As the performance of training algorithms varies between runs, we train each method on each task with 6 different random seeds and report mean and standard deviation of each method's success rate.
4.1 BASELINES
We compare the performance of our method with various single- and multi-agent RL methods illustrated Figure
Single-agent RL (RL): A vanilla RL method where a single policy takes as input the full observation and outputs all agents' actions.
Multi-agent RL (MARL): A multi-agent RL method where each of N policies takes input the observation of the corresponding agent and outputs an action for that agent. All policies share the global critic learned from single task reward (Lowe al. 2017).
Modular Framework (Modular): A modular framework composed of a meta policy and N sets of primitive skills (i.e. one more primitive skills per agent). Every Tiow time steps, the meta policy selects a primitive skill for each agent based on the full observation. Then, the chosen skills are executed for T Tow time steps.

---

Published as a conference paper at ICLR 2020
1.0
0.6
0.6-
2.6
0.4
13.4
0.44
0.
0.2
02
0.0
5.0
0.0
4.3
3.
(1.)
staps
(a) JACO PICK-PUSH-PLACE
(b) JACO BAR-MOVING
(c) ANT PUSH
Figure 5: Success rates of our method (Modular-SBD) and baselines. For modular frameworks (Modular and Modular-SBD), we shift the learning curves rightwards the total number of environment steps the agent takes to learn the primitive skills (0.9 M. 1.2 M, and 2.0 M. respectively). Our method substantially improves learning speed and performance on JACO TICK-PUSH-PLEY and ANT PUSH. The shaded areas represent the standard deviation of results from six different seeds. The curves are smoothed using moving average over 10 runs.
Single-agent RL with Skill Behavior Diversification (RI-SBD): An RL method augmented with the behavior diversification objective. A meta policy is employed to generate behavior embedding for low-level policy, and the low-level policy outputs all agents' actions conditioned on the behavior embedding and the full observation for Thow time steps. The meta policy and the low-level policy jointly trained with the behavior diversification objective described in Equation (4).
Multi-agent RL with Skill Behavior Diversification (MARL-SBD): A MARL method augmented with the behavior diversification objective. A meta policy generates N behavior embeddings. Then, each low-level policy outputs each agent's action conditioned on its observation and behavior embedding for Tiow time steps. All policies are jointly trained to maximize Equation (4).
Modular Framework with Skill Behavior Diversification (Modular-SBD, Ours):
Our method
which coordinates primitive skills of multiple agents. The modular framework consists of a meta policy and N sets of primitive skills, where each primitive skill conditioned on a behavior embedding z. The meta policy takes as input the full observation and selects both primitive skill and behavior embedding for each agent. Then, each primitive skill outputs action for each agent.
4.2 JACO PICK-PUSH-PLACE
We developed JACO
and JACO BAR-MOVING environments using two Kinova Jaco arms, where each Jaco arm a DoF robotic arm with 3 fingers. JACO starts with a block on the left and container the right. The robotic arms need to pick up the block, push the container to the center, and place the block inside the container. For successful completion of the task, the two Jaco arms have to concurrently execute their distinct sets of skills and dynamically adjust their picking, pushing, and placing directions to avoid collision between arms.
Primitives skills. There are three primitive skills available each arm: Picking up, Pushing, and Placing to center (see Figure 4a). Picking up requires a robotic arm to pick up a small block, which is randomly placed on the table. If the block is not picked up after a certain amount of time the arm drops the block, the agent fails. Pushing learns to push a big container to its opposite side (e.g. from left to the center or from right center). The agent fails if it cannot place the container to the center. Placing to center requires placing an object in the gripper to the table. The agent only succeeds when stably places the object at the desired location on the container.
Composite task. Our method Moodllar-SBD) can successfully perform JACO task while all baselines fail to compose primitive skills as shown in Figure 5a The RL and MARL baselines cannot learn the composite task mainly because the agent requires to learn the combinatorial number of skill compositions and to solve the credit assignment problem across multiple agents. Since the composite task requires multiple primitive skills of multiple agents to be performed properly at the same time, a reward signal about a failure case cannot be assigned to the correct agent or skill. By using pre-trained primitive skills, the credit assignment problem is relaxed and all agents can
7

---

Published as a conference paper at ICLR 2020
Jaco Pick-Push-PPaac
Jaco Bar-Moving
Ant Push
0.000 0.000
0.000 = 0.000
0.000 0.000
MARL
0.000 0.000
0.000 = 0.000
0.000 0.000
RL-SBD
0.000 + 0.000
0.000 = 0.000
0.000 0.000
MARL-SBD
0.000 = 0.000
0.000 = 0.000
0.000 - 0.000
Modular
0.324 = 0,468
0.917 = 0.276
0.003 = 0.058
Modular-SBD (Ours)
0.902 = 0.298
0.950 = 0.218
0.323 0.468
Table 1: Success rates for all tasks, comparing our method against baselines. Each entry in the table represents average success rate and standard deviation over 100 runs. The baselines learning from scratch fail to learn complex tasks with multiple agents.
perform their skills concurrently. Therefore, the Modular baseline learns achieve success but shows significantly lower performance than our method (Modular-SBD), This is because the lack of skill behavior diversification makes it impossible to adjust pushing and placing trajectories during skill composition time, which resulting in frequent end-effector collisions.
4.3 JACO BAR-MOVING
In JACO BAR-MOVING, two Jaco arms need to pick up long bar together, move the bar towards target location while maintaining its rotation, and place it on the table (see Figure 4b). The initial position of the bar is randomly initialized every episode and agent needs to find appropriate coordination between two arms for each initialization. Compared to JACO PICK-PUSH-PE this task requires that the two arms synchronize their movements and perform more micro-level adjustments to their behaviors.
Primitives skills. There two pre-trained primitive skills available to each arm: Picking up and Placing towards arm. Picking up same as described Section [.22 Placing towards arm learns to move a small block (half size of the block used in the composite task) the hand towards the robotic arm and then place it on the table. The agent fails if it cannot place the block to the target location.
Composite task. The JACO BAR-MOVING task requires the two For example, the Picking up skill of both arms should be synchronized when they start to lift the bar and two arms require to lift the bar while maintaining the relative position between them since they are connected by holding the bar. The modular framework without explicit coordination of skills (Modular) can synchronize the execution of picking, moving, and placing. But the inability to micro-adjust the movement of the other arm causes instability of bar picking and moving, This results in degraded success rates compared to the modular framework with explicit coordination. Meanwhile, all baselines without pre-defined primitive skills fail to learn JAcO BAR-MOVING.
work very closely together.
4.4 ANT PUSH
We developed multi-ant environment, ANT-PUSH, inspired from Nachum et al. (2019), simulated in the MuJoCo (Todorov et al. 2012) physics engine. We use the ant model in OpenAI Gym (Brockman al. 2016). In this environment, two ants need to push a large object toward green target place. collaboaatig with each other to keep the angle of the object as stable as possible (see Figure 4c).
Primitives skills. We train walking skills of ant agent in 4 directions: up, down, left, and right. During primitive skill training, a block (half size of the block used in the composite task) and an ant agent are randomly placed. Pushing the block gives an additional reward the agent, which prevents an ant to avoid the block. The learned primitive skills have different speed and trajectories conditioned on the latent behavior embedding.
Composite task. Our method achieves 32.3% success rate on ANT PUSH task while all baselines fail to compose primitive skills as shown in Figure and Table The poor performance of RL, MARL, RL-SBD, and MARL-SBD baselines shows the difficulty of credit assignment between agents, which leads one of the ants moves toward a block and pushes it but another ant does not move. Moreover, the Modular baseline with primitive skills also fails to learn the pushing task. This result illustrates the importance coordination of agents, which helps synchronizing and controlling the velocities of both ant agents to push the block toward the goal position while maintaining its rotation.
8

---

Published as a conference paper at ICLR 2020
1.
1230
0.6
902
0.6
9.6
632
and aol pollpaaca
2.4
000
0.0
Instonmmnss sieza 13P0
Dwrtzement dapo (19)
Snw.rrmentateea GP0
Invrrrmans SINDa (20)
(a) Success rate
(b) Reward
(a) PUCK-PUSI-PLACE
(b) BAR-MOVING
Figure 6: Learning curves of our method with
Figure 7: Success rates of our method with dif- ferent Toow coefficients on Jaco environments.
different diversity coefficients X2 on ANT PUSH.
4.5 EFFECT OF DIVERSITY OF PRIMITIVE SKILLS
To analyze the effect of the diversity of primitive skills, we compare our model with primitive skills trained with different diversity coefficients X2 {0.0, 0.05, 0.1, 1.0] in Equation 4 on ANT PUSH. Figure 6 shows that with small diversity coefficients X2 f0.05, 0.1}, the agent can control detailed behaviors of primitive skills while primitive skills without diversity (A2 0) cannot be coordinated. The meta policy tries to synchronize two ant agents' positions and velocities by switching primitive skills, but cannot achieve proper coordination without diversified skills. On the other hand, large diversity coefficients A2 {0.5, 1.0) make the primitive skills often focus demonstrating diverse behaviors and fail to achieve the goals of the skills. Hence, these primitive skills do not have enough functionality to solve the target task. The diversity coefficient needs to be carefully chosen to acquire primitive skills with good performance as well as diverse behaviors.
4.6 EFFECT OF SKILL SELECTION INTERVAL Tiow
To analyze the effect of the skill selection interval hyperparameter Tlows we compare our method trained with Too = {1, 2, 3, 5, 10} on Jaco environments. The success rate curves in Figure demonstrate that smaller Tioe values in range [1,3] lead to better performance. This can be because the agent can realize more flexible skill coordination by adjusting the behavior embedding frequently.
In addition to the fixed Tiow values, also consider the variation of our method in which the skill behavior embedding is only sampled when the meta policy updates its skill selection. Concretely, set the value of Thow to 1 but update (=/ 2/ only if (ct,,. cf') A (cl-1--... ci 1). We observe that in this setting, the meta policy at times switch back and forth between two skills two consecutive time steps, leading to slightly worse performance compared to our method with small Tiow values. This indicates that the meta policy needs to adjust the behavior embedding in order to optimally coordinate skills of the different agents.
5 CONCLUSION
In this paper, we propose a modular framework with skill coordination to tackle challenges of composition of sub-skills with multiple agents. Specifically, we use entropy maximization with mutual information maximization to train controllable primitive skills with diverse behaviors. To coordinate learned primitive skills, the meta policy predicts not only the skill to execute for each agent (end-effector) but also the behavior embedding that controls the chosen primitive skill's behavior. The experimental results on robotic manipulation and locomotion tasks demonstrate that the proposed framework is able efficiently learn primitive skills with diverse behaviors and coordinate multiple agents (end-effectors) to solve challenging cooperative control tasks. Acquiring skills without supervision and extending our method to visual domain are exciting directions for future work.
ACKNOWLEDGMENTS
This project was funded by SKT. The authors would like to thank Karl Pertsch and many members of the USC CLVR lab for helpful discussion.

---

Published as a conference paper at ICLR 2020
REFERENCES
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pp. 166-175, 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Association for the Advancement of Artificial Intelligence, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 115.:::6606.11444 2016.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https 2017.
7/qithub. hwtpa//ccoooopeeaaaaaaelllll
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference Learning Representations, 2019,
net. wtt.:eeeooooummld5555533.33
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Association for the Advancement of Artificial Intelligence, 2018.
Kevin Frans, Jonathan Ho. Xi Chen, Pieter Abbeel, and John Schulman. META LEARNING SHARED HIERARCHIES. In International Conference on Learning Representations, 2018. URL
https://openrevial
nnt peee//oorumidd-SyXieeeet
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and-
conquer reinforcement learning. In International Conference Learning Representations, 2018. 8.
URL htttps:///ppereviieeee net.eeewweeeeeeeeeeeeeeeeeeeeeee wtw.:wwuneeeeeeeememmmmmmmm
forum? id=
Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng. Sergey Levine, and Yoshua Bengio. Reinforcement learning with competitive ensembles of iiformation-constraii primitives. In International Conference Learning Representations, 2020. URL
net met/forund fttt/forum?ddde EXxTTTETE TYXgJTEXDr
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems, pp. 66-83, 2017.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361, 2017.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In IEEE International Confer- ence on Roboties and Automation, pp. 6244 6251, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, pp. 1856-1865, 2018b.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Advances in Neural Information Processing Systems, pp. 7254-7264, 2018.
Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 3675-3683 2016.
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram. Edward Hu, and Joseph J. Lim. Com- posing complex skills by learning transition policies. In International Conference on Learning Representations,
10

---

Published as a conference paper at ICLR 2020
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight. In International Conference on Learning Representations, 2019. URL htttps:/ net
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. Advances in Neural Information Processing Systems, pp. 6379-6399 2017.
Pattie Maes and Rodney A Brooks. Learning to coordinate behaviors. In Association for the Advancement of Artificial Intelligence, volume 90, pp. 196-802, 1990.
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. In International Confer- ence on Learning Representations, 2019. URL Setttps:/oppnreee BJFYvo09Y7
net/ forum? d=
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313 2018.
Ofrr Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. In Conference Robot Learning, 2019.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning, pp. 2661-2670, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detooo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Advances in Neural Information Processing Systems Autodiff Workshop. 2017.
Peng Peng. Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent srrrrcctionallyycorddieeee nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint 1703. 10069, 2017.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. composable hierarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems, 2019.
Mep: Learning
Ahmed H. Qureshi, Jacob J. Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C. Yip. Composing tusk-agnostic policies with deep reinforcement learning. In International Confer- ence on Learning Representations, 2020. URL /ttt:s:://opeeevviee. net/ / forum?? d=
H1 eeFRRRtH
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de
Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning, 2018.
Jürgen Schmidhuber. Towards compositional learning with dynamic neural networks. Inst. für Informatik, 1990.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representationss 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.06347, 2017.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation.
In Advances in Neural Information Processing Systems, pp. 2244-2252, 2016.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. decoppoicoooiiiionnn networks for cooperative multi-agent learning based on team reward. In International Conference Autonomous Agents and Mulli-Agent Systems, pp. 2085-2088 2018.

---

Published as a conference paper at ICLR 2020
Richard S Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts, 1984.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)1811-11,, 1999.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp- 5026-5033, 2012.
12

---

Published as a conference paper at ICLR 2020
A ENVIRONMENT DETAILS
The details of observation spaces, action spaces, number of agents, and episode lengths are described Table
All units in this section are meters unless otherwise specified.
Jaco Pick-Push-Plan
Jaco Bar-Moving
Ant Push
Observation Space
88
88
100
Robot observation
62
82
Object observation
26
26
18
Action Space
18
18
16
Number of Agents
2
Episode length
150
100
200
Table 2: Environment details
A.1 ENVIRONMENT DESCRIPTIONS
In both Jaco environments, the robot works on table with size (1.6, 1.6) and top center position (0, O, 0.82). The two Jaco arms are initialized at positions (-0.16. -0.16, and (-0.16, 0.24, 1.2). Left arm and right arm objects are initialized around (0.3, 0.2, 0.86) and (0.3, -0.2, 0.86) respectively in all primitive training and composite task training environments, with small random position and rotation perturbation.
In the Jaco Pick-Push-Plac task, the right jaco arm needs to pick up the object and place it into the container initialized at the other side of the table. Success is defined by contact between the object and the inner top side of the container.
In the Jaco Bar-Moving task, the two Jaco arms need to together pick the long bar up by height of 0.7, move it towards the arms by distance of ().15 0.15,, and place back the table. Success is defined (1) the bar being placed within 0.04 away from the desired destination both in height and in xy-position and (2) the bar having been picked 0.7 above the table.
In the Ant Push task, the two ant agents need to push a big box together to the goal position. The box has a size of 8.0 x 1.6 x 1.6. The distance between ants and the box is 20 and the distance between the box and the goal is 30 cm. Initial positions have cm of randomness and the agent has randomness of 0.01 each joint. The task considered as success when both the distances between left and right end of the box and the goal are within 5 cm.
A.2 REWARD DESIGN
For every task, add control penalty, 00.001 ||a|2, regularize the magnitude of actions where a is torque action performed by an agent.
Jaco Pick: To help the agent learn reach, pick, and hold the picked object, we provide dense reward to the agent defined by the weighted sum of pick reward, gripper-to-cube distance reward, cube position and quaternion stability reward, hold duration reward, success reward, and robot control reward. More concretely.
R(s) Apick (Ahou Zinit) + Adist dist(Pgrippers, PbennerrrrPP ) + Apos dist(phor,,Pinn
Aquat abs abs(A))at) Ahold thold + Asuccess 1success + >etrrilaa|?,
where Apick 500, Adist 100, Apos 1000, Aguat 1000, Akold 10, Asuecess 100, Actr! 1 10-1
Jaco Place: Reward for place primitive defined by the weighted sum of xy-distance reward, height reward (larger when cube close floor), success reward, and robot control reward.
R(s) Azy distry (pooz Pyoot) ) + Xe |Ebox - Zgoat| + Aeucvees 1 + Accrella|l?,
where Ary 500, A₂ 500. Asuecess 500, Actri 1 10-4
13

---

Published as a conference paper at ICLR 2020
Jaco Push: Reward for push primitive is defined by the weighted of gripper reaching reward, box-to-destination distance reward, quaternion stability reward, hold duration reward, success reward, and robot control reward.
R(s) Areaching dist(Prripper: Pooz) + Apos dist(Phozs Pdest)
Aguat abs( guat) + Ahold thold + Asuccess success + Actrillal?
where Areaching 100,
100,Apos Apos 500, Aquat 30, Anold 10, Asuccess 1000, Actrt 10
Jaco Pick-Push-Place: Reward for Pick-Push- Place is defined by the weighted sum gripper contact reward, per-stage ceach/pick/push/plel rewards, success reward, and control reward. We tune the reward carefully for baselines.
R(s)= Acontact (1ieft gripper touches container + right gripper touches box
Areach 1reach (dist(pieft grippee Peontainer) + dist (rightttttrrr Phox ))
Apick pick dist(Pbox: Pboz.targee sarget) + Xplace place dist( Pborttarget +
Apush dist(Poontainer: Peontainer target) + Asuccess Lauccess + Actrl ||all?
where Areach 10, Acontact 10, Apick >xplace Aplace
10, Asuccess 50, Actri 0, and reachingg lpick, and plcce are indicator functions specifying whether the agent is reaching, pick or place stage. Agent stages are determined by how many multiples of 25 steps the agent has stepped through in the environment.
Jaco Bar-Moving: Reward for Bar-Moving is defined by the weighted of per-stage reach/pick/move/place rewards, success reward, and control reward.
R(s) Areach Treach (dist(pieft grripperrrrrrrrrrrrooorr Plaft handte) + dist(priight.gripper grippprrrtggrpprrrr.rerr. Pright handhe) Apick 1pick dist(pbars Pbar Pbarsarget) sarget) ) + Amove Iplace distay (Phors Pbarstrget)
Aplace ppacce dist. (Pbars Pbar barget) + Asuccess success + Actr ||a|?
where Areach 10, Apick 30, Amove 100, Aplace 100, Avuceess 100, Actri 10 and pick and place are indicator functions specifying whether the agent is in pick or place stage. Agent stages are determined by whether the pick objective fulfilled or not.
Ant Push & Ant Moving: Reward for ANT PUSSH is defined by upright, velocity towards the desired direction. We provide a dense reward to encourage the desired locomotion behavior using velocity, stability, and posture, as following:
R(s) = = abs abs(Atant) Aboxvel abs( (TOoz)) Aupright cos(()) Aheight abs(0.6 h) Agoal dist(Pgont, Pbox):
where Avel 50, Aberve! = 20, Aupright 1, Abeight
0.5. For ANT PUSH, we provide an additional reward based distance between the box and the goal position with Agoat 200.
B EXPERIMENT DETAILS
We use PyTorch Paszke et al. 2017) for our implementation and all experiments are conducted on a workstation with Intel Xeon Gold 6154 CPU and 4 NVIDIA GeForce RTX 2080 Ti GPUs.
14

---

Published as a conference paper at ICLR 2020
B.1 HYPERPARAMETERS
Parameters
Value
learning rate gradient steps batch size discount factor target smoothing coefficient
3e-4
50
256
0.99
0.005
reward scale (SAC)
1.0
experience buffer size (# episodes)
1000
Tow
1 for JAcO, for ANT
N₂ (dimensionality of
5
Table 3: Hyperparameters
B.2 NETWORK ARCHITECTURES
Actor Networks: In all experiments, we model our actor network for each primitive skill as a 3-layer MLP with hidden layer size 64. The last layer of the MLP is two-headed action distribution and the other for the standard deviation of it. We use ReLU as activation function in hidden layers. We do not apply any activation function for the final output layer. The action distribution output represents per-dimension normal distribution, from which single actions can be sampled and executed in the environment.
for the mean of the
The critic network for each primitive skill and meta policy is modeled as a 2-layer MLP with hidden layer size 128. ReLU is used as an activation function in the hidden layers. The critic network output is used to assess the value of a given state-action pair, and is trained by fitting its outputs to the target Q-value clamped by + +00
Critic Networks:
The meta policy is modeled as 3-layer MLP with hidden layer size of 64. Since the actions meta policy are sampled from N categorical distributions for each end-effector/agent and N normal distributions for behavior embeddings, the output dimension of the meta policy is Ei= (m' N₂). The meta policy uses ReI.U activation function for all layers except for the final output layer.
Meta Policy:
B.3 TRAINING DETAILS
For all baselines, we train the meta policies using PPO and the low-level policies using SAC. We use the same environment configurations, composite task reward definitions, and value of Tiow across all basclines.
For Jaco tasks, we train total of 4 primitive skills right arm pick, right arm place-to-center, right arm place-towards-arm and left arm push to be composed by meta meta-policy. For Jaco Fick-Push- Place, provide the meta-policy with right arm pick and right arm place-to-center primitives and left arm push as left arm primitives; for Jaco Bar-Moving, we provide the meta- meta-policyy with right pick and right arm pick and right arm place-towards-amm as left arm primitives. We obtain left arm primitives for bar-moving task by using the learned right arm primitives directly.
right arm
plccetooardssaam
both right and left arm primitives and left
To obtain the 4 primitives skills described above, we train right arm pick with diversity coefficient A = 0.01 and the other three primitives with A = 0.1. The destination of right arm push set to (0.3. -0.03, 0.86), which is slightly left of the center of the table. After pick primitive is trained, we train the two right arm place primitives where episodes are initialized by intermediate states of successful right arm pick episodes where the height of the box is larger than 0.94 (0,01 higher than the target pick height). The place destinations for towards-arm and to-center primitives are (0.15, - 0.2, 0.86) and (0.3. -0.02, 0.86), respectively.
For non- modular baselines that incorporates skill behavior diversification, we use X 0.01 for both Jaco Pick-Push-Plac and Jaco Bar-Moving because both tasks require picking skills, which can only be trained with a small value of A2.
15


Reviews:
[{"review_id": "r1eDu0Dk5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a hierarchical reinforcement learning method for coordination of multiple cooperative agents with pre-learned adaptable skills. These skills are learned via a maximum entropy objective where diversity of behaviour given each skill is maximised and controlled via a latent conditioning vector. This allows controllability of the variation in skill execution by the meta-policy via changing the skill-specific latent vectors. The paper presents empirical results in manipulation (pick-push-place and moving a long bar by coordinating two Jaco arms) and locomotion (two Ants pushing a large block to a goal location). The method proposed outperforms the baselines reported. \n\nOverall, this paper addresses an interesting problem and can be impactful with the caveat for some clarifications and analysis. Given that the authors address my concerns, I would be willing to increase my score.\n\nThe main novelty of this work lies in how one can learn sub-skills that can be leveraged and adapted for down-stream tasks. The problem setting used to test the method is a multi-agent setting where it is crucial that skills are adapted to enable cooperation. I found the environments and the problem setting generally interesting and important for testing the proposed method. I have a few concerns that I listed below:\n\n\n1) I found the notations at times inconsistent and confusing. It would have helped to see some more details on the diagram (Figure 2) to understand how everything fits together. \n\n2) The set of skills for the two agents are selected by the meta-policy in every T_low steps. It looks like in the Jaco environments T_low = 1. Can you comment on this? This seems slightly concerning since it seems like the meta-controller is treating these skills as primitive actions rather than temporally extended behaviour.\n\n3) Looking at the training curves in Figure 4, there seems to be a really high variance in performance of the method. Can you comment on this as this seems concerning. Could you run more seeds to improve this?\n\n4) It is nice to see in section 4.5 how the hyper-parameters balancing diversity in combination with external reward (equation 4) is tuned and how sensitive that is to achieving adaptability for downstream tasks. The only criticism I have is that it is difficult to understand from \"Episode reward\" on y-axis what the success rate is (similar to Figure 4)? It would’ve been nice to report results in a consistent way throughout the paper for these environments. \n\n5) Given that all the tasks in the experiments are cooperative multi-agent settings, I would have liked to see more in depth discussion regarding alternative multi-agent methods. The multi-agent baseline provided (which is using a decentralized policy with a shared critic, inspired by Lowe et al., 2017) seems fair, but I wonder if there has been more recent work in this direction that could have been highlighted? \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJxR4YcnYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to achieve multi-agent coordination by composing diverse skills learned by augmenting individual subtask objectives with DIAYN-style diversity bonuses. Once individual diverse skills are learned for the subtasks, the agents are combined by a meta-agent to coordinate multiple distinct robots to achieve a shared goal.\n\nThis is a good application of low-level skill learning to multi-agent coordination. I have settled on a weak acceptance, because the approach is simple and seems scalable, but the acceptance is weak because the method relies on specifying the subtasks in advance.\n\nThe approach is well-motivated in that learning individual skills in isolation is generally more tractable than learning their combined application from scratch, and the building blocks of this system are well-chosen. The results demonstrate the importance of the diversity objective, and find a good sweet spot for the diversity weight.\n\nI do have some criticisms, related primarily to the decision to pre-train with both a continuously-parameterized diversity conditioning as well as a discrete set of concrete subtasks. Because these subtasks must be specified in advance, this limits the wide applicability of the resulting approach to those that can be broken down a-priori into components. Did the authors consider using the DIAYN objective on its own to encourage sufficiently diverse behaviors? If this didn't work, would perhaps a larger latent skill vector, or a large discrete set of DIAYN skills, have made it work?\n\nI also don't see the size of the latent skill embedding reported anywhere. How big is this vector; that information should be added to the paper.\n\nHowever, the approach is generally good. I think the paper would be improved if it included a strong baseline that uses DIAYN only (no a-priori subtasks), so we can evaluate how important that expert knowledge is to the final performance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eyGV_aYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a specific way of incorporating temporal abstraction into the multi-agent reinforcement learning (MARL) setting. Specifically, this method first discovers diversified skills for every single agent and then train a meta-policy to choose among skills for all agents. \n\nOverall, this paper is well presented so I can understand it well. Unfortunately, this paper didn't give me too much scientific insight. As maybe this is because I don't know too much about MARL, I would like to ask the author to help me address the following questions. \n\nMy first key question is, should we treat temporal abstraction (TA) under the multi-agent setting different from it under the single-agent setting? If they are the same, why do we bother discussing TA under the multi-agent setting? Why not just discuss it under the simpler single-agent setting? If they are not, what are the differences? \n\nThe second key question is if TA under the multi-agent system is special, then why the DIAYN method, which is proposed under the single-agent setting, could be directly used in the multi-agent setting? Why should we consider the DIAYN method, instead of other skills discovery methods? \n\nFurthermore, I would also like the author to help me address three more concrete questions. \n\n1. Section 1, paragraph 2, the author wrote: \"However, all these approaches are focused on working with a single end-effector or agent with learned primitive skills, and learning to coordinate has not been addressed.\" Does the author mean there is no temporal abstraction method for multiple collaborative agents? \n\n2. Section 3.3, the author wrote, \" the prior distribution p(z) is Gaussian.\" I.e., Z is continuous r.v. I would like to know how the author could learn q(z|s) to approximate p(z|s), which is an arbitrary continuous distribution. Maybe I am wrong, but I don't see a way to do this. \n\n3. In algorithm 1, a skill, once being chosen, will be executed for T_{low} steps, where T_{low} is fixed and pre-defined by the algorithm designer. I would like to hear to author analyzing the pros and cons of this critical design choice.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with multi-agent hierarchical reinforcement learning. A discrete set of pre-specified low-level skills are modulated by a conditioning vector and trained in a fashion reminiscent of Diversity Is All You Need, and then combined via a meta-policy which coordinates multiple agents in pursuit of a goal. The idea is that fine control over primitive skills is beneficial for achieving coordinated high-level behaviour.

The paper improved considerably in its completeness and in the addition of baselines, notably DIAYN without discrete, mutually exclusive skills. Reviewers agreed that the problem is interesting and the method, despite involving a degree of hand-crafting, showed promise for informing future directions. 

On the basis that this work addresses an interesting problem setting with a compelling set of experiments, I recommend acceptance.
Decision: Accept (Poster)