Title: LEARNING TO COORDINATE MANIPULATION SKILLS VIA SKILL BEHAVIOR DIVERSIFICATION

Abstract: When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together. Similarly, a robot with multiple end-effectors can perform complex tasks by coordinating sub-skills of each end-effector. To realize temporal and behavioral coordination of skills, we propose a modular framework that first individually trains sub-skills of each end-effector with skill behavior diversification, and then learns to coordinate end-effectors using diverse behaviors of the skills. We demonstrate that our proposed framework is able to efficiently coordinate skills to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside a container while pushing the container with two robot arms, and pushing a box with two ant agents. Videos and code are available at https://clvrai.com/coordination.

1. INTRODUCTION:

Imagine you wish to play Chopin's Fantaisie Impromptu on the piano. With little prior knowledge about the piece, you would first practice playing the piece with each hand separately. After independently mastering the left and right hand parts, you would move on to practicing with both hands simultaneously. To find the synchronized and non-interfering movements of two hands, you would try variable ways of playing the same melody with each hand, and eventually create a complete piece of music. Through the decomposition of skills into sub-skills of two hands and learning variations of sub-skills, humans make the learning process of manipulation skills much faster than learning everything at once. Can autonomous agents efficiently learn complicated tasks with coordination of different skills from multiple end-effectors like humans? Learning to perform collaborative and composite tasks from scratch requires a huge amount of environment interaction and extensive reward engineering, which often results in undesired behaviors (Riedmiller et al., 2018) . Hence, instead of learning a task at once, modular approaches (Andreas et al., 2017; Oh et al., 2017; Frans et al., 2018; Lee et al., 2019; Peng et al., 2019; Goyal et al., 2020) suggest to learn reusable primitive skills and solve more complex tasks by recombining the skills. However, all these approaches either focus on working with single end-effector manipulation or single agent locomotion, and these do not scale to multi-agent problems.

To this end, we propose a modular framework that learns to coordinate multiple end-effectors with their primitive skills for various robotics tasks, such as bimanual manipulation. The main challenge is that naive simultaneous execution of primitive skills from multiple end-effectors can often cause unintended behaviors (e.g. collisions between end-effectors). Thus, as illustrated in Figure 1 , our model needs to learn to appropriately coordinate end-effectors; and hence needs a way to obtain, represent, and control detailed behaviors of each primitive skill. Inspired by these intuitions, our method consists of two parts: (1) acquiring primitive skills with diverse behaviors by mutual information maximization, and (2) learning a meta policy that selects a skill for each end-effector and coordinates the chosen skills by controlling the behavior of each skill.

The main contribution of this paper is a modular and hierarchical approach that tackles cooperative manipulation tasks with multiple end-effectors by (1) learning primitive skills of each end-effector independently with skill behavior diversification and (2) coordinating end-effectors using diverse Figure 1 : Composing complex skills using multiple agents' primitive skills requires proper coordination between agents since concurrent execution of primitive skills requires temporal and behavioral coordination. For example, to move a block into a container on the other end of the table, the agent needs to not only utilize pick, place, and push primitive skills at the right time but also select the appropriate behaviors for these skills, represented as latent vectors z 1 , z 2 , z 3 , and z 4 above. Naive methods neglecting either temporal or behavioral coordination will produce unintended behaviors, such as collisions between end-effectors.

behaviors of the skills. Our empirical results indicate that our proposed method is able to efficiently learn primitive skills with diverse behaviors and coordinate these skills to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside the container on the right side, and pushing a box with two ant agents. We provide additional qualitative results and code at https://clvrai.com/coordination.

2. RELATED WORK:

Deep reinforcement learning (RL) for continuous control is an active research area. However, learning a complex task either from a sparse reward or a heavily engineered reward becomes computationally impractical as the target task becomes complicated. Instead of learning from scratch, complex tasks can be tackled by decomposing the tasks into easier and reusable sub-tasks. Hierarchical reinforcement learning temporally splits a task into a sequence of temporally extended meta actions. It often consists of one meta policy (high-level policy) and a set of low-level policies, such as options framework (Sutton et al., 1999) . The meta policy decides which low-level policy to activate and the chosen low-level policy generates an action sequence until the meta policy switches it to another lowlevel policy. Options can be discovered without supervision (Schmidhuber, 1990; Bacon et al., 2017; Nachum et al., 2018; Levy et al., 2019) , meta-learned (Frans et al., 2018 ), pre-defined (Kulkarni et al., 2016; Oh et al., 2017; Merel et al., 2019; Lee et al., 2019) , or attained from additional supervision signals (Andreas et al., 2017; Ghosh et al., 2018) . However, option frameworks are not flexible to solve a task that requires simultaneous activation or interpolation of multiple skills since only one skill can be activated at each time step.

To solve composite tasks multiple policies can be simultaneously activated by adding Qfunctions (Haarnoja et al., 2018a) , additive composition (Qureshi et al., 2020; Goyal et al., 2020) , or multiplicative composition (Peng et al., 2019) . As each policy takes the whole observation as input and controls the whole agent, it is not robust to changes in unrelated parts of the observation. For example, a left arm skill can be affected by the pose change in the right arm, which is not relevant to the left arm skill. Hence, these skill composition approaches can fail when an agent encounters a new combination of skills or a new skill is introduced since the agent will experience unseen observations. Instead of having a policy with the full observation and action space, multi-agent reinforcement learning (MARL) suggests to explicitly split the observation and action space according to agents (e.g. robots or end-effectors), which allows efficient low-level policy training as well as flexible skill composition. For cooperative tasks, communication mechanisms (Sukhbaatar et al., 2016; Peng et al., 2017; Jiang & Lu, 2018) , sharing policy parameters (Gupta et al., 2017) , and decentralized actor with centralized critic (Lowe et al., 2017; Foerster et al., 2018) have been actively used. However, these approaches suffer from the credit assignment problem (Sutton, 1984) among agents and the lazy agent problem (Sunehag et al., 2018) . As agents have more complicated morphology and larger observation space, learning a policy for a multi-agent system from scratch requires extremely long training time. Moreover, the credit assignment problem becomes more challenging when the complexity of cooperative tasks increases and all agents need to learn completely from scratch. To resolve these issues, we propose to first train reusable skills for each agent in isolation, instead of 

Primitive Skills:

Selected Skill Index

Figure 2: Our method is composed of two components: a meta policy and a set of agent-specific primitive policies relevant to task completion. The meta policy selects which primitive skill to run for each agent as well as the behavior embedding (i.e. variation in behavior) of the chosen primitive skill.

Each selected primitive skill takes as input the agent observation and the behavior embedding and outputs action for that agent.

learning primitive skills of multiple agents together. Then, we recombine these skills (Maes & Brooks, 1990) to complete more complicated tasks with learned coordination of the skills.

To coordinate skills from multiple agents, the skills have to be flexible; hence, a skill can be adjusted to collaborate with other agents' skills. Maximum entropy policies (Haarnoja et al., 2017; 2018a; b) can learn diverse ways to achieve a goal by maximizing not only reward but also entropy of the policy.

In addition, Eysenbach et al. (2019) proposes to discover diverse skills without reward by maximizing entropy as well as mutual information between resulting states and latent representations of skills (i.e. skill embeddings). Our method leverages the maximum entropy policy (Haarnoja et al., 2018b) with the discriminability objective (Eysenbach et al., 2019) to learn a primitive skill with diverse behaviors conditioned on a controllable skill embedding. This controllable skill embedding will be later used as a behavior embedding for the meta policy to adjust a primitive skill's behavior for coordination.

3. METHOD:

In this paper, we address the problem of solving cooperative manipulation tasks that require collaboration between multiple end-effectors or agents. Note that we use the terms "end-effector" and "agent" interchangeably in this paper. Instead of learning a multi-agent task from scratch (Lowe et al., 2017; Gupta et al., 2017; Sunehag et al., 2018; Foerster et al., 2018) , modular approaches (Andreas et al., 2017; Frans et al., 2018; Peng et al., 2019) suggest to learn reusable primitive skills and solve more complex tasks by recombining these skills. However, concurrent execution of primitive skills of multiple agents fails when agents never experienced a combination of skills during the pre-training stage, or skills require temporal or behavioral coordination.

Therefore, we propose a modular and hierarchical framework that learns to coordinate multiple agents with primitive skills to perform a complex task. Moreover, during primitive skill training, we propose to learn a latent behavior embedding, which provides controllability of each primitive skill to the meta policy while coordinating skills. In Section 3.2, we describe our modular framework in detail. Next, in Section 3.3, we elaborate how controllable primitive skills can be acquired. Lastly, we describe how the meta policy learns to coordinate primitive skills in Section 3.4.

3.1. PRELIMINARIES:

We formulate our problem as a Markov decision process defined by a tuple {S, A, T , R, ρ, γ} of states, actions, transition probability, reward, initial state distribution, and discount factor. In our formulation, we assume the environment includes N agents. To promote consistency in our terminology, we use superscripts to denote the index of agent and subscripts to denote time or primitive skill index. Hence, the state space and action space for an agent i can be represented as S i and A i where each element of S i is a subset of the corresponding element in S and A = A 1 ×A 2 ×• • •×A N , respectively. For each agent i, we provide a set of m i skills, Π i = {π i 1 , . . . , π i m i }. A policy of an agent i is represented as The vanilla RL method considers all agents as a monolithic agent; thus a single policy takes the full observation as input and outputs the full action.

(b) The multi-agent RL method (MARL) consists of N policies that operate on the observations and actions of corresponding agents. (c) The modular network consists of N sets of skills for the N agents trained in isolation and a meta policy that selects a skill for each agent. (d-f) The RL, MARL, and modular network methods augmented with skill behavior diversification (SBD) has a meta policy that outputs a skill behavior embedding vector z for each skill.

π i c i t (a i t |s i t ) ∈ Π i

, where c i t is a skill index, s i t ∈ S i is a state, and a i t ∈ A i is an agent action at time t. An initial state s 0 is sampled from ρ, and then, N agents take actions a 1 t , a 2 t , . . . , a N t sampled from a composite policy π(a

1 t , a 2 t , . . . , a N t |s t , c 1 t , c 2 t , . . . , c N t ) = (π 1 c 1 t (a 1 t |s t ), π 2 c 2 t (a 2 t |s t ), . . . , π N c N t (a N t |s t )

) and receive a single reward r t . The performance is evaluated based on a discounted return R = T -1 t=0 γ t r t , where T is the episode horizon.

3.2. MODULAR FRAMEWORK:

As illustrated in Figure 2 , our model is composed of two components: a meta policy π meta and a set of primitive skills of N agents Π 1 , . . . , Π N . Note that each primitive skill π i c i ∈ Π i contains variants of behaviors parameterized by an N z -dimensional latent behavior embedding z i (see Section 3.3). The meta policy selects a skill to execute for each agent, rather than selecting one primitive skill for the entire multi-agent system to execute. Also, we give the meta policy the capability to select which variant of the skill to execute (see Section 3.4). Then, the chosen primitive skills are simultaneously executed for T low time steps.

The concurrent execution of multiple skills often leads to undesired results and therefore requires coordination between the skills. For example, naively placing a block in the left hand to a container being moved by the right hand can cause collision between the two robot arms. The arms can avoid collision while performing the skills by properly adjusting their skill behaviors (e.g. the left arm leaning to the left side while placing the block and the right arm leaning to the right side while pushing the container) as shown in Figure 1 . In our method, the meta policy learns to coordinate multiple agents' skills by manipulating the behavior embeddings (i.e. selecting a proper behavior from diverse behaviors of each skill).

3.3. TRAINING AGENT-SPECIFIC PRIMITIVE SKILLS WITH DIVERSE BEHAVIORS:

To adjust a primitive skill to collaborate with other agents' skills in a new environment, the skill needs to support variations of skill behaviors when executed at a given state. Moreover, a behavioral variation of a skill should be controllable by the meta policy for skill coordination. In order to make our primitive skill policies generate diverse behaviors controlled by a latent vector z, we leverage the entropy and mutual information maximization objective introduced in Eysenbach et al. (2019) .

More specifically, a primitive policy of an agent i outputs an action a ∈ A conditioned on the current state s ∈ S and a latent behavior embedding z ∼ p(z), where the prior distribution p(z) is Gaussian (we omit agent i in this section for the simplicity of notations). Diverse behaviors conditioned on a random sample z can be achieved by maximizing the mutual information between behaviors and states M I(s, z), while minimizing the mutual information between behaviors and actions given the state M I(a, z|s), together with maximizing the entropy of the policy H(a|s) to encourage diverse behaviors. The objective can be written as follows (we refer the readers to Eysenbach et al. (2019) Algorithm 1 ROLLOUT 1: Input: Meta policy π meta , sets of primitive policies Π 1 , ..., Π N , and meta horizon T low 2: Initialize an episode t ← 0 and receive initial state s 0 3: while episode is not terminated do 4:

Sample skill indexes and behavior embeddings (c 1 t , . . . , c N t ), (z 1 t , . . . , z N t ) ∼ π meta (s t )

5:

τ ← 0 6:

while τ < T low and episode is not terminated do 7:

a t+τ = (a 1 t+τ , . . . , a N t+τ ) ∼ (π 1 c 1 t (s t+τ , z 1 t ), . . . , π N c N t (s t+τ , z N t ))

8: t ← t + τ 13: end while for derivation):

s t+τ +1 ,

EQUATION

where the learned discriminator q φ (z|s) approximates the posterior p(z|s).

To achieve a primitive skill with diverse behaviors, we augment Equation (3) to the environment reward:

EQUATION

where λ 1 is the entropy coefficient and λ 2 is the diversity coefficient which corresponds identifiability of behaviors. Maximizing Equation (3) encourages multi-modal exploration strategies while maximizing the reward r t forces to achieve its own goal. Moreover, by maximizing identifiability of behaviors, the latent vector z, named behavior embedding, can represent a variation of the learned policy and thus can be used to control the behavior of the policy. For example, when training a robot to move an object, a policy learns to move the object quickly as well as slowly, and these diverse behaviors map to different latent vectors z. We empirically show that the policies with diverse behaviors achieve better compositionality with other agents in our experiments.

3.4. COMPOSING PRIMITIVE SKILLS WITH META POLICY:

We denote the meta policy as π meta (c 1 , . . . , c N , z 1 , . . . , z N |s t ), where c i ∈ [1, m i ] represents a skill index of an agent i ∈ [1, N ] and z i ∈ R Nz represents a behavior embedding of the skill. Every T low time steps, the meta policy chooses one primitive skill π i c i ∈ Π i for each agent i. Also, the meta policy outputs a set of latent behavior embeddings (z 1 , z 2 , . . . , z N ) and feeds them to the corresponding skills (i.e. π i c i (a i |s i , z i ) for agent i). Once a set of primitive skills {π 1 c 1 , . . . , π N c N } are chosen to be executed, each primitive skill generates an action a i ∼ π i c i (a i |s i , z i ) based on the current state s i and the latent vector z i . Algorithm 1 illustrates the overall rollout process.

Since there are a finite number of skills for each agent to execute, the meta action space for each agent [1, m i ] is discrete, while the behavior embedding space for each agent R Nz is continuous. Thus, the meta policy is modeled as a (2 × N )-head neural network where the first N heads represent m i -way categorical distributions for skill selection and the last N heads represent N z -dimensional Gaussian distributions for behavior control of the chosen skill.

3.5. IMPLEMENTATION:

We model the primitive policies and posterior distributions q φ as neural networks. We train the primitive policies using soft actor-critic (Haarnoja et al., 2018b) . When we train a primitive policy, we use a unit Gaussian distribution as the prior distribution of latent variables p(z). We use 5 as the size of latent behavior embedding N z . Each primitive policy outputs the mean and standard deviation of a Gaussian distribution over an action space. For a primitive policy, we apply tanh activation to normalize the action between [-1, 1]. We model the meta policy as neural network with multiple heads that output the skill index c i and behavior embedding z i for each agent. The meta policy is trained using PPO (Schulman et al., 2017; 2016; Dhariwal et al., 2017) . All policy networks in this paper consist of 3 fully connected layers of 64 hidden units with ReLU nonlinearities. The discriminator q φ in Equation ( 4) is a 2-layer fully connected network with 64 hidden units.

4. EXPERIMENTS:

To demonstrate the effectiveness of our framework, we compare our method to prior methods in the field of multi-agent RL and ablate the components of our framework to understand their importance.

We conducted experiments on a set of challenging robot control environments that require coordination of different agents to complete collaborative robotic manipulation and locomotion tasks.

Through our experiments, we aim to answer the following questions: (1) can our framework efficiently learn to combine primitive skills to execute a complicated task; (2) can our learned agent exhibit collaborative behaviors during task execution; and (3) can our framework leverage the controllable behavior variations of the primitive skills to achieve better coordination?

For details about environments and training, please refer to the supplementary material. As the performance of training algorithms varies between runs, we train each method on each task with 6 different random seeds and report mean and standard deviation of each method's success rate.

4.1. BASELINES:

We compare the performance of our method with various single-and multi-agent RL methods illustrated in Figure 3 :

Single-agent RL (RL): A vanilla RL method where a single policy takes as input the full observation and outputs all agents' actions.

Multi-agent RL (MARL)::

A multi-agent RL method where each of N policies takes as input the observation of the corresponding agent and outputs an action for that agent. All policies share the global critic learned from a single task reward (Lowe et al., 2017) .

Modular Framework (Modular)::

A modular framework composed of a meta policy and N sets of primitive skills (i.e. one or more primitive skills per agent). Every T low time steps, the meta policy selects a primitive skill for each agent based on the full observation. Then, the chosen skills are executed for T low time steps. Figure 5 : Success rates of our method (Modular-SBD) and baselines. For modular frameworks (Modular and Modular-SBD), we shift the learning curves rightwards the total number of environment steps the agent takes to learn the primitive skills (0.9 M, 1.2 M, and 2.0 M, respectively). Our method substantially improves learning speed and performance on JACO PICK-PUSH-PLACE and ANT PUSH.

The shaded areas represent the standard deviation of results from six different seeds. The curves are smoothed using moving average over 10 runs.

Single-agent RL with Skill Behavior Diversification (RL-SBD): An RL method augmented with the behavior diversification objective. A meta policy is employed to generate a behavior embedding for a low-level policy, and the low-level policy outputs all agents' actions conditioned on the behavior embedding and the full observation for T low time steps. The meta policy and the low-level policy are jointly trained with the behavior diversification objective described in Equation ( 4).

Multi-agent RL with Skill Behavior Diversification (MARL-SBD)::

A MARL method augmented with the behavior diversification objective. A meta policy generates N behavior embeddings. Then, each low-level policy outputs each agent's action conditioned on its observation and behavior embedding for T low time steps. All policies are jointly trained to maximize Equation ( 4).

Modular Framework with Skill Behavior Diversification (Modular-SBD, Ours): Our method which coordinates primitive skills of multiple agents. The modular framework consists of a meta policy and N sets of primitive skills, where each primitive skill is conditioned on a behavior embedding z. The meta policy takes as input the full observation and selects both a primitive skill and a behavior embedding for each agent. Then, each primitive skill outputs action for each agent.

4.2. JACO PICK-PUSH-PLACE:

We developed JACO PICK-PUSH-PLACE and JACO BAR-MOVING environments using two Kinova Jaco arms, where each Jaco arm is a 9 DoF robotic arm with 3 fingers. JACO PICK-PUSH-PLACE starts with a block on the left and a container on the right. The robotic arms need to pick up the block, push the container to the center, and place the block inside the container. For successful completion of the task, the two Jaco arms have to concurrently execute their distinct sets of skills and dynamically adjust their picking, pushing, and placing directions to avoid collision between arms.

Primitives skills. There are three primitive skills available to each arm: Picking up, Pushing, and Placing to center (see Figure 4a ). Picking up requires a robotic arm to pick up a small block, which is randomly placed on the table. If the block is not picked up after a certain amount of time or the arm drops the block, the agent fails. Pushing learns to push a big container to its opposite side (e.g. from left to the center or from right to center). The agent fails if it cannot place the container to the center.

Placing to center requires placing an object in the gripper to the table. The agent only succeeds when it stably places the object at the desired location on the container.

Composite task. Our method (Modular-SBD) can successfully perform JACO PICK-PUSH-PLACE task while all baselines fail to compose primitive skills as shown in Figure 5a . The RL and MARL baselines cannot learn the composite task mainly because the agent requires to learn the combinatorial number of skill compositions and to solve the credit assignment problem across multiple agents.

Since the composite task requires multiple primitive skills of multiple agents to be performed properly at the same time, a reward signal about a failure case cannot be assigned to the correct agent or skill. By using pre-trained primitive skills, the credit assignment problem is relaxed and all agents can Table 1 : Success rates for all tasks, comparing our method against baselines. Each entry in the table represents average success rate and standard deviation over 100 runs. The baselines learning from scratch fail to learn complex tasks with multiple agents.

perform their skills concurrently. Therefore, the Modular baseline learns to achieve success but shows significantly lower performance than our method (Modular-SBD). This is because the lack of skill behavior diversification makes it impossible to adjust pushing and placing trajectories during skill composition time, which resulting in frequent end-effector collisions.

4.3. JACO BAR-MOVING:

In JACO BAR-MOVING, two Jaco arms need to pick up a long bar together, move the bar towards a target location while maintaining its rotation, and place it on the table (see Figure 4b ). The initial position of the bar is randomly initialized every episode and an agent needs to find appropriate coordination between two arms for each initialization. Compared to JACO PICK-PUSH-PLACE, this task requires that the two arms synchronize their movements and perform more micro-level adjustments to their behaviors.

Primitives skills. There are two pre-trained primitive skills available to each arm: Picking up and Placing towards arm. Picking up is same as described in Section 4.2. Placing towards arm learns to move a small block (half size of the block used in the composite task) in the hand towards the robotic arm and then place it on the table. The agent fails if it cannot place the block to the target location.

Composite task. The JACO BAR-MOVING task requires the two arms to work very closely together. For example, the Picking up skill of both arms should be synchronized when they start to lift the bar and two arms require to lift the bar while maintaining the relative position between them since they are connected by holding the bar. The modular framework without explicit coordination of skills (Modular) can synchronize the execution of picking, moving, and placing. But the inability to micro-adjust the movement of the other arm causes instability of bar picking and moving. This results in degraded success rates compared to the modular framework with explicit coordination. Meanwhile, all baselines without pre-defined primitive skills fail to learn JACO BAR-MOVING.

4.4. ANT PUSH:

We developed a multi-ant environment, ANT-PUSH, inspired from Nachum et al. (2019) , simulated in the MuJoCo (Todorov et al., 2012) physics engine. We use the ant model in OpenAI Gym (Brockman et al., 2016) . In this environment, two ants need to push a large object toward a green target place, collaborating with each other to keep the angle of the object as stable as possible (see Figure 4c ).

Primitives skills. We train walking skills of an ant agent in 4 directions: up, down, left, and right. During primitive skill training, a block (half size of the block used in the composite task) and an ant agent are randomly placed. Pushing the block gives an additional reward to the agent, which prevents an ant to avoid the block. The learned primitive skills have different speed and trajectories conditioned on the latent behavior embedding.

Composite task. Our method achieves 32.3% success rate on ANT PUSH task while all baselines fail to compose primitive skills as shown in Figure 5c and Table 1 . The poor performance of RL, MARL, RL-SBD, and MARL-SBD baselines shows the difficulty of credit assignment between agents, which leads one of the ants moves toward a block and pushes it but another ant does not move. Moreover, the Modular baseline with primitive skills also fails to learn the pushing task. This result illustrates the importance of coordination of agents, which helps synchronizing and controlling the velocities of both ant agents to push the block toward the goal position while maintaining its rotation. (b) BAR-MOVING Figure 7 : Success rates of our method with different T low coefficients on Jaco environments.

4.5. EFFECT OF DIVERSITY OF PRIMITIVE SKILLS:

To analyze the effect of the diversity of primitive skills, we compare our model with primitive skills trained with different diversity coefficients λ 2 = {0.0, 0.05, 0.1, 0.5, 1.0} in Equation ( 4) on ANT PUSH. Figure 6 shows that with small diversity coefficients λ 2 = {0.05, 0.1}, the agent can control detailed behaviors of primitive skills while primitive skills without diversity (λ 2 = 0) cannot be coordinated. The meta policy tries to synchronize two ant agents' positions and velocities by switching primitive skills, but it cannot achieve proper coordination without diversified skills. On the other hand, large diversity coefficients λ 2 = {0.5, 1.0} make the primitive skills often focus on demonstrating diverse behaviors and fail to achieve the goals of the skills. Hence, these primitive skills do not have enough functionality to solve the target task. The diversity coefficient needs to be carefully chosen to acquire primitive skills with good performance as well as diverse behaviors.

4.6. EFFECT OF SKILL SELECTION INTERVAL T low:

To analyze the effect of the skill selection interval hyperparameter T low , we compare our method trained with T low = {1, 2, 3, 5, 10} on Jaco environments. The success rate curves in Figure 7 demonstrate that smaller T low values in range [1, 3] lead to better performance. This can be because the agent can realize more flexible skill coordination by adjusting the behavior embedding frequently.

In addition to the fixed T low values, we also consider the variation of our method in which the skill behavior embedding is only sampled when the meta policy updates its skill selection. Concretely, we set the value of T low to 1 but update (z 1 t , . . . , z N t ) only if (c 1 t , . . . , c N t ) = (c 1 t-1 , . . . , c N t-1 ). We observe that in this setting, the meta policy at times switch back and forth between two skills in two consecutive time steps, leading to slightly worse performance compared to our method with small T low values. This indicates that the meta policy needs to adjust the behavior embedding in order to optimally coordinate skills of the different agents.

5. CONCLUSION:

In this paper, we propose a modular framework with skill coordination to tackle challenges of composition of sub-skills with multiple agents. Specifically, we use entropy maximization with mutual information maximization to train controllable primitive skills with diverse behaviors. To coordinate learned primitive skills, the meta policy predicts not only the skill to execute for each agent (end-effector) but also the behavior embedding that controls the chosen primitive skill's behavior. The experimental results on robotic manipulation and locomotion tasks demonstrate that the proposed framework is able to efficiently learn primitive skills with diverse behaviors and coordinate multiple agents (end-effectors) to solve challenging cooperative control tasks. Acquiring skills without supervision and extending our method to a visual domain are exciting directions for future work. Actor Networks: In all experiments, we model our actor network for each primitive skill as a 3-layer MLP with hidden layer size 64. The last layer of the MLP is two-headed -one for the mean of the action distribution and the other for the standard deviation of it. We use ReLU as activation function in hidden layers. We do not apply any activation function for the final output layer. The action distribution output represents per-dimension normal distribution, from which single actions can be sampled and executed in the environment.

Critic Networks: The critic network for each primitive skill and meta policy is modeled as a 2-layer MLP with hidden layer size 128. ReLU is used as an activation function in the hidden layers. The critic network output is used to assess the value of a given state-action pair, and is trained by fitting its outputs to the target Q-value clamped by ±100.

Meta Policy::

The meta policy is modeled as a 3-layer MLP with hidden layer size of 64. Since the actions of meta policy are sampled from N categorical distributions for each end-effector/agent and N normal distributions for behavior embeddings, the output dimension of the meta policy is N i=1 (m i + N z ). The meta policy uses ReLU as an activation function for all layers except for the final output layer.

B.3 TRAINING DETAILS:

For all baselines, we train the meta policies using PPO and the low-level policies using SAC. We use the same environment configurations, composite task reward definitions, and value of T low across all baselines.

For Jaco tasks, we train a total of 4 primitive skills -right arm pick, right arm place-to-center, right arm place-towards-arm, and left arm push -to be composed by meta-policy. For Jaco Pick-Push-Place, we provide the meta-policy with right arm pick and right arm place-to-center as right arm primitives and left arm push as left arm primitives; for Jaco Bar-Moving, we provide the meta-policy with right arm pick and right arm place-towards-arm as both right and left arm primitives and left arm pick and right arm place-towards-arm as left arm primitives. We obtain left arm primitives for bar-moving task by using the learned right arm primitives directly.

To obtain the 4 primitives skills described above, we train right arm pick with diversity coefficient λ 2 = 0.01 and the other three primitives with λ 2 = 0.1. The destination of right arm push is set to (0.3, -0.03, 0.86), which is slightly left of the center of the table. After pick primitive is trained, we train the two right arm place primitives where episodes are initialized by intermediate states of successful right arm pick episodes where the height of the box is larger than 0.94 (0.01 higher than the target pick height). The place destinations for towards-arm and to-center primitives are (0.15, -0.2, 0.86) and (0.3, -0.02, 0.86), respectively.

For non-modular baselines that incorporates skill behavior diversification, we use λ 2 = 0.01 for both Jaco Pick-Push-Place and Jaco Bar-Moving because both tasks require picking skills, which can only be trained with a small value of λ 2 .

:



Figures and Tables:

Figure 3: Different multi-agent architectures. (a)The vanilla RL method considers all agents as a monolithic agent; thus a single policy takes the full observation as input and outputs the full action. (b) The multi-agent RL method (MARL) consists of N policies that operate on the observations and actions of corresponding agents. (c) The modular network consists of N sets of skills for the N agents trained in isolation and a meta policy that selects a skill for each agent. (d-f) The RL, MARL, and modular network methods augmented with skill behavior diversification (SBD) has a meta policy that outputs a skill behavior embedding vector z for each skill.

Figure 4: The composite tasks pose a challenging combination of object manipulation and locomotion skills, which requires coordination of multiple agents and temporally extended behaviors. (a) The left Jaco arm needs to pick up a block while the right Jaco arm pushes a container, and then it places the block into the container. (b) Two Jaco arms are required to pick and place a bar-shaped block together. (c) Two ants push the red box to the goal location (green circle) together.



Table content: <table/>

r t+τ ← ENV(s t+τ , a t+τ )

Table content: <table><tr><td>9:</td><td>τ ← τ + 1</td></tr><tr><td>10:</td><td>end while</td></tr><tr><td>11:</td><td>Add a transition s t , (c 1 t , . . . , c N t ), (z 1 t , . . . , z N t ), s t+τ , r t:t+τ -1 to the rollout buffer B</td></tr><tr><td>12:</td><td/></tr></table>

Peer review (iclr2020):
The paper presents a hierarchical reinforcement learning method for coordination of multiple cooperative agents with pre-learned adaptable skills. These skills are learned via a maximum entropy objective where diversity of behaviour given each skill is maximised and controlled via a latent conditioning vector. This allows controllability of the variation in skill execution by the meta-policy via changing the skill-specific latent vectors. The paper presents empirical results in manipulation (pick-push-place and moving a long bar by coordinating two Jaco arms) and locomotion (two Ants pushing a large block to a goal location). The method proposed outperforms the baselines reported. Overall, this paper addresses an interesting problem and can be impactful with the caveat for some clarifications and analysis. Given that the authors address my concerns, I would be willing to increase my score. The main novelty of this work lies in how one can learn sub-skills that can be leveraged and adapted for down-stream tasks. The problem setting used to test the method is a multi-agent setting where it is crucial that skills are adapted to enable cooperation. I found the environments and the problem setting generally interesting and important for testing the proposed method. I have a few concerns that I listed below: 1) I found the notations at times inconsistent and confusing. It would have helped to see some more details on the diagram (Figure 2) to understand how everything fits together. 2) The set of skills for the two agents are selected by the meta-policy in every T_low steps. It looks like in the Jaco environments T_low = 1. Can you comment on this? This seems slightly concerning since it seems like the meta-controller is treating these skills as primitive actions rather than temporally extended behaviour. 3) Looking at the training curves in Figure 4, there seems to be a really high variance in performance of the method. Can you comment on this as this seems concerning. Could you run more seeds to improve this? 4) It is nice to see in section 4.5 how the hyper-parameters balancing diversity in combination with external reward (equation 4) is tuned and how sensitive that is to achieving adaptability for downstream tasks. The only criticism I have is that it is difficult to understand from *Episode reward* on y-axis what the success rate is (similar to Figure 4)? It would’ve been nice to report results in a consistent way throughout the paper for these environments. 5) Given that all the tasks in the experiments are cooperative multi-agent settings, I would have liked to see more in depth discussion regarding alternative multi-agent methods. The multi-agent baseline provided (which is using a decentralized policy with a shared critic, inspired by Lowe et al., 2017) seems fair, but I wonder if there has been more recent work in this direction that could have been highlighted?