Learning to Coordinate Manipulation Skills via Skill Behavior Diversification

1 INTRODUCTION . Imagine you wish to play Chopin ’ s Fantaisie Impromptu on the piano . With little prior knowledge about the piece , you would first practice playing the piece with each hand separately . After independently mastering the left and right hand parts , you would move on to practicing with both hands simultaneously . To find the synchronized and non-interfering movements of two hands , you would try variable ways of playing the same melody with each hand , and eventually create a complete piece of music . Through the decomposition of skills into sub-skills of two hands and learning variations of sub-skills , humans make the learning process of manipulation skills much faster than learning everything at once . Can autonomous agents efficiently learn complicated tasks with coordination of different skills from multiple end-effectors like humans ? Learning to perform collaborative and composite tasks from scratch requires a huge amount of environment interaction and extensive reward engineering , which often results in undesired behaviors ( Riedmiller et al. , 2018 ) . Hence , instead of learning a task at once , modular approaches ( Andreas et al. , 2017 ; Oh et al. , 2017 ; Frans et al. , 2018 ; Lee et al. , 2019 ; Peng et al. , 2019 ; Goyal et al. , 2020 ) suggest to learn reusable primitive skills and solve more complex tasks by recombining the skills . However , all these approaches either focus on working with single end-effector manipulation or single agent locomotion , and these do not scale to multi-agent problems . To this end , we propose a modular framework that learns to coordinate multiple end-effectors with their primitive skills for various robotics tasks , such as bimanual manipulation . The main challenge is that naive simultaneous execution of primitive skills from multiple end-effectors can often cause unintended behaviors ( e.g . collisions between end-effectors ) . Thus , as illustrated in Figure 1 , our model needs to learn to appropriately coordinate end-effectors ; and hence needs a way to obtain , represent , and control detailed behaviors of each primitive skill . Inspired by these intuitions , our method consists of two parts : ( 1 ) acquiring primitive skills with diverse behaviors by mutual information maximization , and ( 2 ) learning a meta policy that selects a skill for each end-effector and coordinates the chosen skills by controlling the behavior of each skill . The main contribution of this paper is a modular and hierarchical approach that tackles cooperative manipulation tasks with multiple end-effectors by ( 1 ) learning primitive skills of each end-effector independently with skill behavior diversification and ( 2 ) coordinating end-effectors using diverse behaviors of the skills . Our empirical results indicate that our proposed method is able to efficiently learn primitive skills with diverse behaviors and coordinate these skills to solve challenging collaborative control tasks such as picking up a long bar , placing a block inside the container on the right side , and pushing a box with two ant agents . We provide additional qualitative results and code at https : //clvrai.com/coordination . 2 RELATED WORK . Deep reinforcement learning ( RL ) for continuous control is an active research area . However , learning a complex task either from a sparse reward or a heavily engineered reward becomes computationally impractical as the target task becomes complicated . Instead of learning from scratch , complex tasks can be tackled by decomposing the tasks into easier and reusable sub-tasks . Hierarchical reinforcement learning temporally splits a task into a sequence of temporally extended meta actions . It often consists of one meta policy ( high-level policy ) and a set of low-level policies , such as options framework ( Sutton et al. , 1999 ) . The meta policy decides which low-level policy to activate and the chosen low-level policy generates an action sequence until the meta policy switches it to another lowlevel policy . Options can be discovered without supervision ( Schmidhuber , 1990 ; Bacon et al. , 2017 ; Nachum et al. , 2018 ; Levy et al. , 2019 ) , meta-learned ( Frans et al. , 2018 ) , pre-defined ( Kulkarni et al. , 2016 ; Oh et al. , 2017 ; Merel et al. , 2019 ; Lee et al. , 2019 ) , or attained from additional supervision signals ( Andreas et al. , 2017 ; Ghosh et al. , 2018 ) . However , option frameworks are not flexible to solve a task that requires simultaneous activation or interpolation of multiple skills since only one skill can be activated at each time step . To solve composite tasks multiple policies can be simultaneously activated by adding Qfunctions ( Haarnoja et al. , 2018a ) , additive composition ( Qureshi et al. , 2020 ; Goyal et al. , 2020 ) , or multiplicative composition ( Peng et al. , 2019 ) . As each policy takes the whole observation as input and controls the whole agent , it is not robust to changes in unrelated parts of the observation . For example , a left arm skill can be affected by the pose change in the right arm , which is not relevant to the left arm skill . Hence , these skill composition approaches can fail when an agent encounters a new combination of skills or a new skill is introduced since the agent will experience unseen observations . Instead of having a policy with the full observation and action space , multi-agent reinforcement learning ( MARL ) suggests to explicitly split the observation and action space according to agents ( e.g . robots or end-effectors ) , which allows efficient low-level policy training as well as flexible skill composition . For cooperative tasks , communication mechanisms ( Sukhbaatar et al. , 2016 ; Peng et al. , 2017 ; Jiang & Lu , 2018 ) , sharing policy parameters ( Gupta et al. , 2017 ) , and decentralized actor with centralized critic ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) have been actively used . However , these approaches suffer from the credit assignment problem ( Sutton , 1984 ) among agents and the lazy agent problem ( Sunehag et al. , 2018 ) . As agents have more complicated morphology and larger observation space , learning a policy for a multi-agent system from scratch requires extremely long training time . Moreover , the credit assignment problem becomes more challenging when the complexity of cooperative tasks increases and all agents need to learn completely from scratch . To resolve these issues , we propose to first train reusable skills for each agent in isolation , instead of learning primitive skills of multiple agents together . Then , we recombine these skills ( Maes & Brooks , 1990 ) to complete more complicated tasks with learned coordination of the skills . To coordinate skills from multiple agents , the skills have to be flexible ; hence , a skill can be adjusted to collaborate with other agents ’ skills . Maximum entropy policies ( Haarnoja et al. , 2017 ; 2018a ; b ) can learn diverse ways to achieve a goal by maximizing not only reward but also entropy of the policy . In addition , Eysenbach et al . ( 2019 ) proposes to discover diverse skills without reward by maximizing entropy as well as mutual information between resulting states and latent representations of skills ( i.e . skill embeddings ) . Our method leverages the maximum entropy policy ( Haarnoja et al. , 2018b ) with the discriminability objective ( Eysenbach et al. , 2019 ) to learn a primitive skill with diverse behaviors conditioned on a controllable skill embedding . This controllable skill embedding will be later used as a behavior embedding for the meta policy to adjust a primitive skill ’ s behavior for coordination . 3 METHOD . In this paper , we address the problem of solving cooperative manipulation tasks that require collaboration between multiple end-effectors or agents . Note that we use the terms “ end-effector ” and “ agent ” interchangeably in this paper . Instead of learning a multi-agent task from scratch ( Lowe et al. , 2017 ; Gupta et al. , 2017 ; Sunehag et al. , 2018 ; Foerster et al. , 2018 ) , modular approaches ( Andreas et al. , 2017 ; Frans et al. , 2018 ; Peng et al. , 2019 ) suggest to learn reusable primitive skills and solve more complex tasks by recombining these skills . However , concurrent execution of primitive skills of multiple agents fails when agents never experienced a combination of skills during the pre-training stage , or skills require temporal or behavioral coordination . Therefore , we propose a modular and hierarchical framework that learns to coordinate multiple agents with primitive skills to perform a complex task . Moreover , during primitive skill training , we propose to learn a latent behavior embedding , which provides controllability of each primitive skill to the meta policy while coordinating skills . In Section 3.2 , we describe our modular framework in detail . Next , in Section 3.3 , we elaborate how controllable primitive skills can be acquired . Lastly , we describe how the meta policy learns to coordinate primitive skills in Section 3.4 . 3.1 PRELIMINARIES . We formulate our problem as a Markov decision process defined by a tuple { S , A , T , R , ρ , γ } of states , actions , transition probability , reward , initial state distribution , and discount factor . In our formulation , we assume the environment includes N agents . To promote consistency in our terminology , we use superscripts to denote the index of agent and subscripts to denote time or primitive skill index . Hence , the state space and action space for an agent i can be represented as Si andAi where each element of Si is a subset of the corresponding element in S andA = A1×A2×· · ·×AN , respectively . For each agent i , we provide a set of mi skills , Πi = { πi1 , . . . , πimi } . A policy of an agent i is represented as πi cit ( ait|sit ) ∈ Πi , where cit is a skill index , sit ∈ Si is a state , and ait ∈ Ai is an agent action at time t. An initial state s0 is sampled from ρ , and then , N agents take actions a1t , a 2 t , . . . , a N t sampled from a composite policy π ( a1t , a 2 t , . . . , a N t |st , c1t , c2t , . . . , cNt ) = ( π1c1t ( a 1 t |st ) , π2c2t ( a 2 t |st ) , . . . , πNcNt ( a N t |st ) ) and receive a single reward rt . The performance is evaluated based on a discounted return R =∑T−1 t=0 γ trt , where T is the episode horizon . 3.2 MODULAR FRAMEWORK . As illustrated in Figure 2 , our model is composed of two components : a meta policy πmeta and a set of primitive skills of N agents Π1 , . . . , ΠN . Note that each primitive skill πici ∈ Π i contains variants of behaviors parameterized by an Nz-dimensional latent behavior embedding zi ( see Section 3.3 ) . The meta policy selects a skill to execute for each agent , rather than selecting one primitive skill for the entire multi-agent system to execute . Also , we give the meta policy the capability to select which variant of the skill to execute ( see Section 3.4 ) . Then , the chosen primitive skills are simultaneously executed for Tlow time steps . The concurrent execution of multiple skills often leads to undesired results and therefore requires coordination between the skills . For example , naively placing a block in the left hand to a container being moved by the right hand can cause collision between the two robot arms . The arms can avoid collision while performing the skills by properly adjusting their skill behaviors ( e.g . the left arm leaning to the left side while placing the block and the right arm leaning to the right side while pushing the container ) as shown in Figure 1 . In our method , the meta policy learns to coordinate multiple agents ’ skills by manipulating the behavior embeddings ( i.e . selecting a proper behavior from diverse behaviors of each skill ) .

Summary: The paper presents a hierarchical reinforcement learning method for coordination of multiple cooperative agents with pre-learned adaptable skills. These skills are learned via a maximum entropy objective where diversity of behaviour given each skill is maximised and controlled via a latent conditioning vector. This allows controllability of the variation in skill execution by the meta-policy via changing the skill-specific latent vectors. The paper presents empirical results in manipulation (pick-push-place and moving a long bar by coordinating two Jaco arms) and locomotion (two Ants pushing a large block to a goal location). The method proposed outperforms the baselines reported.