DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation

1 INTRODUCTION . Consider the bimanual robot manipulation tasks such as rearranging multiple objects to their target locations in Figure 1 ( a ) . This complex and compositional task is very challenging as the agents will first need to reduce it to several sub-tasks ( pushing or grasping each object ) , and then the two agents will need to figure out how to allocate each sub-task to each other ( which object each robot should operate on ) for better collaboration . Importantly , two robots should avoid collision in a narrow space for safety concerns . While training a single RL agent that can solve such compositional tasks has caught research attention recently ( Chang et al. , 2019 ; Peng et al. , 2019 ; Devin et al. , 2019 ; Jiang et al. , 2019 ; Li et al. , 2021 ; 2020 ) , there are still two main challenges that are barely touched when it comes to tackle bimanual manipulation : ( i ) domination , i.e. , one robot may tend to solve all the sub-tasks while the other robot remains idle , which hurts the task solving efficiency ; ( ii ) conflict , i.e. , two robots may try to solve the same sub-task simultaneously , which result in unsafe conflicts and interruptions on shared workspace . One possible solution is to design a task-allocation reward function to encourage better coordination . However , it is particularly non-trivial and often sub-optimal to manually design such a reward function for complex problems that contain a large continuous sub-task space , such as the rearrangement task in Figure 1 ( a ) . Moreover , even with the reward function described above in hand , it remains unclear how to reduce collisions , particularly for the tasks that require two robots to act simultaneously and safely . For example , in the task shown in Figure 1 ( d ) , one robot needs to push the green door to make space for the other robot to move the blue box to the goal position . However , these two robots can easily interrupt and collide with each other when they perform these coordination actions . We consider an alternative setting using sparse rewards without explicitly assigning sub-tasks to the robots . However , this leads to another challenge : How to encourage the robots to explore collaborative and safe behaviors with limited positive feedbacks ? For bimanual manipulation , an intrinsic motivation is introduced by Chitnis et al . ( 2020b ) , leveraging the difference between the actual effect of an action ( taken by two robots ) and the composition of individual predicted effect from each agent using a forward model . While this intrinsic reward encourages the two robots to collaborate for tasks that are hard to achieve by a single robot , it does not address the domination and conflict problems for efficient and safe manipulation . In this paper , we present DAIR : Disentangled Attention Intrinsic Regularization which encourages the two robots to safely and efficiently collaborate on different sub-tasks during bimanual manipulation . Instead of designing a new intrinsic reward function , we introduce a simple regularization term for representation learning , which encourages the robots to attend to different interaction regions . Specifically , we adopt the attention mechanism ( Vaswani et al. , 2017 ) in both our policy and value networks , where we compute the dot-product between each robot representation and the object interaction region representations to obtain a probability distribution . Each robot has its own probability distribution to represent which interaction region it is focusing on . We define our intrinsic regularization as minimizing the dot product between the two probability distributions between two robots ( i.e. , to be orthogonal ) in each time step . By adding this loss function , different robots will be regularized to attend to different interaction points within their policy representation . This forces the policies to tackle sub-tasks over disjoint working space without interfering with each other . We remark that disentangled attention can be generalized to environments with multiple agents . In our experiments , we focus on five diverse manipulation tasks in simulation environments with two Fetch robots as shown in Figure 1 . These tasks not only require the robots to manipulate multiple objects ( more than two , up to eight ) with each object offering one interaction region , but also a single heavy object with multiple interaction regions ( Figure 1 ( e ) ) . In our experiments , we show that our approach not only improves performance and sample efficiency in learning , but also helps avoiding the domination problem and largely reducing the conflicts for safe coordination between two robots . Moreover , the learned policies can also solve the task in fewer steps , which is the significance of bimanual cooperation compared to single-arm manipulation . We highlight our main contributions as : • Observation for two important problems ( domination and conflict ) in training RL agents for safe bimanual manipulation , and a new robotics task set with one to eight objects . • We propose DAIR , a novel and general intrinsic regularization . It not only improves the success rate in bimanual manipulation , solves the tasks more efficiently , but also reduces the conflicts between robots . This allows the robots to collaborate and coordinate more safely . 2 RELATED WORK . Intrinsic motivation in reinforcement learning . To train RL agents with sparse rewards , Schmidhuber ( 1991 ) first proposed to motivate the agent to reach state space giving a large model prediction error , which indicates the state is currently unexplored and unseen by the model . Such a mechanism is also called intrinsic motivation , which provides a reward for agents to explore what makes it curious ( Oudeyer et al. , 2007 ; Barto , 2013 ; Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Huang et al. , 2019 ) . Recently , it has also been shown that the agents can explore with such an intrinsic motivation without extrinsic rewards ( Pathak et al. , 2017 ; Burda et al. , 2018 ; 2019 ) . Besides using prediction error , diverse skills can also be discovered by maximizing the mutual information between skills and states as the intrinsic motivation ( Eysenbach et al. , 2019 ; Sharma et al. , 2020 ) . While these approaches have achieved encouraging results in single agent cases , they are not directly applicable to environments with multiple agents . In our paper , we propose a novel intrinsic regularization for helping two robots work actively on different sub-tasks . Multi-agent collaboration . Cooperative multi-agent reinforcement learning has exhibited progress over the recent years ( Foerster et al. , 2016 ; He et al. , 2016 ; Peng et al. , 2017 ; Lowe et al. , 2017 ; Foerster et al. , 2018 ; Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ; Wang et al. , 2020a ) . For example , Lowe et al . ( 2017 ) proposed to extend the DDPG ( Lillicrap et al. , 2016 ) algorithm to the multi-agent setting with decentralized policies and centralized Q functions , which implicitly encourages the agents to cooperate . However , the problem of exploration still remains as a bottleneck , and in fact even more severe in multi-agent RL . Motivated by the previous success on a single agent , intrinsic motivation is also introduced to help multiple agents explore and collaborate ( Foerster et al. , 2016 ; Strouse et al. , 2018 ; Hughes et al. , 2018 ; Iqbal & Sha , 2019b ; Jaques et al. , 2019 ; Wang et al. , 2020b ) . For example , Jaques et al . ( 2019 ) proposed to use social motivation to provide intrinsic rewards which model the influence of one agent on another agent ’ s decision making . The work that is most related to ours is by Chitnis et al . ( 2020b ) on the intrinsic motivation for synergistic behaviors , which encourages the robots to collaborate for a task that is hard to solve by a single robot . As this paper has not focused on the domination and conflict problems , our work on disentangled attention is a complementary technique to the previous work . Bimanual manipulation . The field of bimanual manipulation has been long studied as a problem involving both hardware design and control ( Raibert & Craig , 1981 ; Hsu , 1993 ; Xi et al. , 1996 ; Smith et al. , 2012 ) . In recent years , researchers applied learning based approach to bimanual manipulation using imitation learning from demonstrations ( Zollner et al . ; Gribovskaya & Billard , 2008 ; Tung et al. , 2020 ; Xie et al. , 2020 ) and reinforcement learning ( Kroemer et al. , 2015 ; Amadio et al. , 2019 ; Chitnis et al. , 2020a ; b ; Ha et al. , 2020 ) . For example , Amadio et al . ( 2019 ) proposed to leverage probabilistic movement primitives from human demonstrations . Chitnis et al . ( 2020a ) further introduced a highlevel planning policy to combine a set of parameterized primitives to solve complex manipulation tasks . In contrast to these works , our approach does not assume access to pre-defined primitives . Both robots will learn how to perform each sub-task and how to collaborate without conflicts in an end-to-end manner , which makes the approach more general . Attention mechanism . Our intrinsic motivation is built upon the attention mechanism which has been widely applied in natural language processing ( Vaswani et al. , 2017 ) and computer vision ( Wang et al. , 2018 ; Dosovitskiy et al. , 2021 ) . Recently , the attention mechanism is also utilized in multi-agent RL to model the communication and collaboration between agents ( Zambaldi et al. , 2018 ; Jiang & Lu , 2018 ; Malysheva et al. , 2018 ; Iqbal & Sha , 2019a ; Long et al. , 2020 ) . For example , Long et al . ( 2020 ) proposed to utilize attention to flexibly increase the number of agents and perform curriculum learning for large-scale multi-agent interactions . Li et al . ( 2020 ) adopt the attention mechanism to generalize multi-object stacking with a single arm . In our paper , instead of simply using attention for interaction among hand and a variable number of objects , we propose DAIR to encourage the agents to attend on different sub-tasks for better collaboration . 3 PRELIMINARIES . We consider a multi-agent Markov decision process ( MDP ) Littman ( 1994 ) with N agents , which can be represented by ( S , A , P , R , H , γ ) . The state s ∈ S and the action ai ∈ A for agent i are continuous . P ( st+1|st , at1 , ... , atN ) represents the stochastic transition dynamics . Ri ( st , ati ) represents the reward function for agent i. H is the horizon and γ is the discount factor . The policy πθi ( a t|st ) for agent i is parameterized by θi . The goal is to learn multi-agent policies maximizing the return . In this paper , we tackle a two-agent collaboration problem ( N = 2 ) , but our method can generalize to more agents . 3.1 REINFORCEMENT LEARNING WITH SOFT ACTOR-CRITIC . We adopt the Soft Actor-Critic ( SAC ) Haarnoja et al . ( 2018 ) for reinforcement learning ( RL ) training in this paper . It is an off-policy RL method using the actor-critic framework . The soft Q-function for agent i is Qθi ( s t , ati ) parameterized by θi . For agent i , there are three types of parameters to learn in SAC : ( i ) the policy parameters φi ; ( ii ) a temperature τi ; ( iii ) the soft Q-function parameters θi . We can represent the policy optimization objective for agent i as , Jπ ( φi ) = Est∼D [ Eati∼πφi [ τi log πφi ( a t i|st ) −Qθi ( st , ati ) ] ] , ( 1 ) where τi is a learnable temperature coefficient for agent i , and D is the replay buffer . It can be learned to maintain the entropy level of the policy : J ( τi ) = Eati∼πφi [ −τi log πφi ( ati|st ) − τiH̄ ] , ( 2 ) where H̄ is a desired minimum expected entropy . The soft Q-function parameters θi for agent i can be trained by minimizing the soft Bellman residual as , JQ ( θi ) = E ( st , ati ) ∼D [ 1 2 ( Qθi ( s t , ati ) − Q̂ ( st , ati ) ) 2 ] , ( 3 ) Q̂ ( st , ati ) = Ri ( s t , ati ) + γE [ max at+1i ∼πφi Qθi ( s t+1 , at+1i ) ] . ( 4 ) Since we focus on collaborative robotics manipulation tasks , the reward is always shared and synchronized among the agents . That is , if one agent is able to finish a goal and obtain a reward , the other agents will receive the same reward .

Summary: This paper proposes an implicit regularization for bimanual manipulation that enforces two robot arms to focus on different regions, which prevents both arms from performing on the same object at the same time. The proposed method realizes this idea by computing attention between robots and objects and then constraining dot product between attentions of two arms to not overlap. This effectively prevents the conflict between two arms by encouraging two arms to focus on different objects (sub-tasks), leading to efficient RL training and safe behaviors. The empirical evaluation demonstrates that the proposed method is generalizable to unseen situations thanks to the attention mechanism with the regularization.