DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
Authors: Minghao Zhang
Pingcheng Jian
Yi Wu
Huazhe Xu
Xiaolong Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Safe Robotics
Bimanual Manipulation
Attention Mechanism

Abstract: We address the problem of safely solving complex bimanual robot manipulation tasks with sparse rewards. Such challenging tasks can be decomposed into sub-tasks that are accomplishable by different robots concurrently or sequentially for better efficiency. While previous reinforcement learning approaches primarily focus on modeling the compositionality of sub-tasks, two fundamental issues are largely ignored particularly when learning cooperative strategies for two robots: (i) domination, i.e., one robot may try to solve a task by itself and leaves the other idle; (ii) conflict, i.e., one robot can interrupt another's workspace when executing different sub-tasks simultaneously, which leads to unsafe collisions. To tackle these two issues, we propose a novel technique called disentangled attention, which provides an intrinsic regularization for two robots to focus on separate sub-tasks and objects. We evaluate our method on five bimanual manipulation tasks. Experimental results show that our proposed intrinsic regularization successfully avoids domination and reduces conflicts for the policies, which leads to significantly more efficient and safer cooperative strategies than all the baselines. Our project page with videos is at https://bimanual-attention.github.io/.

Under review as a conference paper at ICLR 2022
DAIR: DISENTANGLED ATTENTION INTRINSIC REG- ULARIZATION FOR SAFE AND EFFICIENT BIMANUAL MANIPULATION
Anonymous authors Paper under double-blind review
ABSTRACT
We address the problem of safely solving complex bimanual robot manipulation tasks with sparse rewards. Such challenging tasks can be decomposed into sub- tasks that are accomplishable by different robots concurrently or sequentially for better efficiency. While previous reinforcement learning approaches primarily focus on modeling the compositionality of sub-tasks, two fundamental issues are largely ignored particularly when learning cooperative strategies for two robots: (i) domination, i.e., one robot may try to solve a task by itself and leaves the other idle; (ii) conflict, i.e., one robot can interrupt another's workspace when executing different sub-tasks simultancously, which leads to unsafe collisions. To tackle these two issues, we propose novel technique called disentangled attention, which provides an intrinsic regularization for two robots to focus on separate sub- tasks and objects. We evaluate our method on five bimanual manipulation tasks. Experimental results show that our proposed intrinsic regularization successfully avoids domination and reduces conflicts for the policies, which leads significantly more efficient and safer cooperative strategies than all the baselines. Our project page with videos is at https: //bimanuall attention iion gittubtthh... io/.
INTRODUCTION
Consider the bimanual robot manipulation tasks such rearranging multiple objects to their target locations in Figure 1 (a). This complex and compositional task very challenging as the agents will first need to reduce it to several sub-tasks (pushing or grasping each object), and then the two agents will need to figure out how to allocate each sub-task to each other (which object each robot should operate on) for better collaboration. Importantly, two robots should avoid collision in narrow space for safety concerns. While training a single RL agent that can solve such compositional tasks has caught research attention recently (Chang al., 2019; Peng et al., 2019; Devin et al., 2019; Jiang et al., 2019; Li et al., 2021; 2020), there are still two main challenges that are barely touched when it comes tackle bimanual manipulation: (i) domination, i.e., one robot may tend solve all the sub-tasks while the other robot remains idle, which hurts the task solving efficiency; (ii) conflict, i.e., two robots may try solve the same sub-task simultancously, which result in unsafe conflicts and interruptions on shared workspace.
One possible solution is to design a task-allocation reward function to encourage better coordination. However, it is particularly non-trivial and often sub-optimal to manually design such a reward function for complex problems that contain a large continuous sub-task space, such as the rearrangement task in Figure (a). Moreover, even with the reward function described above in hand, it remains unclear how to reduce collisions, partieularly for the tasks that require two robots to act simultaneously and safely. For example, in the task shown in Figure 1 (d), one robot needs push the green door to make space for the other robot to move the blue box to the goal position. However, these two robots easily interrupt and collide with each other when they perform these coordination actions.
We consider an alternative setting using sparse rewards without explicitly assigning sub-tasks to the robots. However, this leads to another challenge: How to encourage the robots to explore collaborative and safe behaviors with limited positive feedbacks? For bimamual manipulation, an intrinsic motivation is introduced by Chitnis et al. (2020b), leveraging the difference between the

---

Under review as a conference paper at ICLR 2022
(a) Rearaangemet
(b) Stack Tower (c) Open Box & Place (d) Push with Door
(e) Adjust Bar
Figure 1: Five bimanual manipulation tasks. (a) Rearrange the blocks to goal positions. (b) Stack the blocks into tower. (c) Open the box and put the block inside. (d) Open the green door and push the block through the wall to the goal position. There are springs on both the box in (c) and the door in (d), which will close automatically without external forec. Thus it requires one robot to hold the box cover and the door. (c) Lift up
and rotate bar with two arms to a target configuration. where the gripper is locked as closed, so it cannot be done by arm. More details about our environments in Appendix A.
actual effect of an action (taken by two robots) and the composition of individual predicted effect from each agent using forward model. While this intrinsic reward encourages the two robots to collaborate for tasks that are hard to achieve by single robot, it does not address the domination and conflict problems for efficient and safe manipulation.
In this paper, we present DAIR: Disentangled Attention Intrinsic Regularization which encourages the two robots to safely and efficiently collaborate on different sub-tasks during bimanual manipulation. Instead of designing new intrinsic reward function, we introduce simple regularization term for representation learning, which encourages the robots to attend to different interaction regions. Specifically, we adopt the attention mechanism (Vaswani et al., 2017) in both our policy and value networks, where we compute the dot-product between each robot representation and the object interaction region representations to obtain probability distribution. Each robot has its probability distribution to represent which interaction region it is focusing on. We define our intrinsic regularization as minimizing the dot product between the two probability distributions between two robots (i.e., to be orthogonal) in each time step. By adding this loss function, different robots will be regularized to attend to different interaction points within their policy representation. This forces the policies to tackle sub-tasks over disjoint working space without interfering with each other. We remark that disentangled attention can be generalized environments with multiple agents.
In our experiments, we focus on five diverse manipulation tasks in simulation environments with two Fetch robots as shown in Figure 1. These tasks not only require the robots to manipulate multiple objects (more than two, up to eight) with each object offering one interaction region, but also single heavy object with multiple interaction regions (Figure 1 (e)). our experiments, we show that our approach not only improves performance and sample efficiency in learning, but also helps avoiding the domination problem and largely reducing the confficts for safe coordination between two robots. Moreover, the learned policies can also solve the task in fewer steps, which is the significance of bimanual cooperation compared to single-arm manipulation. We highlight our main contributions as:
Observation for two important problems (domination and conflict) in training RL agents for safe bimanual manipulation, and a new robotics task set with one to eight objects.
We propose DAIR, a novel and general intrinsic regularization. It not only improves the success rate in bimanual manipulation. solves the tasks more efficiently, but also reduces the conflicts between robots. This allows the robots to collaborate and coordinate more safely.
2 RELATED WORK
Intrinsic motivation in reinforcement learning. To train RL agents with sparse rewards, Schmid- huber (1991) first proposed to motivate the agent to reach state space giving a large model prediction error, which indicates the state currently unexplored and unseen by the model. Such mechanism is also called intrinsic motivation, which provides a reward for agents to explore what makes it curious (Oudeyer et al., 2007: Barto, 2013; Bellemare et al., 2016; Ostrovski et al., 2017: Huang et al., 2019). Recently, it has also been shown that the agents can explore with such an intrinsic motivation without extrinsic rewards (Pathak et al., 2017; Burda et al., 2018; 2019). Besides using prediction error, diverse skills can also be discovered by maximizing the mutual information between skills and states as the intrinsic motivation (Eysenbach et al., 2019; Sharma et al., 2020). While these approaches have achieved encouraging results in single agent cases, they are not directly applicable to environments with multiple agents. In our paper, we propose a novel intrinsic regularization for helping two robots work actively on different sub-tasks.
2

---

Under review as a conference paper at ICLR 2022
Multi-agent collaboration. Cooperative multi-agent reinforcement learning has exhibited progress over the recent years (Foerster et al., 2016; He et al., 2016; Peng et al., 2017; Lowe et al., 2017; Foerster et al., 2018; Sunchag et al., 2018; Rashid et al., 2018; Son et al., 2019; Wang al., 2020a). For example, Lowe et al. (2017) proposed to extend the DDPG (Lillicrap et al., 2016) algorithm to the multi-agent setting with decentralized policies and centralized Q functions, which implicitly encourages the agents to cooperate. However, the problem of exploration still remains as bottleneck, and in fact even more severe in multi-agent RL. Motivated by the previous success on a single agent, intrinsic motivation is also introduced to help multiple agents explore and collaborate (Foerster et al., 2016; Strouse et al., 2018; Hughes et al., 2018; Iqbal & Sha, 2019b; Jaques et al., 2019; Wang et al., 2020b). For example, Jaques et al. (2019) proposed to use social motivation to provide intrinsic rewards which model the influence of one agent on another agent's decision making. The work that most related to ours is by Chitnis et al. (2020b) on the intrinsic motivation for synergistic behaviors, which encourages the robots to collaborate for a task that is hard to solve by a single robot. As this paper has not focused on the domination and conflict problems, our work on disentangled attention is a complementary technique to the previous work.
Bimanual manipulation. The field of bimanual manipulation has been long studied as a problem involving both hardware design and control (Raibert & Craig, 1981; Hsu, 1993; Xi et al., 1996; Smith al., 2012). In recent years, researchers applied learning based approach to bimanual manipulation using imitation learning from demonstrations (Zollner et al.; Gribovskaya & Billard, 2008; Tung et al., 2020; Xie et al., 2020) and reinforcement learning (Kroemer al., 2015; Amadio et al., 2019; Chitnis et al, 2020a;b; Ha et al., 2020). For example. Amadio et al. (2019) proposed to leverage probabilistic movement primitives from human demonstrations. Chitnis al. (2020a) further introduced high- level planning policy to combine a set of parameterized primitives to solve complex manipulation tasks. In contrast these works, our approach does not assume access to pre-defined primitives. Both robots will learn how to perform each sub-task and how to collaborate without conflicts in an end-to-end manner, which makes the approach more general.
Attention mechanism. Our intrinsic motivation is built upon the attention mechanism which has been widely applied in natural language processing (Vaswani et al., 2017) and computer vision (Wang et al., 2018; Dosovitskiy et al., 2021). Recently, the attention mechanism is also utilized in multi-agent RL to model the communication and collaboration between agents (Zambaldi et al., 2018; Jiang & Lu, 2018; Malysheva et al., 2018; Iqbal & Sha, 2019a; Long et al., 2020). For example, Long et al. (2020) proposed to utilize attention to flexibly increase the number of agents and perform curriculum learning for large-scale multi-agent interactions. Li al. (2020) adopt the attention mechanism to generalize multi-object stacking with a single arm. In our paper, instead of simply using attention for interaction among hand and variable number of objects, we propose DAIR encourage the agents to attend on different sub-tasks for better collaboration.
3 PRELIMINARIES
We consider a multi-agent Markov decision process (MDP) Littman (1994) with N agents, which can be represented by (S, A, P. 7). The state s S and the action a; E for agent are continuous. P(s |s of ..., axx) represents the stochastic transition dynamics. (s*, aa}) represents the reward function for agent i. H the horizon and 7 is the discount factor. The policy no, (a'ls') for agent i is parameterized by fz- The goal is to learn multi-agent policies maximizing the return. In this paper, we tackle a two-agent collaboration problem (N- = 2), but our method can generalize to more agents.
3.1 REINFORCEMENT LEARNING WITH SOFT ACTOR-C -CRTTC
We adopt the Soft Actor-Critt (SAC) Haarnoja et al. (2018) for reinforcement learning (RL) training in this paper. It is an off- off-policy RL method using the actor-critic framework. The soft Q-function for agent i is Qa, (s',, a!) parameterized by 0g, For agent i, there are three types of parameters to learn in SAC: (i) the policy parameters dh: (ii) temperature T;; (iii) the soft Q-function parameters 0,. We can represent the policy optimization objective for agent i as,
J (Ø.) = EyEDD Bottaa, [T. log To. (afls") Qo. (e',,aa))) ad]]].
(I)
where T; is learnable temperature coefficient for agent i, and D is the replay buffer. It can be learned to maintain the entropy level of the policy:
J [(r)) EagSTTO
[-rs log na, (atls") TiHl.
3

---

Under review as a conference paper at ICLR 2022
where 7. is desired minimum expected entropy. The soft Q-function parameters 0 for agent can be trained by minimizing the soft Bellman residual as,
(Q) ((s)) 2l,0 -
20s ((s( a()))
(3)
Q Rs' at) + +
max NNTT:
Qo. .a .att))
(4)
Since we focus on collaborative robotics manipulation tasks, the reward is always shared and synchronized among the agents. That is, if one agent is able to finish goal and obtain reward, the other agents will receive the same reward.
3.2 CRITERIA FOR MEASURING EFFICIENCY AND SAFETY
Our core contribution is to ensure the manipulating efficiency and safety by reducing the problems of domination and conflict in the process of manipulation, which also improves the speed of finishing the task. We define three criteria which are all lower the better, and evaluate our approach using them in our experiments: (i) Domination Rate: We count how many steps an arm is interacting with an object in one episode the manipulating steps. We compute the ratio of agent's 's manipulating steps over the two agents" total manipulating steps. We select the maximum ratio as the Domination Rate. Ideally, we hope the Domination Rate to be close 50% which indicates both robots are actively interacting with the objects. (ii) Conflict Rate: It counts the percentage of the "conffict step" over all steps. We consider it conflict step when the distance between two grippers is smaller than a small threshold. This means two robots are interrupting each other's action. (iii) Finish Steps: How many steps do two agents take to finish the task successfully. The maximum episode length both environments is 100. Note that it's reasonable only when we consider all the metrics together, since robots can use trivial solution to reduce one of them, for example, only complete the task with one robot (extreme domination behavior) to avoid confticts.
4 METHOD
Our goal to design a model and introduce a novel intrinsic regularization to better train the policies for bimanual manipulation tasks. We hope the agents can learn to allocate the workload efficiently and safely. In this section, we will first introduce our base network architecture with the attention mechanism motivated by (Vaswani et al., 2017). Based on this architecture, we will then introduce DAIR and how to perform reinforcement learning with it.
4.1 NETWORK ARCHITECTURE
Policy and Q-function networks share the same architecture follow. For simplicity, we omit superscript time t when there is no ambiguity. We can then represent the state as s = [s1....,8y) SN. SN+1, SN+M] where the first N entities represent the state of the robot arms, the next M entities represent the states of the interaction regions. Here we define an interaction region as space object that robot can manipulate on (grasp push, etc) handle the task. In Figure 1, for the first four tasks with smaller objects such as cubes, door, and cover, we define one interaction region located around each object. For the last task of adjusting a large heavy bar, we define it with two interaction regions with one region corresponding to one face. Note that the interaction regions can be specified according to the tasks flexibly, and our method can be generalized to any numbers of interaction regions.
For agent i, we have of state encoder functions (fi.1)),..... ....)) f... f..4.. (-)))MM))))))) (-))
corresponding to the input states, as shown in Figure 2. Each state encoder function (i) takes the state 8j as the input and outputs a representation (512-D) for the state in our policy network. We use 2-layer multilayer perceptron (MLP) to model ft)((). While there are N + M state encoder functions, there are only three sets of parameters (visualized by three different colors in Figure 2): (i) the parameters of the state encoder for agent i itself f..(-);; (ii) the parameters of the other agents fr,()), (1 < j < N,1 + j) (shared); (iii) the parameters of all the interaction regions f.,())) (N + +15 & N + M) (shared). In this way, model can be extended environments with different number of interaction regions and agents. We represent the policy for agent i as,
To. (a:|s) ha(Se,(8s)) LayerNorm(
4

---

Under review as a conference paper at ICLR 2022
where (.))) is one fully connected layer further process the attention embedding U/ which encodes the relationship between agent and all the state entities (including all agents and interaction regions). Adding attention embedding to f(,($:) with LayerNorm operator serves residual module to retain agent i's own state information. The combined features are fed to a 2-layer MLP (.))). The output of h.).) is the action distribution. Note that the parameters of (x()).(()) are not shared across the agents.
Motivated by Vaswani et al. (2017), we further define the attention embedding vy for agent as,
N+M
exp (Br.) Σ exp (Bi,)² B W, Wa.fe(os))
2/2 Σ as)f
oij
ddg
(6)
where W,, represents one fully connected layer to encode the query representation f(,((::) and W represents another fully connected layer encode the key representation (e.(())) dy is the dimension of the query representation. 3r, represents the correlation between agent i and all the other entities.
It is then normalized by a softmax function to oi.j as the probability value, which indicates where agent i is "attending"" or focusing in the current time step and (; E R i2 is computed via weighted sum over all the state encoder representations. For Q-function, there are two modifications: (i) The state encoder [,2 (8; 0g) for the agent also takes in the action as inputs; (ii) The final layer of the Q-function network outputs scalar value instead of action distribution Qe, (s", of), which is used in Eq. and Eq 3.
4.2 DISENTANGLED ATTENTION AS INTRINSIC REGULARIZATION
We propose disentangled attention as in- trinsic regularization (DAIR) improve the state encoder representations for solv- ing the problem domination of single agent and the conflic/ between the agents. Each manipulation task can be decomposed to multiple sub-tasks, each with differrnnt interaction regions. Our key insight to en- courage different robots attend focus on different interaction regions, and conse- quently to work on different sub-tasks.
RL bjecttvv
Intrinsic Regularization
2-layer MLP
LayerNorm
1-layer FC
(0
Specifically, we look into the softmax prob- ability Cij [0, 1] from the attention mechanism in Eq. 6. This variable repre- sents how much attention agent is putting on interaction region/agent, j. To encour- age the agents to focus on different entities, we propose the following loss function for agent i as,
Attention Module
fu
f.2
f1
Az-i
Interaction Regions
Agent Other agent
Our model framework. We use attention mecha- nism to combine all embedded representations from agents and interaction regions. The output of attention module, to- gether with another embedded vector from s; summed together with b The combined feature fed into a 2-layer MLP hs to output a, The intrinsic loss is computed from the attention probability a and encourages the agents to attend to different sub-tasks.
Figure 2:
Latt (p:)
Σ 1-1,3
Cii aj >
(7)
where <.. > denotes dot product of two vectors. This loss forces the dot product be- tween two attention probability vector to be small, which encourages different agents to attend different entities (including agents and interaction regions). We call this particular attention maps regulated by the orthogonal constraint as disentangled attention.
Recall that a; is predicted via the state encoder functions, parameterized by part of o, Instead of proposing a new reward function, our disentangled attention regularization is directly applied on learning the state encoder representation itself. The training objective for the policy network and
5

---

Under review as a conference paper at ICLR 2022
Q-function can be represented as,
min (+() + Maam(()) min (o,)0) + (O)m $:
(8)
0,
where x = 0.05 is constant to balance the reinforcement learning objective and our regularization.
Implementation Details. The robot state si S i I N) contains the joint positions and velocities and the end-effector positions. Thus each robot can reason the other robot's 's joint state and avoid conflicts. Each interaction region (N + S i S N + M) contains the target interacting position, velocity, pose and its goal position, which are all in (It y, coooodninsss The action representation contains the positional control and the gripper motion information.
5 EXPERIMENTS
Environment and setting. We perform our experiments on complex bimanual manipulation tasks (N = 2) in the MuloCo simulator (Todorov et al., 2012). By further leveraging curriculum learning, we successfully complete the scenarios with up to eight objects with sparse rewards. We evaluate the sample efficiency of training, conffict rate, domination rate, and completion steps across approaches to demonstrate that DAIR can (i) help discover efficient collaboration strategies; (ii) improve efficiency and safety by avoiding domination and conflict; (iii) bring adaptation capability with learned task decomposition knowledge; (iv) retain learning capability of synergistic skills.
Baselines. We compare approach with three baselines: (i) The same architecture as model with the attention mechanism, but without the intrinsic regularization (A (Attention); (ii) SAC with Multi- layer Perceptron (MIP) neural network; (iii) Multi-Agen Deep Deterministic Policy Gradient (Lowe et al., 2017) with MLP (MADDPG + MLP). We also tried replacing DDPG with SAC in MADDPG but observed minor differences. Thus we only report results with MADDPG + MLP for simplicity.
Training details. All networks and learnable parameters are trained with Adam optimizer (Kingma & Ba, 2015) with learning rate 0.0001, 31 = 0.9, .3g 0.999. We set the discount factor as y = 0.98, buffer size as 1M, and batch size as 512 for all tasks. We follow the replay k setting in HER (Andrychowicz et al., 2017), and set k = with the future-replace strategy. We update the network parameters after every two episodes. The episode length equals 50 times the object number for each environment. We train all the methods with seeds and report both the mean and standard derivation for the success rate. More details are in Appendix B.
5.1 SUB-TASK ALLOCATION FOR COLLABORATION
We first perform our experiments on two tasks that requires alternately operating different parts in a certain order to show DAIR balancedly and safely allocates the sub-tasks for solving one task. The first task is Open Box and Place (Figure (c)): The robots need to put the blue block object inside the box with sliding cover, which requires one robot arm to open the sliding cover for the other robot arm to put the object inside. The second task is Push with Door (Figure (d)): The robots need to push the blue object to the goal on the other side of sliding green door that requires one robot arm to open it and clear the way for the pushing arm (with the grasping function disabled). In both cases, we also apply a force on the sliding cover/door for to bounce back its original position if there is no outside forces. The following results show that DAIR not only helps achieve better sample efficiency and performance, but more importantly, addresses the problems of domination and conflict, thus further reduces the steps to finish the task at the same time.
Reward setting. We consider two different reward settings: (i) a sparse reward setting where the agents only obtain a reward 1.0 when the block is on the target position; (ii) a informative reward setting which gives a reward 1.0 when the box/door is open and another reward 1.0 when the block reaches to the goal. If the block reaches its goal in trial, we count it successful trial.
Comparison on success rate. We plot the success rate of all the methods over the environment steps in Figure 3. We can observe that DAIR achieves better sample efficiency and better success rate than the baselines in most cases. We also observe that in both environments, DAIR and Attention achieve better success rate in the sparse reward setting than using informative reward. The reason is that sparse reward offers more flexibility for the agents to collaborate under the guidance with intrinsic disentangled attention, while the explicit informative reward can lead to local minimum more easily.

---

Under review as a conference paper at ICLR 2022
Open Box and Pace Information
Open Box and Place (earsal
Push with Doo (nnfomraioo))
Push with Door (Sparse)
Eestrommnt Steps DM2
Endrosimnnt Steps IMG
Endiosment IN
Attertion
MLP
MADDPG-MLP
Figure 3: Performances of different methods on two bimanual manipulation tasks, Open Box and Place (2 the left) and Push with Door (2 the right). We consider reward settings for each task, (i) sparse reward (right in each group). where agents only receive success reward when all the goals are reached; (ii) an informative reward (left in each group), where agent will additionally receive reward for reaching each individual goal in addition to the final success reward.
Box (nformaiie)
Open and (Sparse)
(nformativs)
with Door (Spamee)
Enveemment Stapo (4]
30
Erstormen Staps 1M1
32
DAIR A=0.02
DAIR
Figure 4: Ablation studies on the value of X. Our method is generally robust to the choice X, when even
large (e.g., x=0.2). In our practice, we choose x=0.05 for all the experiments.
Ablation on X. We set the hyperparameter À 0.05 (defined in Equation for balancing the regularization) in all our experiments. To study the stability of DAIR, we perform ablation on different values of À in Figure 4. We observe that our method is robust to the change of À from 0.02 to 0.2.
Comparison on the three criteria. Table shows the comparison on the three criteria defined in Sec- tion 3.2. We observe significant improvements over all the settings using intrinsic regularization, which proves that using disentangled attention can lead to better collaboration. For example, in the task of Open Box and Place with informative reward, our approach achieves almost half less conflict rate, 24% less dom- ination rate, and 12 fewer steps comparing to the Attention baseline without the intrinsic regulariza- tion. For Push with Door using sparse reward, we reduce more than half the conflict rate.
Table I: Conflict rate (%), domination rate (%) and average finishing steps of our method and the baseline with pure attention different tasks. Lower value is better. Box: Open Box and Place. Door: Push with Door.
Domination Rate
Attention
DAIR
Box (Informative)
77.2+2.9
53. +
Box (Sparse) Door (Informative)
74.5+2.8
62.618.4
83.7144.8
76.515.
Door (Sparse)
68.816.7
66.9
Conflict Rate
Attention
DAIR
We perform ablation by introducing extra collision penalty during training: Two robots will receive -1.0 1.. reward if their grippers collide to each other. Note that such a reward is not realistic in practice since we do not hope the robots collide to get the reward. We show the Conflict rate for both Attention baseline and our approach training with this collision penalty in Table 2. DAIR shows consistent improvements and remains be an effective way to reduce conflicts even with the collision penalty.
Box (Informative)
7.4+0.8
4.0=2.1
Box (Sparse)
6.7+s.0
3. 3.6+2.3
Door (Informative)
35.3#19.0
23. 23.3-11.0
Door (Sparse)
44.1=16.1
1817
Finish Steps
Attention
DAIR
Box (Informative)
33.6+5.5
21.3 21.3+a.2
Box (Sparse) Door (Informative)
39.2.1..
40.0+114
23.0+4..4
22.8
Door (Sparse)
30.3
23.316.
Table 2: Conflict rate (%) of our method and the attention baseline on different tasks with Collision Penalty. Lower value is better. Box: Open Box and Place. Door: Push with Door.
Visualization on attention probability a. We vi- sualize the two tasks in Figure 5. In each task, we visualize the attention 01 and 02 for each robot in two rows. Each attention vector a contains four items that correspond to left arm (Ist column), right arm (2nd column), and the two task-specific inter- action regions (last 2 columns). Figure 5 (a) shows the Push with Door task: The left arm is interact- ing with the object block, so has a high value in the corresponding probability 01,3 (Ist row and 3rd
Conflict Rate
Attention
DAIR
Box (Informative)
10.4
3.9
Box (Sparse)
3.542.1
2.3
Door (Informative)
5.341.19
3.4+1.
Door (Sparse)
12.243.0
4.7+1.1
7

---

Under review as a conference paper at ICLR 2022
Table 3: Success rate (%) on Stack Tower of different methods for each curriculum learning and adaptation stage. a + means adapting the policy trained on a objects to objects. towers means the agents need to stack two separate towers.
#object
2
3
2-+3
3-+4
2-+4 (2 towers)
DAIR Attention
100
98.9
68. s.5
12.5
23.3
17.5:4.3
98.7+0.9
9..
42.0
41.3+9.8
3.7
0.0+0.0
Table 4: Success rate (%) on Rearrange of different methods for cach curriculum learning and adaptation stage. means adapting the policy trained on objects b objects.
#object
2
3
2-+3
3-+4
2-+4
3-+8
DAIR
7
98.9
89.0+1.4
74.3+5.8
64.314.2
53.0+b.4
33.3-12.6
Attention
91.0#6.2
90.7
66.7+3.3
16.5
3.3+4.7
3.3
0.0=o.b
column); the right arm is interacting with the door, it also has a high value in the correspond- ing probability 02,4 (2nd row and 4th column). Similarly in Figure (b) for the Open Box and Place task, a high probability with ag indicates the ith arm is interacting with object j. The two interaction regions here are the block object (3rd column) and the box cover (4th column).
5.2 GENERALIZING TO MORE OBJECTS WITH CURRICULUM LEARNING
We increase the number of objects and train with curricu- lum to show that the regularized representation gains better
learning capacity when task gradually becomes harder. We test on Stack Tower (Figure (b)), where the robots need to stack objects as a tower with indicated goal positions; and Rearrangement (Figure 1 (a)), where the robots need to rearrange the objects to their own goal locations on the table. When manipulating one object in these envi- ronments, it is easy for the arm to perturb other objects without intention. We train RL agents for both tasks in the informative reward setting: the agents will receive a reward 1.0 when each object reaches its goal. We leverage curriculum learning to start training the agents to manip- ulate one object and then gradually increase the objects to three to the end.
Figure 5: Visualization of attention a. Each row corresponds to robot at- tending to four items. (a) Push with Door: one robot holds the door while the other pushes the block; (b) Open Box and Place: one robot opens the box while the other picks the block.
We evaluate our approach on two aspects: (i) How does the approach perform in each curriculum stage; (ii) How does the approach generalize to object numbers that exceed its training number (up to eight objects in the Rearrangement task). DAIR achieves better results in both aspects, especially in generalization to multiple objects.
Results on each curriculum stage. We first compare DAIR to Attention on 3-block Rearrangement and Stack Tower tasks. Note that both MLP and MADDPG+MLP baselines cannot handle flexible number of objects due to fixed dimensions of inputs. Thus it is not applicable in these tasks with curriculum learning. and directly training both of them with 3 objects leads to zero success rate. This also suggests that DAIR and Attention have the flexibility to handle variant number of input objects. DAIR gains significant improvements over Attention in all different training stages. Our gain over Attention even becomes larger the number of objects increases.
Results on generalization. We conduct generalization experiments on both tasks where we test the policies trained with objects on the same environment with i + & objects. We show the results of generalization success rate in Table 3 and 4 with the columns labeled by i + i+ For Stack Tower, DAIR trained with b-block stacking generalizes to stacking towers each with 2 blocks (last column in Table 3), while Attention completely fails. For Rearrangement, DAIR can rearrange 8 objects even we only train it to rearrange objects (last column in Table 4), while Attention fails to generalize to even 4 objects. We conjecture that the reason for improvement is similar to conclusions in continual learning area (Delange et al., 2021): regularizing parameters with simple 7,2 loss avoids overfit to specific stage of task and gains better generalization ability in continually shifted tasks.
8

---

Under review as a conference paper at ICLR 2022
[a] Stack Tower abjects
(b) Stack Towers
[c) Rearrangement objects
Figure 6: Visualization of bimanual manipulation. For each object, we represents its goal as a transparent dot in the same color. (a) Both arms are picking up objects and alternatively stacking them into tower; (b) To stack two lower, each arm is working without conffict pick up the objects to their target locations.
tower that is close to it; (c) We show the
can collaborate
Visualization on stacking and rearrangement. We visualize the three demonstrations for our approach in Figure 6: (a) stacking 3 blocks; (b) stacking towers each with 2 blocks using the policy trained with block-stacking; (c) rearranging 8 blocks to their target positions using the rearrangement policy trained with blocks. For stacking tasks, both robots are able to pick up different objects without interrupting the other robot and the stacked tower. For the rearrangement task, the policy is transferred rearrange 8 objects, far beyond the training object number 3. The two robots are still able to collaborate without confficts to solve the tasks. Please refer to our project page for more policy visualization in videos.
5.3 SYNERGISTIC BEUAVIORS DISCOVERY
Besides manipulation tasks with multiple objects, DAIR retains the ability to help two robots collaboratively ma- nipulate one object with multiple interaction regions. To analyze the ability on such synergistic skills learning, we conduct experiment
Table 5: Adjust Bar task. DAIR improves
baseline over domination rate and finish steps. Conffict rate not computed here since two end effectors will be close when the task solved.
the Adjust Bar task, visualized in Figure (e). This task requires the two robots to lift and rotate a heavy bar target height and orientation. We lock the gripper to the closed state and set the bar's mass and size large so the two robots have to collaborate finish the task. The interaction region state inputs are represented by the 3D position of two sides of the bar. The goal is defined by the target positions of the two sides of the bar. The reward sparse that only an accurate adjustment gives 1.0 to two robots. DAIR maintains advantage on perfor- mance over the Attention baseline as shown Table 5 and Figure 7. Two robots
Attention
DAIR
Domination Rate Finish Steps
56.1 16.3
52.6+-0.3
23.526.0
18. 18.218..
Adjust Bar
0.8
Success nate
0.6
0.4
successfully clamp up the bar
0.2
with the opposite forces with very consistent movements. Such results suggest that DAIR is not harmful for learning synergistic skills when it encourages the robots to look at different interaction regions. DAIR also serves as a mech- anism to avoid overfitting to domination and help robots finish the task efficiently with smaller finish steps.
0.0
30 40
Environment Steps (M)
Figure 7: Training Curves of success rate.
DAIR still retains learning efficiency though augmented with disentangled attention.
CONCLUSION
While previous works consider how to learn collaborative skills like synergistic behavior, we notice two main limitations in complex bimanual manipulation tasks: domination and conflict, corresponding to the efficiency and safety of control. We propose a simple and effective DAIR to solve these problems. We validate our approach on challenging bimanual manipulation tasks with multiple objects (up to objects) or one objects with multiple interaction regions. We demonstrate that DAIR not only reduces the domination and conflict problems but also improves the generalization ability of the policies to manipulate much more objects than in the training environments. We hope our work contributes step towards safe robotics.
9

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide the following illustrations in our paper and appendix:
We provide the detailed description of the environment in Appendix A. Evaluation Criteria: We provide the detailed evaluation criteria for domination, conflict and finsh steps in Section 3.2.
Environment:
Implementation Details: We provide all implementation details and related hyperparame- ters in the end of Section 4.2, the beginning of 5 and Appendix B.
We are committed to releasing the code for our approach, the baselines, and the simulation envi- ronment. We believe the open source of our code. the task sets, and evaluation code will be an important contribution to the robotics community. We have released our videos in project page: https: / /bb manua -attent. on. g ttub. io/, and we will release the code and environ- ment on the same website upon publication.
REFERENCES
Fabio Amadio, Adrià Colomé, and Carme Torras. Exploiting symmetries in reinforcement learning of bimanual robotic tasks. IEEE Robotics and Automation Letters, 4(2):1838-18A) 2019.
Marcin Andrychowiez. Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pp. 5048-5058, 2017. URL http:/ /tap:://ppaprss../ hwtpppppwssssss....pmmmpp 7090- hindsight exper ence replay. 6
Andrew Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learning in natural and artificial systems, pp. 17-47. Springer, 2013. 2
Marc G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. arXiv preprint arXiv:1606.01 2016. 2
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint
2018. 2
Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity- driven learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttp:://apeereet ew net/ forum? d J Ax
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. Automatically composing representation transformations as means for generalization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: http::///peeeeeeeeeeeee wwt. fort//orumm? d= #fOnRcKX.
Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Efficient bimanual manip- ulation using learned task schemas. In 2020 IEEE International Conference on Roboties and Automation (ICRA), pp. 1149-1155 1155. IEEE, 2020a. 3
Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Intrinsic motivation for encouraging synergistic behavior. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. URL https: / //openrvview net/ forum? i SJ1 NNNNNN 3.3
M. Delange. R. Aljundi. M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars.
A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10. 109/TPMM1.202 30057446.
10

---

Under review as a conference paper at ICLR 2022
Coline Devin, Daniel Geng, Pieter Abbeel, Trevor Darrell, and Sergey Levine Compo- sitional plan vectors. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' d'Alché-Buc, Emily B. Fox, and Roman Gareett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14963-14974, 2019, URL https: htLtht::://poooe /ptp:////ccccceeeeeeeeeee wwrroceedingsnneriie.... newrips /ttp 00989c20ff1 6dc 6d8 (ee.///eeeeeeeeeaaa-eeeeaaaa - html.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL https: hittps://opett /tttps://operrevier net/ Eorum? net/forum? 1e=idddiibii cbCTTTTTTT 3
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://
net forum? di- jR
Jakob Foerster, loannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In NIPS, pp. 2137-2145, 2016. 3
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Sheila A. Mcllraith and Kilian Q Weinberger (eds.). Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2974-2982. AAAI Press, 2018. URL https://///... /wwww//www.../..o/////.. www.a.aairgggo....... orgg hess/inex..o php/ AAAI AAAI18 ppppp/vieww///eppppppppppaapeeap 17193. 3
Elena Gribovskaya and Aude Billard. Combining dynamical systems control and programming by demonstration for teaching discrete bimanual coordination tasks to humanoid robot. In 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 33-40. IEEE, 2008.
Huy Ha, Jingxi Xu, and Shuran Song. Learning a decentralized multi-arm motion planner. arXiv preprint arXiv:2011.0200 2020. 3
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsssun Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018. URL http://///// http//oeooee.ns....ooo...o....se w. w.r..press/vvl wwa./////haaaaaee wwwraaarnaa....... rno Bb. html.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. In International Conference on Machine Learning, pp. 1804-1813, 2016.
Ping Hsu. Coordinated control of multiple manipulator systems. IEEE Transactions on Robotics and Automation, 944:400-410, 1993. 3
Sandy H Huang, Martina Zambelli, Jackie Kay, Murilo F Martins, Yuval Tassa, Patrick M Pilarski. and Raia Hadsell. Learning gentle object manipulation with curiosity-driver deep reinforcement learning. arXiv preprint
2019. 2
Edward Hughes, Joel Z Leibo, Matthew G Phillips, Karl Tuyls, Edgar A Duéñezz-uzzmic Anto- nio Gareía Castañeda, Jain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Incquity aversion improves cooperation in intertemporal social dilemmas. arXiv preprint arXiv: /803 0088444 2018. 3
Shariq Iqbal and Fei Sha. Actor-attention-criiic for multi-agent reinforcement learning. In ICML, pp. 2961-2970, 2019a.
11

---

Under review as a conference paper at ICLR 2022
Shariq lqbal and Fei Sha. Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning. arXiv preprint arXiv:1905.12127, 2019, 3
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040-3049. PMLR, 2019. 3
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In NeurlPS, 2018.
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' ''lche-Buc, Emily B. Fox, and Roman Garett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9414-9426, 2019. URL https: / http:///ooccedeeeeeeee neurips cccpappe/2211 91000/9//00 9/hash/ Abstract .tml..
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv.orr. abs 1412. 6980.
Oliver Kroemer, Christian Daniel, Gerhard Neumann, Herke Van Hoof, and Jan Peters. Towards learning hierarchical skills for multi palti-phase manipulation tasks. In 2015 IEEE International Conference Robotics and Automation (ICRA), pp. 1503-1510. IEEE, 2015. 3
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object ma- nipulation using relational reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pp. 4051--055 IEEE, 2020, doi: 109111//4994455...0 2...0999....0999944.... URL https: http//oo..oo/ooo////oo////oo 1109/ ICRA40945. 2020. 9197468. 1,
Yunfei Li, Huazhe Xu, Yilin Wu, Xiaolong Wang, and Yi Wu. Solving compositional reinforcement learning problems via task reduction. International Conference on Learning Representations, 2021. URL
net wttp://wwwwwee...neteeeeeeemeeae forum? wtw.rrummmdd-ee9e0eeeee 9556999om6MW
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL 02971.
http:// /ttt:://arxiv.... org/ brgg/bb.
Michael Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML, volume 157, pp. 157-163, 1994. 3
Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary popula- tion curriculum for scaling multi-agent reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nee 2020. URL https::////
http//peenvveeeeee https://openzeiieess
net wttp://ommm..-SSdi..-o-----
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competttvee environments. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30, pp. 6379-6390, Curran Asso- ciates, Inc., 2017. URL https: 3wt.33711111660010000000011
neurips cc/ cccppaper/2017/EE pdf. 3.6
Aleksandra Malysheva, Tegg Taekyong Sung, Chae-Bong Sohn, Daniel Kudenko, and Aleksei Shpilman. Deep multi-agent reinforcement learning with relevance graphs. arXiv preprint arXiv: 101.:1811.125577 2018.
12

---

Under review as a conference paper at ICLR 2022
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Pierre Yves Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):26-2266 2007. 2
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 20/7, volume 70 of Proceedings of Machine Learning Research, pp. 3778-2787. PMLR, 2017. URL http: BtLLhttp://rroceeddiigg http:///ooceeeennneeeeeee
7 html.
Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionlll--cooddnatee nets for learning play stareraft combat games. arXiv preprint arXiv: 1703. 10069, 2, 2017.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. MCP: learning composable hierarchical control with multiplicative compositional policies. In Hanna M. Wal- lach, Hugo Larochelle, Alina Beygelzimer, Florence A'Alhhé Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3681-3692 2019. URL Utthttps: /ptoccee//////eeee//// ngs. nwwrrpsippp... ccc sac/papr////// 2019 2009//aasis ha/.ah////a// 951 92c 772337777777 1.5ff8e.2 96c 66C f2dad2 -ww.eeaaaaaaaaaaaacccaaaaa html.
Mare H Raibert and John J Craig. Hybrid position/force control of manipulators. 1981.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4292- 4301. PMLR, 2018 URL http: / pttpp/eerceeec.eeee/e////oc...se mlr, mml..presss /v80 1080/rrashie rashidlshlllio..e html. 3
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- unsupervised discovery of skills. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: //openrevis ew net/
o dorum2id==U01124444 2
Christian Smith, Yiannis Karayiannidis, Lazaros Nalpantidis, Xavi Gratal, Peng Qi, Dimos V Dimarogonas, and Danica Kragic. Dual arm manipulation survey. Robotics and Autonomous systems, 60(10):1300-133 2012. 3
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero. and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi- aaeet- reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5887-5896. PMLR, 2019. URL http: / //proceedings... mlr press /v7 /ttp///1////l//// 9a. html.
DJ Strouse, Max Seriman-Weined Josh Tenenbaum, Matt Botvinick, and David Schwab. Learning to share and hide intentions using information regularization. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018, URL ht http::/:///opppepteeeeeeo neurips. ca/paper/200 /201 8 1e/1eeee/ httpe//eeeeeeeeeeeeaaaaaaa22ee..
13

---

Under review as a conference paper at ICLR 2022
Peter Sunehag, Guy Lever, Audrumas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg. Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Flisabeth André, Sven Koenig, Mehdi Dastani, and Gita Sukthankar (eds.), Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, pp. 2085-2087. International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018 URL http: /dl. acm. org/citat ses/iiaaiion cfm 32dddd3220000-- 3
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martin, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Learning multi-arm manipulation through collaborative teleoperation. arXiv preprint 20X2::2012.06638, 2020. 3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL http: /ttt:://papersse nips cttccc///ppp.e / 771111111111111 attention is all y-y----uu---- pou-need. 2, 3, 5,
Tonghan Wang, Heng Dong, Victor R. Lesser, and Chongjie Zhang. ROMA: multi-agent re- inforcement learning with emergent roles. In Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. //proceedings.m.. press / vv19/ /wtpg/nnfffff html.
PMLR, 2020a. URL http:
Tonghan Wang. Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30 30, 2020. OpenReview.net, 2020b. URL hattts:://ppenreviee. net forum?id= BJgy96EYvz.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018. 3
Ning Xi, Tzyh-Jong Tarn, and Antal K Bejezy. Intelligent planning and control for multirobot coordination: An event-based approach. IEEE transactions on robotics and automation, 12(3):
439-452, 1996. 3
Fan Xie, Alexander Chowdhury, M Kaluza, Linfeng Zhao, Lawson LS Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. arXiv preprint arXiv:2010.051344 2020. 3
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv: 01830, 2018. 3
R Zollner, Tamim Asfour, and Rüdiger Dillmann. Programming by demonstration: Dual-arm manipulation tasks for humanoid robots. In 2004 IEFE/RSJ International Conference on Intelligent Robots and Systems (IRES)IEEE Cat. No. 04CH37566), volume 1, pp. 479-484. IEEE. 3
14

---

Under review as a conference paper at ICLR 2022
A
TASK DESCRIPTIONS AND DETAILS
(a) with Door
(b) Open
(c) Figure 8: The environments used in our experiments
(d) Rearrangement
Adjust
A.1 ENVIRONMENT DESCRIPTIONS
Push with Door, Figure 8(a). The two robots are placed on both sides of 100cm x 70em table, opposite each other (all robot manipulation environments
same for this setting). The goal is to push a block through sliding door and make it reach the target position on the other side of the door. We put a spring on the sliding door, such that it will close automatically in the absence of external force. The initial positions of the projections of the two grippers onto the table plane are sampled in 40cm 40cm square on the table (all robot manipulation environments The initial position of block and the goal position are sampled from circle with radius 20cm around the table center. We fix the initial height of the two grippers (all robot manipulation environments are same for this setting), and set the initial position of door at the center of the table.
same for this setting).
Open box and Place, Figure 8(b). The task is pick up the block on the table and place into the box in table center. We put spring on the sliding lid of the box, so it will close automatically in the absence of external force. The initial positions of the block and goal are sampled from circle with radius 20cm at the center of the table, outside the box.
Stack Tower, Figure 8(c). The task to stack several blocks into a tower. All blocks are randomly sampled from circle with radius 20cm around the center of the table. We perform curriculum learning in this environment, with one more block sampled in each stage. In the first stage in the curriculum, have block, sample the corresponding goal with the height randomly from Ocm to 30cm. In each following stage, we sample one more object block and goal. After the first stage, the goals will form into tower.
Rearrangement, Figure 8(d). The task is to push multiple blocks to their corresponding target positions on the table. We perform curriculum learning in this environment, with one more block sampled in each stage. All blocks and goals are randomly sampled from a circle with radius 20cm around the table center. We train our method up blocks curriculum stages) and generalize the approach to up to blocks.
Adjust Bar, Figure 8(c). The task is to adjust a heavy bar to the state that the two sides of it match with the two blue goal positions. That requires the height and the orientation are matched. We randomly sample the two goal positions but make sure the distance between them equals to the length of bar. We fix the gripper to force two robots synergistically the force in opposite direction to pinch up the bar.
A.2 OBSERVATION SPACE
The observation vector consists of object states, robot states, and the goals for the objects. Specifically, the object states consist of the position and velocity of all the objects. The robot stages consist of the position and velocity of the gripper and the robot joints. The goal vector consists of the target position coordinates.
15

---

Under review as a conference paper at ICLR 2022
A.3 ACTION SPACE
For robot tasks, the action is 8-dimensional vector, which is the concatenation of two 4-dimensional action vectors for each robot. For each robot, the first 3 elements indicates the desired position shift of the end-effector and the last element controls the gripper fingers (locked in push with door scenario). For mass point tasks, the action is a 4-dimensional vector, similarly combined with two 2-dimensional vectors for each mass point. The 2-dimensional action vector only controls the the desired position shift of the end-effector in a plane.
B TRAINING SAMPLE NUMBER
For Push with Door and Open Box and Place, we use 10M samples train; For Tower Stack and Rearrangement, we leverage curriculum learning and increase the number of blocks by one in each stage. Specifically, we list the number samples for each stage of training in Table 6.
Table Training samples for curriculum learning in each stage
Num Block
Tower Stack
x 10
x 10
10
Rearrangement
1 x 10"
3 x 10"
5 x 10
Computation. In our experiments, single GPU and 8 CPU cores for all the method on each task.
C VIDEO RESULTS
Please refer to our project page: https: h/win/////mmaaaaeeaaaaa -ttaamaannnalatttttioo... github. io/.
16


Reviews:
[{"review_id": "vJZiWPWZyE6", "reviewer": "Reviewer_v16H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written in a clear way, it is well motivated and it clearly defines the domain challenges that are going to be addressed. The proposed method is formulated in a fairly clear way (see comments below). The experimental results are carried out on a variety of tasks involving the two robot’s collaboration.\n Comments:\n- Please clarify earlier (e.g. in “Method”) what the state of the robot arms is, as this is not clear until much later in the paper. This information could help understanding the method better from the beginning\n- How is an “interaction region” (mentioned in “Network architecture”) defined? Is it defined based on the robots’ states in simulation? How would it be defined in a real world scenario?\n- Regarding state encoders and agents: what do f_{i,j}(.) encode exactly? As far as I understand it, f_{i,j} takes state s_j as input and outputs an encoded vector, “according to” agent i? What is the meaning/role of “agent” in this context? \n- The previous point on state encoders and agents is not made clear in Fig. 2 either: here, all encoders have the subscript “i”, does it mean that there is only an agent “i”? Also, what is s_i with respect to s_1, s_2, etc.? Also, is “interaction regions” equivalent to an agent (ref. Fig. 2)?\n- In paragraph “Implementation details”, what is “target interacting position, velocity, …” referring to? Do they indicate the target position of the two robots + objects?\n- In your experimental results sections, can you specify how many repetitions you performed to obtain the scores reported in the tables? Several times the std (?) of the scores reported is quite high with respect to the means, why is that the case? How do you account for that?\n- Success rates for adaptation stages in Table 3, even though better than the Attention scores, show a significant drop in performance with respect to the learning cases. Can you discuss this drop? How could this be improved?\n- Fig. 7 is missing a legend for the two curves depicted.\n", "summary_of_the_paper": "This paper addresses important aspects of bimanual manipulation, in particular domination and conflict, by proposing a technique called disentangled attentions, which provides an intrinsic regularization for two robots to focus on separate subtasks/objects. The proposed method is evaluated in simulation, using two robot arms to solve a variety of tasks on table-top environments.\n", "summary_of_the_review": "The paper presents a well formed idea and method, but has some points to be clarified that would improve the manuscript.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KoxlWyzaHap", "reviewer": "Reviewer_RR63", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths and Weaknesses\n### Things I liked about this work:\n- **an interesting idea**: I really like the idea of intrinsic regularisation by using attention. It seems like a useful technique that may lead to exciting future work too.\n- **a well structured paper**: The overall narrative and flow of the paper is very well put together so it is easy to read and understand.\n- **a good set of experiments**: I also like the type and number of experiments ran and their overall presentation.\n\n### Things that can be improved:\n- **misleading contribution**: there are a few issues with the current state of the claimed contribution. Claiming as contribution the observation that dominance and collision can occur in dual-arm control is misleading. These are not novel observations and are therefore not a contribution to this work. In addition, the claim that the solution ensures preservation of safety in collaborative shared spaces is also a bit misleading because it is implicitly learnt over trial and error with no follow up guarantees. See below for more details.\n- **insufficient number of seeds - 3 is not enough**: although the paper does a great job at evaluating different aspects of this work, it uses 3 seeds per experiment which is significantly insufficient for model-free RL, especially when there are complex dynamics introduced from the manipulation set up and the bimanual nature of the tasks. I find this the biggest weakness of this paper.\n- **overall clarity can be further improved**: although the paper does a great job at explaining the methodology and presenting the achieved results, there are a number of unclear statements that can be further improved.\n\n## Claim and Contribution\nOne concern I have for this work is the practicality of the solution. A large part of the claim is on preserving safety in collaboration in shared spaces. This is truly an important problem in physical systems although not necessarily a big deal for simulation. Although there is nothing wrong with proposing a solution to this with simulated evaluation, it still seems important to somehow ground it to the context where it practically matters. However, as it stands, it is hard to see how this solution can be at all scaled in practice. Specifically, the approach only implicitly stimulates reduced collision through regularisation. This, however, does not prevent from collision in the process of learning and what is worse is that it does not provide any guarantees that there will be no collision once a policy has been extracted. This seems to deem the solution impractical for part of its claimed contribution. However, if there is another benefit to minimising collision in the context of the simulated task and collaborating agents then this should be explained in more details. Otherwise, the claim seems a bit conflicting.\n\nIn addition, the way the work by Chitins et al. is introduced makes me think this work is directly related to this problem. 'While this intrinsic reward encourages the two robots to collaborate for tasks that are hard to achieve by a single robot, it does not address the domination and conflict problems for efficient and safe manipulation.' Encouraging robots to collaborate seems directly related to addressing the domination problem (e.g. to not collaborate) so this seems like a direct candidate to compare against. If the methods are not comparable for some reason, then its reasons should be explained clearly.\n\nThe concept of robot collaboration describes the act of interacting agents and is thus inherently solving the problem of using a dominant agent, e.g. through coordination [1,2,6,7] for example. There is a lot of work in robotics done on safe collaboration in bimanual settings too [3,4,5] too, so it is not clear why this is a contribution for this work. Moreover, those works were not part of the literature review but perhaps some of them might help put things in perspective. \n\nFurthermore, the summarised contribution on page 2, states that the proposed approach can solve the tasks more efficiently but this is not necessarily true when looking at the evaluation section. Requiring 10M as opposed to 11M environmental steps, for a 100 step long horizons, is hardly more efficient. Perhaps the solution results in faster learning but it does not seem more efficient. \n\n## Experimental Evaluation\nI think this work does a great job at presenting the overall experimentation. I really like the breadth of evaluating the work but I have a few additional concerns. The main one is the small number of seeds used. To be able to confirm the utility of this method, I think it's important to run experiments with at least 10 seeds. Being a model-free approach, it is well known problem that the learning of a Q function can vary drastically even if the tasks were not manipulation heavy and not bimanual. The problem of interest in this work asks for more thorough evaluation with respect to the number of seeds. I would happily reconsider my recommendation if this gets fixed.\n\nIn addition, it seems like the proposed approach works better with a fully sparse reward as opposed to an informative one, is there a reason for this, it seems somewhat counter intuitive. \n\nThe overall study of the required finish steps is great but it seems like it is performed on a not-yet converged policies. In this case it is not clear if the reported improvement against the number steps required is really thanks to obtaining a more effective policy, synthesised with the help of the provided regularisation, or is merely a consequence of the not-yet converged policies. This brings me back to the comment made in claims and contribution, perhaps a more accurate statement would be to measure the speed of learning, e.g. as a function of the per-step normalised reward, as opposed to the speed of solving the task. It seems only meaningful to measure the speed of solving a task between two learning processes that have already converged to near-optimal policies. This is not the case according to Figures 3 and 4.\n\n## Minor comments\n'One possible solution is to design a task-allocation reward function to encourage better coordination.' seems like a statement that needs a reference.\n\n## References\n\n[1] Lee, Y. et al  Learning to coordinate manipulation skills via skill behavior diversification. ICLR 2019\n\n[2] Hu, B. et al. Coordinated compliance control of dual-arm robot astronaut for payload operation. IJARS 2021\n\n[3] Sadeghian, H. et al 2012. Global impedance control of dual-arm manipulation for safe interaction. _IFAC Proceedings Volumes_, _45_(22), pp.767-772.\n\n[4] Vick, A. et al. Safe physical human-robot interaction with industrial dual-arm robots. In _9th International Workshop on Robot Motion and Control_ (pp. 264-269). IEEE.\n\n[5] Gams, A. et al. Coupling movement primitives: Interaction with the environment and bimanual tasks. 2014 _IEEE Transactions on Robotics_, _30_(4), pp.816-830.\n\n[6] Deng, M. et al, 2017, August. Reinforcement learning of dual-arm cooperation for a mobile manipulator with sequences of dynamical movement primitives. In _2017 2nd International Conference on Advanced Robotics and Mechatronics (ICARM)_ (pp. 196-201). IEEE.\n\n[7] Freek Stulp et al. Reinforcement learning with sequences of motion primitives for robust manipulation. IEEE Transactions on robotics, 28(6):1360–1370, 2012.\n", "summary_of_the_paper": "This paper proposes an attention-based solution to dual-arm robot manipulation from sparse rewards that relies on a novel idea for intrinsic regularisation. The proposed regularisation term encourages each robotic arm to focus on separate subtasks and objects. The proposed approach aims to reduce the problem of extracting a dominating agent in collaborative settings and to reduce the number of collisions between operating robots in a shared workspace. This work is evaluated in simulation and the obtained results demonstrate the ability of the proposed solution, DAIR, to not only improve both the success rate and sample efficiency of the learning process but also to reduce the number of conflicts between the two operating arms. The approach is interesting and the result seem promising but I have some concerns and additional questions that I detail below.", "summary_of_the_review": "Overall the paper introduces an interesting and simple solution to a complex bimanual set up. However, there are some details around the claims and contributions that appear not clear from the current state of the paper. The provided evaluation resorts to a very small number of seed runs (3) which makes it hard to asses the significance of the reported results. Therefore, I cannot yet recommend this work for acceptance. I would happily reconsider my score if the authors can clarify the detailed above issues and provide a more thorough evaluation in support of their solution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jbwv5bvzdrc", "reviewer": "Reviewer_jKrv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a simple method for regularising the training of a multi-agent system. An attention mechanism serves as a method for disentangling the representations for each agent helping them to attend their respective task, without intervening with the actions of the other agent, thanks to the regularisation term. I find the idea simple and intuitive. I have strong concerns regarding the plenty assumptions being made, in particular for the pre-computed interaction areas, and the scalability of the method to more than two agents. The authors falsely abuse the term bimanual manipulation. The presented experimental setting considers two agents that collaborate, and they happen to be manipulators. The problem at hand falls into then category of collabarotive multi-agent systems, and not bimanual manipulation. A bimanual manipulator is a single agent with two arms, that observes the same state and has to coordinate both arms for achieving a goal. While it may sound the same, the experimental setting and the whole motivation of this paper is wrong. Especially, because I do not see any reason why the system could not scale to three or four robots. Such overstatements are very problematic, especially because the authors refer to a machine learning conference, and they are transferring false interpretations about problems that have been long studied in robotics. Apart from that the authors abuse the term safety. There is no connection of this work with the broad literature of safe learning, safe RL or safe planning. The fact that you are measuring number of collisions and you are getting some better results do not provide any formal guarantees about safety. At most, you can state that the method is more efficient than a baseline when the number of collisions is lower.\n\nIn Section 3.1 the authors falsely claim that in SAC you have to learn the temperature parameter; the temperature parameter is not learned but is optimised throughout the learning, till a target entropy is reached. In section 4.1, the authors state “ For Q-function, there are two modifications: (i) The state encoder $f^Q(s_i , a_i)$ for the agent i also takes in the action as inputs; (ii) The final layer of i,ithe Q-function network outputs a scalar value instead of an action distribution as $Q^{θ_i}(s^t , a^t_i )$, which is used in Eq. 1 and Eq 3”. This statement is completely wrong. SAC takes as input states and actions and already outputs a scalar. It seems to me that the authors are confusing SAC with DQN, and they have not sufficient experience with SAC.\n\nFor the experimental evaluation, the authors do not compare to the most recent Multi-agent RL methods, like MAAC (http://proceedings.mlr.press/v97/iqbal19a/iqbal19a.pdf). The authors seem to also be using HER for a goal-based RL setting. Are you doing the same implementation for MADDPG, i.e., implementing as MAHER? Moreover, the authors state that they tried to implement MASAC but the results are the same. That is hard to believe, so I request that the authors append the results in the supplementary. The results are not convincing, as they are not statistically significant. This is also evident by the high variance in the learning curves, and the overlaps between the simpler Attention baseline and DAIR. The results in Table 1 do not convince me in particular for the Door (sparse) as from Figure 3 the two compared methods have the same performance. How do you justify this?\nIn Section 5.3, the authors overstate the performance of DAIR when clearly Figure 7, does not show any statistically significant results. \n\nOverall, the evaluation is lacking both for the compared algorithms and for the reliability of the results. I would also recommend that the authors present different multiagent experimental settings, e.g., like the ones in the MADDPG paper, with more than one papers to showcase the representation power of the proposed attention mechanism, and necessarily provide comparisons against the aforementioned method MAAC.\n ", "summary_of_the_paper": "The paper presents an attention mechanism that also serves as intrinsic regularisation for a two-agent reinforcement learning framework of collaborative tasks. The method is evaluated in simulation using two manipulator robots, and has been compared against a simpler version of the proposed method (attention without regularisation) and a standard multi-agent RL method. ", "summary_of_the_review": "The paper provides a very simple, yet seemingly effective approach. They do not provide statistically significant results, as they only evaluated with 3 seeds, and this is evident in the results. Moreover, they do not compare with all possible baselines on the topic. Importantly, the paper reproduces false claims, abusing the term bimanual manipulation and the term safety, showing that the authors clearly are not in touch with the topic of robotic manipulation. If the authors change their narrative and fix their motivation, provide sufficiently more seeds to make their results reliable, and also compare with appropriate baseline, I might consider increasing my score,", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A965PRYuhnX", "reviewer": "Reviewer_a42g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The proposed idea is very intuitive and well-motivated.\n- The paper is well written.\n- The comparisons and experimental settings are interesting and helpful to understand the proposed *disentangled attention* and *intrinsic regularization*.\n- Figure 5 is great to show the disentangled attention learned with the proposed attention model and intrinsic regularization. Can the authors provide the video version of Figure 5 over an episode to make sure it is not cherry-picked? Also, it would be helpful to compare the learned attention with the Attention baseline.\n- The generalization capability and collaborative behaviors shown in the experiments are impressive and significantly better than the baseline approaches, which has the potential to extend to more complicated multi-agent RL problems.\n\n\n### Weaknesses\n\n- The proposed regularization prevents two robots to work nearby each other. Although this can generally prevent collision between two robots, it also limits close collaboration between two robots. It is not clear why the experiment in Table 5 and Figure 7 suggests that DAIR does not prevent synergistic skills. In this experiment, the state of the target object is split into two separate parts so that the intrinsic regularization can work on the same object. To be fair, the results when the state space is not tailored to the proposed method are required. Moreover, the results in Table 5 and Figure 7 show that the Attention baseline is comparable to the proposed method (the error intervals are overlapping) even if the experiment is designed prone to the proposed method.\n- The difference between adding collision penalty (\"Attention + collision penalty\") and adding intrinsic regularization (\"DAIR\") seem not drastically different in terms of the final conflict rate. Given that both approaches require task-specific domain knowledge and the performance difference becomes not significant, it is debatable which one is better. A more interesting question would be whether this method can help reduce collision even during training, not just for the final conflict rate.\n- The Attention baseline shows significantly lower performances in Table 3 and Table 4. Is this mainly due to the conflict between two arms? or more due to the regularization effect for scaling? If the former is the case, comparing it with the \"Attention + collision penalty\" baseline can be helpful to support the claim of the paper in the implicit regularization.\n- The paper points out that the generalization capability of the proposed method may come from regularizing parameters not to overfit. The explanation about the generalization capability of the proposed method can be more thoroughly investigated. Following the suggested reason, will including some regularization techniques (such as spectral norm, L2 norm, mix-up) improve the generalization capability of the Attention baseline? or is the proposed implicit regularization much stronger than these prior regularizations?\n- Figure 4 explains that the proposed method is robust to the choice of $\\lambda$, but models with different $\\lambda$ show noticeable differences in performance except Push with Door (Informative).\n- All experiments are done in the state space, which requires the state space to be hand-engineered for the proposed method. It would be interesting to see that it works with more complicated high-dimensional inputs, such as the visual domain, where the split of the state space is not available.\n- Throughout the paper, one of the main claims is that the proposed method can resolve the \"domination\" issue. However, it is not intuitive how the proposed method resolves the problem of domination.\n- It is unclear how HER is working with the proposed model, especially with rewards. Is HER only used for Stack Tower and Rearrangement tasks? Also, is there any synergetic effect by using HER together with the proposed method, or can the proposed method be used alone without HER in other applications? Clarification on HER is required.\n\n\n### Minor comments\n\n- In the preliminary section, $\\pi$ is parameterized by $\\theta_i$ but it should be $\\phi_i$ to be consistent with Section 3.1. \n- Citation and comparisons missing for a closely related work\n  - Lee et al. Learning to Coordinate Manipulation Skills via Skill Behavior Diversification, ICLR 2020\n  - Tung et al. Learning Multi-Arm Manipulation Through Collaborative Teleoperation, ICRA 2021\n- The maximum episode length is specified as 100 in Section 3.2 while it depends on the number of objects in Section 5.\n- Given the high variance in the proposed method and comparable performance of the \"Attention\" baseline, it would be more convincing to have more random seeds.", "summary_of_the_paper": "This paper proposes an implicit regularization for bimanual manipulation that enforces two robot arms to focus on different regions, which prevents both arms from performing on the same object at the same time. The proposed method realizes this idea by computing attention between robots and objects and then constraining dot product between attentions of two arms to not overlap. This effectively prevents the conflict between two arms by encouraging two arms to focus on different objects (sub-tasks), leading to efficient RL training and safe behaviors. The empirical evaluation demonstrates that the proposed method is generalizable to unseen situations thanks to the attention mechanism with the regularization.", "summary_of_the_review": "This paper proposes a novel inductive bias (the attention network and regularization) for efficient bimanual manipulation, which outperforms the simple MLP and multi-agent baselines. The paper is easy to follow and experiments are well explained. However, there are certain limitations in observation spaces and applications. In addition, analysis of some experiments is not convincing (e.g. generalization, synergistic interactions) and it needs additional experiments to support the claims of the paper. I would like to see the authors' responses and adjust my recommendation accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for collaborative task solving via an attention mechanism. The method is evaluated on manipulation task in simulation. 

The reviewers agree that the paper is well written and the idea is novel and intuitive. They also share concerns about the limited applicability (too many assumptions and only two robots) of the method and that it contains unjustified claims, and therefore does not meet the ICLR bar.  

Constructive feedback for the next version of the manuscript:

- The authors should decide if this is a robot learning (a learning-based method that advances robotics, specifically robot manipulation) or a machine learning paper (a method that advances cooperative multi-agent learning). The decision should drive the publication venue and the baselines. If the target is robot learning, the paper should consider adding on-robot experiments. If the target is ML method, that more baselines and benchmarks from the MARL community should be added to the evaluation section. 
- The authors should be careful not to confuse safety guarantees, which have theoretical and analytical implications, with empirical evaluation without collisions.
- Evaluate the learning on more that 3 seeds.
Decision: Reject